they're real or fake you train it both on real data so photos that come from your training set actual photos of cats and you try to say that those are real you also train it on images that come from the generator network and you train it to say that those are fake as the two players compete in this game the discriminator tries to become better at recognizing where their images are real


or fake and the generator becomes better at fooling the discriminator into thinking that its outputs are are real and you can analyze this through the language of game theory and find that there's a Nash equilibrium where the generator has captured the correct probability distribution so in the cat example it makes perfectly realistic cat photos and the discriminator is unable

Ian Goodfellow: Generative Adversarial Networks (GANs) | Lex Fridman Podcast #19


to do better than random guessing because all the all the samples coming from both the data and the generator look equally likely to have come from either source so do you ever do sit back and does it just blow your mind that this thing works so from very so it's able to estimate that density function enough to generate generate realistic images I mean does it yeah do you ever


sit back yeah how does this even why this is quite incredible especially where Gant's have gone in terms of realism yeah and and not just to flatter my own work but generative models all of them have this property that if they really did what we asked them to do they would do nothing but memorize the training data right some models that are based on maximizing the likelihood the


way that you obtain the maximum likelihood for a specific training set is you assign all of your probability mass to the training examples and nowhere else forgets the game is played using a training set so the way that you become unbeatable in the game is you literally memorize training examples one of my former interns wrote a paper his name is a Vaishnav nagarajan and he


showed that it's actually hard for the generator to memorize the training data hard in a statistical learning theory sense that you can actually create reasons for why it would require quite a lot of learning steps and and a lot of observations of of different latent variables before you could memorize the training data that still doesn't really explain why when you produce samples


that are new why do you get compelling images rather than you know just garbage that's different from the training set and I don't think we really have a good answer for that especially if you think about how many possible images are out there and how few images the generative model sees during training it seems just unreasonable that generative models create new images as well as they do


especially considering that we're basically training them to memorize rather than generalize I think part of the answer is there's a paper called deep image prior where they show that you can take a convolutional net and you don't even need to learn the parameters of it at all you just use the model architecture and it's already useful for things like in painting images I think that shows us


that the convolutional network architecture captures something really important about the structure of images and we don't need to actually use learning to capture all the information coming out of the convolutional net that would that would imply that it would be much harder to make generative models in other domains so far we're able to make reasonable speech models and things like


that but to be honest we haven't actually explored a whole lot of different data sets all that much we don't for example see a lot of deep learning models of like biology datasets where you have lots of microarrays measuring the amount of different enzymes and things like that so we may find that some of the progress that we've seen for images and speech turns


out to really rely heavily on the model architecture and we were able to do what we did for vision by trying to reverse-engineer the human visual system and maybe it'll turn out that we can't just use that same trick for arbitrary kinds of data all right so there's aspects of the human vision system the hardware of it that makes it without learning without cognition just makes it really


effective at detecting the patterns we've seen the visual world yeah that's yeah that's really interesting what in a big quick overview in your view in your view what types of Gans are there and what other generative models besides games are there yeah so it's maybe a little bit easier to start with what kinds of generative models are there other than Gans so most generative models are likelihood


based where to train them you have a model that tells you how how much probability it assigns to a particular example and you just maximize the probability assigned to all the training examples it turns out that it's hard to design a model that can create really complicated images or really complicated audio waveforms and still have it be possible to estimate the the likelihood


function from a computational point of view most interesting models that you would just write down intuitively it turns out that it's almost impossible to calculate the amount of probability they assign to a particular point so there's a few different schools of generative models in the likelyhood family one approach is to very carefully design the model so that it is computationally


tractable to measure the density it assigns to a particular point so there are things like auto regressive models like pixel CN n those basically break down the probability distribution into a product over every single feature so for an image you estimate the probability of each pixel given all of the pixels that came before it hmm there's tricks where if you want to measure the density


function you can actually calculate the density for all these pixels more or less in parallel generating the image still tends to require you to go one pixel at a time and that can be very slow but there again tricks for doing this in a hierarchical pattern where you can keep the runtime under control or the quality of the images it generates putting runtime aside pretty good


they're reasonable yeah the I would say a lot of the best results are from Gans these days but it can be hard to tell how much of that is based on who's studying which type of algorithm if that makes sense the amount of effort invest in it but yeah or like the kind of expertise so a lot of people who've traditionally been excited about graphics or art and things like that


have gotten interested in Gans and to some extent it's hard to tell our Gans doing better because they have a lot of graphics and art experts behind them or our Gans doing better because they're more computationally efficient or our Gans doing better because they prioritize the realism of samples over the accuracy of the density function I think I think all of those are


potentially valid explanations and it's it's hard to tell so can you give a brief history of Gans from 2014 we paid for 13 yeah so a few highlights in the first paper we just showed that Gans basically work if you look back at the samples we had now they looked terrible on the CFR 10 dataset you can't even recognize objects in them your papers I will use CFR 10 we use em NIST which is


little handwritten digits we used the Toronto face database which is small grayscale photos of faces we did have recognizable faces my colleague Bing Xu put together the first again face model for that paper we also had the CFR 10 dataset which is things like very small 32 by 32 pixels of cars and cats and dogs for that we didn't get recognizable objects but all the deep


learning people back then we're really used to looking at these failed samples and kind of reading them like tea leaves right and people who are used to reading the tea leaves recognize that our tea leaves at least look different right maybe not necessarily better but there was something unusual about them and that got a lot of us excited one of the next really big steps was lap gown


by Emily Denton and seemeth chintala at Facebook AI research where they actually got really good high-resolution photos working with gans for the first time they had a complicated system where they generated the image starting at low res and then scaling up to high res but they were able to get it to work and then in 2015 I believe later that same year palek Radford and sumh intelli and Luke


Metz published the DC gain paper which it stands for deep convolutional again it's kind of a non unique name because these days basically all gans and even some before that were deep in convolutional but they just kind of picked a name for a really great recipe where they were able to actually using only one model instead of a multi-step process actually generate realistic


images of faces and things like that that was sort of like the beginning of the Cambrian explosion of gans like you know once once you got animals that had a backbone you suddenly got lots of different versions of you know like fish and right they have four-legged animals and things like that so so DC Gann became kind of the backbone for many different models that came out used as a


baseline even still yeah yeah and so from there I would say some interesting things we've seen are there's a lot you can say about how just the quality of standard image generation ganz has increased but what's also maybe more interesting on an intellectual level is how the things you can use guns for has also changed one thing is that you can use them to learn classifiers without


having to have class labels for every example in your your training set so that's called semi-supervised learning my colleague at open AI Tim Solomon's who's at at brain now wrote a paper called improved techniques for training guns I'm a co-author on this paper but I can't claim any credit for this particular part one thing he showed in the paper is that you can take the gun


discriminator and use it as a classifier that actually tells you you know this image is a cat this image is a dog this image is a car this image is a truck and so and not just to say whether the image is real or fake but if it is real to say specifically what kind of object it is and he found that you can train these classifiers with far fewer labeled examples learn traditional classifiers


so a few supervised based on also not just your discrimination ability but your ability to classify you're going to do much you're going to convert much faster to being effective at being a discriminator yeah so for example for the emne status set you want to look at an image of a handwritten digit and say whether it's a 0 a 1 or 2 and so on to get down to less than 1% accuracy


required around 60,000 examples until maybe about 2014 or so in 2016 with this semi-supervised degan project tim was able to get below 1% error using only a hundred labeled examples so that was about a 600 X decrease in the amount of labels that he needed he's still using more images in that but he doesn't need to have each of them labeled as you know this one's a 1 this one's a 2 this one's


a 0 and so on then to be able to for Ganz to be able to generate recognizable objects so object for a particular class you still need labelled data because you need to know what it means to be a particular class cat dog how do you think we can move away from that yeah some researchers at brain Zurich actually just released a really great paper on semi-supervised de Gans whether


their goal isn't to classify its to make recognizable objects despite not having a lot of label data they were working off of deep minds big gun project and they showed that they can match the performance of began using only 10% I believe of the of the labels big gun was trained on the image net dataset which is about 1.2 million images and had all of them labelled this latest project


from brain Zurich shows that they're able to get away with only having about 10% of the of the images labeled and they do that essentially using a clustering algorithm where the discriminator learns to assign the objects to groups and then this understanding that objects can be grouped into you know similar types helps it to form more realistic ideas of what should be appearing in the image


because it knows that every image it creates has to come from one of these archetypal groups rather than just being some arbitrary image if you train again with no class labels you tend to get things that look sort of like grass or water or brick or dirt but but without necessarily a lot going on in them and I think that's partly because if you look at a large image net image the object


doesn't necessarily occupy the whole image and so you learn to create realistic sets of pixels but you don't necessarily learn that the object is the star of the show and you want it to be in every image you make yeah you've heard you talk about the the horse the zebra cycle Gann mapping and how it turns out again thought provoking that horses are usually on grass and zebras


are usually on drier terrain so when you're doing that kind of generation you're going to end up generating greener horses or whatever so those are connected together it's not just yeah yeah be able to you're not able to segment yeah it's generating the segments away so there are other types of games you come across in your mind that neural networks can play with each other to to


to be able to solve problems yeah the the one that I spend most of my time on is insecurity you can model most interactions as a game where there's attackers trying to break your system and you order the defender trying to build a resilient system there's also domain adversarial learning which is an approach to domain adaptation that looks really a lot like Ganz the the author's


had the idea before the game paper came out their paper came out a little bit later and you know they they're very nice and sighted again paper but I know that they actually had the idea before I came out domain adaptation is when you want to train a machine learning model in 1:1 setting called a domain and then deploy it in another domain later and he would like it to


perform well in the new domain even though the new domain is different from how it was trained so for example you might want to train on a really clean image data set like image net but then deploy on users phones where the user is taking you know pictures in the dark or pictures while moving quickly and just pictures that aren't really centered or composed all that well


when you take a normal machine learning model it often degrades really badly when you move to the new domain because it looks so different from what the model was trained on domain adaptation algorithms try to smooth out that gap and the domain adverse oral approach is based on training a feature extractor where the features have the same statistics regardless of which domain


you extracted them on so in the domain adversarial game you have one player that's a feature extractor and another player that's a domain recognizer the domain recognizer wants to look at the output of the feature extractor and guess which of the two domains oh the features came from so it's a lot like the real versus fake discriminator and ends and then the feature extractor you


can think of as loosely analogous to the generator in games except what's trying to do here is both fool the domain recognizer and two not knowing which domain the data came from and also extract features that are good for classification so at the end of the day you can in in the cases where it works out you can actually get features that work about the same in both domains


sometimes this has a drawback where in order to make things work the same in both domains it just gets worse at the first one but there are a lot of cases where it actually works out well on both do you think gas being useful in the context of data augmentation yeah one thing you could hope for with Kenz is you could imagine I've got a limited training set and I'd like to make more


training data to train something else like a classifier you could train Magan on the training set and then create more data and then maybe the classifier would perform better on the test set after training on those big ERG and generated data set so that's the simplest version of of something you might hope would work I've never heard of that particular approach working but I think there's


some there's some closely related things that that I think could work in the future and some that actually already have worked so if you think a little bit about what we'd be hoping for if we use the gun to make more training data we're hoping that again we'll generalize to new examples better than the classifier would have generalized if it was trained on the same buddy at us


and I don't know of any reason to believe that the Gann would generalize better than the classifier would but what we might hope for is that the Gann could generalize differently from a specific classifier so one thing I think is worth trying that I haven't personally tried but someone could try is what have you trained a whole lot of different generative models on the same


training set create samples from all of them and then train a classifier on that because each of the generative models might generalize in a slightly different way they might capture many different axes of variation that one individual model wouldn't and then the classifier can capture all of those ideas by training in all of their data so we'd be a little bit like making an ensemble of


classifiers and I say oh of gans yeah in a way I think that could generalize better the other thing that gans are really good for is not necessarily generating new data that's exactly like what you already have but by generating new data that has different properties from the data you already had one thing that you can do is you can create differentially private


data so suppose that you have something like medical records and you don't want to train a classifier on the medical records and then publish the classifier because someone might be able to reverse-engineer some of the medical records you trained on there's a paper from Casey greens lab that shows how you can train again using differential privacy and then the samples one again


still have the same differential privacy guarantees as the parameters that again so you can make fake patient data for other researchers to use and they can do almost anything they want with that data because it doesn't come from real people and the differential privacy mechanism gives you clear guarantees on how much the original people's data has been protected that's really interesting


actually I haven't heard you talk about that before in terms of fairness I've seen from triple AI your talk how can an adversarial machine learning help models be more fair with respect to sensitive variables yeah there was a paper from Amos Torquay's lab about how to learn machine learning models that are incapable of using specific variables so to say for example you


wanted to make predictions that are not affected by gender it isn't enough to just leave gender out of the input to the model you can often infer gender from a lot of other characteristics like say that you have the person's name but you're not told their gender well right if if their name is Ian they're kind of obviously a man so what you'd like to do is make a


machine learning model that can still take in a lot of different attributes and make a really accurate informed prediction but be confident that it isn't reverse engineering gender or another sensitive variable internally you can do that using something very similar to the domain adversarial approach where you have one player that's a feature extractor and another


player that's a feature analyzer and you want to make sure that the feature analyzer is not able to guess the value of the sensitive variable that you're trying to keep private right that's yeah I love this approach so we'll yeah with the with the feature you're not able to infer right this sensitive variables yeah brilliant it's quite quite brilliant and simple actually another


way I think that Ganz in particular could be used for fairness would be to make something like a cycle again where you can take data from one domain and convert it into another we've seen cycle again turning horses into zebras we've seen other unsupervised gains made by Ming Yue Lu doing things like turning day photos into night photos I think for fairness you could imagine taking


records for people in one group and transforming them into analogous people in another group and testing to see if they're they're treated equitably across those two groups there's a lot of things that be hard to get right to make sure that the conversion process itself is fair and I don't think it's anywhere near something that we could actually use yet but if you could design that


conversion process very carefully it might give you a way of doing audits where you say what if we took people from this group converted them into equivalent people in another group does the system actually treat them how it ought to that's also really interesting you know in a popular in popular press and in general in our imagination you think well gangs are able to generate


data and use to think about deep fakes or being able to sort of maliciously generate data that fakes the identity of other people is this something of a concern to you is this something if you look 10 20 years into the future is that something that pops up in your work in the work of the community that's working on generating models I'm a lot less concerned about 20


years from now than the next few years I think there will be a kind of bumpy cultural transition as people encounter this idea that there can be very realistic videos and audio that aren't real I think 20 years from now people will mostly understand that you shouldn't believe something is real just because you saw a video of it people will expect to see that it's been


cryptographically signed or or have some other mechanism to make them believe the the content is real there's already people working on this like there's a startup called true pic that provides a lot of mechanisms for authenticating that an image is real there they're maybe not quite up to having a state actor try to to evade their their verification techniques but it's


something people are already working on and I think we'll get right eventually so you think authentication will will eventually went out so being able to authenticate that this is real and this is not yeah as opposed to gas just getting better and better or generative models being able to get better and better to where the nature of what is real I don't think we'll ever be able to


look at the pixels of a photo and tell you for sure that it's real or not real and I think it would actually be somewhat dangerous to rely on that approach too much if you make a really good fake detector and then someone's able to fool your fake detector and your fake detector says this image is not fake then it's even more credible than if you've never made a fake detector in


the first place what I do think we'll get to is systems that we can kind of use behind the scenes for to make estimates of what's going on and maybe not like use them in court for a definitive analysis I also think we will likely get better authentication systems where you know if a match every phone cryptographically signs everything that comes out of it you


wouldn't go to conclusively tell that an image was real but you would be able to tell somebody who knew the appropriate private key for this phone was actually able to sign this image and upload it to this server at this timestamp so you could imagine maybe you make phones that have the private keys Hardware embedded in them if like a State Security Agency really wants to infiltrate the company


they could probably you know plant a private key of their choice or break open the chip and learn the private key or something like that but it would make it a lot harder for an adversary with fewer resources to fake things most of us yeah okay okay so you mentioned the beer and the bar and the new ideas you were able to implement this or come up with this new idea pretty quickly and


implement it pretty quickly do you think there are still many such groundbreaking ideas and deep learning that could be developed so quickly yeah I do think that there are a lot of ideas that can be developed really quickly guns were probably a little bit of an outlier on the whole like one-hour timescale right but just in terms of a like low resource ideas where you do something really


different on the algorithm scale and get a big payback I think it's not as likely that you'll see that in terms of things like core machine learning technologies like a better classifier or a better reinforcement learning algorithm or a better generative model if I had the gun idea today it would be a lot harder to prove that it was useful than it was back in 2014 because I would need to get


it running on something like image net or celibate high resolution you know those take a while to train you couldn't you couldn't train it in an hour and know that it was something really new and exciting back in 2014 shredding an amnesty was enough but there are other areas of machine learning where I think a new idea could actually be developed really quickly with low resources what's


your intuition about what areas of machine learning are ripe for this yeah so I think fairness and interpretability our areas where we just really don't have any idea how anything should be done yet like for interpretability I don't think we even have the right definitions and even just defining a really useful concept you don't even need to run any experiments could have a huge impact on


the field we've seen that for example in differential privacy that uh Cynthia Dworkin her collaborators made this technical definition of privacy where before a lot of things are really mushy and then with that definition you could actually design randomized algorithms for accessing databases and guarantee that they preserved individual people's privacy in a in like a mathematical


quantitative sense right now we all talk a lot about how interpretable different machine learning algorithms are but it's really just people's opinion and everybody probably has a different idea of what interpretability means in their head if we could define some concept related to interpretability that's actually measurable that would be a huge leap forward even without a new


algorithm that increases that quantity and also once once we had the definition of differential privacy it was fast to get the algorithms that guaranteed it so you could imagine once we have definitions of good concepts and interpretability we might be able to provide the algorithms that have the interpretability guarantees quickly to what do you think it takes to build a


system with human level intelligence as we quickly venture into the philosophical so artificial general intelligence what do you think I I think that it definitely takes better environments than we currently have for training agents that we want them to have a really wide diversity of experiences I also think it's going to take really a lot of computation it's


hard to imagine exactly how much so you're optimistic about simulation simulating a variety of environments is the path forward I think it's a necessary ingredient yeah I don't think that we're going to get to artificial general intelligence by training on fixed datasets or by thinking really hard about the problem I think that the the agent really needs to interact and have a variety of


experiences within the same lifespan and today we have many different models that can each do one thing and we tend to train them on one data set or one RL environment sometimes they're actually papers about getting one set of parameters to perform well in many different RL environments but we don't really have anything like an agent that goes seamlessly from one type of


experience to another and and really integrates all the different things that it does over the course of its life when we do see multi agent environments they tend to be there are so many multi environment agents they tend to be similar environments like all of them are playing like an action based video game we don't really have an agent that goes from you know playing a video game


to like reading The Wall Street Journal to predicting how effective a molecule will be as a drug or something like that what do you think is a good test for intelligence in you view it's been a lot of benchmarks started with the with Alan Turing a natural conversation being good being a good benchmark for intelligence what what are what would you and good fellows sit back and be really damn


impressed if a system was able to accomplish something that doesn't take a lot of glue from human engineers so imagine that instead of having to go to the CFR website and download CFR 10 and then write a Python script to parse it and all that you could just point an agent at the CFR 10 problem and it downloads and extracts the data and trains a model and starts giving you


predictions I feel like something that doesn't need to have every step of the pipeline assembled for it it definitely understands what it's doing is Auto ml moving into that direction are you thinking wave and bigger autosomal has mostly been moving toward once we've built all the glue can the machine learning system to design the architecture really well so I'm we're


saying like if something knows how to pre-process the data so that it successfully accomplishes the task then it would be very hard to argue that it doesn't truly understand the task in some fundamental sense and I don't necessarily know that that's like the philosophical definition of intelligence but that's something that would be really cool to build that would


be really useful and would impress me and would convince me that we've made a step forward in real AI so you give it like the URL for Wikipedia and then next day expected to be able to solve CFR 10 or like you type in a paragraph explaining what you want it to do and it figures out what web searches it should run and downloads all the whole unnecessary ingredients so you have a


very clear calm way of speaking no arms easy to edit I've seen comments for both you and I have been identified as both potentially being robots if you have to prove to the world that you are indeed human how would you do it but I can understand thinking that I'm a robot it's the flipside yeah touring test I think yeah yeah the proof prove your human test I mean I lecture so you have


to is there something that's truly unique in your mind I suppose it doesn't go back to just natural language again just being able to so proving proving that I'm not a robot with today's technology yeah that's pretty straightforward too like my conversation today hasn't veered off into you know talking about the stock market or something because in my training data but I think it's more


generally trying to prove that something is real from the content alone it was incredibly hard that's one of the main things I've gotten out of my can research that you can simulate almost anything and so you have to really step back to a separate channel to prove that slang is real so like I guess I should have had myself stamped on a blockchain when I was born or something but I


didn't do that so according to my own research methodology there's just no way to know at this point so what last question problem stands all for you that you're really excited about challenging in the near future so I think resistance to adversarial examples figuring out how to make machine learning secure against an adversary who wants to interfere it in control with it


is one of the most important things researchers today could solve in all domains in image language driving in I guess I'm most concerned about domains we haven't really encountered yet like like imagine twenty years from now when we're using advanced day eyes to do things we haven't even thought of yet like if you ask people what are the important problems in security of phones


in in like 2002 I don't think we would have anticipated that we're using them for you know nearly as many things as we're using them for today I think it's going to be like that with AI that you can kind of try to speculate about where it's going but really the business opportunities that end up taking off would be hard to predict ahead of time well you can predict ahead of time is


that almost anything you can do with machine learning you would like to make sure that people can't get it to do what they want rather than what you want just by showing it a funny QR code or a funny input pattern and you think that the set of methodology to do that can be bigger than you want domain and that's I think so yeah yeah like one methodology that I think is not not a specific


methodology but like a category of solutions that I'm excited about today is making dynamic models that change every time they make a prediction so right now we tend to train models and then after they're trained we freeze them and we just use the same rule to classify everything that comes in from then on that's really a sitting duck from a security point of view if you


always output the same answer for the same input then people can just run inputs through until they find a mistake that benefits them and then they use the same mistake over and over and over again I think having a model that updates its predictions so that it's harder to predict what you're going to get will make it harder for the for an adversary to really take control of the


hi today we're looking at fix match simplifying semi-supervised learning with consistency and confidence by cukes on David birth Berthelot and others of Google research so this paper concerns semi-supervised learning so what does semi-supervised learning mean in semi-supervised learning you have a data set of labeled samples so right you have this data set of X's and corresponding Y

FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence


labels but this data set sometimes is very small now you have a much bigger data set of unlabeled examples just X's with no labels right so you don't know what the labels of the of the unlabeled examples are but what you would like to do is you would like to use this really large data set in order to help you with learning the association between the data points and the labels so for


example in this case you would have something like like an image classification data set and I'm gonna take the example here of medical data so you have a pictures of lungs let's draw a long here that is an ugly long you have pictures of lungs and whether or not they are they have like a tumor in them right so medical data is very hard to get especially labeled medical data


because you need first of all you need the data itself but then you also need like like one at least one but ideally like three radiologists to look at whether or not this is a good or a bad image and label it so it's usually very expensive to collect that data but you might have plenty of unlabeled data right you might just be able to go who you're through through some database and


find like anonymized undiagnosed long scans somewhere lying around the same with image like other images so labeling images is pretty human intensive but the internet contains like a whole bunch of unlabeled images so the task of semi-supervised learning is how do you use this unlabeled data set in order to make your classification on the label data set easier and fix match


combines two approaches to this in a smart way namely the consistency and confidence approach right so what does what does well it will jump right into into the method so basically what you want to do is you want to say my loss that I optimized right this is my loss consists of two parts namely a supervised loss which is your classic classification loss right plus an


unsupervised loss right and then you have like some sort of a trade-off parameter in front now your supervised loss here this is where this is just the the cross-entropy let's call it h between your predicted labels and your the actual true labels right and the predicted labels say they can be you know kind of a distribution over labels now the magic of course is


here in the unsupervised loss and this unsupervised loss this is what's described here in this part right so the unsupervised loss is going to be this age between P and Q and we'll see what P and Q is so if for the unsupervised loss you two of course want to start with an unlabeled example then you have the same sample go into two different pipelines in the first pipeline up here what you


do is you so-called weekly augmented and here we're dealing with images so we have to talk about image augmentation so image augmentation has long been used in supervised learning to kind of give you more it's kind of a cheat to give you more training data so if you have an image right of let's say famous cat you can obtain met more training data if you for example by


random cropping so you can random crop let's say we just take this bottom right corner here and then we enlarge it to the original size right then it is still sort of a cat but it's just a part of a cat right but usually that helps because you you say okay um my image data set is just pictures of animals right it's entirely conceivable that someone held the camera like this or like this right


so technically in terms of generalizing to a test set these both data points should be valid so I'm just gonna add both to my training data so you can see how from one training data point you can get many training data points just by doing this cropping what you can also do is you can flip it left right right you just in swap the pixels left right and usually the these kind of um so a a a


cat that has a little dark spot here is still a cat when it has too little dark spot over there right but to your classifier those are two different samples so you can do many of those things and they have to kind of augmentations they have what they call weakly augmented and strongly augmented right so in the weakly augmented pipeline I think they just they crop and


they they shift and they rotate or something like this so you can see here this this horsey here it is something like it's cropped here about then it is turned slightly to the left and then yeah I think that's it so they crop they rotate and then they also flip horizontally at random in like 50 percent of the time so these are what's called weekly augmented the goal here is


just to kind of obtain a bit more training data alright so you run this through your model through your classification model as you would a regular sample and you get a prediction now from your prediction you can take the highest prediction here and that is going to be your pseudo label so this is P of Y this is your distribution that you estimate right so and this and


this if you just take the max this is going to be your Y hat right and this is what they call a pseudo label sorry you'll see why it is called a pseudo label so the other pipeline here is the strong augmentation pipeline now in weak augmentation we just wanted to get somewhere training it in strong augmentation now the goal is to really screw up that picture to the point where


it's still you know you could recognize it in the same class but you can see here the augmentations they go wild so you play around with the color with the hue you play around with the light intensity right with the contrast you can do many many things you can see this this image looks basically nothing like this image buddied you can still kind of recognize it as a horse but the strongly


augmented data is much more distorted than the weakly augmented data and that's the point so also you send the strongly augmented data through the model and again you get a prediction right and now is that the trick is you take the label from here and you you take that as if it were the true label right you take that as if it were the true label and you form a loss from this


prediction being the model prediction as if this thing here that also comes from the model as if that was the true label right that's why it's called a pseudo label because it is a label that you produce from the model itself now of course if these were to be the same picture it would be kind of pointless right that's why you see there needs to be a weekly and a strongly


augmented pipeline I am pretty sure ammo if you want a more basic version of this make this just clean so no augmentation and make this augment it right that's that's how you can think of it the fact that there is weak and here strong augmentation I think is just a your classic trick to get more training data but in essence you can think of it as this is here the clean


thing you just want to produce a label and then you want the that an Augmented version of the image has the same label now you can think of it shortly what does this model learn if you just have this you remember I think the important thing is always to remember that there are two components here right there is first the supervised loss this is the important one ultimately because we have


the true labels right and then second there is the unsupervised loss which is just an auxiliary loss that is supposed to just kind of tune our model to the nature of the data right so don't forget that this this down here just concerns the unsupervised part of that loss so if you think what does the model actually learn when whenever you train it like this it basically learns to revert this


strong augmentation right say basically sells hey model whenever I give you a week augmented image and I distort it heavily right whenever I give you an image and that distort it heavily I want the label to be the same so the model basically learns that whatever the image the whatever the image the model at the end of the trend will be able to basically map any strongly augmented picture to


the same class as a weekly augmented picture if it comes from the same source right so the model basically learned to ignore these kinds of augmentations that's what this loss over here does it basically says these sorts of augmentations these sorts of distortions of images please ignore those because I always want you to output the same label here in the prediction here as if I had not


distorted or just weakly distorted the image so that's that's what you have to keep in mind that this this loss is designed to make the model not distinguish between differently augmented versions of the same image and interestingly that really seems to help with the with the supervised loss right my kind of hypothesis is is that all these methods what they're kind of


trying to do is to just tune the neural network to the let's say the orders of magnitude of the of the input data and also to the kinds of augmentations that the humans come up with and that's a very important point so the Ottoman tations and here we said you know it's it's kind of a rotation and the crop the kind of augmentation really seemed to play a role


so this paper finds that on C 410 where the state of the art I believe is something like ninety six ninety seven percent accuracy on C for ten with just two hundred and fifty labeled examples right now the usual data set size is about fifty thousand it goes to ninety four point three four point nine percent so almost 95 percent accuracy with the state of the art being like ninety seven


this is incredible with just two hundred and fifty labeled examples crazy right and it with only four labels per class it gets eighty eight point six percent so that's just forty images with labels they get a 8.6 percent of of the of accuracy compared to the 97 percent that you get with like 50,000 images that is pretty pretty cool right simply by having all


other images not labeled but pseudo labeled and consistency regularized right so the the two to two things that are combined by fixed match again or consistency regularization which basically it means that the model should output similar predictions when fed perturbed versions of the same image right this they they're really forthcoming that they are not the ones


who invented this they just combine the consistency regularization with the pseudo labeling now the pseudo labeling they have also not invented the pseudo labeling leverages the idea that we should use the model itself to obtain artificial labels for unlabeled data we've seen a lot of papers in the last few months or years where it's like the teacher teaches the student and then the


student teaches the teacher model again and so on so that they simply combine the two methods in a clever way they have one last thing that is not in this drawing namely they only use the pseudo label they have a break right here and they only use the pseudo label if if the confidence if this P of Y here is above a certain threshold so they don't take all the pseudo labels but they only take


the labels where the model is fairly sure about right so they haven't actually an ablation study where they show that this is reasonably reasonably important and if you go down here where they say ablation or is it ablation ablation study oh yeah something I also find cool if you just give one image per class this one image per class ten images that are labeled


it still gets like 78 percent accuracy I think the images are chosen as good representations of their class but still one image per class pretty pretty cool an important part of this is the ablation study where they say okay we want to tease apart why this algorithm why this on semi-supervised learning technique works so well and they find several important factors they find for


example that they're all mentation strategy is extremely important so how they augment the images is very important you see here the error of this 4.8% and the 250 label split if you change up the if you change up the the augmentation strategies your error gets higher right and so they say we use this cutout and we measure the effect of cut out we find that both cut out and seek T


augment are required to obtain the best performance removing either results in in a comparable increase in error rate we've seen before for example they went they went from there some 93 sorry 93 point something percent to ninety four point something percent from the previous state-of-the-art semi-supervised learning and here they find that simply changing the


augmentation strategy changes the error by more than a percent so you can just see this in context of of what's important here right they say again the ratio of unlabeled data seems pretty important we observe a significant decrease in error rates by losing using a large amounts of unlabeled data right then the optimizer and learning schedule seems to be very important as


well in that they use this they say STD with momentum works much better than Adam and then they use this decreasing learning rate schedule this cosine learning rate schedule so there seem to be a lot of things a lot of hyper parameters that are fairly important here and you can see that the gains are substantial sometimes but they aren't like through the roof substantial where


you can make a good argument that it is unclear how much really comes from this clever combination that fit fix match proposes and how much also just comes from whether or not you set the hyper parameters correctly and exactly how much computation are you able to throw at selecting your selecting your hyper parameters so that that seems to be a bit of a a bit of a pain point for me


they also say we find that tuning the weight decay is exceptionally important for low label regimes right choosing a value that is just one order of magnitude larger or smaller than optimal can cost ten percentage points or more and so that all of that seems to me that this this kind of research where you're you're nibbling for half or single percentage points in accuracy while a


single misstep in a choice of hyper parameter might cost you ten times that gain is is a bit sketchy now I recognize they get numbers like no one else has gotten before but where exactly the gains come from and if the gains really come from this architecture or actually just more from throwing computer at it I don't know all right with that I hope you enjoyed this and I invite you


hi there today we'll look at hopfield networks is all you need by researchers from the johannes kepler university in linz and the university of oslo so on high level this paper proposes a new type of hopfield networks that generalizes modern hopfield networks from binary patterns to continuous patterns and then shows that the retrieval update rule of these new hopfield networks

Hopfield Networks is All You Need (Paper Explained)


is equivalent to the attention mechanism that's used in modern transformers and it's actually a more general formulation of the attention mechanism and therefore it can be used to do kind of a variety of things to improve modern deep learning and uh it also has a companion paper where it applies this to some kind of immunology research and gets uh achieves state of


the art in a task that is specifically suited to this type of attention all right let's dive in together we'll go over what this paper does what it proposes and so on if you like pay if you like videos like this uh consider subscribing you know sharing it out and i hope you're enjoying this all right also thanks to my discord community um for you know very helpful bringing me


up to speed on this paper uh super interesting discussions there if you're not on our discord yet uh i invite you to join it's fun okay so what is a hopfield network a hot field network is a pretty kind of old style old conceptualization of a neural network so in a hopfield network what your goal would be is you can conceptualize it as a bit of a neural network


so let's say we have five neurons or something like this uh your what your goal would be is to have a neural network where you can store so-called patterns and a pattern in this case would be a binary string of size five so for example one zero one zero zero or one one zero one zero and you'd have a list of these patterns and what your goal would be is to store


these patterns in the neural network such that and here you know we'll just consider everything to be sort of connected to everything else and um what your goal would be in this is that you can kind of store patterns inside this neural network and you adjust the weights somehow so this was as i said this was this was this is kind of an old model um you store you you adapt the


weights such that you store these patterns and what does it mean for a pattern to be stored if you have stored a pattern you can you will then be able to retrieve it and you retrieve a pattern in these kind of old style hopfield networks by providing a partial pattern so what you'll say is for example i i want a pattern that starts with one one zero and you give that to the network and


there would be a so-called update rule and the update rule is kind of an internal uh rule so let's just go through this so here this one one zero maybe this is one one zero and then they would kind of send messages around so this update rule would somehow adjust the value of this and this neuron here to what's most compatible with the network weights and if if the network weights have been


adjusted correctly this will turn out then at the end of applying this update rule that this is a one and this is a zero and therefore this pattern here is retrieved now had i input uh one zero one at the beginning then the outcome would be different hopefully this pattern here would have been retrieved okay so you can see the applications of this like you can have the first three digits as


sort of a database key and then the last ones as sort of the value that you store along with it and then you can simply provide the first few you can also provide you don't always have to provide three um so this all depends this is this is sort of an as i said an old conceptualization of neural networks so people were imagining that this is kind of how the brain works


you know fire together wire together and also with research into this it it turns out that you know you might think you know there's there's kind of five neurons so maybe i can store five different patterns you know accurately because if i store too many patterns right if i have many many many many patterns then i can't expect to be able to retrieve all the patterns


again because some of them will just be so equal that you know many will start maybe with this and and i won't have a chance to to retrieve the one i want or if the update rule will make a mistake so you might think this might be like five because i have five neurons or maybe ten because i have ten connections but it turns out that um in modern hopfield networks with the appropriate


update rule you can store exponentially many patterns in these networks exponentially many in the in the dimension of the um in the dimension of the patterns and here i guess that would be the length of the pattern so this is a little bit surprising the kind of storage capacity of these networks and we'll this um this paper here generalizes that to continuous uh


to continuous states so what do we mean with continuous states i guess i mean continuous patterns so no longer is a pattern a binary string but a pattern now is a string of floating point numbers okay like 0.5 1.3 and so on and you know a string of floating or a sequence of floating point numbers is naturally depicted as a vector okay so our patterns are going to be


different vectors that we store and um you know in high dimensions that the the vectors will be kind of separated well from each other as long as we don't have too many but this paper shows that all these properties for the modern hopfield networks that hold for binary strings still hold if you go to these kind of um continuous to these vector patterns that means you can store exponentially


many patterns in the dimensions of the vector which is pretty surprising right because you'd think like you know after you have one vector per dimension that you know after that it might get a bit shaky but no you can actually store exponentially many that's pretty surprising and this paper is a lot about how to do that and the fact that that happens and so on


so we've talked about update rules for these um kind of hopfield networks and i haven't really specified what that is i've just said that you know i enter a pattern and then the network does something and out comes out comes the whatever the pattern that matches my query so this here is called a query you might already um this is on purpose like the kind of overlap between the attention


mechanism lingo and the hopfield network lingo we're going to conflate the two to kind of make clear where the two overlap if you don't know what an attention mechanism is or aren't familiar with it watch my video on attention is all you need uh once you watch that this video will make a lot more sense all right so in what the update rule does is specifically and the update rule


there isn't only one right there are many different proposals of hobfield networks and they all lead to different properties but what an update rule does ultimately is it minimizes what's called an energy so every type of hopfield network is associated with an energy function and this the energy function of the modern hopfield network for binary strings is this energy function right here


so with x x is the pattern um the pattern this is the kind of state of the hopfield network and so these are the the whatever is stored in the network and then design here is the query that you enter into the network and then the energy here tells you this quantity you have to minimize this quantity in order to retrieve the pattern that you want okay now we are never directly


working with the energy as such uh so what you could do is for example use back prop or something to use gradient descent to decrease the energy but usually along with an energy function comes an update function and the update function is what i've talked about here like you do something and then the network does something and then you get the pattern out what the network does is it minimizes


its energy function and the update rule is made such that the corresponding energy function is minimized so the energy function is more like a theoretical consideration that you say okay here is my energy function of my hopfield network and the there will be a corresponding update rule that minimizes that energy function and if you use that update rule maybe multiple times then


the energy function will be minimized and you will have retrieved your pattern or not if if you have too many patterns stored it might also fail right so they they say what the update rules are in the um in the text here for the old hopfield networks but we're not really interested in the old ones we're interested in the ones that this paper cares about namely where


the patterns that you store in the hopfield network are these vectors over our vector patterns and the query is also a vector pattern so you want to store all of these patterns into the hopfield network so i'm going to draw it like this here i'm going to store it into the hopfield network and then after that you want to come up with a query and the query is like this and


in the case of the binary strings we had something like well i sort of know half of my binary string now in the vector hop field network it's more like well i sort of kind of know the direction that my vector should point in okay and you will re what you want to retrieve is the vector that has kind of a large inner product okay so if i enter this query into my


hopfield network what i hope is that this vector here is retrieved now you see it's not exactly the same vector like they do point if i translate that here by i it's maybe something like this but so they are different but you want to say well i kind of know what i want i kind of want something like this and then the hopfield network would answer with ah i have something like this it's this


right here okay so you that the connection to attention mechanism should become pretty pretty obvious right now but you know the um to actually establish this formally is the kind of the point of this paper and you know it's pretty cool to see so they formulate this new energy right here this is the energy of this new continuous hopfield network um specifically they


have to have this term right here because they now have continuous states and continuous queries uh this if you minimize the energy it basically means that your query can never you know go to infinity because you have the the query right here in the energy function um the update rule is this right here and we'll look at that in a moment but remember the update rule is what you actually


implement in code so if i if i have a query right here i plug it in here this is the state of my hopfield network and i apply this rule multiple times and out comes the kind of answer of the hopfield network to my question so the i input this and the out comes this after i update after i apply the update rule maybe multiple times right and interestingly you can already see


that this here if you rewrite a bunch of these quantities so if you rewrite the beta here which is the softmax temperature in a way to be one over square root of d and if you take the query design here to be the query matrix and if you take the x here to be the key matrix then this is equivalent to the update or sorry the attention mechanism of a modern transformer so that's the point of the


paper is that we can look at the transformer attention mechanism as a hopfield network and they have this interesting this interesting diagram at the end right here uh so the appendix you know this is typical i guess sep ho haita i remember this sailor paper had like 60 pages of machine proof appendix this also this has like 70 page appendix crazy but at the end of the appendix


you'll find this diagram right here now usually in an attention mechanism um you have whatever the the input is so you have an input right here so this is attention mechanisms or at least transformers they work on sequences or sets of objects and from these you'll generate three things you'll generate the you'll generate the queries the keys and the values now you can either


generate the queries from the same objects which would be self-attention or you can generate the queries from like a different object over here uh it doesn't it doesn't matter too much for our discussions but either you you know have a reference input or you have you know this kind of same input all the way and then what you do is you use three different heads or you know three


different matrices to transform that input into queries keys and values so i often conceptualize this as you have kind of your input set and each of the input sets outputs a key and also each one which would be a vector and also each one outputs a query so i often draw this here the same sequence and each one outputs a query and the query sort of the query is kind of a request for information


so the key exposes uh sort of what exposes something about the input here so this could be a sentence down here this could be my cat is very pretty and the the the vector the key vector right here could encode something like this is a noun or this is an animal or anything like this right and the query here it could ask for for other things so for example since this is cat


this vector right here the query vector is generated from that you know token cat now it could recognize that cat is a noun and it could ask the other nodes to basically say are there any adjectives around here because um you know adjectives because i it itself is a noun it's the object of the sentence right it could ask are there any kind of adjectives that describe the object


because that would be naturally a thing to ask if you were the noun you would want to know are there any kind of modifiers um for for me so it could output the query and the query here could mean you know this direction could mean adjectives and you see here the word pretty is an adjective so it itself would output a key that says by the way i'm an adjective right so


if the cat asks then if this node asks for an adjective and this outputs uh the adjective vector then because the inner product between the two things is high this will be routed here so attention mechanisms basically information routing that's how i always describe it but in this paper we look at it more like these here are the patterns that are stored in a hopfield network and i


by inputting a query and the dot product being the update rule of the hopfield network i retrieve from the hopfield network i retrieve the appropriate pattern that i ask for okay and then you know the values the values are simply a modification of the keys in this form but a lot of people also do keys and values to be the same thing but this routing of information happens


here where you multiply the queries and the keys and then you put a soft max over them okay so if you just look from the perspective of a single node like this node here this cat node what it would do is it would inner product its own query vector with all of the key vectors right so it would build an inner product with all of these and then it would normalize it would put it through a soft


max which will kind of give it a distribution right so here would give it like uh so this this actually matches because ma well my is also very important for cat this this this is just an accident i did not plan this uh this here this also well many things match but but uh in our example we would just say that this last one it's it's not only higher it's also


wider um it matches very well right and so the information routing would route mostly information from this pretty token to the cat token which makes sense in our case right this is the attention mechanism now sin if we are interpreting this as a hopfield network and the update rule here is the dot product you can actually think of applying this rule multiple times


so what happens now if we and this is where this update rule comes in what happens if we take this distribution and we don't aggregate the values like usually we would aggregate the values by this distribution what if we aggregate the keys by this distribution okay what comes out well if we look at this and you know let's just assume that this key right here matches


really well but the others also match a little bit what would come out would be a weighted average where a lot of weight is put on this particular key so what will turn out would be something like something that's very close to that key you can see um i'm going to draw the old key here in green and i want to draw the old query in blue so you see that it's then whatever comes out is not the query


but it's also not that only key that matches right it's kind of a weighted average but with that key dominating okay now since you know in a hopfield network what we would do is we would go again we would put this new thing the red thing instead of the query vector okay so we would use this aggregated keys this weighted average as a new query vector for that node


right here so duplicate that node over here i'll use that query vector again and do the same thing again okay inner product with all of the query vectors and now since this is already an aggregate of the query vectors what's going to happen of course the distribution that's going to come out is going to be weighted even more heavily into the direction so let's make it even


wider into the direction of that key that matches okay and you can pretty clearly see if i do that iteratively um then that will lead to a situation where everything is like very low except that one key will sort of dominate the distribution and ultra high and ultra wide okay and that's how that's exactly how a hobfield network works right i would input the query


which would be sort of what i want right i kind of know what i want okay and then i apply this rule multiple times right and with each time i refine refine refine until i decide on a pattern right the hopfield network is made for pattern retrieval and these here are the patterns that i want to retrieve so here the patterns aren't kind of stored in the network


beforehand but the patterns are also generated like in an attention layer so the keys are generated by the previous layer or by these matrices but that doesn't matter for the hopfield network update rule so you see here that the attention mechanism can be interpreted as simply one step making one step of this update rule but you can think of making actually multiple steps


and and retrieving the particular key so you know deciding on a sort of a hard routing of particular information now that only works if if there are no other vectors that are close to that particular key right so if the query is this and you know the way i drew it here you can see that there are many there is this one and this one and this one that matches so technically the way i drew it um


what would happen most likely is no many no matter how many times you apply your update rule it would sort of result in kind of the average of the three keys right so because they're all matching and they would all contribute to that weighted average of the query in the next step and then that means basically the convergence would be to something in the middle and that's going to be


a central point of this paper um in which situation we are so they call the first part is retrieving a single pattern and they call the second situation where you have multiple patterns that all match that are not well separated from each other they call this a meta-stable state and it's going to be pretty interesting to look at um transform like bert language models


and look at where they actually are are they actually operating in this single pattern retrieval mode or are they operating in the meta stable state mode all right so here you can see it in the diagram the only thing differing this from a hopfield network sorry from an attention mechanism is this branch right here so here you ask do you want to do multiple updates after you've you've


multiplied the queries and the keys do you want to do multiple updates if yes so if you're in a this hot field network situation you want to do multiple updates then you go back as you can see and you do you use the keys together with the output of the softmax to generate a new query so this query q here is now generated from the output here and the key so the keys are the same


these are this is the same thing it's just put here twice okay this is exactly what we discussed okay i hope that's some somehow clear that these the the attention mechanism is simply a one step uh hub field network pattern retrieval algorithm with a particular update rule that is that is matches this energy function that they propose right here of course they do this you


know particularly because the update rule that turns out is the transformer update rule but um i actually don't know if they backwards engineered the energy function to match the transformer or if they first came up with a continuous help field networks and then just kind of discover that it's like the transformer we'll maybe never find out okay so um let's go there are a couple of


theorems i believe there are four five theorems right here that uh show that kind of make some points about this about this stuff and we'll go through them we won't go through the proofs or any you know super in-depth meaning but it's pretty cool to go through them and they are proved very rigorously as i said there's a 70 page appendix so have a look at that if you're up for it


okay so they say here we have an update rule this is our update rule for our new hop field networks so the first theorem they say is the update rule that we propose converges globally if we apply the update rule repeatedly the energy for t goes equals infinity and the energy will converge sorry the energy will converge to a fixed point this being a fixed point


for t of course sorry for t goes to infinity yeah if this is a fixed point basically saying that if i apply this update rule here over and over and over again it will it will make this energy function converge to a fixed it will make this energy function converge don't want to say anything mistakenly here or claim too much but that basically connects the update rule to the energy


okay so just showing like this really is the update rule for that particular energy function okay now as as itself it's not super duper interesting yet but um now we get to theorem two so theorem two for the iteration that's the update rule that we just looked at we have we have that um this convergence holds as t goes to infinity for some stationary point furthermore


this quantity here goes to zero so that means this is the um the update at t plus one and this is the update at t and the difference between them goes to zero so that means not only does the energy converge but the iterates themselves convert so the algorithm actually converges the individual updates of the algorithm so this e new at some point that will no longer change


because the the norm between it and the previous one will go to zero you can see that either the sequence here converges or in the other case the set of limit points yada yada is a connecting subset this is a bit over the top here they say okay it can either converge to a point or it can converge to a connected subset but if the loss is finite then any sequence generated by the


iteration equation three converges to some fixed point so you know basically saying that here we oh this is not the loss i'm sorry um no this is the domain never mind i'm an idiot this is basically saying that this algorithm will converge okay and they define here what it means for a pattern to be stored and retrieved and that's for establishing what the kind of


storage capacity of a hopfield network is so we've established that the update rule minimizes the appropriate energy and the update rule will converge at some point which means that we can you know if it converges we can retrieve the pattern that it converges to so now we define how many patterns can we actually store for that we need to know what does it mean for a pattern to be stored


so we assume that we have patterns and these patterns are called x okay x i we have n different patterns each one is called um x with a subscript we assume that around every pattern a sphere is given so how do we imagine this um we have these patterns and this is this is just a space now they consider patterns of the uh on on a sphere but we'll just conceptualize it as this we'll have


a space there are patterns we want to store okay and we'll say around every pattern there is a sphere okay sphere like this and naturally the patterns are going to be there's going to be a notion of well separated patterns and you can imagine this a little bit like these spheres won't be touching each other if these spheres aren't touching each other that means that the patterns are kind of


well separated and that means that if we initialize the query remember the query here is a vector that kind of sort of looks like a pattern and that means the query is kind of close to the pattern in some notion of distance so if we initialize the query somewhere in that sphere then it might if it converges to that sphere to that pattern then we retrieve the pattern


okay now it gets a bit more complicated than this but not much more so we say a pattern is stored if there is a single fixed point inside the sphere to which all points that start inside the sphere converge and none of the spheres intersect so the sphere of point i doesn't intersect with the sphere of point j so that's where we say all these spheres are non-intersecting


we say x i is retrieved if the iteration equation three converged to the single fixed point in that sphere the retrieval error is the distance so you'll notice you have two things you have x i this is the actual pattern and you have x i star this is the retrieved pattern so these hopf they don't always have to give you the same thing that you stored that's part of the the nature of


continuous neural networks whatnot so for every sphere we say there is a pattern there is a sphere now we as pattern is stored if every i can start wherever i want in this sphere okay wherever i want it will always converge to a point that's inside the sphere okay and maybe that point isn't the pattern that i stored but actually this point right here but wherever i start i will always converge


to that particular point if that's the case then i have stored this particular pattern now the fact is i don't retrieve this particular pattern i retrieve the blue thing but i can then define the error of retrieval the error of retrieval is simply the distance between the two things ideally this distance is very small right but you know we can't can't guarantee it


now there are going to be theorems that deal exactly with this retrieval error but first you can see that here if if these spheres become larger you you can't accurately store a pattern anymore so this is the kind of ideal situation but there are also situations where these spheres you know if i have these patterns right here these spheres are so large kind of the


the attractions of the patterns are so large that if i start let's say here then i don't converge to either of these two patterns i converge to like something in the middle i converge to maybe this point right here and that's going to be one of these meta stable states okay we're going to encounter situations like this but we're also going to encounter situations like this


and the bottom thing isn't necessarily bad and that's what you have to keep in mind um and yeah as i said we will get to it but just keep this kind of sphere image in mind okay so first we'll just deal with the you know the up the top situation where we store patterns and then retrieve patterns so we'll we'll assume a failure probability which is p and p is going to be you know pretty


pretty low for their example so they have p equals 0.001 you know like a 0.1 percent error probability of retrieving your pattern things like this and randomly chosen patterns on the sphere with radius m we define some constants then with probability 1 minus p the number of random patterns that can be stored and stored in the sense of having these spheres around them so


that you can retrieve them accurately or at least you can retrieve something that's close to them is is bounded lower bounded by this quantity right here so there's the square root of p there is this constant c but then you see that d is in the exponent right here so that means it's exponential in the number of dimensions so that's that's pretty cool so if you


add a dimension you exponentially increase the number of the number of patterns you can store and you know that's that is a kind of i mean it's it's been known for modern hopfield networks with binary strings so it's not uber surprising but if you have you know it's not what you would imagine like that okay so they may give a few examples of these co you have to set


these constants you know in a particular fashion such that this is given and so on um but they say you know examples here are where c is something like three and d is 20. um so if you were to add a 21st dimension then your i guess storage capacity would increase by a factor of three which pretty cool all right so this is how many that we can store infinitely not sorry


exponentially many patterns um in these networks now they deal they say the next theorem states that the update rule typically converges after one update if the patterns are well separated so if we're in a situation where these patterns are well separated which is kind of like this but you can also imagine this in terms of dot products because we operate in the


space of dot products so if the patterns are well separated that sort of means that they all kind of sort of point away from each other and this notion of separation is going to be captured by this quantity right here this is the separation of example of pattern i which is just the inner product with itself uh minus the maximum inner product with any other uh pattern and this


quantity is going to be large when no other pattern is close to it so when the separation is large then the update rule the retrieval rule of um calculating you know i have a query calculate the inner product with all of those then i re-weigh all of the patterns by that inner product by the softmax then i used that new thing as a query again and so on as we discussed


it will converge to the closest pattern but this theorem says it actually converges pretty fast and here i have my problems with saying that it converges after one step um typically converges after one update because that you know genuinely depends on a lot of constants as we'll see but it does converge exponentially fast in this separation constant as the theorem 4 says


with query psi after one update the distance of the new point to the fixed point is exponentially small in the separation delta i the precise bound using the jacobian and its value in the mean value theorem are the following so here you can see this is the distance between the updated psi after one step and the um and the fixed point right here this is what it converges to


is going to be the distance as it was before times this thing right here so you can see since this is a this is a multiplicative update um and in this jacobian so this is expanded down here this is this you can see here you have the you have this sorry yeah this is this so this is bounded by that you have the exponent the exponential function negative this separation right here


so the higher the separation the faster this algorithm converges okay to say that it converges after one step is you know it might be a bit of of bragging i don't know if this is a common thing if you have like an exponential convergence uh that you are allowed to say it's after one step i'm not sure especially what i'm not sure about is that you have n here as linear constants in that


factor okay so if you if you and that's what they do in their code so if you look at their code and the code's available which is pretty cool it's implemented in pi torch as a general module that can you can just drop in so this is not only for transformers this is for you can replace like lstms you can replace pooling mechanisms um you can you know do a whole bunch of stuff


in their paper in the accompanying paper they do this multi-instance learning with giant uh sets um on using these hopfield layers so pretty pretty cool this code is definitely worth kind of checking out and maybe you want to replace some stuff with you but the question is how many of these update steps should you do right because we looked at the diagram at least in the


attention mechanism it seems like you have attention layers right you have a transformer and the transformer consists of you know you have this input right here and you go through layer layer layer layer layer and in each layer there's contained in it and one of these attention mechanism right this entire thing is in this layer okay and now if you interpret this as a hopfield


network and you want to do multiple steps that means you go this branch right here so in each layer potentially you do multiple steps of these things so for whatever computational constraints um transformers had already this will certainly make it worse but also you need to decide how many steps you want to do now you can hard code that of course but they say


you should do these steps until this norm here until the norm between the old and the new is small enough so where is that so you can't measure how close you are to the convergence points right because you don't know in practice but you can measure how far you're away you can measure where did we have it you can measure this quantity right here that's something you can measure how far


twit rates are apart so what you'll simply do is you'll measure that and if that is small enough then you'll you'll stop but that i guess is very related to this so how if you we've already proven it converges to this x star so i guess we can approximate this quantity right here with the quantity above and that tells you how many updates you need to do and that quantity


is linear not only linear but actually here quadratic in n i don't care you know yes it's exponential uh amd uh separation but it's quadratic in n and if i've learned anything from kind of my fast code courses is that constants actually matter when you're not dealing with infinity with an infinite number of steps so the number of the number of steps you need to do i guess will


depend on the sequence length in a quadratic fashion so i'm not sure you can always claim this is converges in one step now i might be super mistaken here and none of this uh will can none of this actually makes a difference in the in the light of the exponential decay here but i would just i'm just a bit worried saying this usually converges in one step it's clear


i guess why they do it right because the attention mechanism in transformers is a one-step application of this rule and this here is kind of a theoretical justification for interpreting this precisely as a hopfield network because you'd say well in a hopfield network you would do multiple steps but wait wait we can actually prove that even if you interpret it as a hub field


network you it can it usually converges after one step so what you're actually doing in a transformer is applying a hopfield network update rule uh to convergence so yeah i'm not yeah i might be bickering on a high level here luxury problems theorem five then says so theorem four is how fast does this converge um theorem five the last theorem right here uh says that the retrieval error


of a pattern and so this is the this is what you converge to and this is what you've stored um is bounded by again something that's exponential in the separation right here as you can see okay so that was the theorem so if we go quickly through them again theorems one and two deal with the convergence of this algorithm and the fact that it actually minimizes the proposed energy


then theorem three says you can store exponentially many patterns in terms of the dimension of your space and theorems four and five say that this update rule will converge exponentially fast after after one step if you believe that and the retrieval error will also go down exponentially fast with the number of update steps that you do okay that sounds pretty pretty pretty


good but we've heard it it's very dependent on how well separated these patterns are and it turns out that it's you know at least in transformers they aren't always well separated and that might be on purpose remember the these states here the the patterns aren't pre-stored like in a classic hopfield network but the patterns if you interpret an attention mechanism


as this are also generated by the network itself so the pattern matrix that you retrieve from and the query are generated by the attention mechanism in in this case as i said this is applicable to many many more uh domains than just this but um yeah so there's another slight modification that you have to do to make this actually equivalent to an attention mechanism and that


is you'll have to recast the value because usually what you'll do is you have some sort of input and then you make queries keys and values from that using different heads the only thing to make it formally equivalent is you have to make the values generated from the keys so the keys give rise to the values as you can see right here that you first multiply with


the key matrix and then with the value matrix i think that's you know that i don't i doubt that this will will change anything um if you if you the only way that could really change anything is if this matrix here would be super low rank like collapse the space of um into like very few dimensions which the value matrix wouldn't do so you know but just letting you know that the technical


equality requires this slight modification okay now we said that um it might not you know be that this is always super well separate and you retrieve a single pattern and that's what they research here in a pre-trained bert model so they take a pre-trained model from i guess from hogging face and they run they just run a data set through it and what they do is so for each for each


query and sorry for each attention head because you have multiple ones of these attention heads right um in each layer so in each layer you have multiple ones of these heads for each head they look at over the course of the whole data set how do these softmax distributions look like so when you believe that this is a hopfield network and you believe that this converges in


one step then if the patterns are well separated what we would expect is a distribution as we said like this okay there would be one dominant pattern that you retrieve you know that's what you want to retrieve that's what comes out but a bang um you retrieve that accurate pattern anything else would mean that the hopfield network sort of failed right it wouldn't give you back


one particular pattern so they have i think that's a pretty it's a pretty smart experiment they look how many bars do we need to add how many of these bars in the soft max distribution do we need to add to reach 90 percent right so it depends a bit on the temperature of the softmax which is hardcoded and attention mechanism pdb is one this squared over d um so they say how many do we need


to add to get to 0.9 to 90 percent of the mass of this distribution and if this is the hopfield network where you retrieve one pattern then one will be enough right one of these bars will probably be i don't know like 99 okay but there are other cases imagine the case where the patterns and the query you retrieve the spheres that it gives rise to are all like


overlapping okay so what that will do is it won't converge to any particular pattern but the attractor space in this kind so you can imagine if you have two spheres that are apart from each other the update rule converges either so if it's closer to here it converge here if it's closer to here it'll converge here but if they are overlapping like this the energy


landscape will actually make it such that it will neither if it starts somewhere it will neither converge to here nor to here it will actually converge to somewhere in the middle okay into the mean of the stored patterns and if we take that to the extreme what could be is it could be that the softmax distribution looks completely uniform okay which would basically mean that


you know i don't care where my information comes from just average and this has its applications so if you for example want to make a sentiment classifier a very cheap way to do that is to simply take pre-trained word embeddings like glove or word to back you know assign each word word embedding and then just average the word embeddings okay and you count on the fact if there


are a lot of kind of negative words in there like bad sad angry the word embedding kind of will you know reflect that and the average word embedding will point more into the bad direction and if there's a lot of happy words the average will point into the happy direction okay so there are applications of averaging information not caring particularly where it comes


from and um in that case what we'd expect is that this number and we'll call that so we'll call that the number k in this case it equals one but in this case k equals i guess n the number of inputs okay because we need well not maybe n but you know approximately we need almost all of them to uh to reach the 90 percent okay and there there is an in between and these are called these meta stable


states where and the in between is something like you'd have a couple of patterns here a couple here and a couple maybe here it's almost like a clustering like and these overlap and these overlap and these overlap but they don't overlap with each other which means that if you start somewhere here you would converge to the mean but not to the mean of all the patterns but


just to the mean of these patterns and here here and here here so this this is like a clustering in latent space right so you can interpret these hopfield update rules as somehow you know getting not going to a particular pattern but going to sort of a cluster and this is if you ask something like hey is there any adjective around right and all of these patterns they


kind of overlap in that space in that query space of adjective they overlap and therefore the update rule would converge to sort of the mean which would basically say yes there is an adjective here right and the information would not be routed so that the distribution if we start here right and we converge to this the distribution would look something like small small small and then you'd have a


couple of large ones right you'd have like maybe two or three or four of large ones and these would exactly correspond to the patterns here so the information will be routed from all of those in that cluster to this particular node that asks the query okay these are these are what's called these meta stable states and what they do is they calculate over the entire data set this number


k and here they show you the distribution so in these plots what you'll see is over the entire data set um k goes into that direction so i guess let's go to tis here this this seems pretty easy so k uh is in this direction and this is simply the amount of like how so in each you you let a data point run through it you measure k for that particular layer one you see this is layer one head


four okay this is one layer one attention head and then you can see that the number k is distributed like this okay so contrast this to this head right here where it's a lot of weight on the number one or like very few numbers okay so these blue ones would be these are your typical like when you retrieve one particular pattern so this attention head we can sort of


conclude in this particular tension head this is very specific it looks at its input it looks at its token and it decides what information do i want and it retrieves one particular thing from the other nodes okay whereas here it's more like kind of an an averaging it's more like i want this kind of information and on average i don't even know what the sequence length is here


i guess it's maybe 512. uh so of the 512 the median this number is always the median in median it collects information from 231 of them okay so you can see that this corresponds this green and orange ones correspond to these meta-stable states where uh there's kind of an implicit clustering done in the in this space of attention whereas the blue ones they correspond to


attention heads that ask for particular information retrieve one particular maybe a few patterns and um happy with that and the red ones here you can see that they often just average they just you know because k is so high means that i need all of the i need all of these bars to get to the 90 or i need almost all of them which basically means it's a uniform distribution right so


it's like i don't care where information comes from just average whatever average i just want the average you know some particular uh space and as we said that also has its uses interesting how this translate through so this here is as we go down the vert model on the bottom you have layer one you see there are a lot of these averaging operations going on so a lot


of the heads are simply doing averaging and as you go up the layers the heads get more and more specific in the types of information they seek but then again in the last layers interestingly you get into a lot of these meta stable states again which i guess again interpret this as you as you want i'm going to leave this up to you but it sort of says like here you want


kind of general patterns at the bottom and then the middle layers are kind of the logical workhorses so you look for very specific things in the input this is i guess this is where i guess this is where the thinking happens um so this is sort of pre-processing i'm just making stuff up here by the way this is this must be in no way true this is maybe thinking and


this this here this might already be output again because you know after that you have language modeling or classification so this might already be like aggregating uh types of information this is how i sort of interpret it okay uh yeah so so this these these experiments are pretty pretty pretty interesting and here they have they do these are the last experiments for this paper


um they do an interesting experiment where they actually replace the attention heads by simply an average mechanism and later they actually replace them by gaussians but in this case they simply average and they show that look if i replace layer one with just averaging the perplexity doesn't rise that much right so it's pretty good um even if i replace an entire layer here with


averaging uh it it perplexity goes more up and you can see the corresponds if you remember the previous plot the correspondence is pretty one to one with how much blue and green uh heads there are as in contrast to how much red uh and orange ones there are so here you have lots of blue ones and you can see that the error kind of goes up and interestingly here you


have more meta-stable states at the end but still the perplexity goes up uh more so i guess you can only really replace the red ones with the averaging so this is always averaging in one particular layer and they go into more detail here where they say look this is this is layer six and this is layer 12. so this is one particular tension head from layer 6 and


layer 12 and the updates don't be confused it goes in this direction okay i was confused at first and you can see right here this number k at first you know it's kind of spread out but then it pretty quickly converges to a very small number and there is this kind of point right here i don't know if the learning rate's decreased i don't think so i think that's just kind of a


a phase transition right here this is the blue line by the way the blue training line a face transition where all of a sudden these just these attention heads they somehow decide okay this is the thing i want to specialize in this is the type of task i want like a sub task of linguistic subtask i want to specialize in and then they concentrate on one particular pattern per input so


they are really specializing whereas in the last layer you see here that even during training they are sort of continuously learning so first they also do this averaging then they go into this meta-stable region right this is this meta-stable region k isn't one but also k isn't a very high number um so they continuously learn and it's even indicative of this


training might not be done here first of all and second of all it would be really interesting to see how this works out with you know sizes of transformers and like especially these these huge transformers just the fact that they can keep learning the more we train them might be you know be interpreted in the light of what kind of states they converge to and the fact that there are tension


heads i don't know how does this go on do they stay in the meta stable states because it makes sense to have metastable states as i said it makes sense to kind of cluster things or are they simply is this simply an intermediate step and if you go really far down they would actually also converge to the k equals one where they really specialize or maybe do we need


more attention heads for this i don't know it it's just i think this is just the the beginning of kind of research in this direction i think just this kind of number k um how it's how it's made it's pretty simple and apparently it's pretty pretty revealing so you know that's pretty cool so that was the paper uh and its experiments it's it's a pretty sizable paper as i said even the paper