hi there today we're going to look at curl contrastive unsupervised representations for reinforcement learning by Aravind Sreenivas Michel Laskin and Petra Biel so this is a general framework for unsupervised representation learning for our L so let's untangle the title a little bit it is for reinforcement learning which it if you don't know what reinforcement learning is I've done a bunch of videos on are L afraid works so it's for general reinforcement learning that means it can be paired with almost any RL algorithm out there so we're not

CURL: Contrastive Unsupervised Representations for Reinforcement Learning


going to you know dive into specific or allowed rooms today it is unsupervised which means it doesn't need any sort of labels and it also doesn't need a reward signal forum RL which is pretty cool because usually the entire RL pipelines rely on some sort of a reward or auxiliary reward signal now there is a training objective here but it doesn't have to do with the RL reward and then in the it is learning representations which means it learns it learns intermediate representations of the input data that is useful and in the end


it is contrastive and that is the the kind of secret sauce in here the training objective it's what's called contrastive learning and that's what we're going to spend most of our time on today exploring what that means alright so here's the general framework you can see it down here sorry about that so you can see that reinforcement learning is just a box which is we don't care about the RL algorithm you use that's just you know what what comes at the end what comes at the beginning oh here is the observation so the observation in an RL


algorithm is kind of fundamental now if someone explains RL to you or reinforcement learning usually what they'll say is there is some kind of actor and there is some kind of environment right and the environment will give you an observation right observation Oh which is some sort of let's say here is an image right so in this in this RL framework specifically the examples they give are of image based reinforcement learning so let's say the Atari game where you have this little spaceship here and there are meteorites up here and you need to shoot


them so there is a little shot here right you need to shoot those meteorites right so this is the observation oh and then as an age as an actor you have to come up with some sort of action and the actions here can be something like moved to the left move to the right press the button that you know does the shooting so you have to come up with an action somehow given this observation and then the environment will give you back a reward along with the next observation like the next frame of the game and you're gonna have to come up with


another action in response to that and the environments going to give you back another reward and the next observation and so on so what you want to do is you want to find a mapping from observation to action such that your reward is going to be as high as possible right this is the fundamental problem of RL and usually what people do is they take this act this mapping here from observation to action to be some sort of function some sort of function that is parameterised maybe and nowadays of course it's often a neural network but


you're trying to learn given the input observation what output action you need to do and you can think of the same here so you have this input observation up here and down here after the reinforcement learning the output is going to be an action right and so this this function we talked about up here is usually implemented sorry is usually implement as you put the observation into the r.l framework and then the RL framework learns this f of theta function to give you an action now here you can see the pipeline is a bit different we don't


want to shove the observation in directly right we don't want the observation directly but what we put into the RL framework is this Q thing now the Q is supposed to be a representation of the observation and a useful representation so if we think of this of this game here of this Atari game up here what could be the what could be a useful representation if if I had to craft one by hand how would I construct a useful representation keep in mind the representation the goal is to have a representation of the observation that is more useful to the


RL algorithm than just the pure pixels of the image right so if I have to craft a representation let's say it's a vector right let's say our our our representations need to be vectors what I would do is I would probably take the x and y coordinates of the little spaceship right x and y and put it in the vector that's pretty useful and then I would probably take the x and y coordinates of the meteorites that are around right let's say there are maximum two XY XY here I would probably take the angle right the angle where my spaceship


is pointing to that should be pretty useful because if I shoot I want to know where I shoot right so theta here and then probably maybe the X and y coordinate of the of the shot here of the red shot that I fired if there is one right also going to put that into my representation so x and y and maybe Delta X Delta Y something like this right so you can see if I had to handcraft something if I I can pretty much guarantee that if I put in this representation right here into the RL algorithm but put this in here it would turn out


guaranteed it would turn out to be a better or L agent that learns faster than if I put in the original observation which is the the pixel image of the game right because of course in order to play the game correctly in order to play the game to win you need to extract this information right you need to get our there's something like a spaceship there's something like meteorites this is all things that are elegant doesn't know her say and would have to learn from the pixels right but if I already give it the information


that is useful it can learn much faster all right so you can see if I handcraft a good representation it's pretty easy for the RL algorithm to improve now we want to come up with a framework that automatically comes up with a good representation right so it alleviates the RL algorithm here that reinforcement it alleviates that from learn from having to learn a good representation right it already is burdened with learning the what a good action is in any given situation right we want to alleviate it of the burden to also


extract useful information from the from the observation space right so how do we do this this is Q here is supposed to be exactly that it's supposed to be a good representation but not one that we handcrafted but a used with a technique that can be employed pretty much everywhere and the goal sorry that the secret sauce here is this contrastive loss thing okay this bombed contrastive learning is this this kind of magic thing that will make us good representations so what is contrastive learning in this case I'm


going to explain in this case for this kind of image based for image based reinforcement learning but just for image based neural networks how can we come up with a contrastive loss so you see there's kind of a two pipeline thing going on here there is like this and this and then one of them is going to be the good encoding all right so let's check it out let's say we have this image that we had before right draw it again this little spaceship this and this and so right and we want to we want to do this what we


need to do is we need to produce three different things from it we need to produce an anchor what's called an anchor so we need to produce a positive sample positive sample and we need to produce negative samples let's just go with one negative sample for now right so the goal is to come up with a task that where we produce our own labels right so we want since we're training a encoder and the encoder is a neural network that's parametrized we need some sort of loss function so the goal is to come up with a method where we can


create our own labels to a task but that we construct the task in a way such that the neural network has no choice but learn something meaningful even though we made the task of ourselves all right I hope this was kind of clear so how are we gonna do this our method of choice here is going to be random cropping now random cropping means that I just I take an image right and I crop a a piece from it so a smaller piece from the image I just take a view inside the image so in case of the anchor right I'm gonna draw the same picture here bear with me


I'm gonna draw the same picture here a couple of times this is all supposed to be the same picture and with the negative sample I'm just gonna leave it empty for now there are two meteorites two meteorites shot shot right so for the anchor we're going to actually not random crop but center crop right so we're going to take here the center image right so the assumption is kind of that if I Center if I Center crop I won't lose you know too much of the image I can actually make the crop bigger such that almost everything of


the image is somewhat contained in this and that yeah all right so this is going to be my anchor and then the positive sample is going to be a random crop of the same image so I'm just randomly going to select a same size same size section from that image let's say this is up right here all right and the negative sample is going to be around the crop from a different image right so a different image might be from the same game right but might be there is a meteorite here right and there is no shot I don't I don't shoot and I'm going


to take a random crop from this let's say I'm going to take a random crop here let's put a meteorite here as well just for fun all right so these are going to be our three samples and now the question is going to be if I give the anchor to the neural network I'm going to say I give you the anchor right but I'm also going to give you this and this thing and I'm not going to give any of this I'm just going to give whatever I cropped right so just just these things so I asked the neural network neural network I give you the


anchor now which one of these two which one of these two crops comes from the same image right so as human you look at this and if you just see the center crop you see oh okay down here there's this this tip of this thing and then there's the shot right and in relation to the shot there is a meteor here right and then you look at the second one and you say okay I don't see the spaceship but there's the same relation here from the shot to the meteor and I can kind of see the meteor up here and this also fits with that right and the the spaceship


must be you know down here somewhere and then I go over here and I try to do the same thing is okay here's the meteor and you know it it might be it might be in the original image it might be over here somewhere so that's possible I don't see it right that's possible but then there should be there should be a shot right somewhere here or sorry further up oops T there should be a shot somewhere here right I'm pretty sure because there's there's one over here and I don't see it right so I am fairly sure mr. tasks asked her that this image here


is the positive sample while this image here is the negative sample right so this is the task that you ask of the neural network give it the anchor and you ask which one of the of these two comes from the same image right this is called contrastive learning now is a bit more complicated in that of course what you do is you encode these things using neural networks and then so each of the things you encode so the anchor you're going to encode all of these things using a neural network right and then this is what's going to


become the query and these are becoming the keys so key one or key two and then you're going to feed it always two of them into a bilinear product right the bilinear product is simply you can think of it as an inner product in a perturbed space that you can learn so you're going to have this you have these two here these go into q WK one and then these two here sorry this and this go into q w k 2 now W here is a learnable parameter right so you have some freedom and then you basically take whichever one of those two is highest right so this might


be this high and this might only be this high and then you say aha cool this one's higher so this one must be the positive right and you train the W specifically to make this higher to make the positive ones higher and the negative ones a lower so this is a supervised learning task right where these things here are going to be the lockets or or the so their inner product but you basically then pick the one that is highest as a in a soft max way and they put this in the paper so if we go down here the objective that they use to


do the contrastive learning is this one so as you can see it's a soft max like in multi-class classification of the inner product the bilinear product with the positive samples over the bilinear product with the positive samples plus the bilinear product with all of the negative samples so you're going to come up with more than one negative sample all right now the only thing left that we don't have here is that the encoding how you're going to come from the image space to this space here is going to be slightly different and depending on


whether you're talking on the anchor or on the what what are called the keys the things you compare to and this is out of a kind of a stability criterion you already maybe you don't you know like something like double q-learning or things like this it sometimes when you train with your own thing so in q-learning you're kind of trying to to come up with an actor and a critic or it's not the same thing but you're kind of using the same neural network twice in in your in your setup and then you compare the output stored to each other


which isn't you know it leads to instability so in our case we took it three times here or multiple times especially for the same objective here we have twice something that was encoded by the same neural networking isn't the two sides of this by linear product so if we were to use the same neural network that tends to be somewhat unstable so we have different neural networks one that will encode the query which is this F Q + 1 which will encode the keys sorry F ok now we don't want to learn to neural networks and that's why


there's a bit of a compromise where we say it is the same neural network but but basically this one is the one we learn and then we always every now and then we transfer over the parameters to that one and in fact each step we transfer over the parameters and do an exponential moving average with the parameters of this momentum encoder from the step before so the momentum encoder parameters are a moving average of the parameters of the query encoder and that is so you get kind of get the best of both worlds you don't


have to learn a second neural network but your second neural network is not the same as your first neural network but it it kind of lags behind but it is also it is also performing almost as well so that is um I don't know if that makes sense but it is the best I can to explain it so to recap you take your observation you encode it as a query sorry you crop crop here for your anchor that gets your query and then you random crop for your keys right into positive and negative samples right so random crop from the same observation or from


different observations right these become your positive and negative samples then you take you take me push this through your encoders for the query and for the keys respectively you end up with the Q which is the encoded anchor and the case which are the encoded positive and negative samples and then you learn you update this encoder here using the contrastive loss right and at the same time you feed the q you feed the q here into the reinforcement learning algorithm and you learn your reinforcement learning algorithm instead


of giving having the observation directly as an input here you now have the Q here as an input right that is it the reinforcement learning works exactly the same but except having the so input Oh you now have the representation input queue and you don't have to worry about anything else in terms of the reinforcement learning algorithm it stays exactly the same right the this whole thing here can actually run either in parallel or you can think of it before you can think of it off policy on policy it is sort of


modular how you how you fit this in it simply comes up with good representations so that is that is basically a deal here right and you hope that the whole procedure of this contrastive learning then gives you good representation of this anchor thing here if you encode that to the queue you hope that this representation now is a good representation as a basis for the RL algorithm and it turns out at least in their experiments it is so here you see the same thing they actually they do something more where in RL usually deal


with a stack of observations not just a single observation because so for example in Atari people always concatenate something like for the four last frames right and their their point is okay if we have this stack here if we do this data augmentation you know these crops we kind of need to do them consistently right we need to crop every single image at the same point for the query and also if we do a random crop let's say a random crop down here we need to do this same random crop for all of the of the stack of images here right


so um that that is kind of the additional thing they introduced it with respect to RL that deals with with stacked time frames but it's kind of the same the same diagram as above here right so they explained the the RL algorithms they use and exactly they're they're their thing and here you can see that anchor is a crop and the positive sample is a random crop from the same image this would be up here somewhere the anchor is cropped from the middle and then the negative would be a random crop from a different


image or a different stack of images they have a pseudo code here where that was pretty simple we'll just go through it quickly right you start off with fq + FK these are the encoders for the query and keys you start them off the same then you go through your data loader you do this random augmentation of your query and you keys and I don't not even sure if the random augmentation needs actually to be a central crop for the anchor or just two different two different crops from the same image that might be as


well so you know just I guess I guess it's a thing you could choose I don't know what exactly is the best thing alright then I forward the query through the FQ and I forward the keys through the FK then important I detach this so I don't train I don't want to train the FK I only want to train the FQ right then I do the bilinear product here with the W these these are the bilinear product and then I put this all of this into a cross entropy loss right in the end I update my fq m IW and i do this exponentially


moving average for my key encoder and they test on two different things they test on the deepmind control tasks and they always test 100 K time steps so their big point is data efficiency right they they claim they can use learn useful representations with not much data so the task is here how good are you at one 100 cases that time steps right so you don't you don't optimize until the end you just you get 100k time steps and then the question is how how good are you and the curl here outperforms all of the baselines handily in the deep mind


control tasks and it also outperforms a lot of the baselines in the Atari tasks and it actually if you look at the results it doesn't outperform everything but for example here the red is curl and the dashed gray is state as a si now state si si the important thing to note here is it has access to the state whereas curl only works from pixels right so that what I said before like if I had to craft the use for a presentation basically state si si has access to that and you see that in many of the tasks that the curl comes close


or performs equally well to the state si si right so that's pretty impressive especially if you've took at pixel si si sorry which is the same algorithm but does not have access to the state just the pixels it often fails terribly right so um that is pretty interesting to see and even to me it's pretty interesting to see that this kind of this kind of algorithm this kind of self labeled algorithm comes up with such useful representations all right so I hope I have explained this satisfactorily and check out the paper for more experiments


on April 21st jurgen schmidhuber tweeted out stop crediting the wrong people for inventions made by others at least in science the facts will always win at the end as long as the facts have not yet won it is not yet the end no fancy award can ever change that hashtag it self-correcting science hashtag plagiarism and links to an article of his own website where he wrote critique of Honda Prize for dr. Hinton so this is on Schmidt Hoover's own website and it's by himself and don't you love this how to pronounce his

[Drama] Schmidhuber: Critique of Honda Prize for Dr. Hinton


name jurgen schmidhuber you again sorry this is this is absolutely great so both actually Schmid over and Hinton are on Twitter you can tweet at them and follow them this article here is a basically a critique of the press release of Honda when they awarded geoff hinton for his achievements and it goes through it step by step and we won't look at the whole thing but just two for you to get the flavor so here honda says dr. Hinton has created a number of technologies that have enabled the broader application of


AI including the backpropagation algorithm that forms the basis of deep learning approach to AI and schmidhuber just goes off its he basically claims him while Hinton and his co-workers have made certain significant contributions to deep learning he claimed above is plain wrong right he did not invent back propagation the person who invented back propagation was settled in linear MA and the many papers he says basically many papers failed to cite linin MA and this who was the original inventor of back prop and so on and he go kind of goes


through a history of this and how it's even earlier I always have a bit of a trouble with claims like who invented what because when it is an algo them really the same thing right and when he when is it a variation on another algorithm and when is it something completely new it's never entirely clear but the the points here made that the things the backpropagation algorithm existed before Hinton and also that some of the papers some of the seminal papers did not cite the correct origin statement to in 2002 he


introduced the a fast learning algorithm for restricted Boltzmann machines that allowed them to learn a single layer of distributor representation without requiring any labeled data these methods allow deep learning to work better and they led to the current deep learning revolution and he is no dr. Hinton's interesting unsupervised pre training for deep neural networks was irrelevant for the current latif learning revolution in 2010 our team showed that the feed-forward networks can be trained by plain backprop do not at all require


pre training and he basically again says apart from this Hinton's unsupervised pretending was conceptually a rehash of my unsupervised pre training for deep recurrent neural networks so he you know as you know she made Ober has done a lot of work in recurrent neural networks and he basically says it it was just a rehash of his algorithm now I I have to say I have so first look first of all he he makes a point here right that we don't really do unsupervised pre-training him or until now of course but you like for to train an amnesty law


fighter you don't have to do that but it's also doubtful that this this was a step even though even if it wasn't on the exact path to the current situation it was a thing that got people excited maybe and so the critique is like half valid and also it doesn't help me to burn that he always compares it to his own things like it just it just like either criticized them for you know in general things but then avoid bringing your own things in because it just sounds like I did this before and also I read some papers


from from these times people just wrote papers sometimes I haven't read this specific one but sometimes people just wrote papers writing down their ideas like one could do this and this and this never doing any experiments or actually specifying exactly what they mean they just kind of wrote down a bunch of ideas and that got published especially like there's some some reinforcement learning papers where people are just like oh one I imagine agents doing this and learning from that so it is again it is never


really clear in ideas or just had by everyone I think people people mistake this that think that the ideas are unique it's not ideas that are unique many people have the same ideas but some there's also execution and exact formalization and so on and exact level of specificity this all of this is really hard and then the honda says in 2009 dr. Hinton and two of his students used multi-layer neural nets to make major breakthrough and speech recognition that led directly to greatly improved and this of course Schrader who goes off by


this because speech recognition is of course prime LS TM territory so you don't want to go near this and the Honda further says revolutionized computer vision by showing that deep learning worked far better than existing state of the art and again he says the basic ingredients were already there and so on and the our team in Switzerland already used his first superior award-winning GPU based CNN and so on that's what it's called dan net was produced by his group and again this seems correct right this seems when he lays it out like this but


it doesn't change the fact that Alex net1 imagenet in 2012 and that was like the start of the deep learning revolution it was like wow you can cut the learn like the error rate by something like 30% simply by doing this deep learning stuff so again even if Dan that he says it blew away the competition it just seems it it always seems like Schmidt Hooper's kinda right but then also he's not he's like a cadet exact academic work and and the idea being there on a paper isn't the only thing that drives progress and says to achieve


their dramatic results dr. Hinton also invented a widely used new method called dropout which reduces overfitting no like no and like no just no like randomly dropping parts in order to make something more robust that is surely not a new thing and he also says much early it is there's this stochastic Delta rule and so on and he also critiques that this paper did not cite this they just gave it the name right this is an idea that is kind of so simple that you you wouldn't even necessarily think about researching whether that has existed


already I think they just did it and then because it's a natural idea and then they gave it a name and the name stuck right it's not about the idea itself and then lastly they say of the countless AI based technological services across the world it is no exaggeration to say that few would have been possible without the results dr. Hinton created I love this name one that would not have been possible and he just gives a list of their own group and that are basically possible without Hinton's contributions and this is just it's a


bit of a cheap shot right clearly honda if they're not saying it would have been you know physically him possible without his contributions its but certainly Hinton has has if even if he hadn't invented any of those things he certainly has created like a spark and his these things created a splash got people excited people thinking about new ways of applying things even you know if this is all true so right and but but I would like you to I'd like you to notice this is a critique of what Honda says about Hinton and if I read


through the statements of Schmidt who were most of them are technically correct right and you know that so that was that and then I thought okay cool but then someone posted II didn't read it and then Hinton replies and this is okay don't you love this so Hinton says having a public debate with schmidhuber about academic credit is not at advisable because it just encourages him and there is no limit to the time and effort that he is willing to put into trying to discredit his perceived Arrivals he is even escorted to tricks


like having multiple aliases in Wikipedia to make it look as if other people agree the patient on his website about Alan Turing is a nice example of how he goes on trying to these are like these are shots fired and he says I'm going to respond once and only once I have never claimed that I invented backpropagation David Romo hard invented it independently after other after other people in other fields had invented it it's true when you first published we did not know the history so he basically says okay we did forget decided when we


first published about rock crop but he doesn't say he invented it what I've claimed is that I was the person to clearly demonstrate that back prop could learn interesting in turn represent and that that this is what made it popular right so this goes into into the direction schmidhuber is very much on academic contributions idea was there before and hint and basically says no what we did is kind of we showed that it works in this particular way and we can have got people excited about it I did is by forcing that blah blah blah and it


is he says it is true that many people in the press have said I invented back prop and I've spent a lot of time correcting them here's an excerpt from 2018 where this is I guess a quote from this book that quotes Hinton where he says lots of people invented different versions of back prop before day with normal heart they were mainly independent inventions something I feel I've got too much credit for it's one of these rare cases where an academic feels he has got too much credit for something my main contribution was to sure you can


use it for learning distributed representations so I'd like to set the record straight on that and then he said maybe Jurgen would like to set the record straight on who invented LST M's boom boom crazy shot shots fired by Hinton here this is I mean this is just great but again look at what Hinton says Hinton basically says yes I have not invented that I have corrected this on public record in the past and yeah so so that's what Hinton says and I mean the the the comments here are just gold I really invite you to read it and then


schmidhuber of course being Schmidt who replies again down here he has a a response to the reply and I don't expect Hinton to reply again so I waited for a bit but but I I believe him when he says he does it only once so he goes into this summary the facts presented in sections 1 2 3 4 5 are still valid so he goes what kind of statement by statements is having a public debate blah blah blah and he says this is an ad hominem attack which is true right this is true and he says he even has multiple aliases in Wikipedia


and he just says another ad hominem attack and then he goes into that schmidhuber tries to discredit Alan Turing and then shmita goes into this big long big long basically claim that Alan Turing wasn't as important as people made him out to be and people invented this kind of Turing machine equivalents before that again it's kind of showing tubers take that the idea basically was already there and these people don't get the correct credit and also he's correct that this is a this is a true it's an ad hominem attack right so you know be it


as it may this is correct and then when when Hinton goes that he doesn't stay and invent backdrop and me to persist this is finally response related to my post which is true right however he does not at all contradict what I wrote and it is true that he credited his co-author Rommel Hart with the invention but but neither cited alanine MA and also the statement lots of people he says it wasn't created by lots of different people but exactly one person so this I find como like can you really say now this is the


exact time when backprop was invented even though it probably wasn't in the current exact current formulation and it probably existed someone like this so but again and he his main claim is dr. Hinton except the Honda Prize although he apparently agrees that Honda's claims are false he should ask Honda to correct their statements and like in the end maybe you're going would like to set the record straight who invented LST M's and you know as we as you may know seppo writer it kind of invented LST ms under jurgen schmidhuber


as a as a PhD advisor but the to summarize dr. Hinton's comments and ad hominem arguments diverged from the contents of my post and do not challenge the facts and so on and i have to say after reading this this this is a this is correct right hinton basically replies to hey i I never claimed I invented back prop and other people have invented it and Schmidt Hoover doesn't criticize hinton in this particular post he may otherwise schmidhuber doesn't create as as Hinton for claiming that he criticizes Honda for claiming that


Hinton did and it doesn't hidden basically agrees with him and also schmidhuber says dr. Hinton accepted the Honda Prize although he apparently agrees that the claims are false he should ask Honda to correct their statements and it is true that Hinton accepted this price under this release right now you might be able to say him Hinton also says he's on the record basically saying he didn't do this and I guess if you're Hinton and you know you've had this you've had a successful career and so on and you have previously


really publicly stated that you didn't invent these things and you know made it clear and then you get a prize and they write this thing maybe you just don't want to go after every single press statement and correcting that but you know in essence basically Hinton and understood this as an attack on himself that he claims he invented back prop and schmidhuber says Honda claims he invented back rub and Hinton accepted the price so agrees with it and he basically agrees with it but doesn't say Honda should have corrected at which I


can understand so this is my take on this issue it's kind of both or correct and they just kind of talk past each other and schmidhuber is always on the the idea existed before and Hinton is correct when he says it's not always just about the idea progress is also made by people being excited people actually getting something to work people you know doing something at the right time in the right place which is also correct but it is fun it is fun so so I just I enjoyed I enjoy this honestly like because ultimately this is


The power of yet. I heard about a high school in Chicago where students had to pass a certain number of courses to graduate, and if they didn't pass a course, they got the grade "Not Yet." And I thought that was fantastic, because if you get a failing grade, you think, I'm nothing, I'm nowhere. But if you get the grade "Not Yet", you understand that you're on a learning curve. It gives you a path into the future. "Not Yet" also gave me insight into a critical event early in my career, a real turning point. I wanted to see

The power of believing that you can improve | Carol Dweck


how children coped with challenge and difficulty, so I gave 10-year-olds problems that were slightly too hard for them. Some of them reacted in a shockingly positive way. They said things like, "I love a challenge," or, "You know, I was hoping this would be informative." They understood that their abilities could be developed. They had what I call a growth mindset. But other students felt it was tragic, catastrophic. From their more fixed mindset perspective, their intelligence had been up for judgment, and they failed.


Instead of luxuriating in the power of yet, they were gripped in the tyranny of now. So what do they do next? I'll tell you what they do next. In one study, they told us they would probably cheat the next time instead of studying more if they failed a test. In another study, after a failure, they looked for someone who did worse than they did so they could feel really good about themselves. And in study after study, they have run from difficulty. Scientists measured the electrical activity from the brain as students confronted an error.


On the left, you see the fixed-mindset students. There's hardly any activity. They run from the error. They don't engage with it. But on the right, you have the students with the growth mindset, the idea that abilities can be developed. They engage deeply. Their brain is on fire with yet. They engage deeply. They process the error. They learn from it and they correct it. How are we raising our children? Are we raising them for now instead of yet? Are we raising kids who are obsessed with getting As? Are we raising kids who don't know how to dream big dreams?


Their biggest goal is getting the next A, or the next test score? And are they carrying this need for constant validation with them into their future lives? Maybe, because employers are coming to me and saying, "We have already raised a generation of young workers who can't get through the day without an award." So what can we do? How can we build that bridge to yet? Here are some things we can do. First of all, we can praise wisely, not praising intelligence or talent. That has failed. Don't do that anymore.


But praising the process that kids engage in, their effort, their strategies, their focus, their perseverance, their improvement. This process praise creates kids who are hardy and resilient. There are other ways to reward yet. We recently teamed up with game scientists from the University of Washington to create a new online math game that rewarded yet. In this game, students were rewarded for effort, strategy and progress. The usual math game rewards you for getting answers right, right now, but this game rewarded process.


And we got more effort, more strategies, more engagement over longer periods of time, and more perseverance when they hit really, really hard problems. Just the words "yet" or "not yet," we're finding, give kids greater confidence, give them a path into the future that creates greater persistence. And we can actually change students' mindsets. In one study, we taught them that every time they push out of their comfort zone to learn something new and difficult, the neurons in their brain can form new, stronger connections,


and over time, they can get smarter. Look what happened: In this study, students who were not taught this growth mindset continued to show declining grades over this difficult school transition, but those who were taught this lesson showed a sharp rebound in their grades. We have shown this now, this kind of improvement, with thousands and thousands of kids, especially struggling students. So let's talk about equality. In our country, there are groups of students who chronically underperform, for example, children in inner cities,


or children on Native American reservations. And they've done so poorly for so long that many people think it's inevitable. But when educators create growth mindset classrooms steeped in yet, equality happens. And here are just a few examples. In one year, a kindergarten class in Harlem, New York scored in the 95th percentile on the national achievement test. Many of those kids could not hold a pencil when they arrived at school. In one year, fourth-grade students in the South Bronx, way behind, became the number one fourth-grade class in the state of New York


on the state math test. In a year, to a year and a half, Native American students in a school on a reservation went from the bottom of their district to the top, and that district included affluent sections of Seattle. So the Native kids outdid the Microsoft kids. This happened because the meaning of effort and difficulty were transformed. Before, effort and difficulty made them feel dumb, made them feel like giving up, but now, effort and difficulty, that's when their neurons are making new connections, stronger connections.


That's when they're getting smarter. I received a letter recently from a 13-year-old boy. He said, "Dear Professor Dweck, I appreciate that your writing is based on solid scientific research, and that's why I decided to put it into practice. I put more effort into my schoolwork, into my relationship with my family, and into my relationship with kids at school, and I experienced great improvement in all of those areas. I now realize I've wasted most of my life." Let's not waste any more lives, because once we know


so then let's get started for today welcome to lecture 10 of cs2 9458 deep unsupervised learning now this lecture will be on compression before we dive into that a couple of logistical things there are main logistical things that are ahead of you are your project milestone which is a three-page Goldbach intermediate report is due on Monday so we must look forward to reading those giving you feedback in the days after the deadline so you can gonna make sure you're maximally on track for your full final project the

L10 Compression -- UC Berkeley, Spring 2020, CS294-158 Deep Unsupervised Learning


other thing that's coming up in two weeks we'll have our midterm which will figure out how to do it remote under the current circumstances but the main thing we'll do later this week is release a set of study materials for you that capture the core of the things covered in the class their very core compressive a little bit because of how much we're going to have you study because of course a more difficult semester than most you do outside circumstances so a relatively short study guide and it'll be a PDF with the questions and the


answers and so you'll know exactly what the questions can be and what the answers are that we expect you to get so that will come out later today or tomorrow for you to study link pause here and see if there's any questions about logistics oh and by the way this lecture is recorded so for some reason you you know don't like your voice to be heard just like with the in-class lectures that were recorded then please be aware of that alright then let's get started with the Contin first day so compression what is


it and why would we care in general and why would we there in this class so what is it it's an data you might want to reduce the number of bits for encoding a message a message could be an image you want to send or part of speech or maybe some music you want to send across on communication line and it's an original in a collisional format might take up a very large number of bits and you might want to be able to get that same information across by seeing last bits in the communication channel so what does it


look like you have some bit stream B on the left here so that's where you start out with then what happens next is you want to compress it and end up with a compressed version of that bit stream and the hope that that compressed version has lost its anatomy original so when you send a compressed extreme over a channel or stored on a hard drive or whatever you want to do with those bits in a more compressed way then it's ideally a lot less good but then when you want to use it later you should be able to expand it back out just the


compression into the original alright so why do we care well you could save time you could save bandwidth over communications channel you could save space when you're storing it so many reasons you might care about this from the AI point of view and part of why it's interesting for this class is that often the ability to compress data reflects understanding of the data by the system that compressed the data so if you throw this in that's really by compressing data that means that system somehow has absorbed an understanding of


the data so now there's two types of compression lossy versus lossless compression in this lecture we'll be fully focused on lossless compression where the original bits can be completely reconstructed on the output now sometimes in practice you might care about lossy compression you say well I don't need all the details back as long as I can save more bits I'm happy to lose some detail that would be loss of compression not the topic for this class but also a topic you might be interested in at some point so I want to make sure


you know it exists now one of the very interesting things with compression there are some prizes associated it so recently hutter I should increase the price used to be a 50,000 euro prize for compressing human knowledge and recently it went up to factor 10 it's now a five hundred thousand euro prize if you can compress human knowledge what does it mean more concretely so there's a one gigabyte file of I believe text and this file here and with nine and if you can compress that to less than one hundred sixteen megabytes you win the prize you


won this thing you cracked it the reason how to read out the surprise is not so much because it specifically wants that one gigabyte compressed into one sixteen megabytes because he believes now we go by it has interesting information that any system that can represent it as compactly as one sixteen megabytes must have made its hopefully that's what it thinks some AI advances to be able to do that it's pretty interesting here because unlike most things we've covered in this class and you'll see in any kind of machine learning there's no train at


st it's not that he asked you to send in a compressor and has a secret test set is gonna test your compressor on to see how it works no it's literally there's a 1 gigabyte file and if you can make it smaller small enough you win the prize what's gonna be able to decompress it so you gotta be able to effectively send him something that's 116 megabytes or less and includes the code for decoding back into the one gigabyte so you'd be sending effectively both the decoder program and some encoding of this monkey bad file together it would be able to


reconstruct the original 1 gigabyte file so very very specific problem there's no test said just that one training example but nobody's got them close to actually making this work so interesting challenge maybe something you want to think about at some point and see if it can make some progress then there's another compression challenge on images so this is often held at CDR the main conference for computer vision and so there's a workshop there that looks at how well your compressor and there it's really about a compressor that you send


in compressor of course and they have a secret asset on which they test how well you can compress and decompress the test examples so two very different challenges but both very much at the core what we're going to be thinking about today in lecture all right so why in this course it turns out that we've studied a lot of generic models in this course it also turns out that compression utilizes generative models so the better it narrative model the better the compression can be and in fact Jonathan who will cover the second


half of this lecture has made several breakthroughs in this PhD research showing how some of the state-of-the-art generative models can be converted into compression algorithms with the CIO narrative models under the hood such that you can get better compression now you might go get otherwise and we'll cover that later but so there's a very close connection between better generative models and better compression there be material we would recommend for this lecture is this PDF overview to the nice write-up that


covers the background on essentially information theory slack impression that we'll be covering in this lecture at least the first half in the second half will dive a lot more in the deep learning aspects and how and tied it into this so some applications you might have seen jarick file compression gz p z 7z a zip file systems various multimedia formats you might have seen a file - fake file gif file mp3 mp4 communications that maybe you don't see news anymore now but where compression played a big role in the


past fax modem Skype and so forth and all of these are examples of where here original information might have been represented with many many bits too large for you to a store on file in that format and because you can reduce them or they can get back out the original you can now store it more efficiently or send it more efficiently over get in line when he said it more official over communication line it can reduce both the amount of does he need to send then in the process also reduce the latency because it might be less


delay assuming you can decode quickly on the other side now maybe you might have followed this TV show called Silicon Valley it's uh well pretty finish I would say with many things that are maybe a little too close to home and too close to true but still pretty funny and if you watch that show on HBO you have noticed that actually the company the central company Pied Piper would they put forward as their product is the well a middle-out compression algorithm nobody knows what middle out this but they put forward a compression algorithm


and that's the secret sauce of their company turns out that some people really do this for their actual company so there's various startups out there that you invent don't disclose exactly what's under the hood but invent new compression algorithms using machine learning under the hood most likely to improve upon past state of garden compression now the specific point I shoulda named itself after the Silicon Valley show which was called the company's called Pied Piper and this is now called pact pie so there's actually


a real thing they presented at TechCrunch in 2015 now first question you might ask is can we have universal data compression so their fundamental question you'll see in this lecture a lot of questions we ask them to be very fundamental where we can give actually very very strong theoretical answers sometimes negative answers so can we come up with universal data impressive would that mean that would be can we come up with something that no matter what let's say filing you give it it can make it smaller and later decompress it


back out to the original well the things that's possible well okay let's see imagine you want to compress every possible to compress every possible possible bitstream they ever encounter okay so that's not possible no longer we can do this what's the intuition that should be simple we'll do a proof by contradiction suppose you have a universal data compression algorithm you that can compress every bit streams no matter what your feet it is gonna make it less bits and then go to decompress it back out later to the original okay now given


bit string B 0 you can compress it to get a smaller bit string B 1 if it strictly last bits otherwise it's not a universal compressor now B you want you can feed into it again it'll turn that into B 2 which is yet smaller you keep doing this you do this especially many times at some point you'll have a big string of size 0 at that point it's obvious you cannot recover what the earth will lost kiss you it could have been anything and everything gets turned into 0 you can get back out will int it so what this shows them assuming


somebody tells you I have a universal data compression already compress everything no problem here's a prove that this is actually not possible to prove it another way also another way to prove it is to do it by Counting you can say okay suppose your algorithm can compress all thousand histories ok how many thousand bit strings are there we throw the one thousand possible bit strings now if we can compress all them that means we can pick every one of them and turn them into something smaller and distinct smaller otherwise we cannot get the


original back out but if we look at what's possible with all possible shorter bit strings actually you cannot encode all two to the 1000 possible thousand bit strings so since we can't include all possible to the 1000 bit strings it means we cannot compress all of them so we have two different proves here to show that the universal data compression just not possible why is impossible and practice though even if you cannot universally compress everything well there are statistical patterns that you can exploit for


example here's a piece of text and I'll give you all a minute to read this text so as you're reading this text you'll you'll notice well likely you'll notice that there's something fun about the text and that the words are mostly misspelled but despite these was being misspelled it's how she's still very feasible to read this and effectively what it says is that most people have no problem reading a piece of text if for every word you keep the first two letters you keep the last two letters but then everything in between you can


randomly permute which means that the ordering of the letters in between has not much real information in it it could be any ordering so you don't need to stick to the original order in us here they do this crammed with we can still understand it so it means there's some redundancy it means that certain sequences are just not very likely and when you read this it's close to a sequence that you're familiar with and so you can easily map it onto that and still understand words that were there originally there's another example from


images so on the Left we see a bunch of real-world images of flowers in this case on the right we see random data the data on the left if your dataset looks like that it's very compressible because there are a lot of regularities at an intuitive level example often maybe pixels have roughly the same value whereas images on the right which are completely random there is no correlation between neighboring pixels that you can exploit to maybe compress how to represent the data on the right so two very different


distributions for the complete random distribution not clear how to compress for the real-world type data you can already intuitively see that there are opportunities to compress for example you could just think of every other bit now every other pixel it wouldn't be perfectly lossless we could probably reconstruct most of the image from that alright so what we've covered so far is what is compression what's the goal in compression and why might we care go from a practical point of view and from a AI point of view we also looked at the


fact that universal lossless compressor is just not possible we looked at some intuition as they're being redundancy in most of the data that we encounter in the real world and because there is redundancy I'm totally speaking there should be a way to exploit that because it's only the did that really occurs in the real world that you need to have a good compression for and the data that doesn't really occur in the real world even though they are also can be represented in principle bit strings you might not care much


about how much that kind of non real-world that gets compressed so for the remainder of this lecture we'll want to look at is a couple of things first thing we want to look at coding off symbols so we'll start looking at okay what does it mean to actually have a compression system and this will only comment with a method called Huffman coding that is actually used in many many of today's systems and it's actually quite intuitive very simple way to understand how compression could effectively work so I'm going to look at


some theoretical limits and from there we'll look at some additional consideration for coding that will help us a bit more than what we get from the simplest version we cover first from there we'll tie this into things that we've covered in this class we'll look at our aggressive models will look at Vee we'll look at flow models and try to understand how these models can be leveraged to do better compression all right let's get this started so here's one way of coding information all right a way and just to be clear there's


no compression and this wave coding so ASCII is a system that for everything that's on your keyboard will assign seven bits so every character you can type can be represented with seven bits so two to seven possible characters could be represented this way what's nice about this very easy for kody and decoded there's a very simple one-to-one mapping always going to the seven bits for that character and back out to the character but if you encode this way you're not exploiting its disco parents is Nagano it's not compressing your


information maybe some key strokes are far less likely than others so maybe the ones that are less likely you should allow it to use more bits and the ones that are very likely you should try to represent with a very small number of bits and overall you might have a win that's the intuition behind law compression schemes but obviously here everything seven bits that's not going to happen let's in at least a reference as a starting point so we'll need a spare be length codes codes down assign different lengths depending on how likely a symbol


is how do we avoid ambiguity a simple way to avoid ambiguity when you have variable lengths is when your fix times is very easy first seven bits per character next sum of its next character December's next target but if it's variable length how you know a character has been fully translated and now the next one is starting one way to do this is to ensure that no code word is a prefix of another code word so as you see bits come across the line at some point you'll have seen all the bits for some letter let's say and because no


code word prefix of another one at that point you know there's nothing else that can continue from this this is the thing is the complete thing sent across and the corresponding character can be decoded so another thing you can do but that might be consuming more bandwidth to space you could also end the stop character to each codeword the Morse decoding does this but this might be a little wasteful we can have a general prefix-free code and we'll look at that very soon so let's look at Morris first so in Morse code in what happens this


very old coding scheme this is where I'm back when you want to send let's say you just have a effectively a line the communication line over which all you could send was let's say voltage going on and back down you can make it go up briefly or go go up for longer for three times as long so a dot is a brief spike in or up in your voltage let's say and then a dash is three times as long and then the spaces in between the dot and dash are also encoding that it's it's quiet the reason it's a quiet time in between


and then between characters there will also be three so between total of three units and then between words there will be Sun units of quiet time so that way you can encode every character alphabet all numbers and all I need to be able to do is send dots and dashes or short and longer signals and pauses be able to get everything across and people use that before before telephones time to go up Telegraph where they could send information across this way so some of the things you can already see here the letter A as a relatively short encoding


same for e same for I and that's because those are frequent frequently used letters and then things that are less frequent like maybe a Z has a longer encoding what else is left frequent there J longer encoding essentially well more or less to the letter sound you get a lot of points for in Scrabble have the long encoding skew here X over here and the letters that don't give you much points in Scrabble have these shorter in CONUS because there's many more words that use them okay that's that's a very specific thing the more


general thing that people tend to use a cynical prefix-free codes which can be represented as binary tries or binary trees so what is a binary try or tree it's a tree where one thing's split you hard-code ahead of time that the left will be a zero the right will be a one and so you can build a tree and you don't even have to put the zeros and the ones on it that I'm putting on here because you always know those side is a zero the right side is a one that says their specific type of data structure the reason it's spelled with ie is


because comes from retrieval it's a data structure for easy retrieval of certain information so that's also why it's often pronounced binary trees because it comes from retrieval at the same time there's also threes this way that also our data structure and so it can be a little confusing if they're pronounced the same way so some people will still pronounce this as trust to distinguish from trees so that's a binary tree the way we're going to use it is that the symbols will always live in the leaves and a code word is a path from root leaf


so let's look at some example here's an example we have a code word table we have one two three four five six characters each character has an encoding with a sequence of bits of sometimes only one bit and you can see that there's a crisp honest adapt in this binary tree where all the characters are sitting in leaves of the tree because every character sitting in a leave of a tree what it means lets him I'm getting some message across say I'm receiving this message over here I'm receiving a zero what will I do I will


go down this path I'll say oh I hit a leaf that means I'm at the end nothing left to go ID code an A then I get this one over here we just restarted means good this way I get another one you this way get another one Musa go this way get another one means go this way I hate to leave I know I'm ready to decode and it's a B and so because all the symbols live in the leaves I always know when I hate to leave what's him I need to decode and then come back to the top to start decoding the rest a message that's coming in now


you can of course ask yourself the question for a given set of symbols that you want to send are there multiple binary trees and in fact there are there are many many trees you could put forward to come up with a podium for these six symbols here is another example here the tree is set up a little family and we see them the same string being compressed twice and on the left it requires 30 bits on the right it requires 29 bits and so the name of the game here is can we find a binary tree set down as I tried to encode my message


my bitstream sir as I try to put my original symbol message into a bit stream that a bit stream is as short as possible and race where you could search over all possible binary trees but there would be many many many binaries and then apply decide which one is most efficient but we'll see better schemes than that but you have a very naive way to put is just try all possible binary trees that have the symbols at the leaves and just see for each one of them how long the bitstream is and take the best one we'll see an efficient method


to get very close to that I should we get to the optimal one okay so the efficient method to find the optimal one without needing to do that exhaustive search I just described is something called Huffman codes and right now but we'll cover how Huffman codes work procedurally and then later once we've seen a bit more foundation on information theory we will also prove the fact that they are optimal but for now we're not going to yet prove that there are more you're going to look at the procedure okay so how does it work


Huffman algorithm said they're very simple consider the probability P I of each symbol I that's in your input so you have maybe a belong text file if you're encoded characters you would for each character do a count and then see what's the probability for each character to appear once you've done that you can start with one node corresponding to each symbol so for each of these symbols you have a node so starts as a disconnected tree just a bunch of separate Leafs really but not really connected up to anything yet and


you associate with it a weight P I which is the probability of that symbol from there you repeat the same process over and over until it's all connected together in a single tree what is this process you selected two trees with min probabilities P kmpl initially when each symbol is its own thing what that means is you find the two symbols with the lowest probability later on once you've done some merges it'll be the trees that at the root have the lowest problem then you merge those two into a single tree with associated probability as the sum


of the original probabilities and that's it that's all I need to do so let's take a look at an example of how this works on some example data here we have six symbols each symbol has its own probability associate with it and so let's step through what Huffman coding does we have six symbols each your own probability we have a with a probability of zero point two we have B with probability zero point one C per building 0.05 T with probability zero point two one e with probability zero point three six and F with probability


zero point zero eight let's follow the algorithm what a to lowest probability thinks it's C and F so what do we do we connect them up CNF get connected up and together they have the sum of the probabilities which is zero point one three what's lowest in probability now it's the zero point 1 here and the zero point 1 3 over here so we'll connect this up and the top here now is probability 0.23 what's lowest now we have a zero point two and a zero point two one let's go point two one is somewhat inconveniently


located so I'm going to move over here so be zero point two one and moving it off to the side zero point two one and DNA connect together for a zero point four one what are the two lowest now is the zero point two three dots here and a zero point three six over here is 0.36 is incomming located I'm going to relocate e over here all right then connecting these and then together they have zero point five nine the two lowest ones are the only two left is zero point or one and the zero point five nine and here is our Huffman encoding and then


what we do is like the left side is 0 and the right split is 1 there we go and now we have an encoding we want to know what is D D is 0 0 what is a a is 0 1 what is B B is 1 0 0 1 0 0 what is C 1 0 1 0 E is 1 1 M F is 1 0 1 1 and to say uniquely decodable code every symbol once you have sent the bits across you hit the leave of dizzy coding tree you know you've got an entire symbol and then you start again at the top of the tree to decode the next symbol so we haven't covered why this is optimal but hopefully the procedure is clear that is


a relatively simple procedure they can do for any symbol table that you have and it relies on these probabilities and you might already start foreshadowing here chorus this is what your narrative models might be handy very good generative models might allow us to build good probability estimates that we can then use to find a real good encoding because of course if these probabilities are wrong then this tree will not be a very good tree for encoding the data so here's the same another example that you can work


through solving another example and especially under many of the applications that are used on the internet Huffman codes are used to compress their data all right so maybe let me pause here for a moment to see if there's any questions so have you any questions feel free to type them into the chat window or to just speak up or raise your hand oh hi drew yeah I had a question so I guess something that I noticed about Hoffman codes and stuff would be that you're kind of the number of symbols or number of values that you have is fixed


but if you're trying to let's say encode more complex data structure so if you have something like you know maybe maybe images have like a fixed dimensions to audio can be multiple dimensions for example so is there a way other than discretizing or is the notion just to make chunks and then compress them effect sized chunks which take discrete values yeah very good questions so chunking is between an option and then just send over in chunks which by the way often can be desirable for other reasons also even if you had a fixed


size thing but you wanted this let's say you had a video you wanted to watch on home if somebody first encodes the entire video sensitive crosses one file and only then you can decode and play it's not great you want to be able to stream it across so you have there are reasons to chunk where you're actually for bill optimality of compression but you reduce latency and getting things across we will look at some other codes a little later and it's a really good question that actually fits very well with her describing so the coding systems we'll


look at later are thematic coding and asymmetric numeral systems are able to encode streams in effectively a continuous way such that if the stream is longer it can keep encoding and it just on-the-fly continues to encode as you go along now in practice often people will still chant and stop at some point because otherwise you might have to wait too long before you can decode sometimes but in principle they can work with arbitrary length and not knowing the length ahead of time so we'll cover that but you're


absolutely right for Huffman codes do is that strong assumption that you have an alphabet of symbols and you build it encoding for a doubt alphabetical and you don't encode that specific symbols make sense thank you well compression usually be done in terms of bits like the encoding will be the output of the encoder will be like a lookup table and it won't you say yes the way we think of compression and the way it's ultimately done on computers is that only what comes out is the sequence of bits you can think of a single bit as


the sense of the minimal unit of information like a single bit can either be 0 or 1 and sometimes the minimum that you can send across in terms of information just an outer 0 or a 1 because there's two options you can send information across if you have only one option while there's nothing you can do there's no information be transmitted so in fact often as we'll see the way information gets better than the size of well the amount of information your system gets measured is by bits the mineral number of bits required to


represent the original is the amount of information in that piece of data I say one quick thing to add maybe it is the case that when you actually transmit over certain lines that are not let's say a computer storing zero on bits there are transmission schemes where you send maybe two bits in one go to using potentially something close it to a continuous channel that you then discretize on the other side to get out several bits in one go so that was happen also under the hood but in terms of kind of the information theoretic


properties we tend to think of it as turning everything into a sequence of bits alright great questions thank you let's move to the next part which is threat of the limits and what we're going to cover here to me some of the most beautiful math than any discipline has to offer somehow well we're going to cover we can cover in just a few slides we've been covering quite comprehensively in just a few slides and get very kind of deep throughout up on inside to guarantees across so very excited about getting to talk about that


day so one thing you might have heard in the context of information fear this thing called entropy DeShannon a sort of a measure of information so what is entropy by definition so it's just a mathematical definition not talking about properties yet entropy of X what is X X is a random variable right and so X is just really some distribution P of X and so whoo I measure entropy till the entropy of the random variable not a specific institution of that variable but anyways the entropy of the distribution or entry of the random


variable as a whole and this is definition use sum over all possible values the random variable can take on and you take a weighted sum of the probability of take on that value and then log 2 of 1 over P exile ok so this might seem look a little coming out of water but maybe let's get a little bit of intuition for this of why this might be a meaningful way of how you measure entropy which is the amount of uncertainty you have in a distribution and hence there's a lot of uncertainty about the random variable you need more bits to send the cross we


hope to tell the other person what the outcome us so there I have been unbearable I read an experiment I see the outcome of that random variable I want to communicate the outcome to you how many bees do I need to send on average and this effect doesn't just talk about that but it also then kind of hints at an encoding scheme it kind of says the number of this you can I use for a outcome X is going to be log to 1 over P X I so let's look at an example distribution here's an example distribution random variable can take on


five values and will compute the entropy of this thing so it isn't like 1/4 1/4 1/4 1/8 1/8 and we can compute the entropy entropy is 2.25 then let's look at another distribution much more peaked beautiful these are three quarters and then 1/16 for everything else compute the entropy its 1.3 so 1.3 2.25 entropy is a lot larger on the left and on the right why is that because if I run an experiment on the random variable on the left and then want to communicate the outcome Q actually there's many possible outcomes that are pretty likely and so


it's not like you can come over to very efficient encoding scheme because you need to encode everything pretty much with some reasonable probability you're gonna have to send it across whereas here on the right what happens is this first outcome over here scroll ugly so if you encode that first outcome with a very small number of bits then most of the time you have to send almost nothing and then yet sometimes you have to send more bits to get the other things across but most of the time is very cheap and so that that's effectively what's going


on in this equation and for now a building intuition will make this a lot more formal very soon so let's take a look at another example so think back to think back to our binary trees to encode some set of symbols we have symbol a b c and d if these are the probabilities 1/2 1/4 1/8 1/8 then this thing over here is a optimal way of encoding that have the time you sent just one bit for a the other half of the time you got to cover the rest and to say that you're covering the rest you have to send the one then


in the other half of the time half the time you have to send what it say it's be in the other half of the time you have to signal with the one that it's one of the other two and then at the end you decide which one it is when you're down here so what we can this even though we haven't proven this intuitively it should make sense that this is a very good scheme for encoding this kind of distribution over symbols because you can't say anything less than one symbol for a otherwise you have not communicated anything a is the most


frequent symbol I'll have to do is send one symbol and for being well you know if the first signal it's not a and then you send one more something symbol to communicate it to be similar than for CMD this encoding scheme over here uses a length that is the to log of 1 over P of X I and so you could imagine a world if in the in your world every probability associate with any symbol you have some symbol X I and the probability P of X I is equal to 1 is equal to 1 over 2 to the length of sorry no let's see if ya the paralytics ID can


be expressed as 2 to the power L I then you can encode using the same scheme that symbol X I with a length L I bitstream into a tree that would be built up the way the tree was built up over here haven't threw in this but that's kind of the rough intuition and we'll see is of course things that generalize this to simple we're P of X is not necessarily one over two to the power of something that could be any probability different from a power of 1/2 okay so that's some high-level intuition let's now take a look at some of the theory that we can


put down first main theorem is the Kraft McMillan inequality what it says is that for any uniquely decodable code c okay so this is somebody tells you I have a code it's uniquely decodable and if it's not uniquely decodable you can't really use it to do lossless compression so codes do need to be uniquely the code well unless we're not going to consider them for losses fresh when it comes up with a uniquely decodable code see what does that mean it means a mapping from symbols to bits bit strings corresponding to each symbol if it's


indeed uniquely decodable then it's the case that this property holds true what is this saying it's saying for each symbol and corresponding ain't coding so the word the bit word over you we can look at the length of the encoding and there's some property satisfied by these lengths so funny and somebody's your table symbols and bitstreams it's gonna be a and some bits string here then be some other bits here and so forth you give you a code that's uniquely decodable then the lengths that you encounter here will satisfy this


property this says it's smaller that one has to be smaller than one what does that mean these are negative powers here so it's effectively saying that the lengths have to be large enough they always going to be of a certain length otherwise it's not be satisfied so swimming out this thing is saying if someone is uniquely decodable code I can guarantee you that the encodes have to be relatively long they cannot be shorter than a certain amount because otherwise they would not satisfy this property with Boston


actually held in the opposite direction opposite direction says if you have a set of length L that satisfy that same thing then there is a code you can build in fact the prefix code which is very convenient to deal with which is uniquely decodable of the same size as these links so it's a back and forth kind of mapping if something uniquely decodable this is satisfied if don't satisfy this you can build uniquely decodable code that in fact the prefix to a tree that allows you to encode symbols with feet word lengths what does


this mean this means that since there's a party that holds true for any usually decodable code someone can give you include the decodable code this property will be true when this profit is true there is also a prefix code with the same lengths so it means that we never need to resort to anything but prefix codes if my said have a very clever scheme to make the bitstream uniquely decodable you might have to look ahead and look and look at many places of decode you can say no need i can use the same encoding length and build a prefix


code that will have the same efficiency as your other uniquely decodable code which will be more troublesome to decode so we're going to strict attention to prefix codes all right so what's under the hood here let me give you a quick brief sketch one direction for any prefix codes d and that's kind of a subset of what's on previous slide for our prefix codes C we have down this is satisfied that's what I stated what's the sketch order all the lengths of your code words then setup we prefix code right so we have all these


lengths favorite prefix code who a previous code we can build a tree look at a tree we can look at the tree will actually end initially over here at those red dots because that's where the code words are but we expand the tree to be of equal depth everywhere so even though maybe your symbol a would be encoded here you continue because you want to make it all equal death then after you've done that you can do a simple count you can say each code word for example this one over here how many leaves are covered by it well the whole


trees of depth 4 so the whole thing is depth for it is at depth - so what's under here is due to the four bonds to leaf nodes under here we have 2 to the 4 minus 1 as 8 leaf nodes living here and so forth so every Congress there's any leave of the expanded tree since it's a pretty code let me know overlapped it's a clean tree so the total number of leaves will be in this case 2 to the 4 or in general 2 to the L + 4 n Ln the maximum length code that you're considering so the opposite equality here that the number of leaves


covered is smaller than the total number of leaves you could have in a tree you just divide both sides by 2 to the L 1 and you get this thing over here so not too hard to prove the details of the proof don't matter too much but it can be done in one slide that's the first part how about the second part second part says for any set of lengths if this is satisfied then we can build a prefix third tree with those lengths how is this done you consider a full tree of depth L M which is the longest length and for each eye you pick any node of depth li


still available so you're going a tree say a depth of Li is anything still available okay I picked this once you pick that you consider everything below it average so this is all done at this point nothing below there is available anymore this will consume tuna alin- li leaves of the expanded tree you can you know that as you count together how many leaves you're going to cover in this process is going to be this many on the left here we're told that this thing on the top holds true which means that this is smaller than 2 - L M which means that


we can fit this inside a tree so we have be able to fit all the code words inside a tree okay so two quick proofs we don't need to know to be able to prove going forward but I wanted to get across it these are actually relatively simple to prove consequence from this is probably something you've heard many many times and that will now be able to prove very very easily then for any message distribution P of X there's some distribution over symbols then you have an Associated uniquely decodable code see then the average encoding length so


the expected length of your code we encode a symbol will be always larger than the entropy of the distribution there's a janitor in 1948 entropy is lower bound on how many bits need to encode symbols coming from a certain distribution so let's step through what the key things are to get there this is just what we're starting from the difference between entropy and exit code length entropy is this thing here expected code did this look at all possible symbols and look at the length and take the weighted sum we have px I


here P of X I over here we can come bring this together we have this over here then to bring us more closer together we're going to say well L I equals log of 2 to the L I we have difference of two logs we can bring out together the log things behind multiplied together or divided by each other if you have a negative sign so they've been negative appearing here then what is this thing over here what are we doing we're replacing let me expand it we're essentially replacing it with by bringing this thing over here


we're the expected value of the log of something to make it into log of expected value that's Jensen's inequality we've seen that in variational auto-encoders see it in many many places in machine learning we just applied Jensen's inequality expected value of the log smaller than log of the expected value and noise brought along we have the expected value of the log is over here whereas the log of the expected value is above and so we have just an equality applied here how about the next step we do this is going to be


the craft McMillon in a quote assess if we have uniquely decodable code and this thing over here has to be smaller than equal to one and then log of one is zero and we're done so to prove Shannon theorem all we needed is Jensen and then craft mill and inequality and we're good to go we have the full proof let me maybe pause here who's a pretty big result and see there's any questions all right so at this point we've proven that and uniquely decodable but anybody can come up with with certain lines know the code words you can use a prefix code


if you want to so that makes it very convenient and also that it's never going to be better than the entropy and expected encoding length for she might have next n as well how close can we get to entropy can we find the code that achieves that achieves H of X or it close to it because if we can and we know we're doing optimal okay so here's one way to think of it expected code length would be entropy if we take the lengths of all of them exactly this thing over here on the inside the coding is log to 1 over P of


X I we're good to go now I'm practice I might not be a natural number so you might have to round it up to the nearest natural number to actually make it a bit sequence so this is essentially n2 between Shannon coding so how about we proposed this we're going to try to encode with this thing over here the first question you should have is that even possible is this a valid set of lengths or would this be lengths that actually will not correspond to a curve well I'm gonna think craft of billing allows us to check for a given set of


lengths is there a code that corresponds to it so let's check Kim can we find a code that matches up with this well the this thing over here is the thing that we have on the Left sent hand side into credibility and equality and we want to hopefully prove that this is smaller than one so but trying to prove it's smaller than one we have to make the steps to get there if we can prove this then it means that code exists and we're good to go we can actually means looking to enter the coding so this thing is equal to the code lengths are given by


this quantity over here so just then this is more than or equal is because we're running up here and we're getting rid of the rounding up but the rounding up is happening in a negative exponent so by getting rid of the rounding up we end up with something bigger then this thing is easy to simplify to to the log to of something is just something that's what we have here now some of the probabilities is equal to one and we are good to go we have them 2 to the minus L I sum over I saw you go to 1 which we know from


Kraft McMillan implies exist a prefix code dad worked with the links so we now know that we can do entropy coding so be an alternative scheme to Huffman coding right we would have encoding the bill the treaty here would be you look at the probabilities of all your symbols and then you assign the length and then you still define code words that match up with it but assuming you can run some search or some other item to find those code words you know they exist so you just need to find them and then you're good to go


how is this well there is there's a little derivation we can do that this is very close to achieving entropy so look at this over here what's expected length weighted sum of the lines fill in the lines then what is this thing over here well this length over here is rounding up so it could go up by one relative to the real number that's on the inside that's just the one plus over here once you have that in simplified it the one comes up from does summed over all pxi and then in the back here we have entropy and so we have 1 plus entropy so


expected length is entropy plus 1 so this is pretty good we here have discovered out not only the best thing you can do is entropy coding is the entropy in terms the number of expected bits but also you can use directly log 2 of 1 / PXI as with designated coatings and if you do that you're only one away on average from the optimal encoding now the one thing we haven't covered yet in this whole scheme is how do you find that encoding we now know that we could do entropy encoding we know that this will be close to optimal


but running a massive search coming tutorials face might not be that practical check out Huffman codes can achieve the same optimality and we'll show that now so by induction on the number of symbols in our code book so a number of symbols M by induction meaning that in the proof we'll assume that if we have to encode only n minus 1 symbols and we use the Huffman encoding scheme we would end up with an optimal prefix code for those n minus 1 symbols and now I'm going to show that under that assumption it's also true for M and of


course with only two symbols or one symbol wherever you want to start it's clear Huffman codes are optimal so we're good to go okay this is actually a little intricate but it's not too long Huffman coding always looks at these lowest probabilities symbol so we'll start there we look at the two lows for those symbols X and y there's always going to be two lowest probability symbols mater is a tie but that's fine they arbitrarily break ties and be the two lowest probability symbols in your original code book optimal prefix codes


will have two leaves in elos double branch why is that you have a prefix code if symbol here maybe a symbol here some more symbols here in the lowest level branch which is this one over here there are two leads in higher levels that's not always true here that's not true here that's not true but the lowest level it's always true why is this always true imagine you didn't have two symbols left anymore only had one symbol after you didn't have this one here what do you do you would actually get rid of us all split here you put C up here and now


it's gonna be true again if it's not two symbols at the bottom you only have one we just a bit more okay so at the bottom there is always gonna be two leaves at that lowest branch then without loss of generality we can assume that symbols x and y have the same parent does I have to meet a kids just bill imagine your tree looks like this it can be an X's here y is here and there's a Z here and a W here could be it they don't have the same parent but because they're the lowest probability symbols they will always sit at the lowest lowest level


and at the lowest level you can interchange where they live and you can always make x and y appear together and put W over here and so there now have the same errand it's effectively the same code you just move things around at the bottom so x and y have the same current then every optimal prefix 3 will have x + y together at the lowest level with the same parent so that's what we're out now the steps we've made allow us to include this line over here no matter the tree structure the additional cost of having x and y rather than just


a simple parent symbol Xena needed sending that with my n minus 1 symbols now this M resembled x and y the extra cost will be px plus py y is down the number of times you have to go down that extra level in a tree to reach x and y is P of X plus P of y if you only have to go to the parent of x + y you wouldn't have to go that extra level and whenever you have to go to extra level it cost you one extra bit it happens P of X plus P of y fraction of the time now the end symbol Huffman code tree adds this minimal cost to the


optimal n minus 1 symbol Huffman code tree which is optimal by induction so this here it's a final part approve it's saying no matter what tree you build you'll always pay a prize of P of X plus P of Y when you need to split on x and y you can't just get away with apparent Z that's unavoidable the Hoffman code tree will have them appear that way together so the Hoffman code tree is incurring the minimum possible cost for being a n symbol tree prison minus one symbol tree it's having that minimal cost to when


it's built so far which is a 10-1 symbol tree which we know is optimal by induction and we're good to go alright so click recap of everything we covered entropy is the expected encoding length when encoding each symbol with this length and so there's the equation for entropy atmosphere for 1948 so that every data source P of X is an order 0 Markov model which means there's no dependencies between symbols like that you are accounted for or able to come for then a compression scheme that independent codes each symbol in your


sequence must use at least entropy bits per symbol an average Huffman code is able to do that with an overhead about most one how do we know that because entropy coding doesn't leave an overhead up at most one and we prove that Huffman codes are optimal so given entropy coding has an overhead about most one Huffman codes provide a constructive way of achieving something that also has an overhead about most one beyond the entropy cost any questions I have a question so for the competition you mentioned in the


beginning 500,000 euro competition if we just take the photo and yeah if we compute the entropy of that file that's provided that will provide like the minimum number of bits right if we just turn that can we just turn that to see if it'll be like 116 megabytes like that should would that give like a lower bound on what can be achieved so yeah that's a really good question and that gets exact what you're getting at is a in some sense exactly this thing over here so for now we've assumed the order 0 Markov model so what that


assumes is done let's say that is a sequence of symbols let's say there's only 26 letters and nothing else in that file of course there's other symbols too but you could just look at the frequencies of each of those letters you could then look at the entropy encoding or look at the entropy and say okay this is the entropy and now if I want to compress this by doing each of my 26 letters a bit sequence as its encoding what's the best I can possibly do I can actually find that number and you'll find that it is gonna be more than that


one in a 16 megabytes because otherwise somebody would have long done it the reason there is a hope to be able to do better and well we'll get into how to do this soon is down in reality these letters in that file are not independent when you see first three letters you might have an easy time predicting the fourth letter because there's only so many reasonable completions of that word now you already starting to saw the first three letters off and so then the calculus becomes a little different than we get that in a moment


and that's where things get combat is then all saying like oh it's not as simple as doing counting of trillions of each of the symbols you really need effectively a generic model that can understand how to predict the next law small previous symbols and start measuring the entropy and batma and then the question is how good a maliki you built and yes if you can build the world's best generative model to predict the next character in that sequence you look at the entropy of that then you might have a lower bound


roughly speaking on I mean - I think of a few details to be sure it's exactly true but they don't give you a pretty good estimate of what the optimal encoding might be and we'll look at a few examples soon like three slides from now we'll get to a few more things that touch upon exactly what you're asking about really good question other questions okay let's move on then so a couple of coding considerations we want to look at here what happens when your frequent accounts or maybe some more complicated estimate the distribution over symbols


is not precise you have an estimate P hat but really the tradition is P what's gonna happen with your performance of the compression scheme higher order models we pick the next symbol from previous Emal's how can that help you and what about that plus one didn't ask innocent as it seems or is actually very bad sometimes and what can we do about it so the expected code length when using p hat to construct the code is going to be expected code length but in reality our expectations with p is the way we encounter symbols is governed by


p for though your symbol is p i but the code length we assign is based on p hat a lie so then this is our expected code length simply don't need to round up to get no natural numbers for the encoding so simple calculation then we add and subtract the same quantity now this quantity over here in the front we recognize as KL divergence the thing in the back we recognize as entropy so we see the expected coloured length when we use a distribution estimate P hat is going to be the entropy plus we know it's always going to be more any


encoding is gonna cost you at least entropy may be more it's gonna cost you an additional decay all the versions in P and P hat so the price you pay is a cal divergence we know that the log likely objective when we learn a genera t'v model I should comes down to minimizing the KL divergence between the data distribution and the model that you learn and selectively when we're maximizing log likelihood we're minimizing this calorie version effectively trying to find a distribution now incur minimal overhead if we use it for an encoding


to encode our data notice two ways you can prove Cal is positive we can prove it because we know every encoding has to be hired Andrew could call it done this means that this thing is positive because we know that already or you could prove me from first principles using Jensen's inequality which is shown at the bottom here but so we also will pay a price corresponding to the KL divergence so the better our generic model is the better our encoding scheme can be and so when we think about including with generative models there's


really two things going on you want to somehow figure out a good encoding scheme but the other part is you want to do really well at this part over here which is a maximum likelihood estimator because that's going to help you ensure encoding scheme is actually good on the data now what if P of X is high entropy if P of X is high entropy that would gain a very long code length which you might not like you might be able to decrease the entropy by considering conditional entropies if you condition X on the context let's say what has come before


they may be able to reduce the entropy in fact this it's easy to prove that the conditional entropy of X given context C is always smaller than the unconditional entropy H of X in fact auto regressive models do exactly this in an automatic model you predict the next symbol based on everything you've seen so far and often the prediction of the next symbol or the next pixel is gonna be a lot more lot easier to predict so a lot lower entropy than just independently predicting each pixel and so going back to the price thing that we were talking