about effectively this is saying that if you don't assert each symbol independently but you train a conditional distribution maybe well you will not you should not do worse and likely you should do better then come then when you would do it with each independent symbol being encoded separately all right how about the +1 Matt seem pretty innocent no entropy is optimal n to be +1 why not up up their price of 1 let's look at an example and let's look an example where I might I should be pretty bad and it's not going to be


uncommon if we have a good predictive model which makes H of X very low then it could be very high overhead for example our distribution over symbols the three symbols very peaked mostly on the first symbol because we can predict the next letter maybe very easily the next pixel very easily we're very peak distribution 90% of math then 5% 5% the entropy of this thing is roughly 0.5 but it will pay a penalty of +1 and in fact we send us a lot of time each thing is gonna cost us at least one because sending a bit sending anything across

L10 Compression -- UC Berkeley, Spring 2020, CS294-158 Deep Unsupervised Learning


will be at least one well I should pay a price that's pretty high so here's the optimal code for this we could use just year over 0.9 and then 1 1 1 0 expected code length will be 1.1 so we're going to pay a price here that's actually pretty big that almost twice the length of the code compared to what entropy cover what entropy predicts has the lower bound so this first one gets expensive you send their long sequence of symbols essentially sent twice the sequence length compared to what in principle you wish you could be getting


how can we get around this let's take a moment to think about that anybody any suggestions could you use larger trunks can you use larger chunks exactly why would you care about larger chunks the reason your this price is expensive the plus one is expensive is because when you only have three symbols where you send one symbol you still need use at least one bit but one symbol doesn't have much information in it in this case very little information it is deceiving the first symbol if we send multiple symbols in


one go let's say we turn this into a distribution over we have three symbols ABC chosen distribution over there could be a a a there could be a a B there could be a a C and so forth now we have 3 to the 3 is 27 possible combined symbols that we're trying to send not which friend this will work out a lot more nicely and the overhead will become a lot less than if when we try to send just one symbol of time so let's take a look at this in action one way people do this in actually sending faxes well Tom there keep you have used faxes but essentially


before there was emailed with something called faxes where you could send documents over a phone line and the way it was encoded was by Senshi he's sending he naively would be uttered is fix less wide or block and as you step through the entire page naively you have to send one bit per pixel white or black very expensive because usually actually it's going to be a lot of light in a row or a lot of blacks in a row so you can instead encode it as the number of whites then the number of blocks number of lights it's called a thin coating and


that's what they came up with so what are your symbols now your symbols are I run off I run off let's say one one I run off two whites and run up three wise one of four wives Sam for black you list out old possible run wings that you might want to care about encoding and then you can look at the probabilities of each of those run lines and then build a Huffman code and then you get the encoding so you're going to be using and you get a very compressed representation of that page that you're trying to send across even then also has


a question about the English language how much entropy is there in English language and people have done this experiment so question here is was the entropy conditional entropy of X let's say X a position and given X 0 through X and minus 1 how predictable is the next character so Shannon ran his experiment and he concluded that the English language is only one bit per character so if you train a conditional model that predicts the next care could give it everything before you can get an entropy of 1 bit how do you even figure that out


the way you did it is you actually ask people to do completions so you would say okay here's the beginning of some text now predict for me the next character and then the person predicts a character and then me Shannon would say right or wrong it is right you're done if it's wrong you get to guess again 79% of the time people get it correct in the first guess 8% of time it takes two guesses 3% of time takes three guesses and so forth whenever you get communicated back whether your guess was right or wrong effectively one bit of


communication was communicated about what the underlying character is and so you can just stick the weight of some here you'll see that lands are roughly 1 which means that you need one bit of information for character on average people have not gotten to that by the way I mean that depth of at least ball automatic fix compression schemes have now gotten to that lower yet but things are getting closer and closer over time so looking at practical schemes if you use just 7-bit encoding fixed encoding well then you have seven bits per


character if you use entropy encoding off individual characters of your two to the seven this is 128 or simply 128 characters you use entropy encoding you know I need 4.5 bits per character that's of course the ball it is if you look at entropy but you can't perfectly achieve that because you have to brown to a finite number of bits so - code which is optimal achieves four point seven now if you look at the entropy of groups of eight symbols and then look at the average entropy per character you can line up 2.4 now some


thought is good asymptotically this goes to 1.3 so what you want to do instead of encoding one character at a time when I maybe could eight characters at a time and huffing code will be something probably slightly above 2.4 for that employee ok I propose we take a five-minute break here and when we start at 625 and we'll start looking at how some of these ideas can tie into the generative models we've been studying you alright let's restart any questions before we go to the next talks alright then let's take a look at how we can


combine all the rest of models which we covered in one of the first weeks of this class with coding the key motivation here is that we want a flexible system to group multiple symbols to avoid the potential +1 over head on every symbol including going back to Jerusalem who long this thing is going to be and want to be able to encode that on the fly so question we might have is how many symbols which symbols to group in a naive system that's what you'd have to do yes okay how many similar I'm going to group in


groups of 3 or 10 or whatever and then make some decisions about how to grid them Ciotti hue is actually done but we no need to decide on how many symbols or which symbols win we're doing this for we're going to encode every possible symbol sequence by mixing into a distribution it'll show an example very soon and this works for training over there and is extremely compatible with all aggressive models so let's take a look let's take a look at an example we have a alphabet with two symbols a and B probability of a is 0.8 probability of E


is 0.2 if we individually encode this we'll have to maybe send the 0 for a and a 1 for B and is gonna be a lot of overhead because a cost us just as much as B even though it's way more likely really there should be a way to make it cheaper the second most naive thing would be only talked about earlier gives you a head of time decide what is 3 a is going to be three B's two A's and a B and so forth we're gonna do something quite different we're going to do is we're going to say okay we get something to come in a a PA


let's say first and code the first symbol here a we have a distribution that's available to us the malls the probability of these symbols I'm gonna say okay 80% chance it's a 2% chance it's B so we're gonna actually map the fact that I have an A to this interval over here it means we landed you can think about the all possible random events that couldn't happen in the world is the ones that lie on the 0 to 0.28 interval that's the event that has happened then when the next a comes in we're going to take that interval that


we're working with now 0-3 0.8 and say okay it's again an a so it must have fallen within the first 80% of that new interval then it's a B which means that's fall in the last 20% the last 20% of that interval there and then it's a a which again means the first 80% so end up with is let me take a different color that's more visible is a lot of green already end up with the notion that this string a ABA gets mapped to a very specific interval within the 0 to 1 interval and the way we do this it should be clear that is unique for every


string will have a unique interval he end up in and so that a different sequence we end up with a different interval the idea behind arithmetic code is that what we're going to communicate is the interval so way to communicate the this thing over here now we still have to decide how we're going to commute but that's the idea and you don't need to know I had I'm how long your bit string is or your symbol string is going to be because this interval one boom maps to whatever simple sequence you receive so just need to encode this and


we're good to go if there were more symbols coming in if there was another B after this they would have split this thing again I would have had the small end of all was another a after this with him split this up a bit more and your the smaller interval and so forth so one-to-one mapping between simple sequences of arbitrary length and intervals okay how do we code an interval let's start with a naive attempt naive attempt of encoding interval so K represent each interval by selecting the number within the interval


which is diffused bits and binary fractional notation and you set up the code so for example if we had these intervals we could resent those with point zero one for the first interval point one for the second one and point one one for the third one because those are binary numbers that fall into each of those respective intervals it's not too hard to show you for interval size s so the width of the interval asked we have the most negative log two of us bits rounded up to represent such a number which is great because that means


that because the width here s s is really probability of the civil sequence so that's achieving entropy coding up to the rounding out the problem here is done these codes are not a set of prefix codes for example we have one here that we would send for a second symbol but after we receive one we wouldn't know did they send us a second symbol or was it the third symbol so the second symbol sent twice or the third symbol sent once there's no disambiguation and so this scheme while we might seem reasonable at first and it's efficient it actually


does allow you to decompress correctly so what else can we do we have each binary number correspond to the interval of all possible completions so for the above example when we say point zero zero it means the interval from zero to 0.25 will say point one zero zero it means the interval from 0.5 to 0.625 will see a point 1 1 that means interval from point 7 5 to 1 so we're gonna we're gonna want it to be the case that remember on the previous page any simple sequence that I want to send will result in an interval we want to send we're


gonna find a bit sequence such that when you look at the corresponding interval that we map to it which is given here by example that entire interval should fall inside end of all we're trying to encode leaving no I'm bagheera T which in the ball it belongs to and will not be a prefixed or anything else to work out the details of this it turns out you get an overhead of to possibly instead of 1 but that's actually pretty good because when we do this there's kind of arithmetic coding we can code arbitrary many symbols so the overhead of plus two


is only in count incurred once for the entire sequence instead of incurred for every symbol we send the Cross so it's a one-time overhead for the entire sequence that we encode this way obviously we'd like to avoid the plus two but it's not that bad any remaining challenges well sometimes when you file this scheme what'll happen is that the interval that you're finding as you go to your a be a be a and so forth sequence and you start from that interval from zero to one you might find that you know at some point you have


this interval like this but zero point five is here just marking this next thing you realize oh you're actually here next thing you realize they be you're here the next thing that the way it works out is that you always end up with that interval that's centered around 0.5 if the case you're never able to send that first bit till your entire sequence is complete and so the solution to that is to even though in principle to minimize the number of bits you need to send you need to go to the end of your simple sequence and code the whole thing and


then send all your bits if you want to minimize latency not wait till the end of the whole thing before you can send anything at all you'll split into smaller blocks such tab he keeps traveling 0.5 it's something to say okay I'm done there's a bigger block I'm sending it across another thing down this scheme as I described it assumes this infinite precision it assumes that you can actually compute these intervals precisely and this interval becomes always small small over time and so you could imagine that you're starting under


flow if you just do standard floating-point calculations to compute those intervals and then of course you would start losing information because the floating point system couldn't like encode the information you need to encode there is a solution to that you can actually convert this all into a scheme where you're only computed with integers and the blow-up compression survey then I linked at the one of the very first slides explains how I can turn this into an integer implementation rather than relying on real numbers now that we know how to


encode let's think about how all the rest of models can play well in this so far we said we have P of a P of B but actually no need in this entire scheme that we described at the distribution used for P of X one has to be the same decision as we use for it P of X 2 for X 3 and X 4 we can instead use conditionals that are more precise and more predictive of the next symbol and some lower entropy and a more effective encoding scheme and so this arithmetic coding scheme is perfectly compatible in all aggressive models you can just work


your way let's say put a pixel after pixel or get the distribution for next pixel next pixel next pixel and encode with arithmetic encoding accordingly working your way through an image the better the log probability the further compression will be so better likelihood off your recive model will mean better compression of that data and these two schemes couldn't be any more compatible perfectly lined up predict one symbol at a time and encoded one at a time and keep going so let me pause you're gonna see our questions about arithmetic


coding and then we'll switch to a very different kind of coding scheme okay now I'll switch to think about how I can use a variational auto encoder something called bits back coding and a symmetric numeral systems to encode information this at least to me is one of the most was one of the most mind-boggling thinks how does this even possible they're confusing at first but I hope that you know - the way we laid it out in the slice at all again be clear how does exactly works but there's this notion somehow you get bits back and hence you


send bits but it's actually not as expensive you thought it was because he got bits back and we'll make it more precise soon the references for this part of lecture are listed here with the initial bits back paper by Freund Hinton from 97 actually that's the same one the first one is here which was on using it in the context of man description length for the way to know I'm at work but then start looking at source coding then there was this paper here the bits back ans paper so law to refer it do it that way bits back 10s was the session let me


let me restart the slide for a moment so first first thing that happened with bits back is the thing at the bottom here this wasn't a comics of pure vision learning was not in the comics of coding next thing that happened was in the comics of Ashley making this practical as a coding scheme this idea and but this used arithmetic coding it turns out that the scheme we're going to look at is not very compatible with arithmetic coding unlike autographs of models which are almost designed to do arithmetic coding the when you have a via it's not very


compatible with that not in the same way so this result into lots of overhead lots of extra bits you need to be communicated chunking has to happen a lot when you usually don't want to chunk as you lose efficiency this isn't 97 then in 2019 this beautiful paper came out by concentrated barber who's sure they can do this with NS rather than automatic coding so the underlying information theoretic scheme used in their approaches NS rather than arithmetic coding we haven't covered in us yet but higher-level thing is that I


think the coding looks at your data as a stream you go literally through it ans doesn't including that acts more like a stack and putting things popping things from stack is the way things get encoded and that matches much better with the ideas we're going to step through here and next is actually possibly practical in fact the NS used in many place but physically here very well matched with BAE type coding schemes then in our work Berkeley Jonathan let a lot of this work today with Freesat kima and myself we


looked at essentially this paper here and made it more made it more efficient in by looking at hierarchical latent variable random just single it in variable auto-encoders all of this builds on this ANS TIG invented by chaired Duda in 2007 which is using many coding schemes but is very just encase a lot of information here was invented in authorities in the 50s 1940s 1950s right Shannon's theorem 1948 Hoffman code 1952 ans third aqua coding scheme today invented in 2007 POW at a time where nobody thought you couldn't still invent


really groundbreaking new things that are going to be that widely used in compression and sure they did so quick refresher will covered encoding entropy coding assigns log to 1 over P of X I length for the encoding of a symbol entropy is a lower bound we know and how long do you go to even pass to be the Shannon theorem said that we can't do better than entropy Huffman says the maya with the Huffman scheme you can get to entropy plus one arithmetic coding always have verbal I think many symbols in one go plays a plus two but is not a


plus two or similar so plus do for the entire simple sequence so it's actually more efficient than doing Huffman each separate symbol so that's we've all read so far where some key assumptions it is assumed that we have a model P of X for which we can do the following track to be numerate all X for Huffman otherwise we can do it build that tree to enumerate everything well you want to enumerate all possible images you might want to possibly encode no you can't build a Huffman tree for that arithmetic coding gets around that by you only need


to be able to assign probabilities to the next symbol in your sequence if you can do that you can use arithmetic coding but even that tend to require that there's a relatively small number of symbol values if your symbol can take on infinitely many values it's not really clear how you doing arithmetic coding so when it is fire 11 1 X continuous but that's actually quite fixable we'll look at that on the next slide and then the X is high dimensional and that's the main challenge we'll be looking at the observation here is down


some high dimensional distributions still allow for convenient coding and what we'll see some examples and we will want to do is leverage that to efficiently code mixture models of these easy high dimensional distributions the key wooden that we'll get from this part of lectures down as long as the single the non mixture model can be employed efficiently Whittlesey scheme that from there allows us to encode data coming to the mixture model also very efficiently and of course mixture models are often a lot more expressive than


their individual components which means that we can now have a coding scheme that is designed around a much more expressive distribution class that you can fit to your data oh that seems to be out of order okay well so a real number X has infinite information so we cannot really expect to send a real number across a line in a finite amount of time because the infant that's keeps going forever new information every bit so what we're gonna do we're gonna assume if we have to be able to continuous variables X that we can discretize and


that we're happy with the discretization so this guy's up to some position T you can discretize in two ways imagine you have a Gaussian excellents on the horizontal axis is a Gaussian distribution you can discretize on the x axis or alternatively and often more convenient you can discretize in the cumulative distribution so it still acts the cumulative distribution will run something like this and then this goes from 0 to 1 you can discretize here first of all lets you deal with the notion a just relies on X what were you gonna do with


the tails you probably make one again you make one big interval to go to infinity but still as somewhat inconvenient maybe also if you disregard an X well this thing has a lot of probability mass this one doesn't have much priority mask I see this guitar is based on the cumulative this is just saying every interval has the same probability mass that's how I'm going to disco ties to discretize me located here be here from there then here this interval you go here this interval then you go well it's not perfectly drawn but you could hear you


get the interval here isn't the ball here isn't the ball and so forth so that's that's a way you can discretize continuous variables with equal probability analysis or doing it tacitly on the x axis we can look at something called the discretized variable X this crisis interval with T and then so this is this version of it okay look at the entropy of that variable which will be probability of being in the interval I which is T which is the width times the height P of X I and so this approximation of an integral and then


the value of the function here okay now when we work this out just log of the product is sum of the logs then we see here that this is looks like approximation of integrals so we say okay it's almost the same as integral and we get is here is what is called the differential entropy and then this is some factor that ties into the discretization level so it seems can actually use the differential entropy if we have a functional representation of our of our distribution and we can compute the integral for it we can


understand what the differential entropy is and then the log of our disposition level will determine the overall entropy that would go into representatives as a discrete variable okay so that's a bit of background on how to deal with entropy of continuous variables jump it will be determined by our discretization now let's go to the actual challenge is that we wanted to solve and it was and we'll mostly think about discrete variables now but it also works for contains you can choose so key assumption some high-dimensional


P of X that allows for easy coding exercise I mention all hearings sometimes still used to encode one when X is Gaussian you can do it as an independent random variables along each axis and each and invariably you could efficiently we've said on the previous slide or maybe for each X we can use an auto Russell model and we know how to do a rest of encoding with arithmetic schemes and so forth these are examples of high dimensional situations where we can encode things efficiently might be more but for now let's just let's go


mostly if think about this one over here mixture models allows to have them wider range of the situations and their components for example mixture of Gauss is much richer than a single Gaussian for example a single Gaussian all I can do is look like this but a niche model could have many of these bump mix together and then the overall thing would look something like that which is much more complexed representation that you can capture with this five component mixture that with a single gaussian the key question we want to answer is if P


of X is a mixture model of easily incredible distributions does that mean we can also efficiently encode P of X there we'll look at one the illustrations to get the point across the way that it's easy to draw on slides but keep in mind that we're covering a method that generalizes to higher dimensions and I've always want to do is do including of 1d variables you can use many many methods it's not about 1d that's just a way to draw things on the slide also will not allow ourselves to rely on that the main effects being small because the


main effects is being small we could rely on that we can do many other things so we imagined higher dimensional X take on many many values but somehow an efficiently incurring a single component and mixture that we're using to represent P of X ok let's see what we can do now our running example is going to be a mixture model P of X has a weighted sum this is choosing the mode there's different modes index by I there is a distribution of x given I so supposed to I think is down maybe when we sample X we first sample a mode and


once you sample the mode from sample mode one this is the distribution we sample mode to be this is a distribution we sample mode three maybe this is a distribution sample mode or maybe this is a distribution and so forth assumptions of each of these modes themselves easy to encode East to encode means that we have a scheme that will give us close to this because that's what a good encoding would do it would cost you a number of is equal to log 1 over P of x given I is that's one week more than oh it's mode I our


distribution is P of X given mine ok first scheme we might consider is max mode encoding so max mode encoding what do we do we say ok we have a mixture distribution in this method - code X well we don't know how to correctly from P of X but we know how to code from P of x given AI so what if we could get D on that was used to generate X then we can encode X there efficiently so we find an item maximize B of AI given X so imagine we're back to this mixture model thing our X falls here then we might sit home this mode over here is the one and this


is mug one two three and say okay mine is three that's the most likely mode to have generated this X but of course if we know how to encode x given I we still to send I across otherwise the other person cannot decode with that scheme because I don't know of what we're coding relative to so if the semi which will cost us log of one over P of I then we have to send X which will cost as log of one over P of x given I and so the expected third length shown on the right here is well there's an expectation for possible X's we need to send when we


send an X we look at the I that minimizes we're coming here is both I and X so we're really coding log 1 over P of I comma X willing to choose our I we get to choose and we're picking the one that minimizes that quantity another way to write a second equation here same thing okay so the schema straightforward and we know how much is going to cost us is it optimal it's not optimal because effectively we're using a different distribution Q of X which I'll have a cost H of X plus the kr between P and Q what do I mean with that when we use


this encoding scheme imagine we have two modes this is P and you see P is running over here and both in those two modes when we use a scheme above effectively what we're doing is we're fitting the distribution queue to our to our original situation and we're encoding based on cue because editing the falls on this side will use mode one everything calls and that's how we'll use mode two and this is not the same as P it's different and will pay the price will pay the KL between the two in extra bits now I might say do we care do I


care about paying this KL divergence for all all the pants and this drawing here yeah you probably care it's a pretty big kale if your distribution was such that your modes are completely separated from each other well they're not KL between P and Q will be almost zero you might not care let's think about what we often care about in it our scenarios which is might have a variational auto encoder with a latent code latent variable Z so not just the I would M be Z that Z can take on a continuum of values so there'll be a continuum of modes and if


I pick only one of them instead of somehow using the continuum we're losing a lot because because it's a continuum they're all going to be very close together and so we are going to lose a lot by using Q instead of P in this situation so we have a scheme we can do coding but we're paying a price question is can we somehow get it done without paying that K odd well let's think about it some more so we looked at max mode what if we do posterior sampling in posterior sampling or say as well we still have the same situation as before


but the current is up to taking the I that remember before it was high that maximizes P i given X here we sample might not sound smart up first and in fact when we're done with this slide you'll see that the coding scheme we're covering on this slide is worse than the one we covered on the previous slide but in the process of covering this scheme we'll build up some new concepts that allow us on the next slide to get the best scheme better than the previous one this one so bear with me for a moment here so we sample I from P of Y given X


we stand i same cost as before using encoding based on the prior like the I why not P I give an X you might say isn't that more peak can we just send P are given X well the recipient doesn't have X so they cannot decode it against a given X they have nothing else we send I is the first thing we say well they have to decode a dist on the prior and you have to encode it based enough then we send backs using same encoding scheme as before this is probably efficient but not necessarily as efficient as using the best I remember imagine we have


these distributions here and let's say our X landed over here let's say there's mode 1 mode 2 and we're unlucky and when we sample I from PRI given X we end up with our I equal to somehow well encoding X code X from P of x given I equal 2 is going to be very expensive because there's a low probability here that code is not going to be very efficient at getting X across so it makes it less efficient than what's on the previous slide in fact the difference is that here we have log 1 over P I comma X whereas in the previous


one we had a min over I sitting in front of it ok so we lost some things here but it's all for a good reason so now what we're going to now be able to do is earn bits back which is the key concept we want to get to so it's an optimal yes and no it's yes it's optimal if we like to send I and X but we don't care about sending I we just want to send X is something we made up X is the real thing is the symbol I is just a mode then you know we have an ordered distribution we're fitting so optimal the same both but it's a waste to send I and so how


much the bottle is well it's about more according to the entropy are are given acts effectively because that's where we send that's that's wasted so what can we do what can we do to avoid this overhead the very interesting idea that the base back idea is that somehow we we send too many bits what we can earn them back and so the higher level that's what's gonna happen I say all things gonna happen we say we acknowledge we said too much we're gonna somehow earn them back and not have to pay for them so let's take a


look at that legs back coding solves a scheme on the previous slide we sample I from PR given X the cost descendant is log of 1 over PI then we send X cost is log 1 over P of X given are all the same as in the previous line base back idea with exact difference between approximate inference later what will it do the recipient decodes I and X but knows the distribution for a given X because they have the corresponding model on their side so what that means is that there see piant actually can recover the random seed that you used to


sample i from p i given X they can go to the reverse profit you do the sampling here you use random seed what is it random seed that is really use sequence of random bits that was used since the recipient knows the distribution knows I know sex they can back out the sequence of random bits that caused you to sample I so can reconstruct the random bits use a sample PI given x those findings were also sent those are log 1 over PI given X random bits which we now don't have to count what do I mean with that imagine


you're trying to send X and somehow we have a friend is also trying to send Brandon bits you can take your friends around the bits use them for this sampling send them across through this process and they'll be able to be decoded on the other side and those are your friends base so you don't have to pay the price for that that's their bits they happen to come out on the other side that's their cost to pay so one way to think of it all you have to pay is the X give an eye and that's it and we'll make that more concrete even if


they're your own bits so bits by coding cost you send you'll because of log 1 over P I to send I then cause of log 1 or P of X I just an x given I and then you earn this back because those bits actually it's a bunch of random bits that we're sitting there in or sent across but they're not yours you don't have to pay the price for them and if you do the math you actually have log 1 over P of X so you'll get encoding of our want to send X at the entry rate for X so we've got optimal encoding great we're optimal now what does it look like


you have some symbol data this is what you want to send and that's an auxilary data this is random bits sequence the sender will do lossless compression the through the schema is fine receiver will get back out the symbol data and also get back out the auxilary data because you get them back out on this side you don't count it against your budget for encoding assumptions we make you we can compute P of Y given X which can be a strong assumption being able to find that distribution in your mixture model it's the distribution that you don't


break it up available and then if something you have auxilary random data we'd like to transmit and we set it across we don't have to pay a profit and somebody asked carries that cost so you've actually did something with approximate inference in a VA II we don't find the exact kathiria for Z given X we have an inference network or here Q I give an X in first network we sample hum q are given X otherwise everything is saying we go through the whole process what happens is what we get back is log 1 over Q I given X and


when we see here if then the cost of transmitting data is not a is a little higher than log 1 over P of X because effectively we have the wrong distribution here we have Q instead of P this is the evidence lower bound we applies with the BAE so if you use a via e to do bits back coding by optimizing the loss of the VA e there directly optimizing the compression capability of this big back coding approach so perfect match between the VA objective and compression so how about that source of random bits that we also like to send


where does that come from in practice it's actually your own bits so imagine you already have some bits sitting here you have some zeros ones you know maybe you've already done some compression of something else it's a random sequence it's sitting there ready to be transmitted then the first thing you have to do and the the notation here is slightly different Wyatt responds to our i M s corresponds to our X ok so keep that in mind that's the notation they use in this paper here from which we took the figure so in


cutting the mode why we do it with their infants attribution why given the simpler one code s0 to do that we need to grab random bits to do that sampling well that means we consumed these random bits from our string that we want to send across next thing that happens is we start encoding we code SEO remembers our X so our symbol given the mode gets encoded so this grows the number of bits you want us in then the encode de mode from its prior and this grows again and so what happened here is that in the process


when coding one symbol we have first consumed some bits now we're on the stack of things to be send now we've added more bits to encode asking them why it's a symbol given mode and added more bits to code the mode itself well overall this thing will have grown typically not guaranteed but typical half grown and now we could repeat this process what we had here as the extra information it's now sitting here you can get our next symbol s1 will find what our y1 is and repeat and so we see what actually happens is we were


building up and some kind of popping the stack by pushing through the stack the sequence of bits that it code a sequence of symbols with this mixture model or bits back coding so we really see is the bits that we're getting back or not visitor sitting off to the side necessarily they're bits that came onto our stack from encoding the previous symbol that we encoded this way and you might wonder well if we took it off here but put other things on have we lost the ability to get those bits back no that's the whole idea in the decoding as we saw


on the previous one two slides back sorry when we when we decode we have we can reconstruct the random bits that are used to sample the mole given the symbol and so we get them back out at that time so still get everything on the other side this is not lost will be decoded and bits back all right so the last thing I want to cover and I'm going to hand it off to Jonathan and maybe we'll take a very short break and then hand it off to Jonathan is how do we have to get those bits back I've been telling you you're


going to get these this back you're going to have sampled your mode from Q R given X and then later you're going to get them back how this work so let's say you have a distribution I have an X and I'm going to draw the distribution of Qi given X is going to be discrete for you I'm doing here it's gonna be discrete and so I'm going to look at the cumulative distribution so let's say I lives here and then I could be maybe one two three or four come the distribution we'll say okay maybe one has a probability of let's say 0.2 or


something then once I hit two maybe two is a probability of zero point one so we hit level zero point three over here and three might have a probability of maybe 0.5 to go away to zero point eight and then for weight of the probability of zero point two all the way to one what does it mean to sample i given X I have this bit stream so I have a bit stream sitting there I'm going to start from the end here and work my way so the first thing I see is zero zero tells me so I have a zero one interval here zero tells me that I am in the zero to zero


or interval but in that interval I can still be either so it has to be either one two or three I don't know yet what I'm going to be so simple the next zero at which point out in the zero to 0.25 and I still don't know what I'm going to be I've consumed I've consumed this era have consumed this zero now I'm going to consume this one now as I consume this one it means I'm gonna be in the 0.25 in the top half maybe here I still don't know what I'm going to be I could be a 1 or a 2 I don't know and I'm gonna have to


consume this zero then next now I'm in the bottom half of this and now I actually know once I sampled those four bits I know I won now I can go sample from P X given I equal 1 2 then encode my X right and I can also have my prior P I that I used to encode I equal hold on clear this for a moment so I need to stand X I need to send I how am I going to set I well you could say well I have a distribution here over for pop values and I couldn't either die by maybe building a Huffman code or something over those four possible values that's


you can do something much simpler you can say to get the point across that I need to be I equal 1 well I achieved that by this sequence I achieved by the 0 1 0 0 sequence so I can actually just send across arrow 1 0 0 so not across and that signals that I have I so that way I'm also trivially getting those bits back because the person who receives this gets to read off the bits just like that oh here are the bits I can just read off and then I can all so use a temp to decode X all right so let's see I think that's it for me


let's take maybe a two three minute break as I know Jonathan has a lot to cover and let's maybe restart around 712 713 for the last part of lecture Jonathan do you want to try to take control of the screen here um yeah sure okay um let's see can you hear me okay yeah my I might turn off my camera too so that my internet connections were reliable but but we'll see just let me know if it's not working well okay um I guess I can just jump in and talk about more about bits back it's possible to address a question on chat oh yeah questions on


ChaCha I think it's part of lecture and then we'll dive in with with that later so first question is there's also we can have P of X and P is a mixture of gaussians how you can simply of X to begin with yeah it's a very observation it's not exactly our assumption the assumption more precise is that we have a mixture model and that four components individual components in a mixture model we know how to encode efficiently but for the mixture model as a whole we might not know how to encode and now we have a scheme to do that especially if


you know how to encode each component let's back her to your way to encode it against a mixture model which likely there better fit your data distribution and as we know the closer you are to distribution the smaller K diverges the more efficient your coding will be and so it allows us to the mixture model which might be a better fit which in turn would result in higher efficiency encoding another question there is the new party between Edinburgh Nubia that's a really really good question so one of the big things that I think Jonathan


will be you know Jonathan's covering that paper so the 2019 paper tube it's like a NS paper by a thousand at all investigated exactly that assumption so we'll see bar that but the notion down if you already put bits on your bit stream from encoding the previous symbol previously previous symbol and you'd work with those bits is that really a deficient or real the question is all those bits really random enough to achieve the efficiency that we declare here and so chuckling we'll get to that question maybe five or six slides for


now so all that for now I know it should be clear a few slides no all right um right okay so I'll just talk about how one more about bits back and some more modern instantiations of bits back coding into real algorithms that we can actually download and use and also in particular how fits back coding place with new types of degenerative models like a ease and hierarchical PA ease and flows instead of say just Gaussian mixture models right so the the core algorithm that that all these mute bits back papers are based on is this thing


called a symmetric numeral systems so this is an alternative to arithmetic coding as Peter was saying and it's especially appealing because it it's well first of all it's very simple and you can implement it on on you can implement it in a very efficient way which makes it actually practically usable and it also has some nice stack like properties that make it compatible with bits black coding so I'll just first take some time to describe what ENS actually is so again ans which is something like just like you're a thematic coding is a way of


taking a sequence of data and turning it into a bit stream where the bits greens length is something like the entropy of the data times the number of symbols and so I'll just jump right in and describe how this thing works and so so let's say this source that sorts of things that we're trending odhh is just two symbols it be each occurring with probability 1/2 and so you might imagine that the the naive way to code stuff like this is to just assign a to the number 0 and B to the number 1 and then you just get a string


of A's and B's just turns into a string of zeros and ones and that pretty much is the best that you can do but let's see how ans does this um so ans describes a bit stream not not represented it doesn't represent it exactly as a sequence of bits but it represents it as a natural number so there's an s stores this thing called a state s and we start at 0 and so ans defines an encoding operation so there's this encoding operation that takes in a current state and takes in the current symbol that you wish to encode so let's say you start at some


state s and you want to encode the the symbol a in this case in this very particular case what a in this will do is produce the number 2's 2 times X so remember the state s is a natural number and if you wish to encode B it produces the this state 2's plus 1 so that this is ans for for this very simple source um of course the ans will generalize more but just for in this case this is all it does and so you can see that really what this is doing is its appending numbers zeros and ones on the right of a binary representation of the


state s and that's how this is algorithm stores data that's how it stores a and B and a very important property of any reasonable coding algorithm like ans is that you should be able to decode the data that you encoded so given some state s you want to be able to tell what was the last symbol that was encoded and so that's very easy to check so if s is even then you know the last symbol was was a if it's odd then you know it's B and if you know it's then you can just divide by to take the floor and then you get the previous state so so that's how


this algorithm works and you can already see just based on this very simple example that this algorithm has the stack like property if you encode a sequence of symbols a B V then the next thing that you decode if you wish will be the last thing that you encoded so it's sort of a first in last out type of type of stack ok can I ask a question here yeah so sorry for this simple example what is the capital P of X and the mixture of Gaussian can you explode right in terms of this example and also I don't see why the stack is being used


here thank you yes so in this case we haven't gotten to the mixture yet where we're gonna talk about that soon this is just for this very simple source over here it's just a coin flip but we just want to store coin flips there's no latent variables or anything like that the second question was where does the stack come in it comes in the fact that so let's say we but let's say we encode a sequence of symbols say B a B and so that's if we follow this encoding rule then that's gonna produce a sequence of states it's gonna be like s1 s2 s3 and


so s3 is the final state that we have after encoding these three symbols and then what ans lets us do is decode from that state and when we decode from that state and s will tell us the last symbol that was encoded and then tell us the the previous state that came before that so so that's why it's it's like a stack because if you ask me an s what was the lesson well that was encoded it's gonna be beat or it's gonna be the this beat not the first one hopefully this will be more clear as I've got some more examples


okay um right mmm it's not letting me advance [Music] okay so let's see how this generalizes to the setting of not just the binary source or not just the the coin flip but something more interesting so here we again have two symbols a and B but become the problem the probabilities aren't one-half anymore instead it's gonna be one-fourth for a and three-fourths for B so B is more likely so we're going to now think about how to generalize ans to this setting and and the way the way it's done is like this so you take all the natural numbers so


here here's all the natural numbers and what we do is we partition it into two sets one set for a and one set for B and so I'll just write down what those sets are and then and then talk about why we chose those sets so we're gonna write down one set for a and this is going to be 0 4 8 and so on and this is a partition so the the separate B is just all the other numbers so that's 1 2 3 5 6 7 1 so on so just a draw draw it out here these numbers here for an 8 correspond to 8 and these all the other numbers correspond to be I'm saying


correspond to a meeting correspond to ending in a or correspond to ending in be right um I guess I've been just I haven't defined what corresponds to mean yet I just mean that we're defining these two sets s a s sub a is gonna be all the numbers divisible by 4 and s would be is gonna be the others and so so we we've just defined these two sets and then I'll just describe how we encode some spring so let's say we want to encode the string be a B so again ENS builds up with some big natural number which is a state so we start at


the state start at s equals zero and what we want to do is encode onto the state zero the symbol B so the way we do this is we look for the zeroth number in B's set so that this might sound a little bit weird maybe I'll just write out the general rule if when we encode a state s with say the symbol a we look at the s number and press sobayed so this is s and this is this number and that's so B okay so let's just go through this so when we encode 0b we look for the zeroth number and be set so so be set is this one two


three five six seven all the numbers that are not divisible by four and a zeroth number starting indexing at zero is one that's the first number so that's what we get here so that's just writing it down in this table here okay now the next character we want any code is eight so we want to encode the new state as one and we want to include the number a or the symbol a so we look for number one in a set so a set is 0 for 8 and so on so number 1 is 4 so that was here and then finally the new status for and then


we retreading could be again and so that's 6 so what this says is that ans has turned the string EAB to NSS turned the string BAE into this number 6 and this number six stores these three characters which is kind of cool okay so first of all the this might seem like a weird set of rules to play by but first let's check that this is actually decodable otherwise this would be useless so to see that is it possible to take the the number six and and see which was the last character that was encoded and the answer is yes because


these two sets si in SP were defined to partition the natural numbers so for any number any natural number like 6 you know which said it belongs to so you know that 6 belongs to s of B there's no and and so you know the last character that was encoded was B and then you can also recover the last state the previous state before he was encoded and the way you do that is just by looking at the position of six and S sub B a so you see that six is the fourth number and SOP so that's the previous day and you can just


keep repeating this and you can recover the characters that were encoded on so hopefully that convinces do that this is decodable and kind of the point of this is that we actually chose these sets as sub ans of B so that their density in the natural numbers is approximately well it is pretty much the probability of the symbols so you know if you take a lot of natural numbers the fraction of the numbers which Lyoness of a is about one-fourth and the fraction of the numbers that line s of B is about three reports and so this out this encoding


operation here where we look for the s number in one of these sets on that operation will advance us by a fraction but by a factor of about one over P but that's just what happens because this thing is distributed like a fraction of P over the natural numbers so when you index into it you you you increase by a fraction of one over P so that means that every time you encode a symbol onto on to a state I guess it's called X here you end up multiplying your natural number by about 1 over P that's generally what happens approximately so


here if the SLA is powers of three it'll also work powers of three yeah so we want as SMA so we we just want them to occur about one fourth of the time like zero comma three comma nine etc that'll also work um so that doesn't really occur one-fourth of the time in if you pick some long sequence of natural numbers those numbers don't occur one-fourth of the time for that long sequence oh I see so we want the density of these things to be to be any partition that needs the criteria that first 1/4 is going to work right this is


not a neat partition right so so there are actually a lot of choices for this so this particular choice is true is it so that it's very easy to implement the encoding and decoding operations so you can just do it with some modular um but if you have some crazy choice maybe it'll work but it might be very hard to to compute B encode and decode operations well it seems like the set of natural number that is also chosen like can be chosen otherwise here like we don't have to we only restrict a natural number because of the index is zero so


it's convenient is that why well at the end of the day this is something that we want to turn into a binary string so I guess I haven't described that at the end but so so once you in covered everything you have this big natural number that describes all your symbols and then you turn it into a binary string and then you can in the binary representation and you can ship that off to the to the receiver they start at the end you just have one number right right and then you from this one number you can back word


generate all the three views right right but here we have this number six and now we want to send six to the receiver and the receiver you know the all our communication protocols work in bits so we have to turn six into a binary string and then send that to the receiver but the point the point is that actually secure here's here's the property of this scheme that we basically keep dividing by P of s every time we encode s so that means that if we encode a bunch of symbols we get some starting symbol divided by the


product of the probabilities of all the symbols that we that we encode it and so if we so this is some natural number and if we code the natural number the number of bits needed is about the log of the number that's the log base 2 of the number that's how many bits we need to code it so we see that this is this is the code length it's the sum over T of log 1 over P for all the symbols and so so if we take this and we divide by the number of symbols so if we take this divided by the number of symbols you see that this goes to the entropy of this of


this source so that so this is like an optimal thing to do I roundabout way of answering the question of why use natural numbers but but I think the stack here is just a conceptual framework right we don't know the actual implementation we don't need stack yeah that's absolutely true with this we say it's a stack just because it has this property that every time we decode something it we just get the last thing that was encoded we don't get the first thing that I was encoded so we just call it a stack but but yeah you


don't actually need a real stack I mean essentially it's just a partition of a lookup table like before we have a general lookup table but now you just partitioning the lookup table um sure right I guess like maybe the point here is that yeah ENS is really these rules and you can implement them efficiently this was what judo found and and it seems to work in practice and it kind of helps this stack like behavior that's the best the point of this and it's also optimal okay so what so returning to more interesting models which are not just


two characters a and B but rather things like distributions over images represented by latent variable models so there's this very nice algorithm introduced in 2019 called bits back with ans or BB ans which is its back coding using ans as a back end and the the reason to use ans is because it turns out that the the staff like property of ans where the last thing you decode or the wear whatever you decode is the last in Unicode makes it very compatible with the concept of getting bits back so let's just see how that works so here


we're gonna think about winged variable models so Peter talked about Gaussian mixture models which are one case of this so here's the is the latent variable P of Z is the prior and that's the marginal distribution so this is how bits back coding works and we're gonna talk about how it works exactly what AMS so in BBA in s the first thing you do if you wish to send X so the goal here is the send X the first thing you do is you you you start off with a non-empty bit screen so you start with so s so we can just call it a bit scream


because that that's just how we think about it and so the first thing the encoder does is it decodes Z from the bit stream so the encoder knows X so the encoder can compute Q of Z given X this is just the approximate posterior of this latent variable model and it can use this distribution to decode from the bit stream and we assume that this bit stream was full of random bits and so that this is a question that came up and I'll talk about the consequences of that later so that's the first thing you do but the point is that if you decode from


random bits then you get a sample and then the next thing the encoder does is it encodes X using this using P of X given Z which is actually called the decoder and then it finally encodes e ok so what actually happened here so if we just visualize this in a spit stream like this so so that this is what we started off with in the first phase when we decode Z we actually remove a little bit of this bit stream from the right so imagine this is a stack where we keep adding things on the right so in this first phase we remove a little bit and


then we get a little shorter bit stream then we encode X so that increases the length of the bit stream a little bit more but let's say by this much then then we encode Z again so so that that increases by a little bit so now we can you can just look at this diagram and see how much how long did this bit stream get what was the net change in the length of this bit screen well we we have to add in these two parts right because the bit stream went right there but but then we also subtract how much well I guess not there but we subtracted


a little bit at the beginning so the netcode links though are the net amount of change to this to the length of this bit stream is well that's a negative log P of X given Z minus log P of Z so that was furred for these two parts two and three but then we had to subtract the amount that we we decoded from the bit stream at the beginning I was so that's plus log excuse me for the next and the first part Z gives you some sample from Q so the actual code length on average is is the average of this of received Ron from the approximate posterior so


you can see that this is the v AE bound this is just the variational bound on negative log likelihood so so I guess this is can I ask sorry if you have a stream of let's say oh just lowercase letters A through Z then would P of Z here just be 1 over 26 and then the P of X given Z would be the number of times it occurs in divided by the total length right so it just depends on what your latent variable model happens to be I'm so the case that I'm actually thinking about it is view is that this is a V a and so P of Z is like standard normal


quite better but my confusion is why would it be like a normal distribution like isn't each key represented by a single value in the lookup table it's a constant right so why would it have a distribution um so I'm not really sure what like if you're given a string you just count right and then out of the count you that's a constant let's say it's all just restrict to a through Z then for each of the character you have a basically the probability occurs in this stream and that's a constant value so why would that have a distribution so


maybe let let's back up for a bit and just like talk about what we're trying to do so what we're trying to do is to turn the latent variable model into a compression algorithm so just starting from square root 1 we have a ve of this what's what's the input of the Dae an image it's a stream right let's say for 1d case is it a stream time I propose we can write questions here offline because we've got a lot of cover yes okay yeah happy to talk about this later yes ok so here is a description of the same same thing so during the encoding


phase the decoder decodes from the bit stream then encodes X and Z and you can also check that this is decodable so if you just run everything in Reverse you you just end up getting yeah getting X so you decode Z you decode X and then you can re encode see I'm using using a should actually be Q and re-encoding part is this getting bits back here so once you rien code Z once a receiver re-encode Z then the receiver gets now a slightly longer bid stream from which I can start to decode the next C so that's so so those are exactly


the bits that were gone back right here okay so there are two sort of two points that we should talk about when when getting B ba and s working with continuously variable models like via East which is that disease these are continuous so Z is comes from a standard normal distribution and so we can't really cope continuous data but what we can do is district ice it to some high precision and so if you take Z and you discretize it to some level Delta Z then you pretty much turned a probability density function by P of Z and you turn


it into a probability mass function for capital P of Z which is Z times Delta Z so what you get by integrating a density over this small region of volume Delta Z and so you can do that for both the posterior in the prior so you do that for the prior and you do it for the posterior and you see that these deltas use cancel out and so so we get is that this bits back code length with the discretization being the same between the prior and the posterior still gives you the same KL divergence term in the BAE the second point was that that somebody


brought up is that we we decode Z from the bitstream and that's how we sample from the bit stream is by decoding from it that's how we sample Z but in order for that to really give us a good sample the bits that we that we decode from have to be actually random and so that's not necessarily true and so in a VA II the last Z if you just sort of work out what's going on basically if this KL divergence between this aggregate posterior Q of Z and the prior is small then that means those bits will be random or pretty good and that'll be


good enough to get a good sample but of course in practice for a VA that's not trained exactly well this isn't gonna be nonzero but in practice it seems like this doesn't seem to matter too much I think one thing that might actually work to ensure that the bits are random which I haven't seen explored is to just encrypt the bit stream and that'll make the bits look random and then you can decode anything from it so I think in practice is not a problem and with nice is that this the scheme fits back with ans seems to work pretty


well um so the authors of this paper implemented this it's back ans algorithm for ve strain on em mist and they found that the numbers that we got were very close are pretty much the same as this as the variational is negative variational the variational bound on the negative block likelihood which is exactly what you want that's what is predicted so this thing works as well as as as advertised you right so in our work what we did was we looked at weight in variable models which are not just one layer so we know


that the the more powerful the the model the better the log likelihoods that we get out of it so we should get better compression so here we're looking at a setting where the model is has a Markov chain structure over the wing variables so there lady variables ZL the L minus one up and seal the one the necks so this is the graphical model of a sampling path and and the the inference path accuse go going the other way and they're both Markov chains so this is a particular type of model that we're looking at and so if you had this


particular model which is this this sort of chain structure there are two ways to view it you can just view it as a via e with this block of latent variables as just one latent variable and then you can run bits by B ba and s1 and then that works perfectly fine but another way to view it is to view it as a leak variable model with just a little just draw the layers again so here's X 1 Z 2 3 so you can just view Z 1 as the one and only latent variable but then you see that it's prior is a VA with the same structure so it's prior is P of Z 1


which is a VA II where it's prior is P of Z 2 which is another V and so on so these are two equivalent ways of looking at the same model so in terms of log likelihood they're the same because that if you just write down v variational bounds they're equal but they suggest slightly different compression algorithms with different practical consequences so the idea is that instead of just treating disease as one single block of one large latent variable you can actually recursively invoke its back coding into the prior so you can just


code the first variable so here this is the algorithm for just as usual so here this is basically um decode Z and then encode X and then encode the prior so this is P of X given Z mmm-hmm and here Z given X what you can do instead is code just the first layer and then recursively invoke its back coding into the C the subsequent layers right and so the the consequence of doing this is that I won't go through these the exact steps but the consequence is that you no longer have to decode the entire block of latent


variables at the very first step rather you you just need to decode one of them and then you get some you can add more to the bit stream and then decode more and so on and what that means is that you need fewer auxiliary bits to start bits back coding so what this means is that remember before BBN has to make sense you need a bit stream with some bits on it to even sample Z in the first place and those bits must be sent across and if you you don't have any and if there are no bits there then you end up wasting them and so if you're able to


not have to decode it too many link variables in the first go then you can save on transmitting those logs iliad bits so you can see that in these experiments for especially deep latent variable models we're able to get better code lengths compared to just decoding the entire block of latent variables at once you right so that was via E's so let's just move on to how to trim flow models into compression algorithms so in this class we went through a series of likelihood based models like Auto regressive models


reason flows and we were seeing in this lecture that really any likely hit based model is as a compression algorithm so what about flow models they should also be compression algorithms and so what what's particularly appealing about them is that you get this we can write down the exact the exact log likelihood flow model this is not a bound this is just the real thing so hopefully we should be able to get some really good compression with this so let's think about what that actually means it turns out that it


doesn't really make sense to say that to get a compression algorithm that achieves this code length which is that which is just this flow log likelihood formula and the reason is that flows are density models and it doesn't make sense to code continuous data because it just needs you need infinite precision to do that so rather well we're gonna say is that will code data discretize the high precision so you you have your space of data like this let's say this is the space of all possible images and then we just tile it with these with this very


fine grid like this and then we're just going to discretize every possible data point so instead of coding one data point exactly we'll just coat the biddin that it lies in like that so that's G of X is like the cube bed that's some data point lies in and the point of doing this is that if if you define a probability mass function given by integrating this density given by the flow over these cubes and then you get then you get a negative log likelihood that looks like this it's just this the density times Delta negative plug


density times Delta so this is now a probability mass function and now it makes sense to say that we can compress up to this cook length so actually the code length that we're going to look for when we compress with flow models is this it's negative like this the flow times Delta so it's really just the same thing plus this additional term here so this is just the number of bits of discretization so it can actually be a lot of bits but then we can recover them later right so yeah so now we have a probability mass function I can Chris


wants to flow so can we just run Huffman coding so the answer is no because to do that we need to build a tree that's as big as the number of possible data points but these are we're working with large images here so that's that's exponential in the dimension so that's not tractable so we we need to harness the model structure we actually we actually have to make use of the fact that this is a flow model so one naive attempt to do this maybe this just the most intuitive thing is to take the latent that you get out of the


flow let's say we want to code X so why not just compute Z when that just compute Z by passing it through the flow and coding back using the prior so so maybe that's very simple but unfortunately doesn't work so you can just write down the code link that you get it's just negative log P of Z times let's say Delta but if a flow model is its trained well then the distribution of Z's will match the prior so you end up just let's say the prior is Gaussian so you end up coding Gaussian noise using a Gaussian prior so that's no


compression at all and so if you compare this expression with this expression up here you see that this is the missing term so somehow this this naive approach does not take into effect take into account this jacobian the the fact that the flow changes the volume so we have to somehow deal with that right okay so how do we do this well the claim is that we can turn any flow model into a VA actually we can locally approximate it using a VA so we have this flow model that takes here's F here's the full model and


here's here's a point X and we turn it into f of X this is just what a flow does and so we can define this distribution here so this distribution on the left this like this this ellipsoid and we're going to define it to be a standard normal where the mean is just f of X it's just the length but we give it this covariance matrix which is Sigma squared is just a small number like 0.0001 or something like that just some hyper parameter to this algorithm times the Jacobian times Jacobian transpose of the flow model and and so


this is what we call the encoder and this is a decoder so we so on the float on top of the flow model we define this encoder and decoder and the decoder is just the inverse of the flow with this identity covariance such as very small so that's what these two is ellipsoid on the left and their small circle on the right are so what why did we define this well the point is that if for a flow model is represents a differentiable function so if if you have some data point X and we had a very small amount of noise to it and then you


map that to the latent space that small that see that small amount of Gaussian noise that you added at the beginning will also be Gaussian it'll just have this twist there down covariance and that's given by how the flow behaves linearly and that's just the Jacobian so we know that if you take a multivariate Gaussian and you multiply by a matrix you also get a multivariate Gaussian so locally the the flow behaves like a linear transformation and that matrix is the Jacobian so that that's what that's where this comes from and the point is


that if you then run bits back coding using these two distributions so this was Z given X here Q this P of X given Z so if you run Kota using these two distributions the code link that you get from bits fat coding will be exactly what we wanted plus this little error term which is this second-order error term so so that this is how you turn or this is a way of turning a flu model into a compression algorithm is to convert it into is to locally approximate it with a certain beauty to find like this and then the


code link that you get from from bits back coding on that Wilmette that's what we wanted plus a very small error term so what what's nice about this is that it turns an intractable algorithm into a more tractable one so if you wish to directly implement this algorithm it turns out you do have to compute the Jacobian of the flu model and you do have to factorize it in a certain way and so that's that's polynomial time it's better than exponential time but it's still not good enough for high dimensional data and so the


solution to that is that we can actually specify we can we can specialize this algorithm you even further and so for a lot of regressive flows for example it turns out that we can just code one dimension at a time without ever constructing that Jacobian so that works in linear time if we have a composition of flows like we like we do in real MVPs and then you can just code one layer at a time and we recursively invoke this coding into the next layer but just like we can with hierarchical Yogi's so all together for real MVP type flows if you


implement it correctly you don't need to compute the Jacobi and ever and you get a you actually get a linear time compression algorithm you so the so that's nice so and we achieve this code length here which is negative log density times Delta but if you look at this this this suffers by terms of negative log Delta X which can be like actually quite bad like 32 bits or something like that so this is because we had to discretize the data a lot so that we can actually approximate the integral that defines a probability mass


function easily so that seems like a huge waste of bits especially if we want to transmit say integer data like images from CFR for example or it specifies its are specified as integers and we don't want to have to transmit lots of bits after the decimal point so the solution to this is to use those extra bits four bits back again and so if if you want to do that it turns out that there is an optimal way of doing this and it's and this sort of encoder that you use for that is a D quantizer which I think we talked


about and so if you plug it's back coding to the D quantizer to get those extra bits and then altogether the code length you get is the variational D quantization bound which is what you explicit you train to to be small on on the data set so it ends up being a reasonable you and so with all this stuff we tried it for one of the models that we trained and we found that we were able to get some code links that are very close to what is predicted by the variational T quantization bound and this sort of holds across all these data sets


um and there is a caveat which is that this algorithm doesn't need a lot lots of auxiliary bits actually much more than a via II type methods and that shows up in the fact that we need something maybe like 50 bits per dimension actually to just send one image and so that means that this algorithm really does not make sense if you just want to send one data point but say if you wanted to send like use this algorithm for each frame in a long video in a movie or something like that then the initial overhead can be amortized


across all the different dreams so that is a caveat of this algorithm right so finally let's talk about some other things which are not exactly about bids back so well all these algorithms that we talked about so far basically fall into the to the framework of you pre-trained a generative model on on some training set which you assume is you know drawn from the same distribution as the test set that you want to compress and then and then you just devise a coding algorithm that matches the negative log likelihood of


that and that's how you go but they're actually the other types of algorithms which are quite successful in text compression which we actually all use like in gzip and zip and so on which learn online so they they don't you don't really pre train them on a certain data set they just just give it a file and it learns how to compress it online and it turns out that maybe theoretically these types of algorithms at least if you get them lots of resources can actually learn to compress any distribution so we call them


Universal codes so there's one algorithm called problem if LZ somebody's and which works like this so I'll just try to very quickly describe it you're basically trying to so here's a long string that you're trying to compress and the way it works is that when you try to compress but you're at some position in the file let let's see we're at this position of the file and we want to code the future so what you do is you basically try to find a string starting at this position eight which has already occurred in the past so we


here we have this screen AAC and then we see out in the past AAC occurred so let's just store the index into the past so this occurred one two three times steps into the past so let's just store this number three in the past and then we we also add into the next character which is B so that's that's basically how this works at this point C we see oh there's a string CAE in the future but of that string occurred in the past so let's just store the number three which indicates that you just need to jump three into the past to just copy that


string the past over so this is roughly how Lempel Ziv works you just look for matches that you're trying to compress and past and copy them over and so what why is this a good idea okay so why is this a good idea so just very roughly why this was a good idea if the source of symbols you see is independent then whatever symbol you're at right now will reoccur that will will actually reoccur if you wait long enough and the reoccurrence time is as a geometric distribution so the average reoccurrence time is just one of the probability of


the symbol so that's that and so the local civil rhythm says to just write down the time that the the write down the time that it that you have to look back to find the same symbol again so that's going to take log t bits where T is the time so on average it's just log log T bits which is log 1 over P of X so you can see that this goes to the entropy of the source so this is an interesting algorithm it's basically nearest neighbors and it's saying that if you run if you just memorize tons of data over time and you run nearest


neighbors then this is like the best learning algorithm that you can or this learning algorithm does work um it just might take a very long time to learn and you can see that it does take very long time to learn because template matching does not generalize okay so that was simple as if so I'll conclude by talking just giving you a taste of some very recent research on on deep learning and compression so by no means is this comprehensive or anything like that it's just to give you an idea of what might be out there so


this the authors of EBA NS released some new work and I clear this year where they show that you can train a fully convolutional deep latent variable model on small images and just because it's fully convolutional you can just run it on large images make sure that this works very well so this these are I think some of the best numbers on full resolution image net just by using this full accomplishment property these authors here describe a very intriguing alternative to bits back coding so they described what they called minimum


minimal random code learning which is a coding scheme for latent variable models which achieves the bits that code link without needing bits back so the way that works is that you know the encoder samples a lot of weights the number of leads and samples is 2 to the KL divergence between the encoder and D and and prior and then picks a random one I did just a uniform random one and the decoder can do the same thing if they share the same random number generator and so it turns out that this is a way to basically get a sort of a low bias


sample from cube by sampling a lot of these lanes and picking a random one and the number of bits you need to encode the index the the random one that you picked out of them it's just the KL divergence just log K so the visionary log K bits um so this achieves the bits back cut length without many bits back the trade-off is computational complexity because the concurrency could have to collect a lot of samples and finally there's this other paper which has a very different flavor from the ones that we were talking about this is a paper about


lossy compression where they come up with a recurrent encoder and decoder architecture for lossy compression on sequential data like videos say and the way it works is quite interesting um just the very high-level idea is that the encoder simulates the decoder so normally you would think that the encoder and decoder just operate independently and the encoder you know just doesn't worry about what the decoder is doing but if there's this time structure then the encoder can simulate what the decoder is doing sort


of one time step behind and based on that you can send extra information that will help the decoder or just send the right information that will help the decoder reconstruct the data just in the right way and they show how to write down a neural network architecture that captures this idea and optimizes for the resulting code length in this end-to-end way so that's quite a cool idea right yeah so that's that's all I have to say hopefully that was helpful that was great Jonathan what's it's all over time but I'm thinking maybe we can


spend a couple more minutes if people have some questions didn't want to ask as we wrap up here anyway also value of lecturing and I also did a bunch of questions in the chat to be able to do that in parallel to you making progress on lecture I had a question about ans so I still don't see what the connection between like oh it seems like ans was just like a little add-on to this lecture like what's the connection feels like I don't I don't really see why we need ans like why you can just use another yeah it's a there are ways of


combining Ritz back with arithmetic coding it's just sort of the popular recent thing to do is to combine bits back with ans and in the reason we do it is because you get a very clean algorithm that works very well so that's that's just them that was the motivation can't you use it sorry can't you use base back so do any encoding scheme yeah yeah you definitely can just particularly convenient because of the stack structure of ans and also because the NS does work well in practice um to use it with that social practical


reasons why a net using mmhmm yeah maybe also dumping in here if you look at the Brendan frame geoff hinton paper it managed to do compression with a VA e and arithmetic coding but it had a bunch of overhead that you encountered because I think they could a nice look cute and this back axle or like a stack and so there's an overhead occurred then look at the Townsend on paper you can see how to make it all compatible through using a Dennis and get much better efficiency and compression efficiency tanam the previous paper that uses arithmetic


hi everyone welcome to lecture 12 of deep unsupervised learning spring 2020 today we'll cover representation learning in reinforcement learning first before I start I wanna give a big thank you to many colleagues and friends who have contributed to this lecture through sharing their insights illustrations slides and videos that all very directly contributed to well I'll be sharing in this lecture here today thank you so Terry what class has been about unsupervised learning today we're actually going to look at how one

L12 Representation Learning for Reinforcement Learning --- CS294-158 UC Berkeley Spring 2020


surprise learning and reinforcement can be brought together in a way that inspires learning to make reinforced learning more efficient but we haven't covered reinforcement learning yet in this class and so what I'm first gonna do is step through some of the very basics of reinforced learning obviously can't cover everything it could be a course in itself but we'll go through some of the basics from the recent successes and then from there look at successes where unsurprised learning and then reinforced winning or brought


together so what is reinforcement lis reinforce learning is a problem setting where you have an agent agents supposed to take actions in the world ask the agent take actions the world will change so for example the world could be the robot body and the environment around the robot body and after the world has changed because of the agents action this process repeats over and over and over and the goal for the agent is to maximize reward collected in the process for example imagine our agent is supposed to control a self-driving car


then the reward might be positive for reaching destination and negative forgetting to an accident maybe our agent is a robot chef and then the road might be positive for a good meal even more positive for an excellent meal and negative for making a total mess in the kitchen and the goal in reinforcement learning is for this agent to figure out through its own trial and error how to get high reward so as the human designer who gives a specification with the statement I'd like high reward and you say reward is high for the things I just described


and then the agent specific items on how to achieve that another example could be a video game the score in the video game could be the reward and so the agents supposed to figure out how to play that game to maximize reward where are some challenges in reinforcement learning man let me contrast it with supervised links and supervised learning what happens is you have an input and a corresponding outputs and the way you supervise your learning systems bias named for this input that should be output for this other input that should be up and so


forth in reinforcement learning your robot chef might be busy in the kitchen for half hour comes out with its meal and you might say good or bad meal but that's not reflective of the last action their robots have took it's reflective that whole half and half an hour of working a kitchen that somehow result in a higher board or a low reward and now when that chef robot chef cooks multiple times and sometimes has good outcome sometimes bad outcomes you could start looking at what's common between the good outcomes what's common


among the bad outcomes that process is a seizing apart what might have positively concrete or might have negatively con treated that's solving the credit assignment problem it's one of the big challenges for a reinforcement winning agent another big challenge of stability when let's say you have an agent learned to fly a helicopter well helicopters are naturally unstable so if you're not careful during the learning process you might crash your system and I might just stop the whole thing another big challenge is


exploration for a reinforced training agent to learn to do things it actually if it actually doesn't know how to do anything has to try things it's never done before it has to explore and this is many many challenges when you try things you never tried before well how do you even know what you should be trying it could be so many things to try what's more interesting what's less interesting and it also brings back the stability challenge is how do you make sure we try something you don't destroy the system and so forth now one example


that many people would know in real life all three enforce planning is how to train a dog when you train a dog the dog is the reinforced running agent and you as a human provide rewards you might give the dog positive reward when it does well and negative reward when it does poorly and you don't control what the dog does it's not supervising against you cannot tell the dog do this do that do that all just all its muscle muscles will follow your commands no the dog will just do some stuff and you'll say good or bad depending on how happy


you are with what the dog did so one of the things that we want to do in today's lecture is give you a bit of overview of successes of reinforcement learning but then also from there look at limitations and take a look at how representation when it can help out so one of these successes of probably success that foot deep reinforcement on the map was in 2013 when deepmind came out with the DQ end results deepmind showed that it's possible for a neural network to learn to play a wide range of Atari games from its own trial and error


now this was a big surprise until then if you looked at reinforced learning results they would typically be on relatively small simple environments and the input would not be images the input to the agent would be a very well-crafted representation of whatever world the agent is in summarizing in a small set of features or state variables and so a big surprise all of a sudden reinforcement works with pixels as input from there a lot of progress was made of course including the progress list on this slide here a lot of it coming out


of also deep mind Berkley open AI and much much higher scores and faster learning has been achieved in the Atari was on on the Atari benchmark sense wasn't just Atari deep mind also should even learn to play the game of go long-standing challenge many people thought it would be another 20 years if we asked him in 2013 2014 but sure enough in 2015 a computer beat the world champion in doe then the first version alphago was a combination of imitation learning and reinforcement learning the second version alphago zero was pure reinforcement


learning it just learned from playing us itself and over time became better than the best human players and then was a big result in more advanced gameplay or video game play opening showed that the game of dota 2 can be mastered by reinforced learning engine into 2017 he was shown a reinforcement agent to master the one-on-one version of the game and be some of the best human players and then Lola was shown that reinforcement learning enables playing a very competitive not necessarily beating the human world champion team just yet but


at a very competitive level with some of the best human teams through pure reinforcement learning at Berkeley in parallel we're exploring reinforcement learning for robotic control and so here is actually some reinforcement in action thus far we just talked about results what does it look like what you see it in action here we see an agent that's learning to control and this kid is learning to run and we give it positive reward the more positive the more moves to the right and it's negative reward for falling up to


the ground and what we see is that over time it figures out a strategy a control policy that gets it to run off to the right now the beauty here is that it's able to learn this in a way that is not specific to this two legged robot meaning that we can run the exact same deep reinforced landing code and run it on the four-legged robot it'll learn to control this four-legged robots and in fact it can also learn to play at our games the exact same code in this case is precision policy optimization TRP o combined with generalized advanced


estimation tae that's able to learn to master all their skills in this case that the robots learning to get up the reward is based on how close the head is to standing head height the closer the head of standing head height the higher the reward and then this was generalized to a much wider range of skills so what you see here is reinforced winning edge that has massive very wide range of locomotion skills and then here we see it in action on a real robot this is Bret the Berkeley robots for the elimination of the tedious tasks because


we you must don't want to do the tedious tasks with want robots to do those students tasks for us when we see here is this robot is learning to put the Block in it imagine opening and indeed over time it figures out how to get the block did the matching opening now what it's doing under the hood it's learning a vision system and a control system all together to learn to complete this task what's the catch in all of this data inefficiency well monasteries achieved an Atari Ingo in robot locomotion robot manipulation


and so forth this mastery requires an enormous amount of trial and error and so the big question is can we somehow bring that down and reduce the amount of trial and error that's required to master these skills it turns out I believe in many others believe that representation learning can play a big role in getting there to get to much more efficient reinforcement learning and it's not something that is fully understood yet this is a domain with a lot of room for more research and so we'll cover today is a pretty wide range of highlights of


relatively recent results people have achieved by looking at combining representation learning with reinforcement to make RL reinforce learning more efficient we'll look at is long for directions auxilary losses state representation exploration and unsurprised field discovery and we'll unpack these as we go along one thing you'll notice is that it's not some kind of linearly building up thing and at the end you know it culminates in what is the most important piece really what we're going to be covering is a wide


range of highlights that each has their own interesting factors to it and probably a solution that will be the final solution in the future we'll combine ideas from many of the research results that we cover today into one system so let's start with those hilary losses the paper wanna start with is the unreal paper by deep mine so the idea here is done reinforce money as a sense can be very data hungry especially there's only sparse rewards and the question is can we make an agent learn more efficiently by having auxiliar


prediction and control test by having it not just learn from reward signal because it might be very sparse but have the agent absorb other signals of course nothing else we want to supervise as a human because then it becomes very tedious but self supervising those things that are available in the environment that the agent could try to learn from even if it's not exactly rewards signal so the unreal agent which stands for unsupervised reinforced planning and auxiliary learning showed it tenants improvement in data


efficiency over a three C which was can be standard RL approach deep mind abuses at the time on the 3d deep mind lab which is a navigation first-person vision navigation task and sixty percent improvement the final scores so faster learning and converging to better final scores so what does the architecture look like when we see at the top here in the middle is the base a QC agent so again this is not a reinforcement in lecture let me give you a little bit of background what's going on here in reinforced when you have experiences the


agent without any given time and has to make a decision taking the current input process it and then through the policy output try to make a decision on what to do and through the value function output try to predict how much reward is gonna get in the future from this moment in time onwards so there's two output predictions here and that's the standard base a through C editor already predicts two things how much reward that's V value the cumulative reward over time that's coming and policy PI action it should be take so both of those are


outputs of the same network that's the basis of this edge then all the data gets put into a replay buffer and it's reused in other ways and the same neural net that is the a through C agent will give it multiple heads even more head so it has to make even more predictions and by giving an additional prediction tasks if these prediction paths are related to learning to solve the original problem which is achieve high reward and these prediction tasks are real ended and hopefully it'll learn something that will transfer over to the real task who


care about and be able to learn the real task more quickly so what are these auxiliary tasks the first one is auxilary q functions so the idea here is you're given additional heads to the neural network that are q functions a q function is predicting for the current situation how much you work well I get in the future if I currently take a specific action so for each possible action I'll predict how much reward might I get now the interesting thing about Q function learning is that you can do Q function learning of policy