and if that's not the case i'm not going to wire them together so that would be the situation where you have damage instead if they are not wired together i'm going to and can do this in the next layer of the neural network wire these other two things together you know if if the first thing is not the case i'm going to wire these other two things together to make up for that loss and


there you can see there is kind of this logic built into the network now again i know you can do this with learning a fixed policy you can achieve the same effects the point here is just to show that um given kind of the same size networks and so on there you that there might be there might be like a qualitative difference in certain situations again by no means

Meta-Learning through Hebbian Plasticity in Random Networks (Paper Explained)


this is meant to out compete rl or anything like this okay so we'll we went there now how are these rules actually learned and there we have to again make a distinction that is completely separate from the hebbian non-habian way okay so the heavy and non-habian distinction was do we learn the weights of the policy network directly or do we learn the rules to update the


weights now the question is whatever we learn how do we learn it and again we have to draw the distinction this time between i'm going to say classic or even though the terminology is not really correct classic rl and evolutionary methods okay so in classic rl what i would do is i would use my weights in order to obtain a reward and then i would update my weights


so my delta w would be proportional to the gradient of w of the reward okay so in the classic rl especially this is a policy gradient method right now so i use my policy my weights to get the reward and then i would calculate a gradient and you know usually the reward isn't differentiable so you have this uh reinforced trick in order to pull the reward out and you you can


read all of this up if you look at policy gradient uh the basic policy gradient methods but this here tells me i need a gradient usually this is going to be the reward times the gradient of my fw of my input so what this means is what this means is that if my reward is high then i i just want to know what do i need to do to make more of what i just did okay and the gradient ensures that for


every single weight in your neural network you know what to do so the gradient means that i have an exact handle on how do i need to change this weight how do i need to change that weight how do i need to change this weight in order if the reward is high and because of this multiplication here i want to make more of what i just did and the gradient tells me how


if the reward is low on the other hand i want to make less of what i just did but also the gradient tells me how that can be achieved i simply go into the other direction than i would if the reward is high in evolutionary methods we don't have we don't do this gradient calculation okay now there can be advantages to not doing radian calculation sometimes back propagation simply isn't


possible even if it is possible and this is maybe the case where we are now what we need to learn in our case is these rules to update the rules and imagine you have an episode and that's kind of episode so you have step step step step and in each step these rules are applied right in each of these steps the rules are applied and at the end you get a reward so what


you need to do is to back propagate that reward through all the steps and then through all the rules okay and that might be just computationally not feasible or the rules the rules right here are pretty um pretty easy but the rules might not be differentiable you actually have the same problem in general in classic rl as well but you know you can cut off time steps and so on there are various


hacks in any case there can be advantages to not having that gradient and evolutionary methods are a way to do that in evolutionary method usually you are don't train one agent you train a population of agents so you have a bunch of these uh neural network agents in here and the way you update the neural network agent is you simply let them run you know you let them run your app


the episode so this is your w one of them you let them run the episode they get a reward and then you can do multiple things so this depends on the evolutionary method so you can either pick out the best performing agent or you can update each agent according to some rule the goal here is simply to basically you always want to take your weights you want to add some noise to them


and you want to see does it get better or worse if it gets better good if it gets worse not good okay the difference is without the gradient you don't have a handle on how do you need to change each individual weight all you can do is basically random walk and observe what happens and if the random walk is you know turns out to be good you go more into that direction of that random


walk so it's sort of a sort of a poor poor man's gradient method in these evolutionary methods again completely independent of what we learn you can use the evolution evolutionary method to learn the fixed weights and that's what actually what happens in the table i've shown you uh below or you can use the other evolutionary method to learn the hebbian update rules


as well you can use rl to learn the fixed weight or the update rules in this paper they use evolutionary methods to learn the hebian update rules and they compare mostly with using evolutionary methods to learn the fixed weights okay the exact evolutionary step they use right here is the following so ht here is going to be the thing that you learn you know as compared to w being the


network weights h is going to be the hebian weights since we learn the hebian weights so how they'll update um each agent is going to be they'll take the hebian weights and this this here is how you update right this is your delta h how do you update the heavy and weights well what you do is you you perform n random perturbations so i take my weights and i add noise i just add noise okay so i i'm


here and i just make a bunch of versions of it and then i observe how well are these versions doing so how well are my random perturbations doing this is going to be the fitness fi right here is going to be the fitness and then i'm just going to perform a weighted average so this is my weighted average of these new solutions okay so if this solution here did pretty well


and this solution did pretty poorly i want to walk you know in this direction and then again i do the same thing here from here i do a bunch of perturbations and maybe this one did pretty well and this one did pretty poorly i want to walk in this direction and so on okay so that's how you you'll change the um you'll change weights or rules or whatever you want in an


evolutionary method it's you know it's pretty easy it's easier than reinforcement learning no back prop no nothing basically black box optimizer there are more complicated evolutionary methods but no we don't go into those here right now okay so again i've already shown you these results now i said these static weights are also with evolutionary method they also report what you would get with


like a rl approach like ppo you would get kind of the same thing as they get um as they get here so oh sorry this is not the same as the table yeah i was confused for for a second this here is for the car environment okay this is this vision based environment so with their method they get like an 870 rewards with the hebian based approach with the static weight but still


evolutionary method they get a much lower reward in fact the hebbian based approach is about the same as you get here with an rl algorithm and as we said url algorithm more complicated and if you use like a if you use like a state-of-the-art rl algorithm not just ppo you get a bit of a better performance but not that much if you look at if you look at the actual


numbers so you know pretty cool to see that again this is not outperforming anything this is simply showing that um you can do that they do a number of experiments where they go in the episode and they kind of change stuff in the episode and one cool thing here is that they go and you know this is an episode so at the episode you start with a random network each time in


this hebian setting and then pretty quickly the rules adapt for a high performing right so it it starts to walk it reconfigures itself and starts to walk the reward here again it doesn't have access to that but we can measure it of course and then at this step a right here they simply go to the weights and zero them out so they just delete these weights right here


and only 10 time steps later it has reconfigured itself as you can see right here in order to walk again so 10 time steps later reconfigures itself reconfigures itself and after a short while right here it's back to its kind of original performance as you can see so that's i i'd say that's fairly um fairly impressive uh in this very short amount of time able to recover from such


and such an intervention if you do this i mean of course if you do this to your policy network that's statically learned it's going to be garbage but i guess the fair comparison would be to delete the habian rules themselves and you know so it's not like it's not like this can adapt to new situations or something like this this is still learned for particular environments


right but the the point here is that you learn the rules and this is kind of a study on neuroplasticity now my question actually would be why this diagonal pattern appears and i have not seen a like a clear explanation um especially is this anti-diagonal pattern it's not so much here in the output layer right this is the output layer there are what 21 actions or so and this


one is this this dimension um so not that much there but there seems to be this rule and this is not the case at the beginning right you saw the beginning you saw at the beginning it was pretty random matrix so why why yeah here pretty random and then there's this diagonal pattern i don't know why if you know let me know i mean it's anti-diagonal maybe it it is


actually diagonal and the forward the fully connected layer is just defined as something like wt times x and um but maybe this also depends on the random initialization but there is no inherent way why a particular neuron would you know care about sending information to like the same height of neuron on the other side or is there i don't know i'm so is this a property of the evolutionary


or of the learning rules it seems not because the learning rules don't depend on the position i'm genuinely confused about this and maybe you know maybe they've written it somewhere and i've just overlooked it though i they they do reference it they say oh there's this diagonal pattern appearing but i don't think they ever say why it is diagonal um okay i might just be i might just be


real dumb yeah so they also you know they do some more experiments they show for example that if you just have random hebbian coefficients then your algorithm just jumps around kind of um in in weight space around the zero point however if you actually learn these having coefficients as they do you have like this clear attractor here and you have these kind of


oscillating curves uh when you know when when you do that and you can see here in the different situations where things are damaged and so on so all in all i think it's a pretty interesting study and i think this neuroplasticity is it's a different way you know it's unclear to say if it will ever deliver the the performance that rl delivers but certainly there are situations where


such plasticity is desired and if we can also combine this with greater generalization performance then you know we have agents that can quickly kind of reconfigure and a lot of work by these this kind of open-ended learning community also plays into these roles all in all pretty pretty cool uh non-standard way of doing things last thing the broader impact statement uh every now


and then we'll look at a broader impact statement since these are new just to get kind of an overview of what they look like so they say the ethics computer societal consequence of this book are hard to predict but likely similar to other work dealing with more adaptive agents and robots in particular by giving the robot stability to still function when injured could make it easier from them


being deployed in areas that have both a positive and negative impact on society okay well again this it's it's not really giving robots the ability to still function when they're injured i first i thought first i thought okay they train it when it's fully functioning but then they damage it during test time but as i understand it as i understand the paper they already train it with the damaged


versions they just don't tell the algorithm in which version it is right now so um it's not the same as being able to work when injured unless you've specifically trained for it in this case again i could be wrong about this yeah in the very long term robots that can adapt could help in industrial automation or help to care for the elderly on the other hand more adaptive robots could also be more


easily used for military applications the approach presented is papers far from being deployed in these areas but is important to discuss its potential long-term consequences early on now okay so let's evaluate the broader impact statement let's well the first check to do is always to simply replace um whatever their method is with the word technology okay so let's do that


in the very long term technology could help in industrial automation or help to care for the elderly check on the other hand technology could also be more easily used for military application check the technology is far from being deployed in these areas okay i guess some technology isn't but advanced technology yeah so again the rule for broader impact statements seem to be


you take whatever your method is and you go up until uh you find you know you're basically at technology or something equivalent uh because no one actually i've never seen a broader impact statement that writes about the actual thing in the paper they always go up like one layer or two and then it basically regresses to technology even even though very few papers actually would be able


to discuss their particular thing but you know um and that and then in terms of guidelines on broader impact statement this one is missing there's there's always this um the holy trifecta so the holy trifecta is you go like a you know like you're a you're a catholic uh you go with your finger to your head chest left and right and you say technology good technology bad technology biased okay


so if you want to write a broader impact statement go up the layers technology good bad bias and we're missing the bias here so that's you know i'm just following what these guidelines two broader impact statements are i don't make the rules i'm sorry the the hebbians make the rules apparently um i'm not having okay i've i hope you've enjoyed this paper and this video


hi there today we're going to look at curl contrastive unsupervised representations for reinforcement learning by Aravind Sreenivas Michel Laskin and Petra Biel so this is a general framework for unsupervised representation learning for our L so let's untangle the title a little bit it is for reinforcement learning which it if you don't know what reinforcement

CURL: Contrastive Unsupervised Representations for Reinforcement Learning


learning is I've done a bunch of videos on are L afraid works so it's for general reinforcement learning that means it can be paired with almost any RL algorithm out there so we're not going to you know dive into specific or allowed rooms today it is unsupervised which means it doesn't need any sort of labels and it also doesn't need a reward signal forum RL which is pretty cool


because usually the entire RL pipelines rely on some sort of a reward or auxiliary reward signal now there is a training objective here but it doesn't have to do with the RL reward and then in the it is learning representations which means it learns it learns intermediate representations of the input data that is useful and in the end it is contrastive and that is the the


kind of secret sauce in here the training objective it's what's called contrastive learning and that's what we're going to spend most of our time on today exploring what that means alright so here's the general framework you can see it down here sorry about that so you can see that reinforcement learning is just a box which is we don't care about the RL algorithm you use that's just you


know what what comes at the end what comes at the beginning oh here is the observation so the observation in an RL algorithm is kind of fundamental now if someone explains RL to you or reinforcement learning usually what they'll say is there is some kind of actor and there is some kind of environment right and the environment will give you an observation right


observation Oh which is some sort of let's say here is an image right so in this in this RL framework specifically the examples they give are of image based reinforcement learning so let's say the Atari game where you have this little spaceship here and there are meteorites up here and you need to shoot them so there is a little shot here right you need to shoot those meteorites


right so this is the observation oh and then as an age as an actor you have to come up with some sort of action and the actions here can be something like moved to the left move to the right press the button that you know does the shooting so you have to come up with an action somehow given this observation and then the environment will give you back a reward along with the next observation


like the next frame of the game and you're gonna have to come up with another action in response to that and the environments going to give you back another reward and the next observation and so on so what you want to do is you want to find a mapping from observation to action such that your reward is going to be as high as possible right this is the fundamental problem of RL and


usually what people do is they take this act this mapping here from observation to action to be some sort of function some sort of function that is parameterised maybe and nowadays of course it's often a neural network but you're trying to learn given the input observation what output action you need to do and you can think of the same here so you have this input observation up


here and down here after the reinforcement learning the output is going to be an action right and so this this function we talked about up here is usually implemented sorry is usually implement as you put the observation into the r.l framework and then the RL framework learns this f of theta function to give you an action now here you can see the pipeline is a bit different we don't


want to shove the observation in directly right we don't want the observation directly but what we put into the RL framework is this Q thing now the Q is supposed to be a representation of the observation and a useful representation so if we think of this of this game here of this Atari game up here what could be the what could be a useful representation if if I


had to craft one by hand how would I construct a useful representation keep in mind the representation the goal is to have a representation of the observation that is more useful to the RL algorithm than just the pure pixels of the image right so if I have to craft a representation let's say it's a vector right let's say our our our representations need to be vectors what


I would do is I would probably take the x and y coordinates of the little spaceship right x and y and put it in the vector that's pretty useful and then I would probably take the x and y coordinates of the meteorites that are around right let's say there are maximum two XY XY here I would probably take the angle right the angle where my spaceship is pointing to that should be pretty


useful because if I shoot I want to know where I shoot right so theta here and then probably maybe the X and y coordinate of the of the shot here of the red shot that I fired if there is one right also going to put that into my representation so x and y and maybe Delta X Delta Y something like this right so you can see if I had to handcraft something if I I can pretty


much guarantee that if I put in this representation right here into the RL algorithm but put this in here it would turn out guaranteed it would turn out to be a better or L agent that learns faster than if I put in the original observation which is the the pixel image of the game right because of course in order to play the game correctly in order to play the game to win you need


to extract this information right you need to get our there's something like a spaceship there's something like meteorites this is all things that are elegant doesn't know her say and would have to learn from the pixels right but if I already give it the information that is useful it can learn much faster all right so you can see if I handcraft a good representation it's pretty easy


for the RL algorithm to improve now we want to come up with a framework that automatically comes up with a good representation right so it alleviates the RL algorithm here that reinforcement it alleviates that from learn from having to learn a good representation right it already is burdened with learning the what a good action is in any given situation right we want to


alleviate it of the burden to also extract useful information from the from the observation space right so how do we do this this is Q here is supposed to be exactly that it's supposed to be a good representation but not one that we handcrafted but a used with a technique that can be employed pretty much everywhere and the goal sorry that the secret sauce here is this contrastive


loss thing okay this bombed contrastive learning is this this kind of magic thing that will make us good representations so what is contrastive learning in this case I'm going to explain in this case for this kind of image based for image based reinforcement learning but just for image based neural networks how can we come up with a contrastive loss so you see there's kind


of a two pipeline thing going on here there is like this and this and then one of them is going to be the good encoding all right so let's check it out let's say we have this image that we had before right draw it again this little spaceship this and this and so right and we want to we want to do this what we need to do is we need to produce three different things from it we need to


produce an anchor what's called an anchor so we need to produce a positive sample positive sample and we need to produce negative samples let's just go with one negative sample for now right so the goal is to come up with a task that where we produce our own labels right so we want since we're training a encoder and the encoder is a neural network that's parametrized we need some


sort of loss function so the goal is to come up with a method where we can create our own labels to a task but that we construct the task in a way such that the neural network has no choice but learn something meaningful even though we made the task of ourselves all right I hope this was kind of clear so how are we gonna do this our method of choice here is going to be random cropping now


random cropping means that I just I take an image right and I crop a a piece from it so a smaller piece from the image I just take a view inside the image so in case of the anchor right I'm gonna draw the same picture here bear with me I'm gonna draw the same picture here a couple of times this is all supposed to be the same picture and with the negative sample I'm just gonna leave it


empty for now there are two meteorites two meteorites shot shot right so for the anchor we're going to actually not random crop but center crop right so we're going to take here the center image right so the assumption is kind of that if I Center if I Center crop I won't lose you know too much of the image I can actually make the crop bigger such that almost everything of


the image is somewhat contained in this and that yeah all right so this is going to be my anchor and then the positive sample is going to be a random crop of the same image so I'm just randomly going to select a same size same size section from that image let's say this is up right here all right and the negative sample is going to be around the crop from a different image right so


a different image might be from the same game right but might be there is a meteorite here right and there is no shot I don't I don't shoot and I'm going to take a random crop from this let's say I'm going to take a random crop here let's put a meteorite here as well just for fun all right so these are going to be our three samples and now the question is going to be if I give the


anchor to the neural network I'm going to say I give you the anchor right but I'm also going to give you this and this thing and I'm not going to give any of this I'm just going to give whatever I cropped right so just just these things so I asked the neural network neural network I give you the anchor now which one of these two which one of these two crops comes from the


same image right so as human you look at this and if you just see the center crop you see oh okay down here there's this this tip of this thing and then there's the shot right and in relation to the shot there is a meteor here right and then you look at the second one and you say okay I don't see the spaceship but there's the same relation here from the shot to the meteor and I can kind of see


the meteor up here and this also fits with that right and the the spaceship must be you know down here somewhere and then I go over here and I try to do the same thing is okay here's the meteor and you know it it might be it might be in the original image it might be over here somewhere so that's possible I don't see it right that's possible but then there should be there should be a shot right


somewhere here or sorry further up oops T there should be a shot somewhere here right I'm pretty sure because there's there's one over here and I don't see it right so I am fairly sure mr. tasks asked her that this image here is the positive sample while this image here is the negative sample right so this is the task that you ask of the neural network give it the anchor and


you ask which one of the of these two comes from the same image right this is called contrastive learning now is a bit more complicated in that of course what you do is you encode these things using neural networks and then so each of the things you encode so the anchor you're going to encode all of these things using a neural network right and then this is what's going to


become the query and these are becoming the keys so key one or key two and then you're going to feed it always two of them into a bilinear product right the bilinear product is simply you can think of it as an inner product in a perturbed space that you can learn so you're going to have this you have these two here these go into q WK one and then these two here sorry this and this go into q w


k 2 now W here is a learnable parameter right so you have some freedom and then you basically take whichever one of those two is highest right so this might be this high and this might only be this high and then you say aha cool this one's higher so this one must be the positive right and you train the W specifically to make this higher to make the positive ones higher and the


negative ones a lower so this is a supervised learning task right where these things here are going to be the lockets or or the so their inner product but you basically then pick the one that is highest as a in a soft max way and they put this in the paper so if we go down here the objective that they use to do the contrastive learning is this one so as you can see it's a soft max like


in multi-class classification of the inner product the bilinear product with the positive samples over the bilinear product with the positive samples plus the bilinear product with all of the negative samples so you're going to come up with more than one negative sample all right now the only thing left that we don't have here is that the encoding how you're going to come from the image


space to this space here is going to be slightly different and depending on whether you're talking on the anchor or on the what what are called the keys the things you compare to and this is out of a kind of a stability criterion you already maybe you don't you know like something like double q-learning or things like this it sometimes when you train with your own thing so in


q-learning you're kind of trying to to come up with an actor and a critic or it's not the same thing but you're kind of using the same neural network twice in in your in your setup and then you compare the output stored to each other which isn't you know it leads to instability so in our case we took it three times here or multiple times especially for the same objective here


we have twice something that was encoded by the same neural networking isn't the two sides of this by linear product so if we were to use the same neural network that tends to be somewhat unstable so we have different neural networks one that will encode the query which is this F Q + 1 which will encode the keys sorry F ok now we don't want to learn to neural networks and that's why


there's a bit of a compromise where we say it is the same neural network but but basically this one is the one we learn and then we always every now and then we transfer over the parameters to that one and in fact each step we transfer over the parameters and do an exponential moving average with the parameters of this momentum encoder from the step before so the


momentum encoder parameters are a moving average of the parameters of the query encoder and that is so you get kind of get the best of both worlds you don't have to learn a second neural network but your second neural network is not the same as your first neural network but it it kind of lags behind but it is also it is also performing almost as well so that is um I don't know if that


makes sense but it is the best I can to explain it so to recap you take your observation you encode it as a query sorry you crop crop here for your anchor that gets your query and then you random crop for your keys right into positive and negative samples right so random crop from the same observation or from different observations right these become your positive and negative


samples then you take you take me push this through your encoders for the query and for the keys respectively you end up with the Q which is the encoded anchor and the case which are the encoded positive and negative samples and then you learn you update this encoder here using the contrastive loss right and at the same time you feed the q you feed the q here into the reinforcement


learning algorithm and you learn your reinforcement learning algorithm instead of giving having the observation directly as an input here you now have the Q here as an input right that is it the reinforcement learning works exactly the same but except having the so input Oh you now have the representation input queue and you don't have to worry about anything else in


terms of the reinforcement learning algorithm it stays exactly the same right the this whole thing here can actually run either in parallel or you can think of it before you can think of it off policy on policy it is sort of modular how you how you fit this in it simply comes up with good representations so that is that is basically a deal here right and you hope


that the whole procedure of this contrastive learning then gives you good representation of this anchor thing here if you encode that to the queue you hope that this representation now is a good representation as a basis for the RL algorithm and it turns out at least in their experiments it is so here you see the same thing they actually they do something more where in RL usually deal


with a stack of observations not just a single observation because so for example in Atari people always concatenate something like for the four last frames right and their their point is okay if we have this stack here if we do this data augmentation you know these crops we kind of need to do them consistently right we need to crop every single image at the same point for the


query and also if we do a random crop let's say a random crop down here we need to do this same random crop for all of the of the stack of images here right so um that that is kind of the additional thing they introduced it with respect to RL that deals with with stacked time frames but it's kind of the same the same diagram as above here right so they explained the the RL


algorithms they use and exactly they're they're their thing and here you can see that anchor is a crop and the positive sample is a random crop from the same image this would be up here somewhere the anchor is cropped from the middle and then the negative would be a random crop from a different image or a different stack of images they have a pseudo code here where that


was pretty simple we'll just go through it quickly right you start off with fq + FK these are the encoders for the query and keys you start them off the same then you go through your data loader you do this random augmentation of your query and you keys and I don't not even sure if the random augmentation needs actually to be a central crop for the anchor or


just two different two different crops from the same image that might be as well so you know just I guess I guess it's a thing you could choose I don't know what exactly is the best thing alright then I forward the query through the FQ and I forward the keys through the FK then important I detach this so I don't train I don't want to train the FK I only want to train the FQ right then I


do the bilinear product here with the W these these are the bilinear product and then I put this all of this into a cross entropy loss right in the end I update my fq m IW and i do this exponentially moving average for my key encoder and they test on two different things they test on the deepmind control tasks and they always test 100 K time steps so their big point is data efficiency right


they they claim they can use learn useful representations with not much data so the task is here how good are you at one 100 cases that time steps right so you don't you don't optimize until the end you just you get 100k time steps and then the question is how how good are you and the curl here outperforms all of the baselines handily in the deep mind control tasks and it also outperforms a


lot of the baselines in the Atari tasks and it actually if you look at the results it doesn't outperform everything but for example here the red is curl and the dashed gray is state as a si now state si si the important thing to note here is it has access to the state whereas curl only works from pixels right so that what I said before like if I had to craft the use for a


presentation basically state si si has access to that and you see that in many of the tasks that the curl comes close or performs equally well to the state si si right so that's pretty impressive especially if you've took at pixel si si sorry which is the same algorithm but does not have access to the state just the pixels it often fails terribly right so um that is pretty interesting to see


Thailand GNN learning to tile with self supervised graph neural network many problems in computer graphics face combinatorial optimizations which are typically solved by approximation algorithms are heuristic search methods in this work we explore whether a learning-based approach can solve a classical combinatorial geometric problem tiling our problem specifically we focus on tiling the

TilinGNN: Learning to Tile with Self-Supervised Graph Neural Network (SIGGRAPH 2020)


interior of an arbitrary 2d shape using a given tile set while avoiding holes in tile overlaps our trained network can help produce tilings in time roughly linear to the number of candidate tile locations significantly outperforming traditional combinatorial search our learned to tile approach given an input tileset we first enumerate candidate tile locations


then we generate random shapes to crop and locate candidate tile locations after that we create a graph to describe each tile placement and train our network to predict tile placements with our self supervised loss at test time we applied the Train Network to predict tile locations for arbitrary shapes where we first locate tile placements and progressively fill


the shape with the health of our network overall this work has three technical contributions first we model this tiling problem as an instance of graph learning second we design a graph convolutional neural network to predict tile placement to be a graph convolution here we call our network tile and GNN third we define loss terms directly on the network output so Thailand GNN can be trained


morning everybody and welcome to day two of hike on our first two speakers are Angela liking and Lady moody say and they are from a major telco and there are data scientists and are going to be talking about anomaly detection using also encoders hello okay that's good morning everyone today we'll be sharing a talk about anomaly detection using what encoders I'm melody moody say hi

Anomaly Detection using Autoencoders


I'm Angela and we are both data scientists obviously at a telco and this is also our first time speaking at PyCon yeah so just to take you through some of the content we'll be sharing with you today we'll give you an introduction of auto-encoders how the algorithm actually works a brief history about them thereafter we will give you some architectures different type of popular

Anomaly Detection using Autoencoders


architectures of autoencoders which may be useful for you for your use cases also particular use cases that are out there where auto-encoders are good to solve we'll also be sharing some Python packages popular Python packages that you could use we'll take you through at repeater notebook we will introduce the notion of fraud anomalies and how to actually implement that then right after

Anomaly Detection using Autoencoders


we'll have a quick sense visualization to show you how assets data scientists interpret the results as well as for business stakeholders then lastly we'll be sharing key takeaways from our experience from implementing this type of problem so how many of you are aware of neural networks I'm sure most of us were there yesterday at Alex's talk so I'm sure you are familiar with

Anomaly Detection using Autoencoders


convolutional neural it works I mean he went into quite a lot of details feed-forward neural networks recurrent neural networks and all these type are solving for particular problems like computer vision machine translations and so forth Auto encoders are a part of the family of neural networks so yeah so as milady mentioned before right Auto encoders are a type of neural

Anomaly Detection using Autoencoders


network whose goal is to determine an output based on a similar input so as you can see right the goal of the input data is to be compressed so that it's in a lower dimensional space such that when the decoder comes along it takes that learn representation of that data the pattern such that it's able to replicate this learned image of this mushroom so now just to get a bit more in depth in

Anomaly Detection using Autoencoders


terms of the algorithm of the auto-encoders so it's split up into an encoder and a decoder the encoder is simply just a function of your input and your decoder is a function of your hidden layers now as you can see overall your algorithm is represented by the G f of X is equal to R now you want R to be as close as possible to your input layer so you want that data to be very very

Anomaly Detection using Autoencoders


close to each other so and that's exactly why the objective of an auto encoder is to minimize the loss function now what the loss function means is that you want to reduce and minimize the error between your input and your output the way that these neural networks are trained they are trained through back propagation and what that means is that is the recursive process such that it's

Anomaly Detection using Autoencoders


able to minimize the error between your input and your output and also something just really interesting but I think maybe you might find interesting what I do so autoencoders have been around for decades now people such as the laocoon and Hinton have used it are you all familiar with them I mean ok cool now let's move on to uses of auto-encoders right so the first being dimensionality

Anomaly Detection using Autoencoders


reduction that means that you take your data you condense it into a lower dimensional space the reason for doing that is so that your data itself can be more easily represented visually and this will really assist before you applied into a neural network the next example would be denoising of data you can see that initially these images are very hazy fuzzy it you know you can't really see

Anomaly Detection using Autoencoders


what's going on right now but through the power of auto-encoders what happens is that the noise is removed and it's a more crisper image you can see so now a third example is anomaly detection now what anomaly detection is it's basically a technique for identifying patterns within data so patterns that do not follow the norm so for example in autoencoders for example we have this

Anomaly Detection using Autoencoders


idea of reconstruction errors so if an observation right it's passed in and it doesn't seem very similar to its input like there's a drastic change there the difference then that would be considered as an outlier hence it would be anomalous so you would see these red images these red dots that's an outlier and lastly we get a view of feature extraction so auto-encoders give you a

Anomaly Detection using Autoencoders


view of which features in your dataset are useful or not so to take you through some of the different architectures that are out there of auto-encoders a very popular one is the restricted Boltzmann machine and this is actually produced this particular papers produced by our beloved Hinton and a restricted Boltzmann machine is basically a two layer or two encoder so how it works is

Anomaly Detection using Autoencoders


that it has a visible layer and a hidden layer the visible layer is where our input would come in our variable inputs it would use a combination of that to get into the hidden layer then basically what it's learning is the difference between the hidden layer and a visible layer it uses metric called KL divergence to measure that difference between the two this particular paper by

Anomaly Detection using Autoencoders


Hinton I would encourage that you go forward and read it if you want to get into auto-encoders it's basically how he used auto-encoders restricted Boltzmann machine and auto-encoders for dimensionality reduction and he actually compares this with PCA and the results that he gets is that with autoencoders he's able to reduce dimensions of nonlinear type of data so the results

Anomaly Detection using Autoencoders


that he gets the patterns that he under lives were much better than he got for PCA within the field of autoencoders there's two different types of popular architectures there's under complete and over complete so what angela has just described to you is an under complete architecture so remember what she said is that we're trying to find the underlying pattern within our input but

Anomaly Detection using Autoencoders


to do that what we need to do is to ensure that the neurons within our hidden layer are less than the neurons within our input layer to ensure that whatever our reconstruction of our output is it's not a direct copy of the input then it didn't learn the underlying pattern it needs to be a pattern that's how we ensure that so that is under complete so that's usually

Anomaly Detection using Autoencoders


for most use cases that's how we implement an an autoencoder then we have the over completes architecture and there's three different types of them so this parse this denoising and contractor so how many of you are familiar with regularization in neural networks okay a few of you so within your own networks um what we usually do is that within these different if if we find that our neural

Anomaly Detection using Autoencoders


network is overfitting what we sometimes can't do one technique to listen that is to put in a regularizer which means it's to penalize the variables within our weights right but with an autoencoder the spa's auto encoder uses a regularizer but it regularizes the activation functions before getting into the hidden layer as well as here that is to say that you could have an

Anomaly Detection using Autoencoders


architecture of any type however some of the activation functions are not being initialized that means they not all of the inputs would have been necessarily used so if you experiencing if you build your auto encoder and you like oh my gosh still not finding that underlying pattern there's a lot of noise in my data this is a good thing a good technique that you could use

Anomaly Detection using Autoencoders


another problem that occurs when implementing an auto encoder is that you get the exact copy like it's so annoying but what you can't do if you have such a problem is used the denoising so what do you know is ink does is that you add noise to your input layer and then you use the same under complete architecture of an auto encoder so it is quite a lot if your reconstruction layer is exactly

Anomaly Detection using Autoencoders


like your input the contractive is similar to denoising the problem with adding noise to an input is that you really don't know how much noise you should admin so what contractive does is that in your activation functions it finds the derivative of each activation function so it reduces what they as the inputs so what it's it but that in tells is that it's more robust to noise so the

Anomaly Detection using Autoencoders


more noise you have in your input because of those derivatives it's easier to learn that particular inherent pattern so we have many Python libraries available to us if you are interested in building your very own auto-encoders so first being Kerris which is basically just an abstraction level that sits on top of tensorflow then we have pi torch and then we all

Anomaly Detection using Autoencoders


know if I could learn I'm sure so the very well notes I could learn and then we have h2o but for this purposes for today we'll be showcasing h2o alright so now we've reached the stage of the jupiter notebook but before we begin there I just want to ask you all a question so who have you have experienced any fraudulent acts in your life just array the pans cool so that seems like quite a

Anomaly Detection using Autoencoders


few of you right now imagine like within industry as well they must probably be experienced vast amounts of fortune and activities that happen to them on a daily basis for instance we could look at within the banking sector were all very familiar with the tappan go system right so now imagine if a card is being tapped 200 times on the same day isn't that a huge red flag like

Anomaly Detection using Autoencoders


someone's clearly taking your money unless you really like chopping a lot LMO sorry and then telecoms we get for engine cases like SIM swap fraud or delivery jessa4 so for instance right it's your customer information however the product that's being delivered to you is not sent to your address but it's sent to an address that's who knows 200 kilometers away

Anomaly Detection using Autoencoders


from where you stay once again yet another red flag right and then in the retail space you can get fortunate acts like stocktaking or online purchases yeah so earn an example of an actual fraud case that has happened was what's called the Japan ATM scam this affected the standard bank that we know though it happened within Japan so what these fraudsters did it's

Anomaly Detection using Autoencoders


like for real Ocean's eleven but these fraudsters that it was around a hundred people according to the article it's a suspected out around a hundred people went to various of ATMs within Japan and started taking out cash one of the banks that were affected within South Africa was Standard Bank and Stennett Bank lost 295 million rent from this particular activity they did this under three hours

Anomaly Detection using Autoencoders


that's unlike the solutions so we would if I was the CEO of Standard Bank I'll definitely be like okay you fool me once I definitely wouldn't want again criminals are stealing from me the exact same way rights so we find such an emergent type of fraud that occurs and a business get scared so what we do is that we would ensure to reduce that we'd have either supervised learning model or

Anomaly Detection using Autoencoders


rules so that in let's say the first month we get such a big spike of fraud in the next month that we want to reduce that so we would compact that but definitely those guys who stole that money I'm sure they have a new creative way of of stealing from a different type of company all standard bank and so what she wants to do within an organization is to try combats

Anomaly Detection using Autoencoders


that emergent type of fraud so you can have usual fraud cases but also new types of fraud and if we have an actual algorithm that does work well let's say it's 70% accurate maybe 50% of that money could have been saved cool so just like my lady was mentioning you have like the whole idea of emerging fraud and then like a rule based fraud so banks if they really know the kind of

Anomaly Detection using Autoencoders


fraud that they that are happening right now there's just these will based systems that'll combat it so just once you explain the concept behind anomalies first fraud so as you can see in this Venn diagram right something that is anomalous does not necessarily mean that is fraudulent but something that's fortunate may mean that it's anomalous cool so now I just want to ask you guys

Anomaly Detection using Autoencoders


as well have a look at this table what stands out to you what is the anomaly yay you get a chocolate cool so fantastic but now but now if we think about this right imagine in a real-life situation when we we're not only just looking at six rows now we're looking at ten million rows and we want to cater for real-time situations in real time are we able to identify the anomalies in

Anomaly Detection using Autoencoders


the data set and not only what we just have password change occurrence as a variable we'll have millions more so cool that's where anomaly detection using water encoders can play a role so now we move into the Cagle data set apologies for spelling we are data scientist not English teachers cool right so the keidel data set is just the data set which I'm sure you're all quite familiar

Anomaly Detection using Autoencoders


with it's called the credit card data set and it's based on transactions of customers so as we begin you can see that this data set is highly imbalanced as you can see there are very few fraudulent cases which make up 0.17% of the data set now for machine learning algorithm to learn such a thing it makes it really difficult so cool but we'll explain how to combat that later on and

Anomaly Detection using Autoencoders


then we read in our normal import so because we were speaking to h2o we'll be using the h2o deep learning estimator library we read in our normal packages we then begin by initiating your spa context and your h2o context followed by now this is where the fun begins right we read in our data set using spark we transform that spark data set into h2o because remember we're working with h2o

Anomaly Detection using Autoencoders


models right now you can't pass in a spark data frame into an h2o library so hence you need to convert it then over here we defined our features list now because this is an online data set it's anonymized but in real case situations these features could represent things like maybe the number of times you've was drawn from an ATM is your card linked to the app how like how often are

Anomaly Detection using Autoencoders


you in overdraft you know just like those kind of features then you take your data set and you split it into a chain test and then remember before we were showing or showing you how the data itself was highly imbalanced so in order to combat that right you train the model on what looks like normal what is considered normal you chain that so that the model learns so that when it's given

Anomaly Detection using Autoencoders


unseen data and it picks up patterns that don't follow what it learned then it will flag that as an anomaly cool so now we begin with defining our h2o deep learning estimator we pass it through a variety of parameters I'll just go through a few so the one being the model ID which is purely just the name of your auto encoder so when you do save it for reuse of later on you can reference it

Anomaly Detection using Autoencoders


an activation function of ten and a few hidden layers then you chain your model over here now you savor your model cool so now that we saved our model we want to reload it now that we've reloaded the model this is where the fun begins this is where we actually identify anomalous behavior so we apply it to a testing set and we produce these reconstruction errors now if you

Anomaly Detection using Autoencoders


remember these reconstruction errors are how different is the output from the input so as you can see equal this is the overall reconstruction error but now what if you are interested in identifying the reconstruction errors per feature we can view that over here so this over here will show you the reconstruction error per feature so it's just to show you and give you a sense of

Anomaly Detection using Autoencoders


which feature contributed more to a particular observation for a customer yeah and then if you are interested you know after this presentation you can go home and you can build your own auto-encoders you can visit the Kegel comm website and you can get this data set and so just a recap right of reconstruction errors in terms of like a real-life situation with this image

Anomaly Detection using Autoencoders


right the input data would be your pixels and then the output would be the reconstruction errors without the noise cool so I hope that clarifies reconstruction errors all right so now we're going to get into showing you the trick sense dashboard that we've built that we as data scientists and business stakeholders may be interested in might be a bit cheeky with tricking and

Anomaly Detection using Autoencoders


holding the mic so as you can see within this dashboard we chose what you see in over here is what's down then red is the normal pattern that the algorithm caught and what we have above in blue is the anomalies so how we picked the anomalies is was we we picked a threshold of 0.01 and we picked that threshold just based on what we saw from the particular diagram so what the amounts of anomalies

Anomaly Detection using Autoencoders


we caught is a hundred and five anomalies right so let's say I want to check as a data scientist that okay how much was actual fraud and how much did my prediction was one as you can see or if I close predictions as you can see that I my anomaly detection model picked up 69 fraudulent cases out of 89 if I want to check as the data scientists that how many fraud

Anomaly Detection using Autoencoders


cases that my predictor not get so out of all the fraudulent cases it didn't pick up 20 fraudulent cases the anomaly detection model so it's not a bad model it's pretty it's pretty neat for a fraud project well as you can see here we have 0.16 percent of fraudulent cases and more there's more anomalies that we found which goes to the results that we have what you see here is actually the

Anomaly Detection using Autoencoders


reconstruction errors that Angela described for each and every variable so from fraud analysts point of view that first one might be number of possible changes our example our initial example so the fraud analyst would see that okay these are the variables that are most impactful for different types of fraud if the if the fraud analyst wants to see for a particular customer customer one

Anomaly Detection using Autoencoders


seven zero five they experience fraud we picked up that fraud they would see that these are the variables that actually impacts at that particular customer what we added to the dataset Angela added we added places we just added that randomly but usually with a project you want to know maybe a particular area that is experiencing fraud and by how much and see what variables are impacting there

Anomaly Detection using Autoencoders


so that you're able to contact the customer and help them accordingly okay so just to share with you some key takeaways that we have from building this models at scale the first thing is the interpretability so we showed you how to interpret for the particular for this particular model and what does happen that if you have quite a lot of features it can be difficult for a fraud

Anomaly Detection using Autoencoders


analyst or whoever the business stakeholder is to be able to interpret why is this particular case fraud so that's a common problem that we have another problem is that and maybe this is a general machine learning problem is that if there isn't an underlying pattern in your data then the or think ah don't do anything for it so if that's the case you could think about building

Anomaly Detection using Autoencoders


more features that may assist you to get a particular pattern then when it comes to maintainability when we build it at scale a big reason why we chose h2o because we build models at scale with that so if you want to build it you can use the Cagle data set but we also chose it because of that then just the difference with k-means and an autoencoder using an anomaly detection

Anomaly Detection using Autoencoders


problem so we have used k-means before and you'd find the distance between the chester centroid and the observation does show anomalies but maintaining that code so when you have to retrain your model with new data your cluster sense has change your clusters change inevitably what you are trying to find in anomalies to change but with an autoencoder it is much more consistent then with a

Anomaly Detection using Autoencoders


threshold this goes to capacity so you saw we detected 105 anomalies when you are working at scale with much more data it might be 10,000 anomalies now sending 10,000 anomalies to an actual business they call that to work through might be a bit difficult so they might not have the capacity to do that so picking a threshold usually you have to work with the business area to understand what

Anomaly Detection using Autoencoders


threshold is been suited for them and then lastly a feedback loop we would want overtime to know that what we pick up as anomalies was actual fraudulent behavior and sometimes getting that feedback loop is difficulty so just to really like sum it up right so concerned parent if all your friends jumped off a bridge would you follow them machine learning algorithm

Anomaly Detection using Autoencoders


yes so basically all in all just want to sum it up to what we spoke about today just because a model may say that something is anomalous at the end of the day you also need to check does it make sense to the use case now that make sense to the stakeholders I don't just listen to the machine learning algorithm there's still more to it you still need to bring in like you have the human side

Anomaly Detection using Autoencoders


and to understand that it makes sense with business say yeah I mean thank you so much for listening thanks ladies presentation was great with regards to the Reconstruction era have you experienced any cases where you've got like really high variances in the range of your reconstruction errors and then like if you have what approaches have you taken to like scale those or have

Anomaly Detection using Autoencoders


you worked with them just as you know so let's say you've got a reconstruction error on 0.05 on one observation and then you got on another observation a reconstruction error of I don't know like a hundred right so in that case like have you experienced that and if you have like have you dealt with any sort of like normalization of step or standardization reconstruction era the one that is

Anomaly Detection using Autoencoders


higher to us we looking for that it's noise it's the anomaly that we're trying to find but so yeah we haven't we haven't dealt with that yeah even with real with these cases that we've worked on our Styles like ok but I think it goes to those different architectures that if your general generic Reconstruction era it's not finding the pets and there's still quite a lot of

Anomaly Detection using Autoencoders


so then let's get started for today welcome to lecture 10 of cs2 9458 deep unsupervised learning now this lecture will be on compression before we dive into that a couple of logistical things there are main logistical things that are ahead of you are your project milestone which is a three-page Goldbach intermediate report is due on Monday so we must look forward

L10 Compression -- UC Berkeley, Spring 2020, CS294-158 Deep Unsupervised Learning


to reading those giving you feedback in the days after the deadline so you can gonna make sure you're maximally on track for your full final project the other thing that's coming up in two weeks we'll have our midterm which will figure out how to do it remote under the current circumstances but the main thing we'll do later this week is release a set of study materials for you that


capture the core of the things covered in the class their very core compressive a little bit because of how much we're going to have you study because of course a more difficult semester than most you do outside circumstances so a relatively short study guide and it'll be a PDF with the questions and the answers and so you'll know exactly what the questions can be and what the


answers are that we expect you to get so that will come out later today or tomorrow for you to study link pause here and see if there's any questions about logistics oh and by the way this lecture is recorded so for some reason you you know don't like your voice to be heard just like with the in-class lectures that were recorded then please be aware of that alright then let's get started with the


Contin first day so compression what is it and why would we care in general and why would we there in this class so what is it it's an data you might want to reduce the number of bits for encoding a message a message could be an image you want to send or part of speech or maybe some music you want to send across on communication line and it's an original in a


collisional format might take up a very large number of bits and you might want to be able to get that same information across by seeing last bits in the communication channel so what does it look like you have some bit stream B on the left here so that's where you start out with then what happens next is you want to compress it and end up with a compressed version of that bit stream


and the hope that that compressed version has lost its anatomy original so when you send a compressed extreme over a channel or stored on a hard drive or whatever you want to do with those bits in a more compressed way then it's ideally a lot less good but then when you want to use it later you should be able to expand it back out just the compression into the original alright so


why do we care well you could save time you could save bandwidth over communications channel you could save space when you're storing it so many reasons you might care about this from the AI point of view and part of why it's interesting for this class is that often the ability to compress data reflects understanding of the data by the system that compressed the data so


if you throw this in that's really by compressing data that means that system somehow has absorbed an understanding of the data so now there's two types of compression lossy versus lossless compression in this lecture we'll be fully focused on lossless compression where the original bits can be completely reconstructed on the output now sometimes in practice you might care


about lossy compression you say well I don't need all the details back as long as I can save more bits I'm happy to lose some detail that would be loss of compression not the topic for this class but also a topic you might be interested in at some point so I want to make sure you know it exists now one of the very interesting things with compression there are some prizes associated it


so recently hutter I should increase the price used to be a 50,000 euro prize for compressing human knowledge and recently it went up to factor 10 it's now a five hundred thousand euro prize if you can compress human knowledge what does it mean more concretely so there's a one gigabyte file of I believe text and this file here and with nine and if you can compress that to less than one hundred


sixteen megabytes you win the prize you won this thing you cracked it the reason how to read out the surprise is not so much because it specifically wants that one gigabyte compressed into one sixteen megabytes because he believes now we go by it has interesting information that any system that can represent it as compactly as one sixteen megabytes must have made its hopefully that's what it


thinks some AI advances to be able to do that it's pretty interesting here because unlike most things we've covered in this class and you'll see in any kind of machine learning there's no train at st it's not that he asked you to send in a compressor and has a secret test set is gonna test your compressor on to see how it works no it's literally there's a 1 gigabyte file and if you can make it


smaller small enough you win the prize what's gonna be able to decompress it so you gotta be able to effectively send him something that's 116 megabytes or less and includes the code for decoding back into the one gigabyte so you'd be sending effectively both the decoder program and some encoding of this monkey bad file together it would be able to reconstruct the original 1 gigabyte file


so very very specific problem there's no test said just that one training example but nobody's got them close to actually making this work so interesting challenge maybe something you want to think about at some point and see if it can make some progress then there's another compression challenge on images so this is often held at CDR the main conference for computer vision and so


there's a workshop there that looks at how well your compressor and there it's really about a compressor that you send in compressor of course and they have a secret asset on which they test how well you can compress and decompress the test examples so two very different challenges but both very much at the core what we're going to be thinking about today in lecture all right so why


in this course it turns out that we've studied a lot of generic models in this course it also turns out that compression utilizes generative models so the better it narrative model the better the compression can be and in fact Jonathan who will cover the second half of this lecture has made several breakthroughs in this PhD research showing how some of the state-of-the-art


generative models can be converted into compression algorithms with the CIO narrative models under the hood such that you can get better compression now you might go get otherwise and we'll cover that later but so there's a very close connection between better generative models and better compression there be material we would recommend for this lecture is this PDF


overview to the nice write-up that covers the background on essentially information theory slack impression that we'll be covering in this lecture at least the first half in the second half will dive a lot more in the deep learning aspects and how and tied it into this so some applications you might have seen jarick file compression gz p z 7z a zip file systems various multimedia formats


you might have seen a file - fake file gif file mp3 mp4 communications that maybe you don't see news anymore now but where compression played a big role in the past fax modem Skype and so forth and all of these are examples of where here original information might have been represented with many many bits too large for you to a store on file in that format and because you can reduce them


or they can get back out the original you can now store it more efficiently or send it more efficiently over get in line when he said it more official over communication line it can reduce both the amount of does he need to send then in the process also reduce the latency because it might be less delay assuming you can decode quickly on the other side now maybe you might have


followed this TV show called Silicon Valley it's uh well pretty finish I would say with many things that are maybe a little too close to home and too close to true but still pretty funny and if you watch that show on HBO you have noticed that actually the company the central company Pied Piper would they put forward as their product is the well a middle-out compression algorithm


nobody knows what middle out this but they put forward a compression algorithm and that's the secret sauce of their company turns out that some people really do this for their actual company so there's various startups out there that you invent don't disclose exactly what's under the hood but invent new compression algorithms using machine learning under the hood most likely to


improve upon past state of garden compression now the specific point I shoulda named itself after the Silicon Valley show which was called the company's called Pied Piper and this is now called pact pie so there's actually a real thing they presented at TechCrunch in 2015 now first question you might ask is can we have universal data compression so their fundamental


question you'll see in this lecture a lot of questions we ask them to be very fundamental where we can give actually very very strong theoretical answers sometimes negative answers so can we come up with universal data impressive would that mean that would be can we come up with something that no matter what let's say filing you give it it can make it smaller and later decompress it


back out to the original well the things that's possible well okay let's see imagine you want to compress every possible to compress every possible possible bitstream they ever encounter okay so that's not possible no longer we can do this what's the intuition that should be simple we'll do a proof by contradiction suppose you have a universal data compression algorithm you that can


compress every bit streams no matter what your feet it is gonna make it less bits and then go to decompress it back out later to the original okay now given bit string B 0 you can compress it to get a smaller bit string B 1 if it strictly last bits otherwise it's not a universal compressor now B you want you can feed into it again it'll turn that into B 2 which is yet smaller you keep


doing this you do this especially many times at some point you'll have a big string of size 0 at that point it's obvious you cannot recover what the earth will lost kiss you it could have been anything and everything gets turned into 0 you can get back out will int it so what this shows them assuming somebody tells you I have a universal data compression already compress


everything no problem here's a prove that this is actually not possible to prove it another way also another way to prove it is to do it by Counting you can say okay suppose your algorithm can compress all thousand histories ok how many thousand bit strings are there we throw the one thousand possible bit strings now if we can compress all them that means we can


pick every one of them and turn them into something smaller and distinct smaller otherwise we cannot get the original back out but if we look at what's possible with all possible shorter bit strings actually you cannot encode all two to the 1000 possible thousand bit strings so since we can't include all possible to the 1000 bit strings it means we cannot compress all


of them so we have two different proves here to show that the universal data compression just not possible why is impossible and practice though even if you cannot universally compress everything well there are statistical patterns that you can exploit for example here's a piece of text and I'll give you all a minute to read this text so as you're reading this text you'll


you'll notice well likely you'll notice that there's something fun about the text and that the words are mostly misspelled but despite these was being misspelled it's how she's still very feasible to read this and effectively what it says is that most people have no problem reading a piece of text if for every word you keep the first two letters you keep the last two letters


but then everything in between you can randomly permute which means that the ordering of the letters in between has not much real information in it it could be any ordering so you don't need to stick to the original order in us here they do this crammed with we can still understand it so it means there's some redundancy it means that certain sequences are just not very likely and


when you read this it's close to a sequence that you're familiar with and so you can easily map it onto that and still understand words that were there originally there's another example from images so on the Left we see a bunch of real-world images of flowers in this case on the right we see random data the data on the left if your dataset looks like that it's very compressible because


there are a lot of regularities at an intuitive level example often maybe pixels have roughly the same value whereas images on the right which are completely random there is no correlation between neighboring pixels that you can exploit to maybe compress how to represent the data on the right so two very different distributions for the complete random distribution not clear how to compress


for the real-world type data you can already intuitively see that there are opportunities to compress for example you could just think of every other bit now every other pixel it wouldn't be perfectly lossless we could probably reconstruct most of the image from that alright so what we've covered so far is what is compression what's the goal in compression and why might we care go


from a practical point of view and from a AI point of view we also looked at the fact that universal lossless compressor is just not possible we looked at some intuition as they're being redundancy in most of the data that we encounter in the real world and because there is redundancy I'm totally speaking there should be a way to exploit that because it's only the did


that really occurs in the real world that you need to have a good compression for and the data that doesn't really occur in the real world even though they are also can be represented in principle bit strings you might not care much about how much that kind of non real-world that gets compressed so for the remainder of this lecture we'll want to look at is a couple of things first


thing we want to look at coding off symbols so we'll start looking at okay what does it mean to actually have a compression system and this will only comment with a method called Huffman coding that is actually used in many many of today's systems and it's actually quite intuitive very simple way to understand how compression could effectively work so I'm going to look at


some theoretical limits and from there we'll look at some additional consideration for coding that will help us a bit more than what we get from the simplest version we cover first from there we'll tie this into things that we've covered in this class we'll look at our aggressive models will look at Vee we'll look at flow models and try to understand how these models can be


leveraged to do better compression all right let's get this started so here's one way of coding information all right a way and just to be clear there's no compression and this wave coding so ASCII is a system that for everything that's on your keyboard will assign seven bits so every character you can type can be represented with seven bits so two to seven possible characters


could be represented this way what's nice about this very easy for kody and decoded there's a very simple one-to-one mapping always going to the seven bits for that character and back out to the character but if you encode this way you're not exploiting its disco parents is Nagano it's not compressing your information maybe some key strokes are far less likely than others so maybe the


ones that are less likely you should allow it to use more bits and the ones that are very likely you should try to represent with a very small number of bits and overall you might have a win that's the intuition behind law compression schemes but obviously here everything seven bits that's not going to happen let's in at least a reference as a starting point so we'll need a spare be


length codes codes down assign different lengths depending on how likely a symbol is how do we avoid ambiguity a simple way to avoid ambiguity when you have variable lengths is when your fix times is very easy first seven bits per character next sum of its next character December's next target but if it's variable length how you know a character has been fully translated and now the


next one is starting one way to do this is to ensure that no code word is a prefix of another code word so as you see bits come across the line at some point you'll have seen all the bits for some letter let's say and because no code word prefix of another one at that point you know there's nothing else that can continue from this this is the thing is the complete thing sent across and


the corresponding character can be decoded so another thing you can do but that might be consuming more bandwidth to space you could also end the stop character to each codeword the Morse decoding does this but this might be a little wasteful we can have a general prefix-free code and we'll look at that very soon so let's look at Morris first so in Morse code in what happens this


very old coding scheme this is where I'm back when you want to send let's say you just have a effectively a line the communication line over which all you could send was let's say voltage going on and back down you can make it go up briefly or go go up for longer for three times as long so a dot is a brief spike in or up in your voltage let's say and then a dash is three times as long and


then the spaces in between the dot and dash are also encoding that it's it's quiet the reason it's a quiet time in between and then between characters there will also be three so between total of three units and then between words there will be Sun units of quiet time so that way you can encode every character alphabet all numbers and all I need to be able to


do is send dots and dashes or short and longer signals and pauses be able to get everything across and people use that before before telephones time to go up Telegraph where they could send information across this way so some of the things you can already see here the letter A as a relatively short encoding same for e same for I and that's because those are frequent frequently used


letters and then things that are less frequent like maybe a Z has a longer encoding what else is left frequent there J longer encoding essentially well more or less to the letter sound you get a lot of points for in Scrabble have the long encoding skew here X over here and the letters that don't give you much points in Scrabble have these shorter in CONUS because there's many more words


that use them okay that's that's a very specific thing the more general thing that people tend to use a cynical prefix-free codes which can be represented as binary tries or binary trees so what is a binary try or tree it's a tree where one thing's split you hard-code ahead of time that the left will be a zero the right will be a one and so you can build a tree and you


don't even have to put the zeros and the ones on it that I'm putting on here because you always know those side is a zero the right side is a one that says their specific type of data structure the reason it's spelled with ie is because comes from retrieval it's a data structure for easy retrieval of certain information so that's also why it's often pronounced binary trees because it


comes from retrieval at the same time there's also threes this way that also our data structure and so it can be a little confusing if they're pronounced the same way so some people will still pronounce this as trust to distinguish from trees so that's a binary tree the way we're going to use it is that the symbols will always live in the leaves and a code word is a path from root leaf


so let's look at some example here's an example we have a code word table we have one two three four five six characters each character has an encoding with a sequence of bits of sometimes only one bit and you can see that there's a crisp honest adapt in this binary tree where all the characters are sitting in leaves of the tree because every character sitting in


a leave of a tree what it means lets him I'm getting some message across say I'm receiving this message over here I'm receiving a zero what will I do I will go down this path I'll say oh I hit a leaf that means I'm at the end nothing left to go ID code an A then I get this one over here we just restarted means good this way I get another one you this way get another one Musa go this


way get another one means go this way I hate to leave I know I'm ready to decode and it's a B and so because all the symbols live in the leaves I always know when I hate to leave what's him I need to decode and then come back to the top to start decoding the rest a message that's coming in now you can of course ask yourself the question for a given set of symbols that


you want to send are there multiple binary trees and in fact there are there are many many trees you could put forward to come up with a podium for these six symbols here is another example here the tree is set up a little family and we see them the same string being compressed twice and on the left it requires 30 bits on the right it requires 29 bits and so the name of the


game here is can we find a binary tree set down as I tried to encode my message my bitstream sir as I try to put my original symbol message into a bit stream that a bit stream is as short as possible and race where you could search over all possible binary trees but there would be many many many binaries and then apply decide which one is most efficient but we'll see better schemes


than that but you have a very naive way to put is just try all possible binary trees that have the symbols at the leaves and just see for each one of them how long the bitstream is and take the best one we'll see an efficient method to get very close to that I should we get to the optimal one okay so the efficient method to find the optimal one without needing to do that exhaustive


search I just described is something called Huffman codes and right now but we'll cover how Huffman codes work procedurally and then later once we've seen a bit more foundation on information theory we will also prove the fact that they are optimal but for now we're not going to yet prove that there are more you're going to look at the procedure okay so how does it work


Huffman algorithm said they're very simple consider the probability P I of each symbol I that's in your input so you have maybe a belong text file if you're encoded characters you would for each character do a count and then see what's the probability for each character to appear once you've done that you can start with one node corresponding to each symbol so for each


of these symbols you have a node so starts as a disconnected tree just a bunch of separate Leafs really but not really connected up to anything yet and you associate with it a weight P I which is the probability of that symbol from there you repeat the same process over and over until it's all connected together in a single tree what is this process you selected two trees with min


probabilities P kmpl initially when each symbol is its own thing what that means is you find the two symbols with the lowest probability later on once you've done some merges it'll be the trees that at the root have the lowest problem then you merge those two into a single tree with associated probability as the sum of the original probabilities and that's it that's all I need to do so let's take


a look at an example of how this works on some example data here we have six symbols each symbol has its own probability associate with it and so let's step through what Huffman coding does we have six symbols each your own probability we have a with a probability of zero point two we have B with probability zero point one C per building 0.05 T with probability zero


point two one e with probability zero point three six and F with probability zero point zero eight let's follow the algorithm what a to lowest probability thinks it's C and F so what do we do we connect them up CNF get connected up and together they have the sum of the probabilities which is zero point one three what's lowest in probability now it's the zero point 1 here and the zero point


1 3 over here so we'll connect this up and the top here now is probability 0.23 what's lowest now we have a zero point two and a zero point two one let's go point two one is somewhat inconveniently located so I'm going to move over here so be zero point two one and moving it off to the side zero point two one and DNA connect together for a zero point four one what are the two lowest now is


the zero point two three dots here and a zero point three six over here is 0.36 is incomming located I'm going to relocate e over here all right then connecting these and then together they have zero point five nine the two lowest ones are the only two left is zero point or one and the zero point five nine and here is our Huffman encoding and then what we do is like the left side is 0


and the right split is 1 there we go and now we have an encoding we want to know what is D D is 0 0 what is a a is 0 1 what is B B is 1 0 0 1 0 0 what is C 1 0 1 0 E is 1 1 M F is 1 0 1 1 and to say uniquely decodable code every symbol once you have sent the bits across you hit the leave of dizzy coding tree you know you've got an entire symbol and then you start again at the top of the


tree to decode the next symbol so we haven't covered why this is optimal but hopefully the procedure is clear that is a relatively simple procedure they can do for any symbol table that you have and it relies on these probabilities and you might already start foreshadowing here chorus this is what your narrative models might be handy very good generative models might allow us to


build good probability estimates that we can then use to find a real good encoding because of course if these probabilities are wrong then this tree will not be a very good tree for encoding the data so here's the same another example that you can work through solving another example and especially under many of the applications that are used on the internet Huffman codes are used to


compress their data all right so maybe let me pause here for a moment to see if there's any questions so have you any questions feel free to type them into the chat window or to just speak up or raise your hand oh hi drew yeah I had a question so I guess something that I noticed about Hoffman codes and stuff would be that you're kind of the number of symbols or


number of values that you have is fixed but if you're trying to let's say encode more complex data structure so if you have something like you know maybe maybe images have like a fixed dimensions to audio can be multiple dimensions for example so is there a way other than discretizing or is the notion just to make chunks and then compress them effect sized chunks which take discrete


values yeah very good questions so chunking is between an option and then just send over in chunks which by the way often can be desirable for other reasons also even if you had a fixed size thing but you wanted this let's say you had a video you wanted to watch on home if somebody first encodes the entire video sensitive crosses one file and only then you can decode and play it's not great


you want to be able to stream it across so you have there are reasons to chunk where you're actually for bill optimality of compression but you reduce latency and getting things across we will look at some other codes a little later and it's a really good question that actually fits very well with her describing so the coding systems we'll look at later are thematic coding and


asymmetric numeral systems are able to encode streams in effectively a continuous way such that if the stream is longer it can keep encoding and it just on-the-fly continues to encode as you go along now in practice often people will still chant and stop at some point because otherwise you might have to wait too long before you can decode sometimes but in principle they can work


with arbitrary length and not knowing the length ahead of time so we'll cover that but you're absolutely right for Huffman codes do is that strong assumption that you have an alphabet of symbols and you build it encoding for a doubt alphabetical and you don't encode that specific symbols make sense thank you well compression usually be done in terms of bits like the encoding will be


the output of the encoder will be like a lookup table and it won't you say yes the way we think of compression and the way it's ultimately done on computers is that only what comes out is the sequence of bits you can think of a single bit as the sense of the minimal unit of information like a single bit can either be 0 or 1 and sometimes the minimum that you can send across in terms of


information just an outer 0 or a 1 because there's two options you can send information across if you have only one option while there's nothing you can do there's no information be transmitted so in fact often as we'll see the way information gets better than the size of well the amount of information your system gets measured is by bits the mineral number of bits required to


represent the original is the amount of information in that piece of data I say one quick thing to add maybe it is the case that when you actually transmit over certain lines that are not let's say a computer storing zero on bits there are transmission schemes where you send maybe two bits in one go to using potentially something close it to a continuous channel that you then


discretize on the other side to get out several bits in one go so that was happen also under the hood but in terms of kind of the information theoretic properties we tend to think of it as turning everything into a sequence of bits alright great questions thank you let's move to the next part which is threat of the limits and what we're going to cover here to me some of the


most beautiful math than any discipline has to offer somehow well we're going to cover we can cover in just a few slides we've been covering quite comprehensively in just a few slides and get very kind of deep throughout up on inside to guarantees across so very excited about getting to talk about that day so one thing you might have heard in the context of information fear this


thing called entropy DeShannon a sort of a measure of information so what is entropy by definition so it's just a mathematical definition not talking about properties yet entropy of X what is X X is a random variable right and so X is just really some distribution P of X and so whoo I measure entropy till the entropy of the random variable not a specific institution of that variable


but anyways the entropy of the distribution or entry of the random variable as a whole and this is definition use sum over all possible values the random variable can take on and you take a weighted sum of the probability of take on that value and then log 2 of 1 over P exile ok so this might seem look a little coming out of water but maybe let's get a little bit of intuition for this of why


this might be a meaningful way of how you measure entropy which is the amount of uncertainty you have in a distribution and hence there's a lot of uncertainty about the random variable you need more bits to send the cross we hope to tell the other person what the outcome us so there I have been unbearable I read an experiment I see the outcome of that random variable I


want to communicate the outcome to you how many bees do I need to send on average and this effect doesn't just talk about that but it also then kind of hints at an encoding scheme it kind of says the number of this you can I use for a outcome X is going to be log to 1 over P X I so let's look at an example distribution here's an example distribution random variable can take on


five values and will compute the entropy of this thing so it isn't like 1/4 1/4 1/4 1/8 1/8 and we can compute the entropy entropy is 2.25 then let's look at another distribution much more peaked beautiful these are three quarters and then 1/16 for everything else compute the entropy its 1.3 so 1.3 2.25 entropy is a lot larger on the left and on the right why is that because if I run an


experiment on the random variable on the left and then want to communicate the outcome Q actually there's many possible outcomes that are pretty likely and so it's not like you can come over to very efficient encoding scheme because you need to encode everything pretty much with some reasonable probability you're gonna have to send it across whereas here on the right what happens is this


first outcome over here scroll ugly so if you encode that first outcome with a very small number of bits then most of the time you have to send almost nothing and then yet sometimes you have to send more bits to get the other things across but most of the time is very cheap and so that that's effectively what's going on in this equation and for now a building intuition will make this a lot


more formal very soon so let's take a look at another example so think back to think back to our binary trees to encode some set of symbols we have symbol a b c and d if these are the probabilities 1/2 1/4 1/8 1/8 then this thing over here is a optimal way of encoding that have the time you sent just one bit for a the other half of the time you got to cover the rest and to say that you're covering


the rest you have to send the one then in the other half of the time half the time you have to send what it say it's be in the other half of the time you have to signal with the one that it's one of the other two and then at the end you decide which one it is when you're down here so what we can this even though we haven't proven this intuitively it should make sense that


this is a very good scheme for encoding this kind of distribution over symbols because you can't say anything less than one symbol for a otherwise you have not communicated anything a is the most frequent symbol I'll have to do is send one symbol and for being well you know if the first signal it's not a and then you send one more something symbol to communicate it to be similar than for


CMD this encoding scheme over here uses a length that is the to log of 1 over P of X I and so you could imagine a world if in the in your world every probability associate with any symbol you have some symbol X I and the probability P of X I is equal to 1 is equal to 1 over 2 to the length of sorry no let's see if ya the paralytics ID can be expressed as 2 to the power L I then


you can encode using the same scheme that symbol X I with a length L I bitstream into a tree that would be built up the way the tree was built up over here haven't threw in this but that's kind of the rough intuition and we'll see is of course things that generalize this to simple we're P of X is not necessarily one over two to the power of something that could


be any probability different from a power of 1/2 okay so that's some high-level intuition let's now take a look at some of the theory that we can put down first main theorem is the Kraft McMillan inequality what it says is that for any uniquely decodable code c okay so this is somebody tells you I have a code it's uniquely decodable and if it's not uniquely decodable you can't really


use it to do lossless compression so codes do need to be uniquely the code well unless we're not going to consider them for losses fresh when it comes up with a uniquely decodable code see what does that mean it means a mapping from symbols to bits bit strings corresponding to each symbol if it's indeed uniquely decodable then it's the case that this property holds true what


is this saying it's saying for each symbol and corresponding ain't coding so the word the bit word over you we can look at the length of the encoding and there's some property satisfied by these lengths so funny and somebody's your table symbols and bitstreams it's gonna be a and some bits string here then be some other bits here and so forth you give you a code that's


uniquely decodable then the lengths that you encounter here will satisfy this property this says it's smaller that one has to be smaller than one what does that mean these are negative powers here so it's effectively saying that the lengths have to be large enough they always going to be of a certain length otherwise it's not be satisfied so swimming out this thing is saying if


someone is uniquely decodable code I can guarantee you that the encodes have to be relatively long they cannot be shorter than a certain amount because otherwise they would not satisfy this property with Boston actually held in the opposite direction opposite direction says if you have a set of length L that satisfy that same thing then there is a code you can build