So the red line, as you can see, mostly, most of the time, they're going to either sample


mixup: Beyond Empirical Risk Minimization (Paper Explained)

the thing on the very left or the thing on the very right.


That means they either sample the first or the second data point.


But some of the time, they actually sample something in the middle.


And it's fairly uniform in the middle.


So it appears like a good distribution to sample from if you want to sample these mixing


coefficients.


And by adjusting the the actual number of alpha and beta here, you can determine how


many times you sample the original data points versus how many times you sample something


in the middle.


Okay, on this toy data set right here, they showcase what mix up can do.


So in a classic model, you have the orange and the green data points.


And blue is basically where the classifier believes its class one, you see this very


hard border here.


It's quite a hard border.


Now you only have two classes here.


And so the hard border is sort of a problem in itself.


Because if you think of, for example, adversarial examples, all they have to do is basically


get over that one inch and the classifier is already super duper sure it's the orange


class.


Right?


Whereas if you use mix up your border is much, much, much more fuzzy.


It's like, yeah, it's only really sure here, and out out here, everywhere.


But in the middle, it's sort of like me, I don't know.


And so that's kind of a more desirable situation.


Now of course, this here works particularly in this in this linear 2d setting.


But as we can see, the same reasoning applies to sort of higher, higher layers and higher


dimensionality data points, right, I have seemed to lost the ability to zoom on up,


that's back.


Okay.


And that's basically it for this paper.


This is all they do, they propose this method, and then they test it.


They say something interesting here that mix up converges to the classical method as alpha


approaches zero.


So that would push your beta distribution basically in the middle all the way down.


And you would only sample from the very left or the very right.


So you can smoothly interpolate between this mixing and the classic method.


So their main results are, we apply this to classifiers.


And what I like is since again, is also a classifier, so the discriminator is a classifier,


they also apply it to GANs and they outperform, and stabilize the classic training on GANs,


they show that it's more robust towards adversarial attacks, because it's not so sure about intermediate


things.


And they generally outperform other methods.


But also they do this nice investigation here, where they measure the prediction error of


in between data.


And what it means is, they say a prediction is counted as a miss if it does not belong


to yi or yj.


So you have a sample right here, xi and a sample right here xj.


And you look at what the classifier says in between the two data points.


So you just interpolate the two data points and just measure what the classifier says.


And whenever the classifier either says yi or yj, either label of those two data points,


you count it as correct.


And you only count it as incorrect if it says something else.


And you can see here, if you train with the classic method, ERM, these errors happen much


more often.


And this is exactly the situation I pointed out at the beginning, where in the high dimensions,


it can, you know, occur that all sorts of decision boundaries sneak here in between


the two data points.


And by interpolating between them during training, you sort of much reduce that, you reduce that


effect a lot.


Now this, they also say that the gradient norm of the gradients of the model with respect


to input in between training data, it happens the same thing, the norm of the gradients


in the middle is also much, much lower.


And this, yeah, this investigation, I find pretty cool.


I have to say I've seen mix up in practice.


So it might be useful.


I've read a paper where they basically say, oh, it was a big transfer paper.


Yeah, where they basically say, it is useful if you have, for example, if you have little


data and a big model, so you can sort of regularize the model.


And it is also useful to know that they did test this with dropout.


So they compared it with dropout.


And the conclusion is basically that this is something else than dropout.


So it's not doing the same thing.


Dropout of course, it means you drop out some of the data points in intermediate activations,


and that sort of gives you a noisy version of the data point.


This here can actually be combined with dropout, which means that it gives you an additional


benefit you see right here, most of the best numbers happen when you use mix up plus dropout.


So it seems to be just an additional regularization on top of dropout.


Pretty cool, pretty cool investigation.


Awesome.


Alright, so if you like this, I invite you to read the paper.


If you liked the video, please subscribe and like and comment.


And yeah, have a nice day.


Gradients are Not All You Need (Machine Learning Research Paper Explained)

Hi there! The video you're about to see is a bit of a mixed bag. I just wanted to say this to warn you ahead of time.


It's a bit more basic than other videos, so I spend a lot of time deriving backpropagation through time,


which is used for backpropagating through dynamical systems in these papers, or in this paper.


And also I spend quite a bit of time explaining the reparameterization trick and things of that nature.


And then after that, I go into three distinct examples that they give in the paper that all basically show the same thing.


So the video is maybe a bit longer than it needs to be, especially if you're already experienced.


Feel free to skip ahead. Just wanted to let you know such that you can choose the parts that suit you.


Alright, with that being said, this is a current research paper. It's quite cool what it shows.


It shows that you might not always want to backpropagate through things, even though you can, especially if they're iterated systems,


especially if they're noisy and chaotic. And they give some nice demonstrations of when that's actually not appropriate.


So, yeah, enjoy! Bye bye!


In summary, gradients are not all you need. Just because you can take a gradient doesn't mean you always should.


That's how the paper ends. Now, what paper is this? This is a paper called Gradients are not all you need.


And this is by Luke Metz, C. Daniel Freeman, Samuel S. Schönholz, and Tal Kachman.


This is a paper that argues against, in certain cases, against backpropagating through specifically dynamical systems that can exhibit chaotic behavior.


So it treats a bunch of applications of these things.


For example, when people backpropagate through physics simulations, when people backpropagate through inner learned optimizers and so on.


And it shows that very often in these cases, it can happen that the gradients you get have extremely high variance, are extremely poorly behaved and so on.


And that it might be better to just use black box estimators for these gradients rather than actually backpropagating through the inner dynamical system.


This might seem a little bit, this might seem a little bit, you know, far fetched and out there.


But this is actually happening. People are backpropagating through all sorts of things nowadays.


As I said, physics simulations are now backpropagatable. They're completely differentiable.


You can backpropagate through a physics simulation and get a direct gradient.


And the same goes with, as I said, learned optimizers.


So you have an outer optimizer that learns an inner optimizer and so on.


All of this stuff becomes differentiable and people are very excited about this.


But this paper argues that as it says, you may not always want to do that.


And this paper goes into the details of why that is the case, what can be done about it and where you should pay attention.


So they give a bunch of examples right here of these, what they call dynamical systems,


iterated dynamical systems that you are the basis for these observations.


So in a very basic case, in a linear iterated dynamic system, you have a state S and you apply a matrix,


a K, and that will give you the next state, s k plus one right here.


However, if you do that over and over again, let's say you always have the same matrix a and you just keep plugging in s in here and get the next state.


So you sort of plug it, plug it into a, it's a recursive system or a recurrent system one might call it, you simply plug in the same state over and over and over.


Or you put equivalently, you put your state through a neural network that has always the same parameters to get the next state and then you put that state into the neural network, and so on.


And you might get a loss function at some point.


This should remind you, for example, of something like reinforcement learning, where you have a state s one,


that you put through some neural network f in order to get the state s two, I'm sorry, not through a neural network, of course, f in this case might be the environment.


It might also be the inner environment model of your recurrent neural network.


It might also be tracking the state. So you might always get an observation.


You have an observation, you derive a state from it, and that state is being kept track by a neural network. So many things are possible right here.


However, let's say this is some sort of a neural network that in some way, estimates these state transitions, then each state you can technically derive a loss from maybe what kind of rewards did you get or something like this.


So this gives you loss one, this gives you loss two, this gives you loss three, and this gives you loss four, that should be consistent in my else.


Haha.


All of this together would obviously so this would result in a total loss being the sum of all the losses.


So Li. And now the question is, if I now want to, so every one of these, this neural network is always the same, there is a parameter vector, that's part of all of these neural network.


And now I want to know, how do I need to change my neural network? How do I need my to change my estimator of this series, whatever that is a state transition in reinforcement learning problem, for example,


how do I need to change this such that I do a better job at predicting the future, and therefore minimizing all of these losses?


Well, that's as easy as computing a gradient, a derivative, sorry, obviously, of my loss with respect to my parameters, right.


And that's what that's exactly what's happening right here. So this should be familiar to you. If you ever have taken a class on recurrent neural networks, this is the chain rule applied to neural networks, sorry, to recurrent neural networks.


So what you want to do is you can see the loss right here is basically the path to the loss is there are four paths to the loss right here.


So what we want to do is you want to back propagate through all of the possible paths that lead from the parameter vector into the loss.


It's a bit easier if you just consider one of the losses, let's just consider l four right here. So you want to do is you want to back propagate through this node through here, here you encounter the first parameter vector.


So that's one term in your that's one piece in your loss and then but you also want to back propagate through this node right here through it with the chain rule back propagate through this path, that's going to be another one, another piece of your loss right here, and so on.


So you want to back propagate through here up to here, and that's going to be another piece of your loss or of your of your derivative, I should say, not of your loss of your derivative of the loss l four with respect to the parameter vector.


Similarly, you could do for the other losses. So if I did the same for l three, it would be only here not to the right, obviously, because we we l three does not depend on this application right here.


So not that, but to here. So that would be another part of that gradient. And through here, that would be another part of that gradient.


So you'd get these sums of sums. And that's exactly what you have right here. If the first step, we simply back propagate, we use the chain rule to expand this, we back propagate to the step zero.


And from that to the parameters, plus maybe there's a direct influence on the parameters.


The first loss, we have to take two different paths. Okay, so first through the step one, sorry, state one, then back to state zero, which is, if you can see, that's the same as this right here.


So here, and here is the same.


And that means that these two paths overlap, right? So if I look from we don't have l zero here, we have l one. So if I look this path, and the path that goes from here, back one state, and then up here, those two paths partially overlap, that's exactly this.


And then there's also this one right here. And this will be the direct path from here, like, right up here.


Well, okay, I screwed this up a little bit. But you know, no one gets recurrent back propagation right at the first try.


In essence, what you do get is you do get these these big sums of derivatives. And what you can see that the components of these sums, as you go on, so these are the individual parts, you can see here is the general form for loss t.


So little l t. You can see that the individual parts that get longer and longer, right, one element, two elements, three elements, four elements, right here.


And the inside parts here, the inside is always we derive state two with respect to state one, then state one with respect to state zero, and so on.


And the general form of this is that you start at a loss, and you go to its given state, then you go through the chain of states all the way back to state to, you know, state k, where k goes from one to t, but in the worst case, in the longest case, all the way to state one, I guess,


that index is messed up right here, right? I think so. That should be like zero to match up here.


That should be zero. Yes.


Excellent. That should be zero. Good. We made a difference. We found a mistake. Paper rejected. Go.


Okay, so the problem is, obviously, here, this is a single matrix, right? If we're applying it over and over and over again, right, we're, we're deriving from the, we're driving through these state transitions again, and again, and again.


And this can quickly get out of control, namely. So here, by the way, is the sum of sums. So this is the total, the derivative of the total loss is now a sum of sums. And inside each of these sums, you have these expanding product, these telescope products, I think they're called telescope products, not exactly sure.


They say note that this product here, appearing on the right hand side of equation eight, the matrix of partial derivatives that each state derived with respect to the state right before it is exactly the Jacobian of the dynamical system f, that's the neural network.


And this and so the neural network or whatever that function is, right, defines how one state goes to the next one. So if we back propagate through it, we'll get the first derivative of that. And that's a Jacobian if this is a high dimensional map.


This has precisely the iterated structure discussed in the beginning of this section. So the beginning of the section, we looked at what happens if we just have a matrix, we have a state, and the state that comes out, we plug in again.


Thus, one might not be surprised to find that the gradients of loss functions of dynamical systems depend intimately on the spectra of Jacobians.


So what do they mean? They mean that this Jacobian, it has some sort of an eigen spectrum. And what we do care about is notably the biggest eigenvalue. So this Jacobian, it can be decomposed into into two transformations and a diagonal and the diagonal is going to be composed of the eigenvalues.


And the largest eigenvalue here has a special property. Namely, it determines sort of the largest in absolute number. So let's let's just assume we only have positive eigenvalues for the sake of argument.


If the largest eigenvalue here is larger than one, then the product, whatever vector, right, whatever vector I put in here, for almost all vectors, if I put them through this matrix, and then put them in again, and then put them in again, they're going to grow in norm.


And if I do this enough times, then you just over time, if you look at the norm of whatever vector I put in, it's just going to grow exponentially, because every single time, it's going to be essentially multiplied by a number greater than one, at least in in one component of the vector space.


However, if that is smaller than one, then the opposite happens, namely, whatever vector I start with, it's going to essentially shrink to almost nothing. And both of these are problematic.


And in recurrent neural networks, you have heard them as two problems. So this problem here is called the exploding gradients problem, gradients. And this here is called the vanishing gradients problem, vanishing gradients.


And the paper here makes the argument that essentially the dynamical systems that we're back propagating through, it's not only neural networks, but also, as I said, the simulations and so on, they suffer from the same fate right here.


And it, it, it is even a bit,


let's say, a bit more pronounced and a bit more hidden than it might be in recurrent neural networks. So they specifically talk about the re parameterization trick. So what happens if we have such a dynamical system, and the dynamical system also has some noise on it.


And one of the one good example of this is when you apply the re parameterization trick. So what is that? That is, when I have, for example, a variational auto encoder, variational auto encoder takes something like an image right here, puts it through a neural network into now, if it was a regular auto encoder, it would put it into like a latent vector.


That's the encoder. And then the decoder would reproduce the image from that latent vector. And the assumption here is that if that if we train this well enough, this latent vector will be a good description of what's in the image.


It turns out that auto encoders by themselves don't really work. No one knows exactly why because it makes total sense, but might have something to do with the loss function, or with them just being not super robust.


However, variational auto encoders work a bit better. And what they do as their encoder notably does not produce a vector like it doesn't produce the latent representation by itself. But what it does is it produces the distribution of the latent vectors.


So what it does is it produces a whole bunch of mu and sigma parameters, essentially, so mu and sigma mu and sigma, and they define the distributions of each of the components of the of the latent vector.


So what we're saying is that all of the late the latent vector is essentially distributed like a Gaussian, and we are not predicting the latent vector itself, we're predicting the parameters of the distribution that describe the distribution of latent vectors.


So we're somehow inferring from the image, what the distribution of the latent vector might be. And now in order to actually get an image out of that we need to do this step right here, this sampling, sampling step.


And that we can shove into our decoder, and then get an image out here. And all is good. But now we have to train the thing. So how do we train we could do the same thing, we could apply a loss like we do in the auto encoder, compare the output and the input and say these two need to match.


And, you know, we can do that. However, this is fine for the parameters of the decoder, the decoder has some parameters, we can back propagate this loss totally to these parameters. The encoder also has some parameters.


And then we run into the problem that we need to back propagate through the decoder. And we need to back propagate through this sampling step right here, which is not possible.


Now what do people do people have this reparameterization trick, where essentially, if you look at this as a parameterization graph, I have the input x here that goes through the through the encoder, that gives me, let's just let's just say, mu, and a sigma.


Let's write these as computation nodes gives me a mu and a sigma right here. So the parameters are in these two arrows that we need to get through. And now the usual way of doing of describing this is you say we use these two to get the distribution.


And we use the distribution to sample the latent code h, and we use that to produce through the decoder to produce the output. And again, we cannot back propagate through this thing right here.


So what do we do? Otherwise, what we do is, we say there is an interesting property of Gaussians, some other distribution as well, but of Gaussians specifically, namely that there is this thing called a normal distribution that has me mean zero and standard deviation one.


And if I sample a variable x, according to that, and I imagine another distribution that has mu and sigma arbitrary parameters, not zero and one sample y from that, then x and y are related by the fact that y is exactly x times sigma plus mu, this is sometimes called a a z transform.


In statistics, I believe or something like this, essentially, what it says is that I can sample from a distribution with arbitrary parameters by first sampling from a normal distribution, and simply multiplying the output of that sample by mu and sigma.


Now that's interesting, because what we can now do, we can change our computation graph, we can have our sampling, our distribution right here, we can have our distribution that is a normal distribution mu zero, sigma one, we can sample from that we can sample a, let's call it let's call it z just because we can.


And then we can multiply it by sigma and add mu right here we multiply here we add, and that gives us that latent code. And now you see, we don't have to back propagate through sampling because sampling is down here.


And our back propagation path can be through here.


This is called the reparameterization trick. And this turns out to be it's turns out to be very good because we can train variational auto encoders. But it turns out to be a bit of a deception when we look at estimating gradients in these in these systems.


So they make an analogy right here. And the problem by the way is the paper says, is that if I have some my actual objective my actual loss function here has a sort of a smoothing in it, right, because of this sampling step.


So this sampling step, it kind of smooths the loss function, right, there is a certain, certain randomness in it. And if I average over the randomness, then that that gives the landscape a bit of a smooth feeling.


However, as you can see, the gradient flow is not that it is not the smoothed variant, the smoothing comes is down here. However, the gradient flow is straight through all the deterministic route.


And that might screw up your gradients big time, as far as I understand it, I'm actually not sure I understand this paper correctly.


They give an example right here where they say, Look, we have a function right here, that we believe to be quite wonky, which is this sine wave with a bit of a curve in it, you see the square function, those are these things here.


And they change this w parameter. So the higher the W, the more squiggly the line is, that's the that's the initial loss objective. And then they convolve that with a with a Gaussian, which gives them the blue objective.


Now what they do is they say, Okay, can we use the re parameterization trick to estimate the gradients. And the point here is that I believe what the point is, is that the blue thing is the true objective, right, the one that's actually has the noisy parts in it.


That is the true loss. That's the true objective, you want to estimate the gradient from. However, your re parameterization trick gradient, it will be it will be along the red function along the squiggly function.


If that's not if I'm saying something wrong, I might be then I'm really sorry. That's how I understand it.


So if the oscillations are quite low, then the re parameterization tricks works super well. In fact, it works about one or two orders of magnitude better than if we were to use a black box method to estimate the gradient black box method is, I mean, essentially, it's a, you have a you have a function, right, you evaluated at two points like here and here, you draw the line, you say like the gradient is kind of like the, the


steepness of the line right there. It's not it's not that much more, it's just in higher dimensions.


So, obviously, re parameterization trick is going to work better because we can have exact derivatives.


However, the more squiggly the line gets, the more the noisy objective and the objective where the re parameterization gradient flows are going to sort of diverge from each other. And as you can see, the re parameterization gradient is not, it's not the case that it's wrong.


It's just the case that its variance is very high, right? So it's it's not as far if I understand correctly, the gradient is still let's say correct, it's it's unbiased, right?


However, its variance is going to be super high. If we if we look at different samples, if we look at different places along, maybe the x axis, it's going to be very, very, very high variance.


Instead, the reprimit, sorry, the black box gradient, it doesn't, it doesn't really care, it's just going to estimate pretty much the same with the same variance in all of the issues.


And this is what the papers claim ultimately is, is that there are situations where back propagating through dynamic systems is a good idea. And there are situations where backprop back propagating through dynamic systems is a bad idea.


Because the gradients have very high variance, and you'd be better off estimating the gradient using some sort of a black box optimizer. So even though you could back propagate through the system, you're better off just sort of estimating the gradient by something like


what I just said right here or an es and is it an evolutionary step? I'm not exactly sure. They dive into three different examples. So first, rigid body physics. And here they say they they use a Brax, which is a package that provides very, very fast physics simulations.


And on top of that differentiable physics simulations, right? Excellent. This is really exciting, because differentiating through physics simulations means that you could technically optimize some stuff really well.


Instead of doing reinforcement learning, you can now just look at you know, which action would actually bring my loss down because I can factor in how the world would react to my actions.


In this case, they say, we get right. So there is we look at policy optimization of some stochastic policy parameterized by neural network. We test this using the default and environment and default multi layer perceptron policies.


So this is not a big problem. This is not a very complicated problem. But it's enough to show this effect. So this is a stochastic policy parameterized via a neural network, which means that is this is you get the observation.


This goes into a state by a state encoder. This then goes through a neural network that's going to give you an action and the next state, right and the action is going to be stochastic if I can, if I estimate this correctly.


It's giving you an action distribution like maybe this, sometimes this, sometimes this, sometimes this action. Or maybe it's a continuous actually, I think it's continuous and is probably continuous. So it's going to give you some sort of a distribution over actions.


And to get the real action, you actually need to sample. Right? Now, does that sound familiar? Yes, it should. Right? So this action this. So this is the action distribution.


Let's how do I make something into distribution, a squiggly line, double, double barrel thing. Okay, to get the real action you need to sample, and you push that into the environment.


And the environment is going to give you a next observation. And that together with this state, probably, maybe I don't know if the state gets in or not, is going to lead to state two, and then we start again, right?


The important part right here is that if we back propagate through the environment, which we can do with Brax, right, and we can also back propagate through the stochastic policy, we could technically optimize this neural network here directly to change to the actions that actually give a much, much better outcome.


However, is this act does this actually work in practice?


So here is an experiment they do. So what they do is they check, they do different unroll lengths. So they make a plot and say, What if we unroll this policy for one step for two steps for four steps, eight and 16, essentially means how many steps in the environment are we going to wait before we do the back propagation, you can't wait for the whole episode that will blow your memory.


So usually these reinforcement learning tasks, even if they do, if they don't back propagate through the environment, they will stop after a number of steps, and then back propagate through that it is a bit of a limited horizon.


So you want to do as many as you can, ideally, in order to get really good improvements.


So here you can see different lines for different number of unrolls, the randomness is fixed. So this is always essentially starting from the same state. And what they plot here is mean loss over these unrolls.


And what they plot here is shift along a random direction. So in this neural network, that this here is a big vector of parameters. They take one of those parameters, and they just shifted a little bit, they just shifted a little bit, as far as I can understand.


And they show what happens to the loss as they do that, right. Now you can see, if you consider one step look ahead, it's still it's pretty smooth, but still like there is a lot of change in the loss as you move this around.


Yeah, so then, and if you look at more and more and more unrolls, you can see that this becomes more and more noisy, the variance as you shift long becomes heavier and heavier.


And the systems become, I think the paper calls them chaotic, which means that little change in the initial condition will lead to a big change in the sort of in the outcome.


And that's essentially their, their problem right here is that you can't really estimate these gradients through these dynamical systems, because just the variance of the gradients will be really, really high.


And they show right here, what happens if we don't just look at one unroll, but we do a bunch of unrolls, right, we take the average over the randomness over the unrolls. And as you can see, that helps right you.


So this is a fixed I believe this is an eight step on role. So it's just from this eight step on role, which is a reasonable look ahead, they take a bunch of them, and they just average over them.


And that gives you a kind of a smoother line if you can see right here. So even if you take the average over different samples, if you then unroll for more, you can see that a still the gradient variance essentially explodes.


This here is a log scale over the mean gradient variance, that's essentially how many squiggles happen up and down as you shift along these directions.


And you can see that it's it just kind of explodes.


And that's the problem that the paper wants to highlight. They go into two more examples right here.


One is a metal learning an optimizer. So that's when you have essentially an outer, you have an outer optimizers, you have a big optimizer optimizer big.


That is that optimizes optimizer small that optimizes a loss, right? So optimizer small is doing its inner updates for a neural network optimizing a loss.


And the big optimizer is essentially optimizing the parameters of the inner optimizer. So you want to learn to learn. And for that, what you want to do is you want to take this optimizer right here, run a bunch of these steps here, see how much did you decrease the loss, and then learn the parameters of the inner optimizer such that the loss is decreased more in future iterations.


So it's a bit of an, it's a bit of an alchemy field, I feel like this, I'm not I'm not so sure about about inner optimizers and so on. But you can, you know, you can back propagate through the inner unrolling, you can unroll the inner optimizer, you can back propagate through all of it.


Therefore, you could learn the outer optimizer like this. Again, you can see right here, depending on how long you unroll, if you unroll for just eight steps, the the system does not behave that chaotic, you can see that the lines is pretty flat as you again, shift a lot one parameter along a given direction.


However, as soon as you go up to more sort of reasonable things to unroll, like what actually people do in order to learn something, then you can see that the system just behaves quite heavily chaotic, namely, as you shift a little bit, the parameters change.


Again, you can remedy that a little bit by averaging, this is an average over doesn't even over are shown in color, okay, we don't actually know which of these lines we average over, I think, I think it's one of the, like, it's either the 512 or the 256 that they average over.


And it's moves down. However, still, as you can see right here, depending on the shift, there can be situations where the variance as you unroll, and this isn't even like this isn't even for long, right?


So, as the that the variance just explodes, right here.


Again, this is a system with a bit of randomness, because the inner optimizer is trained on mini batches, and the mini batches are sampled randomly, right. And this randomness comes external to the optimizer.


So the optimizer, the randomness essentially enters from a different direction, which essentially gives the same artifact as the reparameterization trick.


The last example they go into is a, a not some sort of a deep learning thing. It's disk packing. So this is like you have a volume, and you want to pack two different sizes of disk, so big disks, and small disks.


And you, you want to figure out like how, how should I pack the disk such that they're packed the most and you can do that via back propagation. And they see the same behavior right here, that if they sort of back propagate, so you can run, I think the simulation here, and you can backpropagate through it.


And the result is essentially the same is that there are, this is that diameter of the smaller particle with respect to the larger particle, you can see that sometimes it's well behaved.


However, as you get to, as you get to like regions where this particle becomes rather small, you unroll for a number of steps, this becomes very unstable, it becomes very chaotic, small change in the initial parameters leads to a big change in the end result.


And same thing right here, if you unroll for a number of steps, the variance of your gradients just becomes huge.


And therefore, it's not really optimal to learn from it. So what does that all tell you they go into different experiments right here. So they say, we go back to the first experiment of the end, and we look at the spectrum of eigenvalues of that policy.


And what they find is they compare two different runs with two different initializations. In it one is initialized in an unstable regime. So in one of these chaotic regimes where they observe the gradients exploding, or the gradient variance exploding, and in it two, which is in a stable regime, and they wonder what's the difference.


So look at the spectrum of the eigenvalues of the Jacobians as they back propagate. And what they find is that in the one initialization, the unstable one, you have quite a number of of eigenvalues that have a norm larger than one eigenvalues can be imaginary.


So everything outside the circle is norm one, everything outside is larger, you can see right here, that if they look at the different steps, you can see that after a while, you can clearly see that the maximum absolute eigenvalue, it shoots up into these are this is again a log scale.


And if you look at the product of Jacobians, right, which is what you would do if you actually unroll for a number of steps, then that product just grows. Essentially, every time it encounters one of these big eigenvalues, it just bumps up, it just grows in in norm.


So this is again the the eigenvalue, but essentially, what you would multiply your loss or your vectors by.


And again, yeah, so the gradient norms correspondingly rise exactly with the rise in the biggest eigenvalue of the Jacobian, this is like a straightforward consequence.


So their conclusion is, if in the well behaved behaved initialization, this doesn't happen.


So So their conclusion is, look, if you can, if you can try to keep your eigenvalues of your Jacobians smaller than one. Now that's easier said than done. So what can you actually do they say pick well behaved systems.


This isn't that helpful because sometimes you actually want to study these not so well behaved systems, right.


So for recurrent neural networks, they say there are initializations that can help. So there is a initialization.


Sorry, they, they initialize the RNN near the identity. This means that the recurrent Jacobian will have eigenvalues near one, and thus be able to be unrolled longer before encountering issues. However, after training progresses and weights update the Jacobian drifts eventually resulting in vanishing or exploding gradients late enough in training.


So this is not that much of a remedy. They also suggest a second solution is to change the problem entirely. The case of an RNN, this is feasible by simply changing the neural architecture.


And I guess this is what everyone learned that those classes on recurrent neural networks is that things like LSTMs and GRUs, they generally avoid this problem.


The recurrent Jacobian of an LSTM was specifically designed to avoid this exponential sensitivity to the hidden state because it has these gates and additions and so on.


And may I say residual connections, and is thus significantly more robust than a vanilla RNN. Nevertheless, it can still happen right but with an LSTM you're sort of more protected.


In rigid body physics, they talk about talk about maybe you have to go to a complicated solution. So instead of if you have particles and they kind of bump into each other, and bump into each other.


Maybe you have to chunk up your simulation into different parts. So into this part where you can back propagate through, then they're in a part where there's a collision. And then once the collision happened, you can again simulate forward and then back propagate through that part, and so on.


So now I want to actually go down here, jump a little bit and discuss these two sections right here truncated back propagation and gradient clipping. And this is an idea that I guess everyone has when you look at these results is that can't we just kind of clip the gradient or like if the gradients too big just kind of tone it down a little bit in order to not run into these issues right during back propagation, we might just


you know, cap the gradient somewhere. And then we don't have these big gradients. The problem is that of course, by doing that, you bias the gradient you it's no longer the true gradient. And they have, for example, done this in this Brax environment right here in this and task.


And they say, in this task, we back propagate the task rewards directly to the policy parameters after 400 steps for truncation length t, a sorry for truncation length t a stop gradient up was inserted every t steps in the 400 step trajectory.


So they truncate the back propagation through time. So they would, instead of back propagating through all the sequence, they would just chunk it into like lengths of let's say three. So they introduce a stop gradient after each three steps.


And that will essentially make it such that the loss from here can only go to here. As I said before, that is already happening when we unroll for sort of not as many steps because of memory constraints. But now we chunk even smaller, because we're afraid that the gradient will explode even if we so for the length that we unroll.


Now, what they find is that there is a narrow band where this actually works. However, I guess I guess that's the band right here where the reward is high.


But they essentially their their, their conclusion is that this disturbs the gradient so much that essentially, you diminish your ability to learn anything because the gradients are no longer good, unbiased gradients.


And I guess the same goes with gradient clipping, they said, if they tried the gradient clipping in. So as before, this calculation of the gradient is biased. To demonstrate this, we took the same and policy and sweep learning rate and gradient clipping strength, I guess swept.


Or, yeah, we found no setting which results in positive performance and thus omitted the plot. Right? Zero, zero positive performance here with gradient clipping in this very simple environment that could actually be optimized fairly easily.


And that also reinforcement learning can optimize fairly easily. So here you can already see the difference. And the difference is their fourth recommendation, just use black box gradients. And by black box gradients, they essentially mean, you know, these estimators that I've shown you or, for example, reinforce, which is this gradient estimator through black box environments that is often used in reinforcement learning.


Reinforce gives you an unbiased gradients. They also say, in addition to the unbiased methods, there are other methods and you might know them from reinforcement learning, for example, proximal policy optimization easily outperforms all of our experiments, training the and policy with gradients that we perform.


So the end policy with gradients, I guess. And there you have it, this is a clear, this is at least one or three demonstrations, where if you back propagate through the environment, even though you can, it is a more efficient to use a black box, let's say reinforcement learning gradient estimator, rather than the true gradient, because in chaotic systems, true gradient estimator,


true gradients variances explodes. As you back propagate through long sequences of these dynamical systems. And that's how they reach their conclusions. They say, we hope this paper says lighting to when gradients can be used, namely when the recurrent Jacobian has small eigenvalues.


In the other cases, when gradients do not work, we encourage readers to try black box methods, they estimate the same quantity, and with less pathological variance properties, especially when it's possible to calculate a smooth proxy for the loss function of interest.


In summary, gradients are not all you need. Just because you can take a gradient doesn't mean you always should. And that's the ending of this paper. I know this was a bit of a bit of a all the way through, starting out from, you know, the re parameterization trick and whatnot.


But I hope you've seen the point that the paper makes is that, you know, things going more and more differentiable can be dangerous, especially in the presence of chaotic systems, especially when there's a component of stochasticity involved.


Reinforcement Learning with Unsupervised Auxiliary Tasks

Hi there. Today we're looking at reinforcement learning with unsupervised auxiliary tasks by Google.


So in this paper, the authors consider a reinforcement learning task, and I can show you what it looks like.


It looks like this kind of a maze, or this is an example that they give, where you have to navigate the maze.


It's 3D, and you have to navigate from pixel inputs. You have to collect apples and reach the goal, and this gives you rewards.


So on the left you can see what the agent is actually seeing. On the right you can see it from a top-down view.


The problem is, of course, that the reward is very sparse, meaning that you have to navigate a lot of maze before you even get a single point.


So reinforcement learning has a big trouble with this, because it relies on constant reward to notice what actions are good and what actions are bad.


So what the authors propose is, in addition to the regular loss that you would have, so your reward, which is this thing,


you would also have an additional set of auxiliary tasks, and here C goes over the auxiliary control tasks that you specify.


Each of those has a reward, and you're also trying to maximize these, each with some kind of a weight here.


The thing is that the parameters that you maximize over control all of the different tasks, so they are partly shared between the tasks.


So what you're hoping is that by learning to do one thing, you also learn to do another thing.


So the difference between this and, let's say, we've seen work like this before, where you do it more like an autoencoder setting.


So, for example, you count the agencies, the input on the left here, and it tries to predict what the next input will be, what the next frame will be.


The thought behind this is if you can accurately predict what the next frame will be, maybe you'll learn something useful about the environment.


In this work, it's different because now we couple a reward to these tasks, and I can show you here what the authors propose as additional rewards.


Sorry, they're further on top. Let me go there.


Especially they consider here these two auxiliary control tasks, so pixel changes, which means that the agent actually tries to actively change pixels.


So it gets a reward for changing the pixels in the input.


So it tries to maximize this. It needs to learn what I need to do to maximize my pixel changes.


And probably that will be moving around. So it will learn to kind of move around, not to move against the wall, because if it moves against the wall, the pixels won't change.


So it will kind of learn to move along the, like how a regular human agent would also move, not into a wall, not like into a dead end or something, such that the pixels always change.


Of course, it's not perfect. You can also change your pixels quite a bit by simply spinning around in a circle.


But this is one auxiliary tasks that they augment the agent with. The other one is network features.


So it's kind of a meta learning here. You actually reward the agent for changing its own internal activations.


So the hope is that it kind of learns about something about itself. How can I activate my internal neural network units?


And it gets rewarded for that. So we might want to activate a lot of them and want to learn how they're activated.


So this kind of self introspection, you also hope that it kind of leads to a network that does more sophisticated tasks or that by nature of trying to get most pixel changes and the most network feature activations, that you also learn something useful for the actual task.


So these are the two tasks they propose. In addition, they also do, and they have a drawing of this over here.


They also do a lot of other things, namely on the top left, you can kind of see here, what's it called? The base agent.


This is an A3C agent, meaning that it's an active critic. So you learn a policy and you learn a value network.


We might go over this in a future video. So just consider this a standard reinforcement learning agent.


You feed its experience into a replay buffer and out of the replay buffer, you do many things.


So for one, you try to learn these auxiliary tasks. Note that these are shared parameters between all of these networks.


That's why the auxiliary tasks actually help.


You also try to better learn your value function, they call this off-policy learning, because you kind of pause the base agent training for a while and then you train the value function some more, just because that helps.


You also try reward prediction in here and the way they do it, as they explain, is kind of in a skewed sampling way.


So out of all the situations you can be in, the agent will have a reward very, very few times.


So what they do is they simply sample out of the replay buffer, out of all the experiences they've had so far, they sample more frequently the experiences where they have actually gotten a reward.


That way the hope is, of course, the agent, if you look at the experience here where you actually get an apple, then the agent might learn a lot faster,


oh, there's some kind of apple there and I move towards getting a reward. So that's the hope that you instantly recognize high reward situations and kind of are not so interested in non-reward situations.


Of course, it does introduce bias in your sampling and you might decide for yourself if that's good or bad. Here it seems to work.


So they have a lot of experiments in this task, this labyrinth task, and they, of course, as is with research, they reach state of the art, they're much better than anything else.


No, I mean, they don't boast as much, so it's actually fair comparisons.


The criticisms, so they also evaluate on Atari games, the criticisms that I have are twofold. First of all, the choice of auxiliary tasks is completely up to the implementer,


which means that I have to decide as an implementer of this algorithm what my auxiliary tasks will be.


And here pixel changes and network features, they seem like fairly general tasks that you could apply to a lot of these kind of problems,


but it always kind of comes down to how much knowledge about the task would you like to code into the actor.


And here, I mean, you can see it makes sense to get at least the pixel changes as an auxiliary task, but it's questionable how much of kind of domain knowledge this already encodes.


So the fact, the choice of these are certainly something that you have to decide as a human.


And I think these are good choices. So they're not too domain specific, but also they do correspond to like some kind of visual moving around game task.


And the other kind of criticism, it's not really criticism, it's just a remark, is that they do a lot of things.


So, I mean, their paper is about the auxiliary tasks, but they also then do these skewed sampling and the off-policy value learning and so on.


And of course, you can kind of argue, yeah, this is all done in other reinforcement learning tasks.


That's why it's a fair comparison. I guess it's a philosophical question.


If you want to reach state of the art, of course, you have to first of all get a better method here.


This will be the auxiliary tasks. This is the new idea. And then implement all the tricks that the other people have discovered, which is good because you kind of reach the highest performance you can get.


But also the problem is you make it harder to compare, you make it harder to see where the improvement is coming from.


Have you simply chosen better hyper parameters for the reward predictions of things?


Have you simply, is there an interaction maybe between the auxiliary tasks and the skewed sampling part?


All of these kind of things wash out and it's not really clear where the improvement is coming from.


On the other hand, if you simply take a basic, basic, basic algorithm like just A3C here on the top left and you augment it with nothing but these auxiliary tasks on the bottom left,


then you see an improvement, you can be relatively sure it's due to your new idea.


But of course, you won't reach any state of the art numbers because everyone that does A3C also does these tricks.


Philosophical question here. I'm standing more on the side of not doing the tricks or maybe doing both.


Regularizing Trajectory Optimization with Denoising Autoencoders (Paper Explained)

Hi there, today we're looking at regularizing trajectory optimization with denoising auto


encoders by Rene Bonet, Norman DiPaolo and others of various places.


But a lot of the people are from curious AI.


And we actually had a discussion with Hari, who is the CEO of curious AI.


And this was on our machine learning street talk podcast.


So this is another YouTube channel for those of you who don't know, where every week or


so we try to have either an interesting discussion or a guest like sort of an interview or talk


about comment on the talk.


So if it is not out yet, I'll link to it as soon as it comes out.


But if you're watching this video later, make sure to check out our conversation with Hari


because it was absolutely fantastic.


And in general, if you like videos like this, consider subscribing, liking, sharing if you're


still here at the end, and liked it.


Okay, so this paper on a high level deals with model based reinforcement learning.


Model based reinforcement learning means that you are using a model of the world to do reinforcement


learning.


So in essence, if you have your reinforcement learning setup where you are an agent, and


you have to interact with the world, you have to do so in many steps in like a round trip


fashion.


So you put an action you act, and the world gives you back an observation.


And you have to act in the world over and over and over such that you will be able to


maximize your reward.


Now what is model based reinforcement learning?


Model based reinforcement learning basically means that the agent here has internally


a model of the world.


So it sort of understands how the world works.


Situations where you have a accurate model of the world are things like chess.


So in chess, the rules are very clear, you know how the world's going to behave if you


perform a certain action.


But in real world applications, it's very, very hard to actually make a model.


So people usually rely on learned models.


So what does it mean?


It means you basically learn a neural network that tries to predict how the world is going


to act.


So this here is going to be a deep neural network that you learn from what you see in


the world.


Now trajectory optimization basically means that you are now you now have this world model,


and you use it to look ahead, as I said, so you are in the state like here, and you can


do let's say three different actions, and you use your world model here, world.


And you see, you think how's the world going to react if I do either of those three things,


and then you get into three different states.


And then again, you after each one, you consider three actions, three actions here, three actions


here, and so on.


So ultimately, you're going to kind of have an overview over a planning horizon, which


here we call H.


You kind of look ahead a couple of steps, or there are various ways of doing this.


But ultimately, you will basically find that this this path here is really good.


So I think I'm going to take this as a first action.


So trajectory optimization considers finding the best green path here in this tree of possibilities


that your world model gives you.


Okay.


Now what does what do these people say?


They say this procedure often suffers from exploiting inaccuracies of the learned model.


What does that mean?


That basically means that if I have a world model, and it is not accurate, then it is


basically, basically the the thing that tries to find the best green path here, the optimizer


is sort of trying to find the the best path against this world model.


Now if that world model is inaccurate, that can lead to devastating consequences.


So what do we mean by this?


I'll give you an example.


If you have a room, right, and the room is, let's take our classic room, like this.


And you are here and you would like to go here.


And so you're a reinforcement learning agent, you do some exploration, right, you explore


a bit here, the next episode, you might go here, and you might go here, and so on.


And over time, in this framework, you're going to build a model of the world.


So at the beginning, we won't tell you how these rooms look, you have to discover it


by yourself.


So maybe at the beginning, we only tell you, there's these four walls, the rest you have


to figure out.


So on and on, you're going to fill in your blanks, you do your first explorations, and


you've had there's a bit of a wall here, right, and there might be some wall here, I crashed


into that, right, you're going to do here, you crash into wall, you saw there's a wall


here and here, ah, there's a wall here, you go maybe here, oh, there's no wall.


So you go further, there's no wall anywhere here, you crash here, okay, we already knew


there's a wall, maybe you crash here.


Alright, so right now, you have, okay, you go here, you have you have a model of the


world in this situation, where there's a wall here, a wall down.


And if you now try to do trajectory optimization, remember, you have to go from here to here.


If you try to do trajectory optimization, what is it going to turn out, it's going to


turn out like look, there you go, that works just fine.


And that's not because you're so good at planning.


I mean, you are good at planning, but because your model is inaccurate here, because it


has never seen this, your entire training distribution that you trained the world model


on only explored the area over here.


Right?


So you see how the more efficient this planning algorithm is, like the blue arrow, the thing


that finds the blue arrow, the more efficient that is, the more consequential it is when


your learned world model has mistakes, because it will exactly exploit these mistakes in


order to get the shortest path possible or the highest reward in that case.


And this, they call this like almost an adversarial attack on the world model, which is a pretty


good way of framing it.


They propose actually to solve this problem.


They say we propose to regularize trajectory optimization by means of a denoising auto


encoder that is trained on the same trajectories as the model of the environment.


We show that the proposed regularization leads to improved planning with both gradient based


and gradient free optimizers.


We also demonstrate that using regularized trajectory optimization leads to rapid initial


learning in a set of popular motor control tasks, which suggests that the proposed approach


can be useful tool for improving sample efficiency.


So in essence, what do they do?


They basically say, okay, we want to regularize this using a denoising auto encoder.


And I think it's best if we if we look at the at the math for doing this.


So the math here starts off as follows saying, you want to learn a world model.


This is f here, f is the world model, it takes in a state and an action, and it gives you


the next state or an approximation to it.


And the parameters here indicate that this is some sort of function that you learn like


a deep neural network.


You can do this in fully or partially observed environments.


Now, when you plan, what you want to do is you say I have a planning horizon H, right,


then I have a reward function.


And the reward function is going to give me a reward for each state action pair.


So if I'm in a state and I do a certain action, I'm going to get some reward, this could be


you have reached the target, or this could be you know, how much money you've collected


or whatnot.


So you're going to look at a horizon H, you're going to look eight steps into the future,


and you want to maximize the sum of all the rewards here.


So in the limit, this reduces to simply like, for example, reaching the target in our rooms


case, if if H is infinite, but you can consider a lower planning horizon.


So you want to find the action sequence that maximizes this reward in the future.


Okay.


And now this reward relies on your environment model.


So here's the the algorithm.


First, you collect some data, okay, that's how you start off, then train the dynamics


model, the world model, using the data you've already collected.


Then for each time step t, you want to optimize this trajectory.


So you want to find the best next action sequence and take the first action, implement the first


action and get the new observation.


And do you do this in a loop until the end and at the end, you say at this data to D.


So that's what that's what you do, you use your world model to get the best action sequence.


That's how you optimize the trajectory.


And then at the end of the episode, you've you've done an episode, right, you went somewhere,


you put all of this into your training data to make the world model better.


Something to note here is that the world model will only learn about things that you have


done, right.


So there is kind of an interaction effect.


That's the green area here, the world model only knows the paths, the world model only


can accurately estimate the world where you have been.


And that's going to turn out to be the entire problem because the these blue arrow finder


can now go away from the from that.


That's explained here, potential inaccuracies of the trained model cause substantial difficulties


for the planning process.


Rather than optimizing what really happens, planning can easily end up exploiting the


weaknesses of the predictive model.


Planning is effectively an adversarial attack against the agent's own forward model.


This results in a wide gap between expectations based on the model and what actually happens.


Okay, and they have this, this example here, where it's like an industrial control process.


And what you have to imagine, there's like some sort of a container here with liquid


liquid in it.


And there are two two pipes that lead to this container, pipe one, and pipe two.


And there are valves here.


So there's this valve right here.


And there's this valve right here.


So these are valve one and valve two.


And there is also an output pipe right here.


And that's a another valve right here.


So you can control these three valves, the in two inputs and one output, and you have


to somehow optimize the reaction in here.


So this is a chemical reaction made up out of the two liquids that flow in here.


And you have to somehow optimize a property of that.


And that's highly nonlinear and has maybe like time, time shift.


So when you open a valve, it's going to take a while and then it's very nonlinear.


And then you are not supposed to break the pressure limit.


So you have to also outflow some stuff.


And if you just do this with a learned model, it looks like this.


So first of all, here is a classic controller, like people have been doing this stuff in


industry, and they basically build controllers for it.


And you can you can do that.


And that works out really okay ish.


As you can see, right here, this is the product rate, what you're supposed to optimize.


And you see some sort of a smooth, you're supposed to actually bring it to this dashed


line right here.


And this is some sort of smooth thing, right?


And you're supposed to, I guess, bring the pressure here and the A in purge, I don't


know what these quantities are, but you're supposed to bring them to the dashed line.


And it's very nonlinear, and very time dependent.


So that works.


And you see here kind of the smoothness by which the variables are manipulated.


Now, if you just learn a world model, and then do this trajectory optimization, basically,


this is some sort of a, yeah, reinforce planning based reinforcement learning with a world


model.


You see right here, it works, but it's super jittery, the pressure spikes here.


And apparently, this here is a pressure limit.


So it spikes the pressure limit.


And you can see that the manipulated variables are up and down and up and down and up and


down.


Because at each step, it basically completely overestimates its its potential reward, it


thinks like, wow, this is really good.


But all it does is find a weakness in the model and not a really good action per se.


Now with their method, to already take it away, you can see that now the control task


super smoothly and very quickly converges to these optimal things.


And you can see that the variables being manipulated are also rather smoothly manipulated.


And that's an indication that the model is accurately estimating the rewards.


Okay, so how do they do it via what they call trajectory, we are regularization of trajectory


optimization.


So in essence, what do we want to regularize here, there are many things we could do to


solve this.


But the way this paper goes is they say we not only do we want the most return, we also


want a high log probability of our of our taken path.


So this here, as you can see, this is observation action, and, and so on observation action.


So this is the future.


This right here is the future.


So this, this sequence here is going is what is going to give me the reward right here,


right.


So G is also dependent on these things, but it's not said explicitly here.


So G is dependent on your plan, maybe let's not call this the future.


This is the plan.


Okay, this is the plan you came up with.


So this is directly going to influence G and G is the reward you're going to get under


your model.


But also you want the log probability of the plan itself to be high.


Now there, I think there is a bit there is something missing here.


And that is conditioned on your training distribution right here.


And I think that's a actually rather crucial part.


Now this, that's the KL thing.


So this is conditioned on your training.


So what you want is you want the plan to be basically in your training distribution.


So you, you want what you, you want your plan that you're going to execute, if that is actually


part of your training data set, then you know, I have already executed this once before.


And it's reasonable to assume that therefore my world model has learned from this experience


and is going to give me an accurate reward.


If we go back to our rooms example, then up here somewhere, if we go back to our rooms


example, right, you see that anywhere in the green area where I have already explored,


the world model is fairly good, right, it's going to give me accurate reflection of the


world.


But as soon as it go outside the green area, it is not.


And inside the green area is basically where my training data is.


Now if I in the future actually take a path here, crash into a wall right here, right,


the you saw in the algorithm, at the end of an episode, I'm going to add my trajectory


to the training data for the world model.


So this green part here expands to include that.


And now, if I go here again, if my plan goes there again, now I can trust the world model,


but also now it has it is actually correct because it has a wall here.


So you see that the regularization basically you not only do I want the biggest reward


possible under my world model, I also want that the plan that I'm about to execute is


has a high probability under my training distribution.


Okay.


And the way we do this is by denoising auto encoders.


So I want the log probability here to be high and you do this via a denoising auto encoder.


What's a denoising auto encoder?


A denoising auto encoder is basically so if you have, for example, an image and the image


is of a our trusty cat whiskers and a denoising auto encoder is an unsupervised method where


you have it's basically an auto encoder.


So there's a bunch of layers compressing to a hidden representation, then uncompressing


it again.


And at the end, you want to output the same as at the beginning.


So it's basically an auto encoder.


But the special part about the denoising auto encoder is that first, you take your input


and you know, you put some noise on it.


So that could mean could mean anything here.


But here, what they do is they do they make some Gaussian noise on it.


Now, I can't really draw Gaussian noise here.


But it would be kind of convolved with Gaussian Gaussian noise.


So I'm just going to add some noise like this.


So noise, noise, noise.


So there's some noise, you see, and then you feed that.


That's now what you feed in here.


And the algorithm is supposed to reconstruct this, this original image so that the algorithm


is basically supposed to take away the noise, it doesn't see the original image, but it's


supposed to produce it.


And you do this with your training data.


So what does that mean, ultimately, for our trajectory optimization?


It means that if I have a trajectory that I did before, and it maybe goes here, right?


What I can do is I can make a noisy version of it, which would be the black one right


here.


So I put some noise on it, some noise.


It's, it's kind of the same, but okay, and the denoising auto encoder is supposed to


give me back the red one.


This will simply give me some sort of a probabilistic model of my training distribution.


So they go through the math here and show that these denoising auto encoders actually


naturally output this log probability, sorry, the gradient of the log probability.


Because optimal denoising theory says that for zero mean, and Gaussian noise, the optimal


denoising function, the optimal, the optimal denoising function for zero mean Gaussian


corruption is this thing right here.


So it is, if, if I if you give me x, and you tell me x is has been corrupted by zero mean


Gaussian noise of size, sigma n, then the best and do you simply tell me give me


back the original image, the best thing I can do is to take what you gave me and add


this gradient of the log probability of x, if, if I can, if I have a model of the log


probability, right.


So that's the best thing I can do.


And that's the best denoising function.


And now you have to think a bit of in reverse, if we train a denoising auto encoder, that


is going to approximate this best function that there is, okay.


So we know that the best possible denoising function is this, we train a denoising auto


encoder, which in the optimal case is going to converge to the best denoising function.


So if we then reformulate, and we do denoising auto encoder of x minus or x tilde minus x


tilde, that is go or divided by the standard deviation.


Sorry, the variance that is going to give us this quantity right here, the gradient


of the log probability, and the gradient of the log probability of x is exactly what we


need to run gradient descent on our function.


So here is our function again, g plus this regularization.


Now they don't regularize over the entire future, but over these windows.


But in essence, it's g plus the log probability of your plan.


If you take the gradient of that, of course, you take the gradient of the sum, so it's


the gradient of g plus the gradient of the log probability with respect to the actions.


And here's simple application of the chain rule will tell you that you have to propagate


through the input through the x and you need this quantity, the gradient of the log probability


with respect to its inputs.


Now as we just saw, the optimal denoising auto encoder is going to output that problem


that thing.


So if we train a denoising auto encoder, and we suppose it's reaches a good accuracy, then


we can obtain this quantity basically for free.


And that's the entire trick here.


So in essence, what does it mean?


In essence, what it means is that if we are in our room again, and we have our partial


model of the world, let's say we have this model, because we are here, and all we've


ever explored is so we've explored this, these things right here.


Okay, and this.


Now when I go and do my trajectory optimization, and my trajectory optimization wants to go


here, I simply say, no, I don't know that I haven't seen that yet, you can only plan


basically within the space where we have already been.


So you can plan like here.


So here now there is of course, there is going to be some exploration.


So some probability that you can go away a bit, but not too much, right.


So in this case, it would result in the planning only to happen in spaces where we've actually


been.


So it might go here, and then here, because okay, here, we haven't been anywhere.


But then that would lead me to take the first step in this direction, and not in this direction.


And if I take my first step in this first direction, then of course, I'm going to be


already a bit on the correct path right here.


Whereas I if I take the first step into this direction, then after that, I'm gonna have


to, if once I crash here, I'm gonna have to correct really hard.


And that's exactly what's going to give you this super jittery control.


Whereas if you only plan where you've already been, you won't the probability that you're


going to have to do like a 180 is going to be much, much lower.


Okay.


That seems like


that's about it.


Let's look at the experiments.


So their experiments.


Basically, I actually want to go down here to this industry, sorry, not the industrial


control process, but to the Mujoko experiments.


So these are kind of continuous control tasks, you might have seen it.


So there's some like one is a, a, the ant here is basically this 3d.


And it's like a blob.


And it has, I think, four legs, and each leg has two joints.


And it just needs to walk as far as possible, or reach some sort of goal.


And the half cheetah is like a 2d thing where I think it's something like this.


It also has these two legs, and it's supposed to walk forward and not fall over.


And you can put force basically on each of the of the joints here.


So you see that their baselines are Gaussian processes.


And this pets thing is a previous baseline to do, to also do model based control with


a learned model.


And here they, theirs is the main, their main one is the red one.


And as you can see that it goes much faster, well, basically outperforms the rest in these


high in these more complicated tasks.


And then card poll or something like this is is lower dimensional, easier tasks.


And you can see that at least it does not hurt.


They make they say here, something they don't, they don't show in the plots, they say that


if you let this run for a while, then basically the their method doesn't make any improvement


anymore, whereas the baseline methods will sort of at some point surpass it.


And the reason that is, and I'm not sure if it's on this exact task, but they mentioned


that which it's it's I respect so far, is because they say since we only plan where


we know, where did I draw it?


Since we only plan where we know, we basically do much less exploration than others, we kind


of stick to what we know when we plan.


So inherently, we do less exploration.


And in our conversation with Hari, he basically said this, this is intended.


And the base, the intention is that you want to do your planning, where you know, and then


explicitly add a component that does exploration.


So you have control over so you can basically say, huh, I, I've never been here, sort of


now, you would be in an exploration phase, you would explicitly go there, rather than


intermingle your planning with your exploration and basically rely on your planning to screw


up and you're exploring because if your plan, if your planning never screws up, then you


won't explore either, right, then you will always reach your goal, or your planning will


always be correct.


And these other methods that don't have this explicitly, they explore every time their


planning screws up, and you don't want that.


You want your planning to be as good as possible.


And they do that by sticking to what they know.


And then they the next step, which is not in this paper would be to add an explicit


exploration policy to reach areas they've never reached before.


Okay, so that's the reason why they don't ultimately reach the best accuracy.


But they do reach a the initial accuracy much faster than the other tasks, because they


plan better.


They have a long discussion here of what still problems are, like local minima, or the planning


horizon problem, open loop versus closed loop compounding errors in planning.


But I'm going to leave this out for now.


And I thank you for being here, I very much invite you to check out the paper for more


details.


It's pretty cool, pretty easy to read, actually, it's very written very well.


And with that, see you next time.


Exchange choice change - Tkinter tutorial Python 3.4 part 20

What is going on everybody welcome to the 20th TKinter tutorial video in our TKinter


mega series so to speak here where we are making an actual program.


So where we left off we've got a nice baseline and we've got all of our options and stuff


that we want in our program but we haven't actually tied in any back end operation to


do them.


So for example we hit SMA and it doesn't really do anything it does set it right we


do set that parameter but we are not actually doing anything.


So in this video in the coming videos we are going to start actually applying some function


and some code to those operations.


So the first thing we need to do is in order to do this we are going to be using non-tick


data and to display non-tick data we are going to be using candlestick charts as well as


we want to be able to plot with dates and that sort of thing and modify the ticker or


the actual axis labels.


So we are going to be importing a few more matplotlib things generally it's a good idea


to go ahead and just kind of group some things together that are like from the exact same


module so I am going to go ahead and move this from matplotlib import plyplot just right


up here and now we are going to import two more things we are going to say import matplotlib.dates


as mdates and then we are also going to go ahead and import matplotlib.ticker as mticker.


So this will allow us to kind of modify the dates and as well as modify our ticker information.


Now scrolling down a little bit right underneath resample size let's go ahead and add in data


case which we start off with say tick data eventually we are going to go ahead and change


that but for now oh whoops here we have database right there Harrison you are blind anyway


just edit in tick instead of one day because that is what we are starting off with.


Okay so now what we want to go ahead and do is we are going to come down into our animate


function and now we need to go ahead and add in where are you animate okay so right now


the only thing our animation function is even capable of doing is animating tick data whereas


really we only want to animate tick data when that is the users choice so now we need to


go ahead and program this block of code or really this function to support various forms


of whether you know data frames basically or data time frames so we are going to go


ahead and global the refresh rate and we are going to global dat counter and not really


sure if we don't really have a refresh rate up here yet but that is okay we will leave


that there for now but we will have to add that in so coming back down to animate so


that just for the record that is not really a syntax error but we don't actually have


anything corresponding to refresh rate but we will deal with that later so the questions


that get asked here are first of all in order to animate we need a few things to happen


first of all if you recall we are offering the choice to either pause or resume the graph


from running so the first question that we should ask in this entire function is if chart


load okay so if chart load is true then go ahead and continue on if it is not true we


don't need to do anything else okay now I am going to make some more space we are going


to have a lot of statements here the next question we are going to ask here is about


pain count we haven't really talked about that the idea is eventually we can add multiple


pains and have comparison charts stuff like that for now we are just kind of putting it


in there for the future possibility but it is not really doing anything at the moment


so anyway if pain count equals one and then the next thing we are going to do is if data


pace equals tick so this is basically where all of this code needs to go of course the


only problem is this is only btce tick data so not only do we need to put this block of


code under there we need to put the code necessary to do all of the other exchanges and as you


will see they are each a little different luckily besides just tick data we are going


to be pulling all of the data from c of btc so that basically normalizes all of the data


so that will be a lot easier to handle and deal with so anyway if data pace equals tick


great we are going to run a try and we are basically going to try this block of code


right so we will take this and now we need to tab this over to fit underneath try so


one two three four tabs were required bring this up like that and then the next thing


that we are going to do is lets come down here and we need to throw in an exception


so we are going to say try and then it does that and then we say accept exception as e


print failed because of comma e if you are on python 2.7 accept exception comma e not


as e moving on so now that should work lets see if we got any issues we do pain count


is not defined great so lets go ahead and just define pain count at the top here as


equals one don't have pain count equals one run it again agree cool so that's all fine


and dandy the only issue at this point is not only do we get tick data from or at least


price data from ticks or price data in the tick data that's what I am looking for not


only do we get pricing data we also get volume data so it's kind of useful to know what volume


was involved in these prices so it's kind of silly not to put it up there so lets go


ahead and talk about how we are going to put that up there so coming down to animate I


am just going to do control f it's popped up in a different window but here we go animate


brings me down here and now we are ready to change up a few things first of all lets go


ahead and we are going to define two sub plots so we are going to have a equals something


and then a2 equals something a is going to equal plt dot sub plot to grid if you are


not familiar with sub plot to grid it's a little different than the typical one where


you basically say the sub plot equals like it's in a two by one and this is number two


something like that sub plot to grid works a little bit different you basically say what


the full grid size is and then you say the starting point and then you say the dimensions


so for example here sub plot to grid the full grid is a six by four so that's the first


parameter notice that that's a tuple as a parameter the starting point of this plot


or sub plot rather is going to be zero zero so the top left corner and then we are going


to say the row span so this is how many rows this is going to span crazy and then colspan


this is how many cols it will span so this is going to span five rows that leaves one


row remaining and the cols it's a full width ok that's why we are talking about pain count


because later on we could say this is part of a twelve by four or something like that


and it could take up less space and then we can put more stuff on there so this way it


really lets you customize very nicely so with some forward thinking here that's how we are


going to do it next we are going to say a two is basically the same thing so I am going


to go ahead and copy this paste it the only difference is it doesn't start in the same


spot and it does not span five rows it only spans one row finally there is one more parameter


we are going to add here and this one is going to be share x and that's going to be a and


what this means is it shares the x axis with a so as you zoom in to one of the sub plots


the other sub plot is affected it will be zoomed in to the exact same degree and if


you move around that kind of stuff it will be identical so they will always be aligned


since we are dealing with date data it's really important but really a lot of times it's really


important so anyway if you want to learn more about the customization definitely check out


the map plot the customizing map plot libs series I have although we are going to be


doing a fair amount of customization here so the next thing that we want to do is we


are going to change let's see I am going to go ahead and because we are going to have


to plot a little different so I come down here I guess and we are going to say we are


going to say data date stamp and that's going to equal np for numpy it's going to be a


numpy array of the data time stamp which is a unix time stamp and then we are going to


say dot as type so this kind of converts it for us and we are going to convert it to numpy's


date times 64 s ok so pretty much the same thing that we did down here we are just we


are basically adding this instead so we have done that now what we want to do is we are


going to say yeah we didn't do that either so all dates all dates equals data date oops


we need to have this date stamp and then dot to list because we are not able to pass a


numpy array through I suppose it would be a series anyway we have done that now we are


going to go ahead and talk to buys here we don't need this anymore and we don't need


this anymore so comment out these lines that's wasted processing next we like I said we get


volume data so now we want to add in the calculation of volume or the basically attribution of


volume volume equals data amount so it's just part of the data json that we are getting


right it's in that json already so and it's just under the label of amount that's how


much the volume is so then we run an a dot clear totally fine now we have got buy dates


sell dates that's fine the only thing that we are going to add now is let's go before


the legend we will say a two dot fill underscore between and this will be all dates because


this is volume data then you specify when you do fill between you it fills ok so you


have to specify what the minimum point of that fill is going to be in our case that's


going to be a zero you can do really cool stuff with fill between by the way this is


obviously not a variable but there are ways you can pass variables through there and you


can make some really cool graphs but maybe more on that way later the data that we are


plotting now is volume and then we are going to set the face color equal to we will use


color here so let's just copy paste boom so that fills it between that's why we need to


set all dates because we already we got the buy and sell dates but volume is basically


every data point is volume so that's why we needed to do that so now the next thing that


we want to do is add let's see we don't really we need to add the axis layer or the conversion


now so the next thing we need to do is a dot x axis dot set underscore major underscore


locator and we are going to do m ticker dot max n locator and then five and what this


does is it sets the maximum amount of actual marks so we are displaying full dates here


so obviously if we displayed 50 of them we wouldn't be able to see them anymore and so


there are some ways that you can have many so the default is to display them horizontally


but you can actually set write some code to make it display diagonally so you can have


diagonal dates and you can have more marks in it like it's a better way so they don't


run over each other because if you program it and you allow it to it will they will text


will just run over itself it's really ugly so we've done that now we want to set the


formatter and then we'll do one more thing but I'll show you why in a moment so we'll


do a dot x axis dot set underscore major underscore formatter and here we're going to format how


the date actually looks and we're going to use m dates dot date formatter and then here


you put the actual date code okay so for a full year you do percent capital y then we


want to just superimpose a dash in there and then a month is lowercase m that will be a


numerical month you could also do an uppercase m for the written out month I believe and


then percent day for the number as opposed to the d for the written out day space and


then we want the hours colon minutes we it is tick data so I suppose we could have seconds


as well that wouldn't really hurt mostly this way so we've done that and now finally let's


go ahead and run that and see how we're looking at the moment what that was an issue we didn't


get any reason for that issue data pays tick we are running hopefully tick I believe up


here oh we did get what do we don't have any like major error let's go animate try


to find this should show up let me just write a simple print statement here


so it's getting through all of the code not really sure why it's not and we're also not


getting our two panes there okay well we have yeah okay so we probably we've done probably


some stupid stuff here so first let's go up to the top here and we have F equals figure


that's fine we're gonna comment out a there and then let's go back down into animate here


we've got like some of this old code basically so we've got a and a to a dot plot is fine


a dot a two that should be fine look if yous why that wouldn't maybe it like won't replace


its own subplots I'm not too sure it's even a bigger mess okay hold on let's close out


of this it's okay the problem is there's basically two forms of figure okay so you've got just


figure that we've imported and been we've been using for Matt from matplotlib dot figure


and then you've got PLT dot figure so what we're gonna go ahead and do is first of all


we need to comment out this we're not going to use that form anymore and then come down


here and where you have F equals figure F now equals PLT dot figure so save and run


that and hopefully we should have a superior graph yes okay so here we go we've got the


volume data here and then obviously tick data this is pretty boring looking data mostly


because you've got you know spikes of volume that are pretty much overshadowing the other


trades most trades are pretty small so there's just really not much volume to see there but


obviously some of these like have some pretty large volume to them anyway so that's it for


this video we've got basically the tick set up and in the next video we'll add some of


the other exchanges so we can view tick data from the other exchanges and then we'll start


adding in the time frames and all that so anyways hopefully you guys have been enjoying


if you have any questions or comments please feel free to leave them below otherwise as


always thanks for watching thanks for all the support and subscriptions and until next


CM3: A Causal Masked Multimodal Model of the Internet (Paper Explained w/ Author Interview)

Today, we'll talk about CM3, which is a model that directly ingests websites, learns the


HTML, it uses a novel objective that does left to right language modeling, but with


a twist that essentially allows it to incorporate bi-directional information into the language


modeling.


It incorporates text, structure, images, hyperlinks, and with clever prompting, it can do almost


anything.


It can do what Dali does, generating images from text.


It can caption images, it can do text summarization, it can do entity linking, and it can do much


more.


I like this paper because of the idea of incorporating the structure of HTML, and also the new objective


is very cool.


So we're briefly going to go over what the paper is and does and how it works.


And then we're going to jump into an interview with Arman, who joined me in talking about


this paper.


It's a very informative interview, and I suggest that you give it a listen.


So this is just going to be a short introduction.


Again, I have to rely on you to tell me how I make the best use of authors coming on because


I think it's so cool.


I want to talk to them about the paper and I want to get the most information out there


for you that is possible.


So please tell me short intros, long intros, how to structure it and all.


Leave a comment down if you like videos like this, leave a like as well.


If you leave a dislike, you know, that's kind of useless now on YouTube, but you know, feel


free.


I'm still going to see it.


So CM3, a causal masked multimodal model of the internet by researchers at Meta.


I'm going to guess this is now.


So this model is, it's a family of models actually, and a family of causally masked


generative models trained over a large corpus of structured multimodal documents that can


contain both text and image tokens.


In fact, much more.


So what this model does, it's a language model and the language model ingests HTML, a cleaned


up version of HTML, but still HTML.


If you don't know what HTML is, HTML is essentially the language your websites are written in


and it consists of tags.


So for example, one tag is a div tag.


That is it's, it has, it had, I think it had a meaning at some point, but right now it


just serves as kind of a container tag.


So div might be something like a container and you close it by saying slash div.


Anything in between is the content of that div.


Other popular elements are, for example, a paragraph.


So inside a paragraph, you can have some text, hello there.


And then what you can also have is hyperlinks.


So hyperlinks start with an a tag.


So you can see these tags can be nested.


These tags can have attributes.


So the a tag can have an attribute like an href.


So that is a URL.


So www dot something and so on.


So it can have URLs.


It can also have URLs within the document.


Then there is the text of the link.


Now we close the a tag.


Oops.


Then we may continue the paragraph or we may close the paragraph, a forward slash.


And the last thing that we're also going to need in these documents right here are images.


So there can also be images and I'm going to write this over here.


After all, white space doesn't matter in HTML.


So images can have a so called source, the two most important attributes are the source.


And the source is it's usually, usually it's a URL, it can be a base 64 blob.


But usually it's also a URL, like, I don't know, like imgur.com slash something, something


dot jpeg.


So the browser would actually go and fetch that image and display it at this position.


And also, an important thing is the alt text, which you put there for screen readers and


other sort of assistive technology that cannot directly make use of the image to see what's


in the image.


So you can already see here that there's a lot of information in HTML.


Now previous work, what they would have done is, if it's a language model, for example,


GPT three, they would simply only take the text bits of that they would take, for example,


here, hello there, they would probably also take the text of the link right here.


And and that would be it, they would scrape the websites for the containing text to do


language modeling.


Other models such as Dali, Dali, I've made a video about Dali, if you don't know what


it is, but essentially a model that you put in text, and it gives you an image.


And the reverse of that is is sort of clip, not the reverse, but clip is a model where


that says whether or not an image or piece of text go together well.


And the reverse of Dali would be like a captioning model, you put in an image and you get a text


describing that all of that you can get by also scraping the internet, and always taking


the following two things, you take the alt text of a an image tag, and you take that


source image.


And these are pairs of images and text that go together, right?


So you can train this is kind of like weak supervision, there are some problems with


that.


But it's weak supervision.


Likewise, there are other tasks.


If you are, for example, doing entity linking or entity, entity disambiguation or something,


what you would do is you would go to Wikipedia.


And on Wikipedia, you would always take the text of a link and the link itself if it points


to another Wikipedia article.


And you know, in this case here, it says like, Romans were captured by Alexander the Great,


Alexander the Great would be a thing you could click on.


And then that link would sort of tell you what entity that is, it lead to the Wikipedia


page of Alexander the Great.


So people have parsed websites for a long time in various ways to achieve different


tasks to collect data for different tasks.


However, there is this new direction.


And it's not the first paper that does this, but it is the first that I've come across.


And the previous work is also by largely the same authors.


So I'm just going to give them credit for some at least some of this.


The the novel idea here is that why don't we use the entire structure of HTML directly


in instead of just scraping subset of them.


Now again, they do clean the HTML, because a lot of HTML is kind of like visual elements,


the cascading style sheets and so on, there definitely would be information there.


But it is a good step to say, hey, the whole thing, you know, the entire thing here, the


structure that is actually super duper important, it has so much structure that we would throw


away.


Otherwise, for example, the image right here, you know, it could be not only described by


the alt text, it could also be described by like the surrounding text, like this stuff


right here, of course, if there's an image on a website, reasonable to assume that the


surrounding text might also have to do something with it, right?


It is reasonable to assume that in order to disambiguate this entity right here, you might


want to take a look at the text around it, you might want to take a look at the images


around it, and so on.


So if we had a model that could directly learn the structure of HTML, we could exploit all


the work that went into creating that HTML, which is essentially what front end programmers


and website programmers do all day.


This is human ingenuity that goes into creating these structures, even if it's a framework,


right, that there's something, someone that has to come up with, you know, what are the


elements, how is the structure, and that is, that is really good data.