that people would like their ideologies to overrule reality. And I don't think that's


[ML News] AI predicts race from X-Ray | Google kills HealthStreams | Boosting Search with MuZero

a worthwhile goal. So that all being said, these problems are of course, incredibly difficult,


but we should look at them with the view of what's going to help the most people and what's


going to deliver the best outcomes for all individuals. And there are probably no easy


solutions for incredibly interconnected problems that are extremely multifactorial and include


things like genetics, environment, society, data gathering, and the entire historical


context of all of that. And that I guess is my rather boring take on that. In related


news, the New York Times writes, Facebook apologizes after AI puts primates label on


video of black men. Facebook called it an unacceptable error. The company has struggled


with other issues related to race. Now the article is about this Daily Mail video about


a couple of black men and the algorithm asks keep seeing videos about primates Yes or dismiss.


So the classification algorithm made a mistake here and this is not a new thing. As the article


states in 2015, Google mistakenly labeled pictures of black people as gorillas. And


the article also said more than two years later, wired found that Google's solution


was to censor the word gorilla from searches while also blocking chimp, chimpanzee and monkey.


The article then goes into some more intercompany things inside of Facebook trying to link this


to the system or something like this, which I find quite shady. Honestly, these systems


have a number of issues. There are issues of course, with data collection, there are


issues with all kinds of other stuff. But ultimately, these systems are trained in a


way that errors are errors. So if you fail to distinguish a yacht from a sailboat, that


is an error to the model in the same way as if you fail to distinguish a human from a


primate, the model has no inherent way of knowing that one is a socially acceptable


error, and one is a totally socially unacceptable error. There are ways to mitigate this, but


they usually require efforts on the part of humans that go there and essentially correct


for all the potential socially terrible errors that the model can do. And very often that


burden is so large, it's combinatorically very, very hard to do this, all you can do


is just block entire pieces of the search space in order to mitigate these mistakes.


This is displayed as some kind of like a negative system like, well, the AI is still biased,


but now we're just sort of censoring it. Yes, I mean, what can you do? It's very easy to


complain about these types of things. Now, of course, many of you might have noticed


that technically, the model isn't wrong, as human are the most abundant and widespread


species of primates. But you know, technicalities aside, I think we can all agree that this


isn't an output that you would want from your system. So what's the solution? I don't know,


probably the best solution would be an attack from multiple sides where the companies in


invest more work into mitigating these types of errors, which means essentially collecting


more training data on these intersections of very socially critical issues such that


the models get more confident about them. And on the other hand, it might also require


a little bit of a rethinking in society where we see a mistake like this not as some terrible


thing happening, but more into the category of mislabeling a sailboat as a yacht and vice


versa. It'd be nice if we get to a point where we think, Oh, cool, the system made a mistake.


Let's go on with my life. But of course, it's not always that easy because we use these


types of systems in situations where it actually matters what the system predicts. So ultimately,


it comes down to close supervision of your products and continuously evaluating their


deployments. Again, it's a hard problem. I'm confident we can make progress on it. Complaining


about it is fine. Just complaining and acting like it's the most terrible thing and it means


something beyond what it actually means is probably not helpful. ML news has previously


reported that Distill is taking a break due to the high load and the very high quality


standards they have leading to kind of volunteer burnout. They released what appears to be


some of the last articles that they're going to release in a while and they are on graph


neural networks. One is a gentle introduction to graph neural networks. The other one is


understanding convolutions on graphs. So the article pretty much contain what their title


says if you're interested in graph neural network, I can absolutely recommend you give


these articles a read. They have very good illustrations of what's happening examples.


And as you are used to from Distill articles, their quality is extremely high. Can definitely


recommend check it out.


Jürgen Schmidhuber announces that he'll be starting as director of the KAUST AI initiative.


KAUST is the King Abdullah University of Science and Technology in Saudi Arabia and is one


of the most well funded universities on the planet. Schmidhuber will remain in all his


other positions and lead the AI initiative there apparently traveling back and forth.


And on his blog, he writes, we hope the new AI initiative will contribute to a new golden


age for science analogous to the Islamic Golden Age that started over a millennium ago. So


quite likely we'll be hearing a lot more from KAUST in the near future.


Not really ml related, but maybe a little bit if you care about codecs and models that


produce code, GitHub has submitted a friend of the court brief, which is essentially an


advisory letter to the courts on DMCA takedown notices of copyrighted material in the space


of programming. Specifically, the brief concerns what they say is claims involving non literal


copying of software. And they give an example case right here where the SAS Institute has


brought infringement claims against world programming software. And specifically, they


claim that it is not specific lines of code that the defendant has copied, but only that


other aspects like the codes overall structure and organization were used. The blog post


here also says after examining the first question, the court found SAS Institute simply repeated


and repeated that their system was creative, but did not point to any specific examples


that would enable the court or the defendant to identify which parts were used in order


to ultimately define those parts that were actually protected by copyright. The court


ruled for the defendant leading to this appeal. Imagine something like you didn't exactly


copy my picture, but you use the same organization of putting paint on the canvas. Now get a


life SAS. Now, of course, I don't know all the behinds like copyright is such a complicated


issue. And there are legitimate cases where people steal from each other. And I can even


see that there are some cases where you can say, well, the structure of my code is so


unique and creative, and they copied it or something like this. Like, can't you just


spend the money on something useful. So GitHub's position on this is that with a DMCA takedown


notice, the noticer should specify in as much detail as possible, what are the parts of


the defendant's work that are infringing on the copyright such that there is even a possibility


of responding. Apparently, it's totally possible to issue a DMCA takedown notice simply by


saying, well, there's something in there. And I agree, that's not helpful, but ultimately


helpfulness and what ultimately results from the legal system and the courts don't always


match. So we'll keep an eye open on how this develops. So this week, there wasn't really


many questions in the news to be answered. But there were some really nice questions


on Reddit, some really good threads, I thought at least going with it. So there was a thread


on how machine learning will revolutionize physics simulations in games. This is almost


like a blog article in a Reddit post seems a little bit wasted, honestly, but it's pretty


cool. It details what kind of models exist for doing physics simulations and what their


advantages and disadvantages are. For example, here's one that's specifically good at modeling


large deformations and tears and so on. This is a piece of bread tearing apart. And it


also details how machine learning is being used in order to speed up the simulations.


Essentially what you want to do is you want to run the simulations, which are very intensive


until you have a data set. And then you want to train the model to sort of predict the


end of the simulation from the beginning, which seems like it should be impossible,


but hey, it's deep learning. So so pretty cool. If you're interested in the intersection


of deep learning and physics, give the Reddit post a read and of course, an upvote. So good


job say Ed HM for contributing to the ML subreddit. aristocratic octopus asks, what are the most


important problems in ML today? And I specifically want to highlight this thread because the


answers are both diverse and really good. They range from diverse environment learning,


catastrophic forgetting modular learning unstructured data, causality, few shot learning, generalization,


and so on. Now, these are things that are researched today. Yet I think if you are coming


into this field and looking for something to do, you don't really have an idea of what


to work on this thread might be a little bit of inspiration for you. Kamwa asks, do you


reproduce a method for state of the art comparison? Or do you just take the result from the paper


of the method for state of the art comparison? It's an interesting question. I've seen people


doing both. But the user says for example, they try to reproduce a method yet they couldn't


get the exact same score saying they only got a 30% accuracy on a task but the paper


claim they can obtain a 70% accuracy. They say they just ran the author's code with maybe


a little modification. Some authors said that they need to tune the hyper parameters. And


they also say they spend almost 90% time just trying to reproduce previous methods. Welcome


to ML research that is Yeah, I don't know what the answer is here. There are also various


opinions in the comments, you can almost guarantee that a lot of these research papers nowadays,


you cannot really count on their numbers, they might leave away from the paper a lot


of tricks that they have done to reach that number or the numbers are just fake altogether.


Of course, it could also be that the code they have on GitHub is kind of old code, which


happens often if you resubmit somewhere you redo some experiments, something changes in


the meantime. So there can be legit and illegitimate reasons why you don't get the numbers you


do. What you can do is you can report both the number they have in the paper, you can


also report the number that you achieved with their method and simply consider this as two


different baselines and explain yourself in the paper. It is a problem that you spend


like ginormous amounts of time reproducing baselines. And as the PhD progressed, I more


and more moved away from trying to get the exact numbers that baselines have gotten and


simply give it my best shot at reproducing them and then reporting that I think it's


up to you as long as you detail in the paper what you do, at least you can't be faulted.


And lastly, Olly Mac P asks, what kind of hyper parameter optimization do you use? And


again, if you are looking for good advice, this thread might be something nice for you.


There are suggestions such as Raytoon, Optuna, Hyperopt, and so on. If you want a cheap method,


I would start with all the hyper parameters on the default setting, then simply take the


one you think is most important and vary it a little bit while keeping the others constant.


Then once you found a good setting for that one, keep that one constant and vary one of


the other ones while also keeping the other one constant. If you found a good setting


for that one, keep going one by one through the parameters until you've tuned all of them


once and start from the beginning. And at some point, you'll converge, you might get


into a loop, but it's kind of unlikely that usually got me to relatively good places in


hyper parameter search. And it takes way less compute than running some kind of big grid


search. Usually these hyper parameters aren't that dependent on each other. So tuning them


individually is okay. Speaking of tuning and reproducing and performances, there is a new


paper from it's the USI and supsi called the devil is in the detail simple tricks to improve


systematic generalization of transformers, which gives a number of hints to what you


might want to tune when you train transformers. So the paper is an in depth investigation


into what it takes to train transformers and what matters and they give some advice. For


example, relative positional embeddings seem to outperform absolute positional embeddings


for certain tasks. Also, you should be careful on how you do early stopping and how you scale


your embeddings among other things. And lastly, the paper highlights the trouble with only


having I ID validation splits and not some sort of test that measures generalization


capabilities beyond the exact distribution that the model was trained on. If this is


of interest to you, give it a read. Also collaboration between Apple and the vector Institute release


unconstrained scene generation with locally conditioned radiance fields at ICC 2021 releasing


code on GitHub as well. And this is pretty cool. So this is scene generation, but with


a freely moving camera. So apparently previous works have sort of focused on small camera


movements, which is already impressive. But with this technique, it allows you to generate


scenes from a generator. So this is essentially a GAN that first creates a latent floor map,


and then based on that floor map generates the 3d environment in which you can then move


around the camera freely. So essentially, you can render that scene from wherever you


want. It still looks a little bit wonky. But I think the possibilities of these techniques


to make it into entertainment into training into simulation into gaming is pretty cool,


and probably not that far away. Again, the code is on GitHub, check it out. Facebook


AI research open sources common objects in 3d, a large scale data set for 3d reconstruction.


So this is a data set for 3d reconstructing what they call common objects. Apparently,


this is a crowdsource data set of objects that people just apparently happen to come


across, which is pretty cool, because these are things that actually appear in real life.


Seems like an extremely challenging data set, but often the most challenging data sets spur


new types of discoveries. If you work in 3d reconstruction, this might be your next challenge.


Salesforce releases warp drive extremely fast reinforcement learning on an NVIDIA GPU. We've


seen a number of libraries recently, such as Brax and Isaac Jim that make reinforcement


learning a lot faster by making use of the accelerators warp drive is especially geared


to do multi agent reinforcement learning. So multi agent reinforcement learning is where


you have many agents in the same world, and they need to interact with each other somehow


cooperating or competing. And the difficult part is, of course, that you need to evaluate


strategies for all of them, they depend on each other. And things like back propagation


become extremely hard, especially if you're limited in compute power. This library makes


optimal use of the power that you have. And I can definitely recommend that you check


it out if you are not a giant corporation. Speaking of giant corporations and reinforcement


learning, there's a new paper called boosting search engines with interactive agents and


look, it's me. So I've worked on this with this team as part of my internships and consultancy


gigs at Google, but I am in no way the main author here. The paper is about developing


agents that search in more than one step. So if you go to a search engine, usually you


enter some sort of query. And if you don't immediately find what you're looking for,


you may look at the top results and then kind of refine your query to find better results.


And that's exactly what we try to do with agents here. So here you might start off with


who won the US Open, you'll see a bunch of sports appearing and you might rephrase saying


that you're specifically interested in tennis and so on until you achieve the answer that


you want. What's specifically cool about this is that there's code to go along with it.


So next to the specific code that powers the search agents, there is a implementation of


mu zero based on a library called seed RL. Now this is also geared at making optimal


use of your accelerators in such as a GPU or TPU while massively distributing the inference


environments. So the mu zero algorithm is a generic I have authored part of it. And


if you are looking to use mu zero, this might be a good implementation for you as the mu


zero paper as well as the pseudo code they released contain various small subtle errors


that nevertheless make the whole thing essentially not work. This implementation right here to


the best of my knowledge contains less bugs. And it works pretty much with gym environments.


So you plug in a gym environment with a little bit of extra information on how your tensors


are shaped and so on. And that's all you have to do to trigger mu zero. So check out paper,


check out code and let us know if something's wrong. And last news AI startups claim to


detect depression from speech, but juries out on their accuracy. This is from venture


beat now time and time again, we see these articles about claims that AI can do something


but it turns out the reality is a little bit more complicated. So there are a lot of examples


of systems claiming to detect something to do with COVID. And then it turns out none


of them is useful. This here is a little bit less bad because with COVID there was a big


academic push to just make use of the hype to get papers published here we're already


a little bit into the direction of actual products being implemented. But still the


article details numerous problems that startups face. Some have only collected their data


from certain parts of the world to be exact just from one city others focus on only native


English speaker and confused not being able to speak English with showing signs of depression.


Still others neglect entire accents even for native speakers. And the list of problems


goes on and on and on. Again, I don't think this is a problem where there is any kind


of easy solution. I'm strongly of the opinion that we need to make progress in this there


is a shortage of mental health professionals. And it's not inconceivable that machines can


assist us and can deliver better lives to people even in the mental health area. But


exactly what shape that's going to take and exactly how we're going to prevent some sort


of dystopian future where some sort of buggy algorithm has way too much power over your


life is I guess one of the big challenges of our generation. Again, a good place to


start is to continuously monitor and evaluate the systems there are and to allow ourselves


to take some risk as we push forward as long as we have it under control. Again, I know


not a super strong opinion but what can I do? I'm boring. Cool. This was it for ML news.


Thank you so much for watching, listening and subscribing. If you know someone who's


not informed about the world of ML, please tell them about ML news. We're about to reach


Predictive Coding Approximates Backprop along Arbitrary Computation Graphs (Paper Explained)

Hi there, this is an LSTM cell or the computation graph of an LSTM cell. It is pretty hideous as you


can see, but what I'm about to show you is even more hideous. This is the computation graph of the


LSTM cell augmented with error units, evincing the connectivity scheme of the predictive coding


algorithm. So you may see that there are appearing these little red arrows right here that are so


called error units. And these are necessary for an algorithm called predictive coding,


which is an algorithm that is a biologically plausible alternative to backprop. And that's


what we're going to look at today, specifically this paper, as you can see, it is quite a thorough


paper. And it is called predictive coding approximates backprop along arbitrary computation


graphs. And have you ever heard a more descriptive title of what's in a paper? So the authors are


Baron Millage, Alexander Chance, and Christopher L. Buckley. This paper, as the title says,


it looks at this predictive coding algorithm. And it shows that this approximates backprop and we'll


see that this approximates is in terms of there is an inner iteration in the predictive coding


algorithm. And the more you run that, and under certain assumptions, this approximates the


backpropagation algorithm. And the new thing in this paper is a long arbitrary computation graphs.


So there have been papers before describing predictive coding this algorithm in various


sub settings like fully connected layers and so on, the fact that it approximates backprop there.


However, this paper shows that that's actually the case for arbitrary computation graphs under


certain assumptions. Predictive coding approximates the backpropagation algorithm. Why is this


important? Because the backpropagation algorithm isn't exactly biologically plausible. So they say


right here in the abstract backpropagation of error, or short backprop is a powerful algorithm


for training machine learning architectures through end to end differentiation. Recently


has been shown that backprop in multi layer perceptrons can be approximated using predictive


coding, a biologically plausible process theory of cortical computation, which relies solely on


local and Hebbian updates. So the difference between backpropagation and predictive coding


is exactly this point, that predictive coding relies solely on local and Hebbian updates.


Okay, and the keyword I think is local. So in a neural network, you have some sort of input x, and


you ship it through many layers, layer, layer, layer, layer, and then you have an output y hat,


and then you compare that output using a some kind of loss function with your with your true output


that you want. And then there is this backwards phase right here. And in this backwards phase, you


want to derive gradients for each of the layers weights. So each of these layers has a weight


associated with it. I'm not going into Greek letters again. So this is w, I don't know, w three,


w two is here, and so on. So what you want to get out as you want to say, how do I need to change w


in order to change my loss for the better. So what you want is this gradient, this gradient right here,


and backpropagation does a very natural decomposition, namely, if you have these


hidden states in here, so x is transformed to hidden state h zero, h one, h two, h three. So


that is the latent representation. If you want, for example, weight, if you want to know how to


change weight, let's say weight two, the backpropagation algorithm decomposes this into


the derivative according to the hidden state at layer two, multiplied by the derivative of the


hidden state by the weight. So this is what you would sort of learn in a beginner's course of


deep learning, this decomposition and of course, in this part right here, this part decomposes into


del L for h three, and then h three by h two. So this is the standard backpropagation algorithm,


you can clearly see in the formula, the computation graph, it goes from the L, it flows


backward to h three, right, so to h three, and then from h three, it flows to h two, and then from h


two, it flows to w two. So that's sort of the flow of the gradient backwards through the network.


And that's pretty cool, because it allows us to run gradient descent on arbitrary computation graphs,


which ultimately enable deep learning, including frameworks like TensorFlow,


Pytorch, or the older ones like Theano or LuaTorch, even AutoGrad, things like this.


It's pretty cool, but it's not really plausible in the brain, because neurons are not bidirectional,


bidirectional like this neurons generally, I'm not a neuroscientist or anything, but these neurons,


they have some sort of soma. And then you have this, this axon, right. And then these axon goes


into many different of these synapses to its to its children, and it kind of docks onto the somas


of or on the dendrites of the other neurons. And this is not bidirectional, this is generally here,


there's unidirectional signal in this direction. And there are so called feedback connections. So


from these neurons to the dendrites of this neuron, but you cannot really send this gradient


information, you cannot send this sort of vector gradient information. And you cannot do so in this


sort of sweep. So in the brain, it's probably not the case that the layer propagates forward,


and then sort of waits for a synchronized backward pass across the network in order to update itself.


All of this needs to happen much more in parallel, much more local so that things are only considering


local information of global information right here. For example, you need the global gradient


in the update of w two. And you need to have that back propagated. That's not plausible. So


predictive coding comes along. And today, we'll look mainly actually at how predictive coding


works. Of course, this paper is about extending it to arbitrary computation graphs, which is cool,


because they do predictive coding for CNNs, RNNs, and even LSTMs. And if you look at their so let's


first jump into the numerical results. If you look at their numerical results, they have lots of these


plots, where they basically show, we did this network, we train it with backprop, and then we


train it with predictive coding, and the lines are just the same. And so it's pretty convincing


evidence, even if you go super duper deep. And they do, I think RNNs with up to 100 layers or 100


time steps unrolled. So the empirical evidence that predictive coding approximates backprop is


certainly here. And we'll look at what predictive coding is, how it works, and how it works along


arbitrary computation graphs. So that's today's paper. And I hope you enjoy it. If you do, don't


hesitate to share it out and subscribe. Alright, so this graphic right here compares the two


algorithms in principle. On top, very much what I've said so far, the backpropagation algorithm


somehow has this signal, it propagates forward, okay, and then at some point, there's an output.


And if you want to train it, there is a label, you compare that to the output, that will give you an


error and by derivation, a gradient, and that gradient is now back propagated according to the


chain rule, according to the backpropagation algorithm, you can see, it's very much what I've


drawn, the predictive coding algorithm is a little bit different. And it's honestly not super clear


from this graphic right here, I find this graphic to be a bit confusing. But you can see, first of


all, there is this introduction of these error nodes in the computation graph right here. And


there also seems to be the introduction of these new hat, whatever that is. So we're sort of first


going to dive into the math. And then we're going to check out how the algorithm works as such. So


the math right here is a little bit, it's a little, you have to think a little bit differently


than you do in backprop. So first of all, they say we define a generative model, which parameterizes


the value of each vertex given the feed forward prediction of its parents, according to this


distribution, and a factorized variational posterior, where P denotes the set of parents


and C denotes the set of children of a given node X. So this is very special. Namely,


this turns the entire algorithm into a sort of a guessing game into a


into a variational approximation algorithm. So what they're basically saying is that signal


in this type of algorithm, signal isn't just forward propagated. But signal is forward guessed,


it's like a bit of a guess. So you have a signal right here, Vi. And this is a node in your neural


network. And when you forward propagate the signal, maybe this is a fully connected layer


right here. So it's simply multiplying it by parameter, you're not, you're not going to obtain


the next layer's signal, what you're going to obtain is a guess for the next layer's signal


right here, you're only guessing, you're assuming that you're sort of assuming that the true next


signal is somewhere in the vicinity of this. So what you do is actually assume this is a Gaussian


with the mean that you predicted, but then there is a fair a good chance it's somewhere around here.


So what you do is you always your guess the next layer's signal by forward propagating your own


signal. And you're, you're so you're not directly computing it. Okay. And the model that we have for


that here, you know, it's, why do we do this, we do this, because we're also not so sure about this


one right here. Okay, so the this entire thing is built upon, we're pretty sure what the input is.


And we're pretty sure what the label is of a data point. But without, you know, we're not,


we assume we're not really sure what the intermediate layers are. And we're going to run


sort of an update procedure on these on our guesses of where these intermediate signals are.


And that's going to be this predictive coding algorithm. So it's called predictive coding,


I guess, because you always only predict where the next layer signal might be. And you refine


that prediction in a series of inner iteration steps. And that all before you even do a parameter


update. So there's going to be an inner iteration to determine what the forward values are. And


this is very different from a back prop, there's just a single forward pass, right, then you know


the values, and then there's a backward pass. Here, there is, as you'll see, there is a single


forward pass, but then there is an inner loop to refine the forward pass. And before there is a


backward pass, and we need this because we only do this sort of local updates, you'll see in a second.


So the Gaussian I just drew, so the assumption is going to be that we


iteratively refine these guesses of where Vi is. And of course, here, you'll see that if I


change Vi to be down here, my next guess, so this is a time step t, my guess is this, my time step


t plus one is this, of course, if I apply the same fully connected layer, my new guess is going to be


down here somewhere. And so the assumption here that we're going to make is that


the value of each vertex is this model right here. This is the generative model, so it's a probability


distribution, depending on the parents. And we're going to approximate that by this variational


posterior, which as you can see, doesn't depend on the parent. So we're going to use the same


variational posterior, which as you can see, doesn't depend on the parents anymore. So it basically


says that the distribution is not conditional, it sort of stays the same.


Not sure if I expressed this quite correctly, but you can see right here, they assume a Gaussian


for the generative model that's dependent on these things. And then the posterior is simply


a factorized Gaussian and the variational approximation algorithm simply makes the KL


divergence between this variational posterior and the true assumed posterior small. And they can


prove that this is equal to these errors and the errors are going to be the errors between


what's predicted and what's guessed. It's best if we...


So if I extend this right here, I have v0. v0, I'm pretty sure what it is because it's my input.


Then what I'm going to do is I'm going to forward guess what v1 is. So this is my guess of v1.


Now from v1, I am going to guess what v2 is. And at the beginning, my guess of v1 is the same


as my forward prediction. I have no other reason. I have no reason to assume it's anywhere else.


So I'm just going to draw this on top of v1 right here. So since it could be anywhere,


it could be anywhere in the vicinity here, but I'm going to assume it's the same. I have no reason


to do so otherwise. And then I'm going to predict v2. And v2, let's say that's already my output


layer. And this is my guess of v2. That's already my output layer. But now we're going to compare


v2 to our true output, what we desire, our label L, and there's going to be an error. So there's


going to be an error right here. And what the predictive coding algorithm does is it basically


says, well, look, v2 could be actually anywhere here, anywhere around this thing, it's most likely


in the middle, but it could be anywhere. And it's actually quite possible that it's closer to this


label than we initially guessed. So it takes this error right here, this red error. And it says,


I'm going to update my guess of v2 a little bit closer into that direction. So


here is a new color. So v2 is going to be a little bit closer here. It's possible, right?


We simply guessed v2, so it could also be there. It's a little bit less likely.


It's a little bit less likely because it's not in the middle of the Gaussian, but v2 could be where


it is. But now I have to sort of communicate this error back to the last one. And the trick here is


that we don't communicate the global gradient, but we only communicate these local error signals.


So this first red arrow here is our first error signal. And we are going to communicate that


thing back to the previous layer. So the difference between v2 and v, and here is a fully connected,


let's say this is a fully connected layer. What we're going to send back to the last layer is this


information of, see, you predicted v2 hat, but actually you should predict v2. Please update


yourself such that that doesn't, you know, that's a bit closer. So now we're going to update our


guess of v1 and say, well, if we moved v1 a little bit over here, that would predict v2 to be up here,


right, with the same fully connected layer. And if that's the case, then v2 would be a little closer


to the true label. So we're going to move v1 over here. Now we're not going to move it fully because


so this is a sort of optimization. There is a force keeping it to where our original guess is,


but there is also a force drawing it in the direction of this error signal. You can see.


So we're going to say, well, if we just move v1 to up here, we would predict the perfect v2.


But also it's less likely. So we're going to find like some sort of a trade off where it's still


quite likely under our Gaussian assumption. But it will predict a little bit more of the correct


label, and so on. So this, if we had a longer computation graph, this would then sort of every


node in the computation graph would ask itself, I am going to guess my own value at a place that is


pretty close to my original guess coming from the forward propagation, but also is consistent with


the output of the next layer. And the output of the next layer, of course, here is this v2, right?


So that the logic isn't I need to make the loss small, the logic is, well, if the next signal is


v2, then I can't be in the middle here, I must be a little bit more up here, because you know, I, my


signal runs through the fully connected layer and outputs v2. So I am probably more up here. So you


can see that if you have a computation graph v0, v1 hat, v2 hat, v3 hat, and so on. If at the end,


you have a loss signal, you're sort of distributing that loss across this entire chain. So you're kind


of building this guessed chain of values, v3, and so on. And sorry, that's the output node,


which is close to the loss. You're moving all of these things. And now once you've done this,


once you've done this, you can do one step of parameter updates. So once you've guessed all the nodes,


what you can go ahead and say, okay, this is this is a configuration that is at equilibrium


in this sort of algorithm. And now, here are here is fully connected layer one. So here is


here is w0, here is w1, w2, and so on w3. So now we can go ahead and actually update these weights


such that the initial guesses that we had, and where we truly think the signal is, are closer


together. Okay, so we're now going to update the weights in order to minimize all of these


individual errors. And this is also can be done locally. So you see that the parameter update


step here is now a local one, because we've computed all of these errors between where we


initially guessed the signal is and where we sort of think it should be. Now we can minimize these


errors. So what I've drawn here is actually not exactly the algorithm, but I hope you get the


point. So step one is you sort of guess where all the stuff is initially, then at the end, you get


an error signal, right, this is an error signal, then you distribute that error signal backwards.


And that is now that is not the same as distributing a gradient, I know it looks the same,


but it is not the same. And so I have to say that, you know, they say, oh, this is only local and so


on, this doesn't require a backward sweep, I think when I look at this algorithm, it very much does


require a backward sweep. So very much it goes from the back to the front. In fact, it goes from


the back to the front many times. Now you can do that in parallel. So this node here can update so


to finish the argument here, as I said before, then you kind of wiggle on these nodes to find out,


this should probably be more here, this one should probably be more here, this one should probably


be more here, this one should probably be more here in order to satisfy, in order to make that


error smaller. And the point is that the parameter update step now is a local one. Okay, so the


parameter update step now only needs these local errors between where you initially guessed and


where your refined iterative guess is after distributing the error through the network. And


this can all happen in parallel, this, this, all of this updating, sending information around and


so on, this can be parallelized, but it does require a backward sweep, if you ask me.


Okay, so there are two equations, so the, the, there's two things right here. There is first,


as we said, there is a phase where the guesses of where our vertex units are, where our hidden


representations are, are refined. And this is given by these dynamics right here. So you see that


Vi changes with time according to this thing right here, F is the variational free energy. So this,


this algorithm sort of falls out from the math of assuming these, assuming these generative models


right here under the assumption that they are these Gaussians. Okay, so under the assumption


so under under this assumption, if you calculate the KL divergence, it turns out to come out to


this algorithm right here. So how does the, how do we need to update the node Vi? The node Vi is


updated according to this gradient and this gradient is, as we said, only computed as properties of


local things. So the first thing is Ei, which is that's, so again, if we have this is our initial


guess of Vi, and then here is our refined guess of Vi, Ei is the error right here. That's, that's


sort of, we need to stay close to our initial guess. But also, we want to go into the direction


such that into this direction right here. So Ej, j is the children of Vi, j are the children. And


this thing right here says, how do we need to change my guess of Vi to make, to make it fall


more in line with Vj. And you see here, that's Vj, the initial thing, but then, of course, the error


is, so the error j is going to be the difference between Vj and Vj hat. So ultimately, you are


guessing, you're saying, how do I need to change Vi in order to make it more commensurate with Vj


after going through the layer? Okay, so this, this derivative right here, this is going to involve


the derivative of whatever the fully connected layer or the conv layer and so on. So there is


not, there's not no derivatives in this algorithm, but there are only sort of these local derivatives.


So Ei is going to be the difference here. And then we'll have the fully connected layer using


w gives you Vj hat, but also your refined guess gives you Vj. And the error j is going to be


this thing right here. Okay, so you want to stay close right here, but also you want to


make Vi such that it outputs Vj such that it also minimizes that error. Okay, sort of.


Yeah, it's, it's hard to, it's hard to draw these things, but I hope I've explained it in multiple


ways right now. It's at least a little bit clear how this works. And at the end, once you've reached


equilibrium of all of your guesses of all of your guesses of where the next nodes are,


what you do is you update your parameters here in a local fashion. You can see right here,


what you need is this error of the if layer and you multiply that by this derivative. And this


derivative is simply the local derivative of your hidden representation with the


expectation with respect to your layer. Okay, so this is very akin to in the back propagation


algorithm, hi to wi. This is just this local derivative. So using the update, the update step


of the weights now only requires local derivatives. And that's the point. So here, it's in this


code, things are a little bit a little bit unclear in this, but we'll do so for the entire data set


x is the data point and l is the label, you fix the start, so you fix v zero, then you go you do


the forward pass. So you do this once you these are your initial guesses, these hat things, and


see the hat things are always computed from the parents, you compute the output error right here,


and then begin backwards iteration phase of the descent of the free energy. So here you see,


there is this inner loop while not converged. And this is just going to work out to be some sort of


in some sort of an inner iterative scheme for a number of steps, this is going to be a hyper


parameter. And this here, this is something you can technically do in parallel, you have to send


a bit of information around. But you can technically do it in parallel, this inner, these inner loops.


But you can, you can just imagine it always going from the back. And you distribute these errors,


you refine your guess a little bit, and you start from the back again, you distribute errors, refine


your guesses, and so on. And you do that, you always start from the back in the actual code. So


you compute these errors. So this is your initial guess. And this is your refined guess of the


current layer. And then you update the vertex values, you say, okay, the my guess for the next


layer is going to be my guess for this layer, plus some sort of a this gradient and this gradient we


get from equation number two, from this thing right here. So my guess is going to be updated


such that I still stay close to my original guess. But I also update, I also predict better what the


next layer is. And at the end, when this is converged, you do the update on the weights and


the updates on the weights is simply again, this, what we saw, it's the error that you want to


correct. So this E is the error you want to correct. Now you have a good approximation of


the error once this is converged, times the derivative, of course, with respect to the weights.


So the error is in terms of how much are your predictions of from what they should be. And the


derivative simply translates that into the how do you need to change the weights such that in the


future, that error is smaller. So then they show that this actually approximates back prop. And


this it's a fairly, fairly simple proof. It's an it's sort of a proof by induction, by iteration


that showing that one such thing like this thing right here at the equilibrium at the last layer


is equivalent to back prop, because you can simply substitute this and then by sort of recursion


that goes back the layers. And this is all dependent on you actually reaching that equilibrium,


which you do, as we said, by inner iterations. So they have a bit of an example right here,


where they have this function of it's a pretty simple function, this function right here. The


output is the tan of the square root and there's parameters in there. So this is an arbitrary


parameter that you might want to learn. And then you give some data sets. So this is equal to


two, but I guess the network doesn't know that. I don't know. So you have to learn it. And they


test that. And you can see the this augmentation by error graphs makes the computational graph quite


a bit more complex. So you have all these error graphs right here. But, you know, ultimately,


you know, ultimately, you could automate this. That is not a problem.


Okay, so they also do this for, as I said, CNNs, RNNs, LSTMs. And the results are quite


remarkable, I think, in that they just follow the same accuracy and loss and performance patterns


of these networks. That's pretty cool. The downside, of course, is that they are way smaller,


sorry, they're way, way slower. And they say this sometimes, due to the need to iterate the Vs until


convergence, the predictive coding network had roughly a 100 times greater computational cost


than the back prop network. Though they say this is a bit misleading, because you can distribute


and parallelize that. However, as we've seen, it's not fully local, like you need to send signal


around every node needs to send signal to its parents or its children. And that, of course,


in back prop, you just need to do that once, right? So I'm not exactly buying this argument


of this is much more local, and so on. So the last thing that I want to point out in the paper,


and then we looked briefly at the code is this thing right here. There's a further simplification,


they say, importantly, if the edge function linearly combines the activities and the parameters


followed by an element wise non linearity, which is most of deep learning layers nowadays,


a condition which we call parameter linear, then both the update rule for the vertices and the


parameters become Hebbian, specifically, the update rules for the vertices and the weights become so


here is if you have a linear operation followed by a non linearity, which you know, is the fact


in RNNs in CNNs in fully connected layers, then this here are these update rules. So the local


layer derivative is simply going to be your forward activations passed through and this is a


bit weird. It's the forward activations passed through the derivation of the non linearity. This


is the non linearity right here, times again, the weights of the forward iteration, and the


update rule with respect to the parameters are very, very similar. And the reason I point this


out because now we're going to jump into the code. And I hope you can see this, you can recognize


this again. So first of all, let's go into the into the CNN. Hello. Alright, so the code is quite


quite ugly, honestly. But you see that they have they've backprop or CNNs, but they have this thing


right here, this, this model, which is the one they train and here is the train function. So in


the train function, they go through the data set. And you can see for each data point is simply


called this infer function right here. So this infer function is what ultimately does the training.


So in the infer function, they get an input, as you can see, and the label and the number of


inference steps. So they start out by and this this is labeled a bit a bit different. So that


if these mus and outs and these prediction errors and the predictions, and we're going to see how


that works. So first of all, they go through the layers right here, and I'm going to use my mouse,


they go through the layers right here. And you can see they simply forward propagate the signal. So


they always take this mu of the last layer, they forward propagate it to get the mu on the layer


plus one, and the outputs are simply cloned from the mus. So these must be our mus before or our


vs, whatever you want to call them. So one is going to be the initial guess, and the other one


is going to be the guess that we iteratively refine. Okay, in fact, the mu here is going to


be the guess that we iteratively refine. At the beginning, we simply set them to be the same. Okay,


and then the last layer here, we put at the label, and then the prediction errors, that's going to be


yeah, that's going to be the error variables. So the last prediction error is going to be the


derivative of our loss function with respect to the last layer. And now we start this iterative


algorithm. So here you see we go through this number of inference steps train, which is going


to be like 100 or so. So 100 times, we're going to update each of our guesses of the intermediate


layers. Then here is what I said, we're going through the layers in reverse order. So 100


times, we're going from back to front, back to front, back to front, back to front. And we do


that. So here you can see what the first thing we do is we compute the current error, which is the


difference between the guess that we currently have and the initial guess that we had during


forward propagation. This is going to be zero for most of the layers at the beginning except


the last layer, right? In the last layer, we've actually put the mu


to something else than the output. And thus this error is going to, it's beginning at zero at each


layer, as the guesses are the same, but then we're going to refine and refine and refine. And sort of


this error of the last layer is going to iteratively propagate through the network


from the back to the front multiple in an iterative fashion, so multiple times.


So once we have the prediction error, we're going to backward this through the layers and this


backward here, that is sort of, that is this backward edge we saw. Where did we see this?


So this backward is going to be this local derivative. In this graph, the backward is


going to be the red thing right here. So we take the error of the next layer and we're going to


see how do we need to change the current guess in order to make the next layer's error


be a little bit smaller. So that's going to be the backward function and we can actually look


at the backward function of let's say, yeah, here. So this is the backward function of a fully


connected layer. This is the projection layer. There is a fully connected layer


here and the F is going to be the nonlinearity and the DF is going to be the derivative of the


nonlinearity. So in the forward, you can see what we're doing is we're multiplying the input by the


weights and then we're going to save the activations and simply propagate them through the nonlinearity.


In the backwards, we're going to take the activations, the forward activation, and we're


going to shove them through the derivative of the nonlinearity. And this is why I pointed out this


is this Hebbian learning rule. So first I was a bit confused. Why do we use the forward activations


and shove them through the derivative of the nonlinearity? But this is exactly, this is simply


because they've derived that this is the correct local gradient. And then we have this,


this is the local gradient of the layer and we're going to multiply that by the weights.


So this completes the formula that we had right here for these Hebbian updates, this thing.


So these are the activations. This is the derivative of the forward layer. We're going to


multiply that by the weight again. So this is now the complete derivative, the complete local


derivative, which is this thing I've already circled 50 billion times right here. And all


we need to do now is we need to multiply this by the error, by the prediction error in that layer.


And then we get an idea of how do we need to change this node such that in this one child,


and there can be many children, such that in this one child, we make a little bit less error.


So that's why we multiply this by E right here. So E is the error. And that will be the backwards


thing. So backwards simply tells the parent how it needs to change the child, sorry, how it needs


to change itself such that the child is a little bit happier. And since this is a forward, you know,


a CNN, we don't have multiple children, we simply have one child per parent. So we have a list. And


these predictions, as you can see, we simply take the prediction error of layer j plus one,


we backward it. So how do we need to change this layer in order to make it a little bit more


commensurate with the child. And then here is this trade off. So the trade off between


the prediction error. So how close am I to my original guess, I don't want to go too far away,


right? Because I assume my original guess isn't too bad. In fact, there's a Gaussian likelihood


model, how I want to stay close to that. But also, I want to go into the direction such that I make


the next layer happier. This is this fundamental trade off, it's computed right here. And it's,


it's this minus sign. And then at the end, this is the inference learning rate. And I simply go


into that direction of this trade off. Okay, so I update the current the guess of the current node


like this. And as I said, I go through the network back to front, back to front, back to front, back


to front, back to front, until I reach some sort of equilibrium. And only when I reach equilibrium,


or in this case, after this many steps, I then update the weights and the update the weights


function. That's very similar, I think here, here is update weights. That is simply I, each layer,


I input the prediction error of that layer. And that layer calculates this function right here


in much a similar way than you just then you just saw. Maybe we can look at one of them.


Let's go. This is layers. Let's go here. Fully connected layer. Okay, and you're going to see


this Hebbian learning rule again. So activations through the derivative. And so now instead of,


so there's a little bit of a difference to before, right, but the difference isn't large.


Right, so activations multiplied by through this and then multiplied by the inputs instead of the


weights. So that's that. So this multiplied by the inputs instead of the weights, then multiplied


by E, which is so this here multiplied by the error term right here. And that's going to be


our local update. Okay, cool. So that's the code. That's predictive coding. And, you know,


the challenge is, it's not that these people propose this as a true alternative to backprop,


but it is a step in the direction of saying, look, the brain with its more Hebbian nature,


and it's more local updates, and so on, it could actually be doing something much more close to


backprop than we thought, because people thought, well, backprop is impossible in the brain,


therefore, the brain can't be doing backprop, right. And now we see that actually, the brain


can do something possibly, it's not proven, but it's possible that the brain does something


that approximates the backprop gradient actually arbitrarily, if you know, if all of these if


these some assumptions are given, but that's sort of the results. And they also show it's quite


robust to learning rate changes, and so on, as we said, we can go pretty deep, even though this is


this kind of iterative guessing algorithm. Under these Gaussian assumptions, and there is variational


approximation, it is fairly robust and all. So this goes, this sort of puts the ball back into


maybe the brain is doing something very close to backprop, or at least getting the same results,


getting the same parameter updates as backprop. So I hope that wasn't too confusing. I've tried


to tackle it from many angles. And maybe after seeing the code, you see it a little bit more


Learning To Classify Images Without Labels (Paper Explained)

Hi there, check out these clusters of images right here.


And just have a look at how all of them are pretty much showing the same object.


So here's balloons, here's birds, here's sharks or other fish.


These are from images from the ImageNet data set.


And you can see that these clusters are pretty much the object classes themselves.


So there's all the frogs right here, all the people that have caught fish.


So the astonishing thing about this is that these clusters have been obtained without


any labels of the ImageNet data set.


Of course, the data set has labels, but this method doesn't use the labels, it learns to


classify images without labels.


So today we're looking at this paper, Learning to Classify Images Without Labels by Wouter


van Gansbeke, Simon van den Hende, Stamatios Georgoulis, Mark Prozemans and Luc van Gool.


And on a high level overview, they have a three step procedure.


Basically, first, they use self supervised learning in order to get good representations.


Second, they do a clustering.


So they do a sort of k-nearest neighbor clustering, but they do clustering on top of those things.


But they do it in a kind of special way.


And then third, they do a refinement through self labeling.


So if you know what all of these are, you basically understand the paper already.


But there's a bit of tricky steps in there.


And it's pretty cool that at the end, it works out like you just saw.


So before we dive in, as always, if you're here and not subscribed, then please do and


if you like the video, share it out and leave a comment if you feel like commenting.


Cool.


So as we already stated, the problem, they ask, is it possible to automatically classify


images without the use of ground truth annotations?


Or even when the classes themselves are not known a priori?


Now, you might seem like you might think that this is outrageous.


How can you classify when you don't even know what the classes are and so on?


So the way you have to imagine it going forward, and they don't explicitly explain it, but


it's sort of assumed that if you have a data set, and you learn to classify it, what basically


that means is you cluster it, right?


You put some of the data points in the same clusters.


And then, of course, the data set, I'm going to draw the same data set right here, the


same data set would have an actual classification thing.


So this would be class zero, this here may be class one, and this here might be class


two.


Now, you can't possibly know how the classes are like called or something, which one is


the first, which one is the second.


So at test time, basically, if you have a method like this, that doesn't use labels,


what you're going to do is you're basically going to find, you're going to be as generous


as possible.


And in the assignment of these and say, Oh, look, if I assign this here, to cluster zero,


and this here to cluster two, and this here to cluster one, and I just, you know, carry


over the labels, what would my accuracy be under that labeling?


So you're, you're as generous as possible with the assignments of the labels.


So that's how it's going to work, right?


That's what you have to keep in mind, we're basically developing an algorithm that gives


us this kind of clustering of the data.


And then if that clustering partitions the data in the same way as the actual labeling


would, the actual labeling with the test labels, then we we think it's a good algorithm.


Okay, so they claim they have a.


Okay, in this paper, we deviate from recent works and advocate a two step approach.


That's actually a three step approach, but we're feature learning and clustering are


decoupled.


Okay, why?


Why is that?


So they argue, what you could do, what people have done is, and I'm going to, well, this


is just a wall of text.


So what you could do is you could just basically cluster the data, like who says you can't


use clustering algorithms, but then the question is, what, what do you cluster them by?


Like you need a distance.


So if I have points in 2d, it sort of makes sense to use the Euclidean distance here.


But if I have images of cats and dogs and whatnot, then Euclidean distance between the


pixels is really not a good, good thing.


But also, so you might think we could actually, we could use a deep neural network, and then


basically send the image, that's the image right here, send the image through the deep


neural network, and then either take these last state right here, so it goes through


and through and through.


And we could get take either of the hidden states, or we could just take you know, the


last state that is the sort of hidden representation right here, and to do the clustering with


that.


But then of course, the question is, what do you what which neural network do you take?


How do you train that neural network?


And there have been a few approaches such as a deep cluster, which try to formulate


basically an objective for that neural network where you first you send all the images through,


right, you send a bunch of images through to get you in embedding space, get you points


and then in embedding space, you think, well, the features that are in the embedding space,


they are somehow latent and they if basically the entire thing is, if this neural network


was used to classify images, you would have a classification head on top.


And a classification head, this is like a five class classification is nothing else


than a linear classifier boundary that you put on top of this hidden representation.


So if you were to use this neural network for classification, it must be possible to


draw a linear boundary between the classes.


And therefore, the either things like the inner product distance or the Euclidean distance


must make sense in that space.


They don't make sense in the picture space, but they must make sense in the hidden representation


space because what you're going to do with them is exactly linear classification.


The last classification head of a neural network is just a linear classifier.


So the assumption is that and the conclusion is, well, in this space, you should be able


to cluster by Euclidean distance.


So what deep cluster does, like is first get the representations, you start off with a


random neural network, then cluster these representations, then basically label, self


label the images in a way.


Now I'm way oversimplifying that technique right here.


But you have this alternative steps of clustering and then kind of finding better representation


and then clustering these representations.


And what it basically says is that the CNN itself is such a is like a prior because it's


the translation environment works very good for very well for natural images.


The CNN itself will lead to good representations if we do it in this way.


And there's some good results there, but this paper argues that if you do that, then the


algorithm tends to focus a lot on very low level features.


So if the pixel on the bottom right here is blue, right, then you can end the neural network


by chance puts two of those images where the blue pixel on the bottom right, it puts them


close together, then in the next step, it will because they're close together, it will


cluster them together.


And then it will basically feed back the new representations should put the two in the


same class, right?


It will feed back that it should focus even more on that blue pixel.


So it's very, very dependent on initializations.


And it can jump super easily onto these low level features that have nothing to do with


the high level task you're ultimately trying to solve, which is to classify these images later.


So what this paper does is it says, we can eliminate this, we can eliminate this, the


fact that these methods will produce will produce neural networks that focus on low


level features.


And how do we do that?


We do that by representation learning.


So representation learning, you might know this as self supervised learning.


And this is the task they solve in the first step of their objective.


So let's go through this, this right here is an image.


Now, the T is a transformation of that image.


And in self supervised learning, there are several methods that you can transform an


image.


So for example, you can random crop an image, you can just cut out like a piece right here


and scale that up to be as large as the original image.


Or you can use for example, data augmentation, which means you take the image, and you basically


so if there is, I don't know the cat right here, you you kind of convolve it with some


things it's there's like a very squiggly cat.


Okay, I'm terrible.


You can, you can rotate it, for example.


So it's like this.


Okay, so these are all these are all sets, including the crop sets of this transformation


T. So you transform it in some way.


And you want after you've transformed it, you send your original image.


So that should be read, you're sending your original image and the transformed image through


a neural network, each one by themselves.


Okay.


And then after this, you say the hidden representation here should be close to each other.


Okay, this is this is basically the self supervised training task is been shown to work very, very


well.


As a pre training method for classification neural networks.


You have an image and its augmented version, and you minimize the inner product or the


Euclidean distance between the two versions in the hidden space.


And the rationale is exactly the same.


The rationale is that this hidden space, of course, should be linearly classifiable.


And so the distance between those should be close.


And the rationale between having these tasks is that, well, if I flip the image, right,


if I flip the image to the right, it cannot focus on the pixel on the bottom right anymore,


because that's not going to be the pixel on the bottom right here.


And I'm not always going to flip it into the same direction.


And sometimes I'm going to crop it.


So it also can't focus on the pixel on the bottom right, because in the crop, that pixel


is like out here, it's not even in the crop.


But basically, what you're looking to do with the self supervised methods is you are looking


to destroy this low level information.


That's that's all you're looking to build a pipeline of a neural network here that destroys


deliberately low level information and you do that by coming up with tasks like this


self supervision tasks that deliberately exclude this information from being used.


I think that's what's going on generally in the self supervised learning thing.


Okay, so this here, as you can see, is the neural network that you train, you send both


images, the original and the augmented version through the same neural network, and then


you minimize some distance, which is usually like the inner product or the Euclidean distance


in this embedding space, okay.


And what you train, you can see right here, you train the parameters of this neural network.


So the transformations are fixed or sampled, and the distance is fixed, you train the neural


networks such that your embeddings minimize this task.


Now this is nothing new.


This has been this has been used for a couple of years now to get better representation.


Self supervised learning is a thing.


But they basically say we can use this as an initialization step for this clustering


procedure because if we don't do that, we we focus on these low level features.


Okay, and notice you don't need any labels for this procedure.


That's why it's called self supervised.


Okay, so the second second part is the clustering.


Now they cluster, but they don't just cluster these representations that would be that doesn't


perform very well in their in their experiments.


What they instead do is they minimize this entire objective right here, and we'll go


through it step by step.


So they train a new neural network.


Okay, this thing right here, this is a new neural network.


So first you have you already have the neural network, which was called, what was it even


called the one that gives you the embedding with the theta, okay, it's called phi theta.


It's the same architecture, and I think they initialize one with the other.


So in step one, you get phi, theta, phi theta goes give from from


x gives you a representation of x, okay, let's call it hidden x.


So that's via self supervised learning.


But in step two, you train an entirely new neural network, this phi, eta here, and you


initialize it with this one.


But now you train it to do the following.


Again, you want to minimize, sorry, you want to maximize the inner product, right here.


See that's the inner product, you want to maximize the inner product between two things.


Now that's the same thing as before, we want to minimize the distance between two things,


and the dot product distance in that case, you maximize the dot product between two things.


And the two things are two images that go through the same neural network as before,


right, this and this.


Now what's different here is that here we input an one image of the data set.


That's the same as before, okay, so we input one image, but here before in the self supervised


learning, we input an augmented version of that.


And now we input something else, we input this k right here.


Now what's k, what k comes from this neighbor set of x, okay, this is the set of neighbors


of x, and these neighbors are determined with respect to this neural network right here.


So what you do after step one is you take your neural network with the good embeddings,


and here is your data set x, your data set x, this should be another, your data set x


is this list basically of all the images in your data set.


And what we're going to do is you're going to take all of them using that neural network


that you just trained and embed them into a latent space right here.


Okay.


This is the latent space where you have done this self supervised training.


And now for each image, right here.


So if this is x i, you're going to find its k nearest neighbors and they use I think they


use five as a benchmark.


So you're going to find its nearest neighbors, its five nearest neighbors.


And you do this for each image.


So this image has these five nearest neighbors, so on.


So in step two, what you're trying to do is you're going to try to pull together each


image and its nearest neighbors in that in this, not in this space directly, but you


determine which ones are the nearest neighbor from this neural network and you keep it constant.


That's how you determine what the nearest neighbors are in the first task.


And that is your n x set for x i.


And in the second step, you're trying to make the representations of any image and its nearest


neighbors closer to each other.


Okay, so with with this thing right here, you maximize the inner product between x in


after this neural network and a nearest neighbor of x that was was a nearest neighbor after


the first task.


Now, the way they cluster here is not just again by putting it into an embedding space,


like we saw before.


But this thing right here, this neural network, as you can see here, is a C dimensional vector


in zero one.


Now, C is the number of classes that you can either know that so you don't know which class


is which you don't have labels, but you could know how many classes there are, or you could


just guess how many classes there are.


And as long as you as you over guess, you can still like build super clusters later.


So this, they simply say, it's in zero one, but they also say it performs a soft assignment.


So we're also going to assume that this is normalized.


So for each for each data point x here, you're going to going to have an image, you're going


to put it through this new neural network, okay, this new neural network new, and it's


going to tell you, it's going to give you basically a histogram, let's say class one,


two, or three, we guess there are three class and it's going to give you an assignment of


the three.


And you also take a nearest neighbor, here is your data set.


You also take a nearest neighbor of that.


So you look for this set n of x, and you take a nearest neighbor, maybe that's, that's a


maybe that's a dog, I can't, I really can't draw a dog.


Yeah, that's the best I can do.


I'm sorry.


And you also put that through the same network.


And you're saying, since they were nearest neighbor in task one, they must share some


sort of interesting high level features, because that's what the first task was for.


Therefore, I want to make them closer together in in the in the light of these of this neural


network right here.


So this is also going to give you an assignment like, maybe like this.


Okay.


And now you object you train this network right here to basically match these two distributions.


Okay, so this is this is now a classifier into C classes, but we guess C, and we don't


have labels, we simply our label is going to be my neighbors from the first task must


have the same labels.


That's our label.


Now, they say they also have this term right here, which is the entropy over assignments.


Okay, as you can see, so they minimize the following, they minimize this quantity, which


has a negative in front of it.


So that means they maximize this log inner product.


And they also maximize the entropy, because, sorry, so they minimize this thing, and but


the entropy is a negative quantity, right?


So they maximize the entropy because here's a plus.


And now they minimize the entropy.


Let's see what they say.


By minimizing the following objective.


Now entropy is the sum of the negative sum of p log p.


And this if this is p, yes, this is the probability that an image is going to be assigned to.


To cluster C over the entire data set.


So they're going to Yes, so it's negative, this quantity, negative minus p log p.


And this is the entropy.


So they're going to minimize the entropy.


We include an entropy term, the second term in equation two, which spreads the predictions


uniformly across clusters C.


Okay, so what we want is a uniform assignment over cluster, which means we should maximize


the entropy.


Oh, yes, okay.


They minimize this thing, and this here is the negative entropy, right?


Okay, so they want, basically what they want over the whole data set that not all of the


images are going to be in the same cluster.


This is cluster one, and then this is cluster two, and then this is cluster three.


So that term counteracts that.


Basically the more evenly spread the entire data set distribution is, the higher the entropy,


the lower the negative entropy, and that's the goal right here.


I'm sorry, this was, I was confused by the too many negative signs, and then you minimize


the entire thing.


All right, now they say a different thing right here.


They say here, this bracket denotes the dot product operator.


As we saw, it's the dot product between these two distributions right here.


The first term in equation two imposes this neural network to make consistent predictions


for a sample x i and its neighboring samples, the neighbors of x i.


And here is an interesting thing.


Note that the dot product will be maximal when the predictions are one hot, and that


means confident, and assigned to the same cluster, consistent.


So they basically say the objective encourages confidence because it encourages predictions


to be one hot, and it encourages consistency because the distributions need to be the same,


they should be in the same cluster.


Now I agree with the consistency.


If you make the inner product high then of two of these histograms, of course they'll


look the same because these are ultimately vectors.


These are three dimensional vectors.


Let's call them two dimensional vectors.


So here is class one, here is class two.


If you make the inner product small or high, they will agree on their predictions.


But I disagree that this encourages anything to be one hot.


In my mind, if you have two vectors that are both zero one times zero one, the inner product


is going to be one.


And if you have two assignments that are 0.5 and 0.5, then it is also going to result in


an inner product of, is it 0.5?


It's also going to be, no?


So what's the inner product here?


The inner product is 0.5 times 0.5 plus 0.5 times 0.5, which is 0.5.


Am I dumb?


An embarrassingly long time later.


Oh, it's because the L1 norm.


Okay, okay, we got it.


We got it.


I am, I am, okay, I am too dumb.


Yes, of course, I was thinking of these vectors being normalized in L2 space where their inner


products would always be one.


But of course, if you have assignments between classes, and it's a probability distribution,


a histogram, then all of the possible assignments lie on this thing right here.


Now, the inner product with yourself, of course, is the length of the vector and the length


of a vector that points to one class, or the other class is longer than a vector that points


in between.


So okay, I see that's where they get this, that's where they get this must be one hot


from.


So okay, I'll give that to them.


It is actually encouraging one hot predictions, as long as these things are normalized in


L1 space, which they probably are because they're histograms, right?


Yes, that was, that was dumbness of me.


I was trying to make a counterexample.


I'm like, wait a minute, this counterexample is a counterexample to my counterexample.


Okay, so yeah, that's that.


So as you can see, they are of course, correct here.


And they now make the first experiments.


So they say basically, after the first step of the self supervised training, they can


already retrieve sort of nearest neighbors.


And the nearest neighbors, the nearest neighbors of these images right here are the ones that


you see on the right.


And after the self supervised one, these nearest neighbors are already pretty good at sharing


the high level features actually crazy, crazy good, right?


This flute here is in different sizes, as you can see, the fishes aren't, aren't all


exactly the same.


The birds, so you can see it really focuses on sort of higher level features, but I guess


it's really dependent on this higher level task.


And they were they also investigate this quantitatively, but I just want to focus on how good is this


after only the self supervised thing.


And now they do this clustering.


And they can already sort of could already evaluated right here, because now they have


a clustering, right?


After this step, they've basically pulled together the neighbors, and they have this


neural network that is not assigning classes.


So they could already evaluate this and they are going to do that, but that's not good


enough yet.


Then they do a third step, which is fine tuning through self labeling.


Now self labeling is pretty much exactly what it's what it says.


It's you label your own data with your own classifier.


Now that might be a bit outrageous.


And you basically saying, wait a minute, if I label my own data and learn a classifier


on these labels, isn't, isn't it just going to come out the same?


And the answer is no, right?


If you have a data set, because your classifier doesn't give you just first of all, if your


classifier is something like this, right, just happens to be and you label and you learn


a new classifier, it is going to be more like this, right?


Because it sort of maximizes a lot of classifiers maximize these distances between the classes.


So even if it's like that, and then the second step they do is they say, okay, there are


some points where we are actually more confident about such as this one, we're more confident


about that one, also this one, and then this one here is pretty close, like we're not super


neither this one, but we're very confident about these two.


So we're only going to use the ones where we are, in fact, confident about to learn


to learn the new classifier, or basically we, you can also weigh them, and so on.


But they go by confidence right here, as you can see, in this final algorithm.


So this is the entire algorithm and I got kicked away.


Our algorithm, there we go.


All right.


So semantic clustering by adopting nearest neighbors, their scan algorithm.


So in the first step, you do this pretext task, this is the self supervision, the representation


learning, okay?


Your entire data set, no, sorry, this is this year, optimize the neural network with task


T. That's just self supervised representation learning.


Okay, then the second thing, we're going to determine the nearest neighbor set for each


X.


Now, they also in that step, they also augment the data, they do heavy data augmentation


and so on.


Also in this, in the third step in the self labeling, they do data augmentation.


There's a lot of tricks in here.


But ultimately, the base algorithm goes like this.


So you find your neighboring sets for each X.


And then what you do while your clustering loss decreases, you update this clustering


neural network by with this loss that we saw.


So this is the loss where you make the nearest neighbors closer to each other while still


keeping the entropy high.


Okay.


And then in the last after you've done this, you go through and you say while the length


of Y increases, what's Y?


Y is all the data points that are above a certain threshold.


Now, you're going to filter the data set that is above a certain threshold.


And that's your data set Y and you train this same neural network, you basically fine tune


it with the cross entropy loss on your own labels.


So now you only have labels Y.


Okay, so it's not labels, you have the cross entropy loss between the assignments of this


and the assignments of your data set.


Okay, so you basically do the same task, but you filter by confidence.


And they use a threshold, I think, of point seven or something like this.


Now let's go into the experiments.


The experiments are look as follows.


So they do some ablations to find out where in their methods kind of the the gains come


from and we'll just quickly go through them.


If they just do these self supervision at the beginning and then just do k-means clustering


on top of that, that will give them on CIFAR-10 a 35.9% accuracy.


So not very good.


So the clustering, you can't just cluster on top of these representations and then be


done.


If they do what they say, so this is sample and batch entropy loss, this basically means


you do not care about the nearest neighbors, you do this entire thing, but you only make


an image close to the prediction close to itself and its augmentations.


So you don't use any nearest neighbor information also doesn't work.


Like I wouldn't pay too much attention that the numbers are 10, 20 or 30.


It just is like doesn't work.


Now if you use the scan loss, you all of a sudden you get into a regime where there is


actual signal.


So this is now significantly above the, this is significantly above random guessing.


And if you use strong data augmentation, as I said, a lot of this has these tricks in


it of what kind of data augmentation you do and so on.


So never forget that these papers besides their idea, they put in all the tricks they


can.


So you get 10% more and then if you do this self labeling step, you get another 10% more.


And this is fairly respectable like 83.5 without ever seeing labels.


It's fairly good.


But of course there are only 10 classes right here.


So keep that in mind, but they will do it on ImageNet later.


And they investigate what kind of self supervision tasks at the beginning are important.


And they investigate things like RotNet feature decoupling and noise contrast of estimation,


which noise contrast of estimation is the best.


And noise contrast of estimation I think is just where you, as we said, you input an image


and then it's kind of noisy versions with augmented in various ways.


And then you classify them together.


This has been like this, these methods have been very successful in the last few years.


Yeah, so this, they have various investigations into their algorithm.


I want to point out this here.


This is the accuracy versus confidence after the complete clustering step.


So this is now the third step, the self labeling.


And you can see right here, as this confidence of the network goes up, the actual accuracy


goes up as well.


So that means the network after the clustering is really more confident about the points


that it can classify more accurately.


There's like a correlation between where the network is confident and the actual label


of the point, which is remarkable because it has never seen the label.


But also see how sort of the range here is quite small.


So with the standard augmentation that goes like from here to here.


So where you set that threshold is fairly important and might be quite brittle here


because you need to set the threshold, right, such that some points are below it and some


are above it and you don't want to pull in points where you're not, because if you pull


in points from here, you only have the correct label for 75% or something like them, of them.


That means if you now self label and learn on them, you're going to learn the wrong signal.


So this step seems fairly brittle, honestly, but I don't know, of course.


They go on and investigate various things such as how many clusters do you need or how


many nearest neighbors, sorry, do you need this number K here.


And you can see that if you have zero neighbors, then you're doing a lot worse than if you


have, let's say, five nearest neighbors.


So the jump here, as you can see, is fairly high in all the data sets.


But after that, it sort of doesn't really matter much.


So it seems like five nearest neighbors should be enough for most things.


And here they just show that when they remove the false positives, that their algorithm


actually converges to the correct clustering, the correct accuracy, which is not surprising.


Like if you remove the wrong samples that are wrong, then the rest of the samples are


going to be right.


I think that's just showing that it doesn't go into some kind of crazy downward spiral


loop or something like this.


But still, it's just kind of funny.


Okay, so they do investigate how much they improve and they improve by quite a lot above


the kind of previous methods.


So they have a lot of previous methods.


But this includes things like k-means and so on, GANs, deep cluster that we spoke about.


And this method, it already gets, as you can see, fairly close to good accuracy.


So you have like 88.6% accuracy.


And that's, you know, fairly remarkable on C410 without seeing the labels.


But we'll go on and now they go into ImageNet.


Now ImageNet, of course, has way more classes.


It has 1000 classes compared to C410's 10 classes.


So if you think, you know, clustering 10 classes might end there fairly apart from each other,


might work with various techniques ImageNet, 1000 classes, that's way more difficult.


Now they do subsample this to 50, 100 and 200 classes and they get okay accuracy.


As you can see, they get 81% in 450 classes where a supervised baseline would get 86%.


In the 200 classes, they get 69% where a supervised baseline would get 76%.


So it's there.


And that's quite remarkable for these low number of classes.


And they figure out that if they look for the samples that are kind of in the most of


the middle of their cluster, they get these prototypes right here.


And you can see all of these images, if you know ImageNet, some of the images, they really


only have a part of the object and so on.


So here with the prototypical things, you really get center clear shot of the object


with clearly visible features and so on.


So this sort of repeats the fact that this clustering really does go on that sort of


semantic information.


Of course, the labels here are from the test label set, the network can't figure that out.


And then they go for 1000 classes and in 1000 classes, it doesn't really work because there


might be just too many confusions right here.


But they do have this confusion matrix of their method.


And it shows that the confusion matrix is pretty much a long block diagonal along these


super clusters right here.


So you can see the dogs, the network confuses the dogs fairly often and the insects with


each other, but not really across here, which is still quite remarkable.


But I mean, you get the same thing for a lot of these methods.


So I don't know how much different this would be in other methods.


But certainly it's interesting to look at.


Now they go into one last thing.


And that is what if we don't know how many clusters there are, right?


If we don't know anything.


So say so far, we have assumed to have knowledge about the number of ground truth classes.


The model predictions were evaluated using the Hungarian matching algorithm.


We already saw this in the DETR by Facebook, if you remember.


However, what happens if the number of clusters does not match the number of ground truth


classes anymore?


So they now say table three reports the results when we overestimate the number of ground


truth classes by a factor of two.


Okay, so now they they build just 20 classes for CIFAR-10 instead of 10 classes.


And we'll look at table three real quick.


Where's table three?


This is table three.


Okay.


So when they over cluster, you get the thing here on the bottom.


And you can see there is a drop in accuracy right here.


Now what I don't actually, they don't actually say how they do the over cluster matching.


So if you imagine if I now have, I don't know, six clusters, but I need to assign them to


three clusters, you know, here.


Do I still use this most optimistic thing?


So do I still use, I think they still use this most optimistic matching right where


you assign everything to its best fitted cluster, right?


You compute all the permutations and then you give it the best benefit of the doubt.


Now if you imagine the situation where I over cluster to the point that I have each image


in its own cluster, and I run this algorithm to evaluate my clustering, I give it basically


the most beneficial view, then I would get 100% accuracy.


Okay, so like in one of these over cluster approach, I would sort of expect that you


actually get a better score because you can, like there is more generosity of the matching


algorithm involved.


Now that's counteracted by the fact that you can't group together things that obviously


have similar features because they are in the same class.


So there's kind of two forces pulling here, but I was kind of astounded that it's going


down and the evaluation method of this matching algorithm, it sort of breaks down when you


have more classes, at least in my opinion.


Yeah, but it's interesting to see that you can just overshoot, but then you need some


sort of heuristic to reconcile that.


In any case, I think this paper is pretty cool.


It brings together a lot of things that were already present and introduces this kind of


this step approach, but what you have to keep in mind, and by the way, there's lots


of samples down here.


What you have to keep in mind is there are a lot of hyperparameters in here.


There are like this threshold and you know, first of all, yeah, the number of classes,


the thresholds, the architectures and so on, and all of this has been tuned to get these


numbers really high, right?


All of these steps, all of the augmentations and so on, the chosen data augmentations,


it has been chosen to get this number as high as possible.


So you know, to interpret this as, oh, look, we can classify without knowing the labels


is, you know, yes, in this case, but the hyperparameter choices of the algorithm are all informed by


the labels.


So it is still very, very unclear of how this method will actually work when you really


don't have the labels, when you actually have to choose the hyperparameters in absence of


anything.


And yeah, I think the future might tell if they continue to work on this.


All right, thanks for listening, looking, watching, and bearing with me through my wrestling


with with various math, basic math in this video.


IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures

Hi there, today we're looking at Impala, scalable distributed deep RL with


importance weighted actor learner architectures by Lasse Espeholt, Hubert


Sawyer, Remy Munoz et al. So this paper deals with a new architecture for deep


reinforcement learning, specifically distributed deep reinforcement learning.


So that means settings where you go beyond one single machine or beyond one


single accelerator like a GPU. So I want to introduce this by showing you this


task here. This is called the DeepMind Lab and the DeepMind Lab is a kind of a


3D environment as you can see here. These are screenshots where there are very


different goals but some of this as you can see are kind of labyrinth style


things where you have to collect apples, some are platformers where you I guess


have to jump around and so on find objects. So DeepMind introduced this as


kind of a as a reinforcement learning environment and what you can do the


agent as you can see here has a camera it perceives pixels and it can get


rewards for performing actions. The actions it can perform is it can you


know walk back and forth it can jump it can crouch it can rotate. So this is


kind of a limited set of actions that it can do but it can move around in this


3D world and it needs to achieve some goals. Alright so that usually this


is kind of a good setting for reinforcement learning and this paper


doesn't do a whole lot of new things in terms of reinforcement


learning but it does a lot of things to kind of make it work in a distributed


setting. So usually what you would like to do is something like A2C. A2C is


Advantage Actor-Critic Learning and it's it's a very successful algorithm in


reinforcement learning. We won't go into this much here but basic elements of it


is you what you have are two things you have a policy and usually this is called


PI. Sorry about that. Usually this is called PI a policy that you input your


current state so your current observation at time T and you want to


score an action right action A. Now you might have maybe as we saw before you


can walk left walk right and so on so you might have 10 actions or so. So in


here you would put action 1 or action 2 or action 3 and for this you would get


probability distributions over each action so maybe in this particular state


so each time with the same state. So you would get a distribution something like


this right so here you should probably go with action 3. That's your policy


function. Policy function PI tells you in this particular state which action


should you take how often kind of gives you a distribution. The second thing you


want is a what's called a value function. So the value function V capital V


usually you input your state and it will output it will output what the value is


of that state and that's usually termed kind of as as lowercase V. The value of


the state is given if you're in a maze right I'm gonna draw a maze from the top


here right doesn't you can't reach there like here so here is the goal and let's


say you are oops you're right here the green right and you have a choice of


going forward to the right or to the left. Now the this would be this would be


your policy here your policy you would ask your policy and a one would maybe be


go forward a to go to the left a three to the right so your policy would decide


what to do your value function however would decide in each of the state so


where you are plus where you could go here here here so basically for each


state in the system it would give you a value in particular in this case it


would probably give you a very very high value here like yeah this is a good


point because you're very close to the goal right here this is probably not so


good a point and this is a very bad point because you you're going to corner


you're actually moving farther away from the goal so if your value function is


trained well then you can you can use that also to assess your situation so


the value function for each state s it will give you a numerical value of how


good that state is in terms of reaching your goal and the A to C algorithm now


deals with the interplay of these the A to C uses actually both of these in an


interplay so it will use one to teach the other one right and this interplay


between those gives makes for a very successful reinforcement learning


algorithm now the way A to C does it is as you can see here what it does is it


has to there are two variants here stink sink step and sink trajectories but in


essence it has to run these episodes and these here are steps in the episodes and


let's say an episode has four steps before it can do the learning part and


the learning part is here the orange thing once it has done a step of


learning it has to run episodes again and then it has can do learning again


and that's because of a limitation of this which is called on policy learning


so on in on policy learning you always want to have your update step which is


the orange part to be fed with data so the this all of these app of these steps


here go into this update steps and it's necessary that the steps that you make


the updates from are computed with kind of the most current version of the of


the agent right so that the agent will go into the world make some steps using


its neural network maybe I should explain so that the agent right is this


box and the agent has this policy right and with this policy as we saw it will


go and it will interact with the world right outside of itself and it will kind