actual task right here. And as the task gets more complex, PonderNet needs more steps to think about


PonderNet: Learning to Ponder (Machine Learning Research Paper Explained)

the task. And this might be exactly, you know, what we saw that you have some sort of a string


of zeros and ones. And you learn during training, you learn a how to take one of these maybe in


multiple steps and get an output. But now you have a longer string, right? Well, so now what you can


do is you can also learn an output for this one. And now you have two outputs, right. And now you


can learn a series of steps to transform the two outputs here into a single output. And that might


just need one or two more computation steps, which is exactly what we see right here happening. So


it's a good, it's a good indication that something like this is happening, I would be wondering,


pondering, one might say, haha, if you know how this actually happens, like, like, what do the


individual computation steps represent? Is it in fact, a, for example, in this parity task,


is the network going about this task in a hierarchical fashion? You know, like, like I've


shown here, is it something different? Is it going about it in sort of a purely recurrent fashion,


where even though we, as I understand it, we input the entire string at the beginning,


does it only look at the string position by position? Or, you know, how does this work?


How does the scaling behave in general, if you know, they only show small strings, large strings,


but how does it behave in general, as you go up the length, and so on, it would be really


interesting to introspect this model a little bit more than simply showing kind of end results here


of the individual tasks. Okay, what they also find is that the hyper parameter, how you regularize


the shape, we've seen this up here, how you regularize this shape is, you know, that is a


hyper parameter, but it doesn't seem to be terribly important. Again, they compare to a CT, which has


another hyper parameter that does the similar thing that regularizes the shape of the desired


halting distribution, which they call tau. Now tau doesn't mean a particular thing in, so they say


it does not have any straightforward interpretation, though, I guess the authors of ACT might disagree.


But as you can see, here, so if I draw the means, there is a region where the tau, where a selection


of tau performs high, though you have to say, see that is all around sort of the same value of like


five e minus four or something like this. And then for the other values that you might set it for,


it simply doesn't work at all. So you, the authors claim you have to hit this tau,


pretty correctly in order to even get the network to do anything. Whereas they claim in PonderNet,


this variable right here, first of all, it's between zero and one and not just an arbitrary


value, right, because it's a probability. And they claim that, you know, it kind of works for,


for most things, except this one right here, where essentially, you bias the network to just output


everything after one step. So the trick is for the geometric distribution, you have to take the


inverse of one over this lambda p, and that will give you the expected number of steps that the


network would compute, according to this prior. So when you put in point nine, that would essentially


be a single step that you ask that work to do. But for all the other things, well, you judge for


yourself, whether whether this here is really good. But what you can say is that look, it goes


from zero to one, so you have a clear range. And for most of that range, the thing seems to work,


okay ish. And what they highlight is even down here. So even if they do this, even if they set


lambda p to one or sorry, to point one, which would essentially bias the network towards 10 steps,


that the prior is please do 10 steps of computation. In this parity task, as I understand it,


even for that point one, you can see the network, it doesn't do 10 steps, it actually also goes


towards three, four or five steps most of the time. So the network learns to be somewhat robust


to this prior distribution. I mean, I guess that's also a function largely of the hyper parameter


here, where you trade it off, we don't know the effect of that just from the paper. But even you


know, even if they set that to really low, it's it, of course, then the network is kind of robust


to the choice of the lambda p. Yet, it's still good news, because that means would mean you


wouldn't have to regularize the model super heavily in order to get the result that you want.


The the model super heavily in order to get it to work. Okay, they go into two other tasks right


here. Again, these aren't tasks that you might necessarily know they are tasks where this type


of computation shines particularly. And yeah, as I said, I see the paper more as sort of an


interesting, an interesting task, an interesting niche task, sub task, you might say of, of


connecting deep learning and classic algorithms. There are a number of things that I think you can


do right here to extend this. So it's completely thinkable that, you know, the loss might be a bit


different that you don't ask the network to output the direct answer at each point. But you know,


you might, you might want to attach memories and so on at at these output nodes. You might want it


want them to output intermediate results or something like this. Another thing you could do


is you could work with sort of adversarial losses instead of, of, you know, kind of reconstruction


losses or whatnot. So you could you could have some sort of a GAN going on inside of this in


order to decide on the on the stopping probability that there's lots of stuff one can fiddle around


with this type of network. You can even think of crazier architectures, I don't know, Hopfield like


structures where you decide, you know, how far you iterate, because you don't, you may not always


want to iterate until fixed points. I don't know, I'm just I'm just talking crap right now. Okay,


one last shout out to the broader impact statement of this paper, what a beautiful, beautiful piece


of, of writing. So essentially, they say, well, this enables neural networks to adapt their


computational complexity to the tasks they are trying to solve. You know, neural networks are


good, but currently, they require much time expensive hardware, they often fail, PonderNet


expands the capabilities, they say, look, it, you know, it can do this, it can do that, makes it


particularly well suited for platforms with limited resources, such as mobile phones, which


is a good thing, right? It can also generalize better. That means it's better for real world


problems. And they say, we encourage other researchers to pursue the questions we have


considered on this work. We believe that biasing neural network architectures to behave more like


algorithms, and less like flat mappings will help developing deep learning methods to their full


potential. And that is indeed the broader impact of this work. Like that is, that's the impact it


had on me. And that's the impact that it should have. Yeah, I'm not like, at today's conferences


that must might be kicked out because of course, it doesn't say technology, good technology, bad


technology bias. But you know, respect for that. And that was it for me. Let me know what you think.


DALL-E 2 by OpenAI is out! Live Reaction

on you. Nice. Cool. Yeah, we'll look at Dali 2. It just came out 10 minutes ago. This is


by OpenAI. If you want to investigate yourself, this is just the OpenAI blog. I have not seen


this yet. So legit, this is the first time I'm seeing it as well. If you see something


cool, let me know in the chat. Also, I have no clue. New camera, very wide angle, makes


my arms really long. See, really long arms. Excellent. All right. I mean, we'll just dive


in. Right. So we'll read this together. As I said, I have like, this is the first time


I'm seeing this as well. So yeah. New AI system that can create realistic images and art from


a description in natural language. That sounds interesting. So and this is typical OpenAI


blog, right? Where they just have their examples. Like there's no way you're going to interact


with this, I guess. There's also a paper. So they don't even put this on archive, right?


Like their papers, they're just on their own website. They're so famous. Hierarchical text,


conditional image generation with clip latents. So I think, let's search diffusion models.


So yeah. So this is now no longer based, I think this is now based on diffusion models,


which was to be expected. Sorry, Esther, this is not live coding. We'll just look at this


new research that just came out. And we'll react to it in the moment. So an astronaut


riding a horse in a photorealistic style. Okay, so this can create original realistic


images and art from a text description can combine content, concepts, attributes, styles.


This is what we're used to, I guess, from the image generation models, but this obviously,


so Dali was always it's different than the clip notebooks, and so on the AI art that


you're used to, because those are all mainly clip based. And then you put some kind of


a model in front of clip, and you optimize the whole thing. But Dali is a forward, so


you input the text, and then there's a forward propagation. And then the result is an image


and that's may or may not be filtered. So teddy bears shopping for groceries as a one


line. This is insane. So this is what this model can do. And there there is a bunch of


different images right here. Now, these are, well, if they're not cherry picked, at least


they're curated, right? So they they want to see that nothing, nothing bad happens when


you look at these things. A bowl of soup that looks like a monster. What is this? Please,


please come on. Okay, this is insane. Really insane. Knitted out of wool, made out of spray


painted on a wall. Wow. I mean, yeah, it depends how cherry picked these things are. But this


is really impressive. Now, I don't know if you've seen but there is a kind of a new lion


plus clip notebook or something like on clip. If fusion, let's just search for that. Because


this is also is very like there's a new notebook. And I've seen this around Twitter. Yeah, exactly.


There's this new notebook. This is not by open AI. This is an clip that has been trained


on this open source data set. And when people combine this with the kind of old methods


of clip notebooks, then this is what happens. So this is like weird. And it's funny because


it can spell but it kind of it is like it is not. It doesn't spell really well. But


if you so there is this this research here and these are already quite insane in quality,


right? Latenton diffusion. I love generative models. That's actually good. A painting of


a squirrel eating a burger, but now we're at this Dalí. So these seem to be very, very


much better. Now this model is based on diffusion, which means that it has some interesting capabilities.


If you know diffusion models, technically what they do is they are creating their own


data set in the sense that a diffusion model essentially iteratively refines a piece of


input in multiple steps until it reaches the output. So it's not necessarily one forward


propagation, although it depends a little bit on how you model it. You can have one


layer for each of the denoising steps or you can have the same neural network and you pass


the input over and over again, maybe with some indication of what time step it is. So


what a diffusion model does is it takes an image and it just adds noise to it step by


step. So it adds a bit of noise and it takes that, it adds a bit of noise, it takes that


until it is completely garbage, like until it is complete noisy. And at that point you


can reasonably assume by some limit theorem that this is now perfectly modeled by Gaussian


noise or whatever noise you use. And because of that, because you have a process that results


in a defined distribution, you can sort of make reverse inferences mathematically, which


leads to some training properties that you can then use to train these diffusion models.


In practice, what that means is that you take an image, you make it continuously more noisy.


That's your training set. And what you train is the reverse. So you train always to go


from one step in this denoising process to go to that little bit less noisy version of


that image. So that's what you train. So this, this direction is easy, right, to make it


more noisy, but then you train a neural network to make it always one step less noisy. And


that essentially means that you can now sample the original thing, which you know is a Gaussian,


so you can sample from that and then you can send it through all of these stages of denoising


and at the end you'll end up with a really nice picture. And you can do that conditionally


on text, which is what they do here. Now, given that this is what they do, they have


some interesting capabilities. Notably, they can edit images so they can start with an


image and then edit that image and they can even edit only parts of that because they


can mask some of the parts and then simply apply the denoising diffusion process to that.


So that's, that's what they do right here. Now I am going to, if you don't mind, just


quickly put on Twitter that I'm live streaming right here and maybe also on Discord. I hope


this opens over here. Yes. One second. Nope. Yep. All right.


Just, just writing the announcement, adding the little picture, picture. And off we go.


All right. Yeah. So they can edit. So now let's see, make realistic edits to existing


images from a natural language caption. And this I think is what really will make a difference


in sort of AI art. But they say we use diffusion models for the decoder and experiment with


both auto aggressive and diffusion models for the prior. Is my mic muted? Is my mic


muted? No, I think I just made the Twitter announcement and then I was quiet for 10 seconds.


Sorry. All right. Let's see. So you can make edits. So what they do is here's the original


image and they, they select a location where they want to edit. As I said, you can edit


images and you can do so in a constrained way. And one is, so you can select the location


and you can prompt it to add a flamingo, right? That's the natural language prompt is to add


a flamingo and you select the location where you want to add a flamingo. Look at these.


This is insane. Look at these results. Add a flamingo to this. Okay. It's interesting


that the flamingo style, right, matches the rest of the picture. It's not just a flamingo


when it's inside. It's this sort of, what is this even? Like, is this a piece of furniture


or so? Like this isn't or a sort of a swimming device flamingo yet when it's, when you select


the, when you select it on the water as well, but when you select it outside, it's actually


real flamingo. So this is, this is quite, quite insane. It doesn't just add a flamingo.


It actually takes the entire picture into account. Let's say add a corgi again. So this


is the original image on the left and then you select the location and the prompt is


to add a corgi. Now notice that the style of the corgi changes depending on where it


is, right? The left is, is this the, is this like the self portrait of, is this the self


portrait of Fungo? I'm not sure. But, and if you add the corgi here, it's actually a


real corgi, right? So it's super powerful because it, it looks beyond like it, it, uh,


swimming device, I'm sorry, I don't know what it's called. So yeah, you can see that this


is, this is all a real corgi. Whereas where if you go into the, if you point into the


image, it actually matches the style of the, of the image drawn. So this is, this is super,


like this demonstrates quite a bit of understanding, I believe, at least like integration, integration


of the entire thing. This is going to be a paid service, isn't it? I, look, I don't know.


The interesting thing is they didn't even make Dolly into a paid service until now.


And because they withheld Dolly, we got all of this, you know, VQGAN plus clip and diffusion


models plus clip things, which is like, I don't think that would have happened if they


were just to release Dolly even as a paid service. Let's see what else they have here.


So we add a sofa. Yeah. So I think this is going to be quite huge for, for making any,


any sort of visuals. Like if, you know, I need to make thumbnails for these videos and


all. So if you do anything like this, this is essentially, it is a, let's say a weak,


weak form of Photoshop, but that's automatic. So if you, you know, you could, you could


give this to someone on Fiverr and that's about what it would turn out, right? You could


give the picture on the left and be like, I'd like to add a sofa in the back corner


there, please do it. And that'll cost you, I don't know, 20 bucks, 50 bucks or so on


Fiverr. And this just works. So yeah, I'm excited to see what they do with it because


they do, they do have a, like investors and so on. Can take an image and create different


variations of it inspired by the original. Yeah. So this is, this is essentially different


variations. I don't know what they prompted with here. They don't say as far as I can


tell, but maybe they just, they just, maybe what they do is they, they sort of back propagate


it or they noise it a bit and then they denoise again from the intermediate stage. So with


the these diffusion models, right? You can make an image more noisy, maybe change it


a little bit in some intermediate layer and then use the diffusion models again to upscale


it and maybe that's how they do these types of things. I'm not sure they don't, they don't


say right here. Maybe they say in the paper. This is though fairly interesting. You can


see that there's two, two people on the left. Is this Klimt? I'm not sure. I'm not an artist


person thing. Image to text embedding to image. That could be, that could be. But you can


see that it is not, it is like the, the variations are really keeping sort of the semantic content


here and I believe that's also one of the powerful things of the, oh, that's one of


the powerful things of training image and text together because text by nature of being


made for humans to communicate with each other, text is much more on the level of sort of


the, what should I say this, type of information and the gradation of information that is important


to humans. So training image and text together, I believe just by the nature of having the


text in there will give representations that are much closer to what humans would consider


different or relevant or something like this. This is, I mean this, come on, this is insane.


Look at that. Look at that. Wow. I think it's taking the generated embedding and then regenerating


the image from that embed. Yeah, yes, exactly. So they, they, oh, you believe, okay, maybe


they run it through the encoder. They don't actually noise it. That's very possible. Yeah.


Would this be possible with audio? I guess. Yeah. You'd have to find a relevant noising


process. I'm not sure if Gaussian noise or so is really appropriate for audio, but since


this, since audio is a continuous signal, I don't see why this wouldn't be possible


with audio, especially if you train it with associated with text. Right. I mean, yeah,


this is crazy. It would be really interesting to see if sort of the representations of this


stuff aligns with the representations of like real pictures of birds or if they are way,


way different because you can see here, these style variations, they all, they all depict


the same subject and they are kind of in the same style. Right. It'd be interesting to


see how much it can actually differentiate, differentiate that from. It's also interesting


that here they have these, these colored boxes in there. So do you think these might be the,


is this, is this an artifact of what is this like an artifact of the generation process?


Like is this the, they sample autoregressively? Are they always the same? Yellow, light blue,


green, red, blue, yellow, light green, green, red, blue. Can't tell what they are. Look


at that. I mean, yeah, this is like just a level of the level of abstraction and, and


I'm not sure what people like, like Gary Marcus make of this. This would be super interesting


to hear because I mean, look at that. So there is, it is clearly not a natural picture, right?


It's not yet. It depicts a girl with headphones on the laptop, like in apparently in some


sort of garden or surrounded with plants overlooking a city in the night sky with the moon and


there's a cat in it. And all of that is in all of these pictures. However, it's just


different, right? It's a bit, look at like, you can't tell me that this thing doesn't


do some degree of abstraction or it just brute forces all of this by training data, but I


can hardly imagine. So, you know, one second. I've learned the relationship between images


and the text used to describe them. It's used a process called diffusion, which starts with


a pattern of random dots gradually alters the pattern towards an image when it recognizes


specific aspects. So yeah, as I, for people who tuned in late diffusion models, they create


training data by making pictures more and more noisy. So they learn to reverse, to reverse


that noise. Let's see. Well, can you hear? You probably can't hear that sound, right?


So yeah, no. No. Okay. Let me quickly try. How about this?


Now I can't hear it. All right. One second.


Yep. This one.


Here's a video idea. I built a real life avocado armchair.


Let's do that. That's a future video. Promise. I should promise things like this. Okay. Promise.


Like in painting, it can even start with an image as an input and create variations with


different angles and styles. Dolly was created by training a neural network on images and


their text descriptions. Through deep learning, it not only understands individual objects


like koala bears and motorcycles, but learns from relationships between objects. And when


you ask Dolly for an image of a koala bear riding a motorcycle and knows how to create


that or anything else with a relationship to another object or action, the Dolly research


has three main outcomes. First, it can help people express themselves visually in ways


they may not have been able to before. Second, an AI generated image can tell us a lot about


whether the system understands us or is just repeating what it's been taught. Third, Dolly


helps humans understand how AI systems see and understand our world. This is a critical


part of developing AI that's useful and safe. The technology is constantly evolving. If


it's taught with images that are incorrectly labeled, like a plane labeled car and a user


tries to generate a car, Dolly may create a plane. It's like talking to a person who


learned the wrong word for something. Dolly can also be limited by gaps in its training.


If you type baboon and Dolly has learned what a baboon is through images and accurate labels,


it will generate a lot of great baboons. But if you type howler monkey and it hasn't learned


what a howler monkey is, Dolly will give you its best idea of what it thinks it could be,


like a howling monkey. What's exciting about their approach used to train the dog is that


it can take what it learned from a variety of other labeled images and then apply it


to a new image. Given a picture of a monkey, Dolly can infer what it would look like doing


something it's never done before, like paying its taxes while wearing a funny hat. Dolly


is an example of how imaginative humans and clever systems can work together to make new


things, amplifying our creative potential.


There's a lot of hedging here, OpenAI. A lot of hedging against people criticizing


you. That's good, good strategy. You just hedge ahead of time. You're just like, hey,


you can't do this. You can't do that. Howler monkeys are a species of monkey that indeed


howls a lot. Well, then it's correct, right?


Dolly 2 generates more realistic and accurate images with 4x greater resolution. Okay. A


painting in the style of Claude Monet of a fox sitting in a field at sunrise. Well, that


is indeed a great, great picture. Yeah. Do you think these things down here are the sort


of initial tokens for some autoregressive prior? I'm not sure. Okay. Which we currently


do not make available in our API as part of our effort to develop and deploy AI responsibly


for studying Dolly's limitations and capabilities with a select group of users. Let me be the


select group of users. Oh no, I'd be terrible. I would be immediately canceled. They're making


it harder to share without knowing where it's from. Well, you can cut just cut off the bottom.


Like that's safety mitigations we have developed include preventing harmful generations. We've


limited the ability to generate violent hate or adult images by removing the most explicit


content from the training data. Okay. You also use advanced techniques to prevent individuals


facing those of public figures. Yeah. So you can do some things, right? You can do this


at training data time, which is where you just say, I'll never expose my model to bad


things and then it doesn't learn bad things, which I don't think it can work, but not particularly


well because you're always going to miss some of that stuff like a lot. Like you're going


to miss most of that stuff, honestly. And therefore, so either you limit your training


data to such a degree or you're going to miss most of the stuff because the number of ways


that humans can come up with to do shady things and harmful things, whatever that means is


it's absolutely insane. So I trust humans to always generate things that you will miss


and you can do it at training time. So you can, if you have some sort of a classifier


that, that detects a harmful content or whatnot, you can discourage the generation like at


training time of, of these things. And then at inference time, you can obviously also


filter simply if it generates something like this, you can, you can filter that out. All


of these things I find suboptimal, but I do understand that if you want to release a product


like that, you, you don't want to necessarily have it output. It's already a stretch, right?


For the company to, to provide a service where they themselves don't exactly know what it


outputs, which is crazy with these new APIs. So I understand, but again, they told us GPT2


was too dangerous to release. You know, you remember that when they told us GPT2 was too


dangerous? Yeah. So curbing misuse or content policy does not allow users. Okay. We won't


generate images if our filters identify text prompts and image uploads them to violate


our policies. Yeah. Okay. That, that makes sense. You just tell people like don't do


it. Phase development, phase deployment. Okay. Previewing, limited number of trusted users.


Well I hope they release it at some point, not like DALI one. There's a system card.


We don't care about system cards. Okay. That's the blog post. So for people who've joined


later, this is, it's pretty insane what you can do. So this is a text to image system,


an astronaut playing basketball with cats in space in a minimalist style. Insane, insane,


insane. Yeah. And what it can do also is edit, which is going to be huge, right? Huge for,


for anyone. Like if this is in some sort of Photoshop or GIMP or something like this,


new world opens for anyone who has to make stuff like a lot. I don't think people realize,


you know, how much you could, you could do if you, whatever you just, you have a presentation


and instead of having some stupid stock images, you just kind of generate something and then


after generation you edit it, right? The cool thing here is that you don't have to, you


don't have to restart and figure out the prompt exactly. You can just generate something and


then edit at some point. You can say, you know, add this here, remove this, change it


to this and so on. And that's just, and make some variations, right? You can just say,


make some variations and that I believe is going to be quite huge for a lot of people


to just use casually if that's how they intend to release it. And I'm very, it'd be very


cool to see if there is an open source replication of that. Given how far we've come with Clip


and with these new data sets, again, for people who haven't been here, there is this Lion


500M dataset and people have been training a Clip models on that. Oh no, so Lion 400M


dataset, this is the old one. There's a new 5 billion one and if you combine that with


diffusion models, you can get, like you can get pretty insane things as well. And then


if you combine that with up sampling models, then you already have very powerful models.


However, I don't know how fast they are. So the stock images of today might be the basis


of all pictures in the future. Yes, pretty much. I think, you know, we are, we are probably


the generation who still makes real images and uploads them somewhere. And before that,


it was just not possible because the internet didn't really exist in the capacity to support


that. And after that, most images will just be AI generated. So yeah, so that's where


we're a special generation, the last real picture generation. Okay, let's just skim


through the paper a bit and figure out how they, you know, a bit more about what they


do. So here, Super Saiyan, Super Saiyan sentient bag of chips, ArtStation, it's, they're using


the okay, okay. Controversial opinion, you know, given that they are using essentially


discoveries of the open source Twitter community, that was explicitly explicitly a reaction


to them not releasing Dolly one, do you think it's appropriate that they are now using the


tricks that that community has figured out to make cooler images in their again, proprietary


system that they don't release and probably end up selling to you? Like arguments for


and against, right? But point of contention, point of contention. Like, you, in my opinion,


you know, go do all the selling you want. But then, you know, this is kind of the ArtStation


at the end is a bit of a that's kind of a thing of the kind of sharing open source community.


And yeah, not sure. Teddy bear on a skateboard in Times Square. This reminds me of that movie


Ted No. Dolphin in a rest, a dolphin in an astronaut suit on Saturn. This this Yeah,


this is 1024 by 1024. So these actually become usable, let's say in presentations and whatnot.


The 256 by 256 was pushing it a bit in terms of resolution, but this actually becomes variations


by encoding with clip and then decoding with a diffusion model. Okay, okay. So this is


again, this is combined. This is, is this is this essentially clip based diffusion?


Okay, here it is. How of unclip. This is okay. Unclip might be the variation thing, right?


That's my problem. Really? So is this the whole method? Or is this just where you make


variations of a thing? Variations between two images interpolating their clip image


embedding and decoding with a diffusion model. We fix the decoder seed. Okay. So clip seems


to play an actual role right here, right? So they have clip here with the clip objective,


right? There's a text encoder and an image encoder, which they can give in. So this will


give them some sort of an embedding. And then that embedding goes, there's a prior and then


there is this deep diffusion decoder. The dotted line we depict our text to image generation


process. A clip text embedding is first fed into an autoregressive or diffusion prior


to produce an image embedding. Clip text embedding is fed through an autoregressive or diffusion


prior to produce an image embedding. And then this embedding is used to condition a diffusion


decoder, which produces the final image note that the clip model is frozen during training


of the prior and the decoder. So there seems to be three components, right? The first component


is clip itself that's above the dotted line here. Hello, Philip Wong. Good to see you


here. So the and they already either have clip or they train clip a new but they first


train clip, which gives them these embeddings of text and image, right? That correspond


well to each other. And then they freeze that. And then there's two more parts to it, which


are these this prior thing. And the prior thing maps a text encoding to an image encoding.


So the clip objective simply makes them align, makes these encodings align. However, the


prior actually transforms one into the other. It's interesting that they need that. And


that you can't adjust have the one I want to say, have the text encoding be the start


of the diffusion process. But apparently they they put this prior here. And maybe that models


that for a given piece of text, there can actually be a lot of different images that


go along with it. But since the two are probably already quite aligned, it might require some


minor modification suit. And then there's the diffusion decoder, right, which we saw,


you can train all of this probably from the same data sets. So the clip you can train


obviously, and once you have clip, you can train the prior to simply translate the text


embedding that you get from the frozen clip encoder to the image embedding that you can


get from the frozen clip encoder. I'm not sure if that what happens, but yeah, and then


you can train the decoder likewise, simply by encoding with clip and then trying to reverse


that process. Although again, I don't know if that's actually what they do. So let's


read into it. Again, this is the I'm seeing this for the first time as well. So I'm as


clueless as you are. Does anybody have a link to the paper you got to go to openai.com click


on Dully and then click on view research. A training data set consists of pairs of images


and their corresponding captions. Given an image x let zi and zt be its clip image and


text embeddings. We design our generative stack to produce images from captions. So


it's image from caption, not the other way around. Using two components, a prior that


produces clip image embeddings conditioned on caption. So yes, so these are these here


are clip image embeddings. They're not newly trained ones. So they, the prior translates


the text encoding to the image encoding again, which is special because the clip objective


already aligns them during training. So that again, that might require some minor minor


modifications to them. It's interesting that they have to have it and a decoder that takes


in text captions again, conditionally and the clip embeddings and produces image condition


on clip. So this is not the same if I have this correctly as like clip guided diffusion


because in clip guided diffusion, we take a diffusion model and we feed that into clip,


right? Let me, we take a diffusion model and we feed that into clip and then we try, we


try to back propagate the clip loss to the diffusion model and try to guide the diffusion


model in this way. Here it's different. Here we actually use the diffusion model, but we


don't start from noise. We actually start from an embedding. That's very interesting.


The I'm going to guess the, yeah, the prior translates to the clip embedding and then


we start from that. That's very interesting. So the decoder, yada, yada, yada, yada, yada,


yada. Okay. Projecting and adding clip embeddings to the existing time step embedding. So this


is a variation of the glide model. Projecting clip embeddings into four and into four extra


tokens of context that are concatenated to the sequence outputs from the text encoder.


We retained the text conditioning pathway, yada, yada, but find that it offers a little


help enable classifier free guidance, which is also nice by randomly setting clip embeddings


to zero 10% of the time and randomly dropping text caption 50% of the time. So this is a


conglomerate of sort of the recent advances by open AI and by the community. It goes back


from the decoder back into clip. It kind of sounds like they just modified the clip embeddings


and output via the image encoder. Zoom, but, um, yes, exactly. They modify the clip embeddings


via the prior first and then via the diffusion model. Okay. The prior, um, to generate high


resolution images, we train two diffusion up samplers, one to up sample images from


this to that, and another one to further up sample them to improve robustness. We slightly


corrupt the conditioning images during training using Gaussian blur. Okay. Is this, I guess,


so it'd be interesting to know if this is end to end these up sampling methods because


to really get good generations, right? The up samplers would need to also be informed


of the, um, of the prompts and so on. Otherwise the details are kind of generic and the big


structure is, um, which is usually what you want, but it's not exactly what you want.


Prior while the decoder can, while the decoder can invert clip image embeddings to produce


images, we need a prior model that produces ZI from the captions to enable image generations


from text captions. So there is an autoregressive prior clip image embedding is converted into


a sequence of discrete codes and predicted autoregressively or diffusion prior the continuous


vector ZI is directly modeled using a Gaussian diffusion model conditioned on the caption.


Okay. That's cool, I guess. For the diffusion prior we train a decoder only transformer


with a causal attention mask embedding noise. Sorry. Um, my reading style is not conducive


to live streaming. I guess I'll adjust. So here is what, how they train the loss. So


the image embedding is fed into this one along with the timestamp


and the text description. The F I'm going to guess is a noising process or something


like this. And then, um, this is the target. So this is now a, let's say true diffusion


model where we continuously noise what we want to achieve. And then we try to reverse


that. Variations. We have given an image. We can obtain its clip image embedding and


then use our decoder to invert producing a new image. To make the variations more similar,


we first perform this inversion with the decoder condition on ZI. Okay. So it seems like clip


has a big part in this, um, which is a bit surprising, but not too much because the original


Dali just use clip as a, a sort of ranker at the end. Whereas this one uses clip intrinsically


in the process. So it uses, if it gets a piece of text, it puts it through the clip text


encoder, which is frozen, right? And then it has the prior and the, um, the fusional


diffusion, exploring the clip latent space using the on clip decoder. Okay. This is again,


the variations, variations of images featuring typographic attack, predicted probabilities


across three labels. Surprisingly, the decoder still recovers Granny Smith apples, even when


the predicted probability of this label is near 0%. So what they do, they project these


images into the clip latent space and then decode them again. Can it generate text? It


cannot. I mean, it can probably generate text as part of an image, as you can see right


here. Um, so this is an apple. And even though the classifier is what kind of classifier


is this? Okay. This is clip. So here you feed this image into clip. This was, I think a


famous experiment. You feed this image into clip and you ask clip, which of the text classes


is the most likely? And clip says the iPod class is very likely. And when you actually


project that and then up sample it again, you can see that it's interesting because


clip sees mostly an iPod, right? That's the class is 99.98% versus Granny Smith's 0.02%.


Now this is normalized, which is why I think that this still works. So they say the surprising


part right here is that it in fact retains kind of the Apple. It retains the Granny Smith


Apple, even though technically clip mostly sees the iPod. But I think that's, that's


kind of because it probably still sees the Apple, but when you let it classify, it's


just that much closer. Oh, sorry. To the iPod text label rather than to the Apple. But it


probably the visual information in here, like the encoding of that is still very much contains


the Apple motive. I guess this is what they want to demonstrate. So funny how it can,


this is the same thing, right? It's the same thing as this thing right here in that it's


kind of bad at spelling. Like it's so interesting that it can in fact write text, even though


it's an image producing model. It can write text in pixels, but it's really bad at spelling.


Like, yeah, why? Nobody knows. Okay, let's spend a whole bunch of time on this prior


right here. Clip image embeddings reconstruct them with progressively more PCA dimensions.


Okay, so we take the image on the right, I'm going to guess, and we project it into the


clip latent space. And then we only retain the top principal axis or the top principal


components and then we recreate them again. So there is, you can investigate a little


bit what is kind of important. So it seems, let me try to make this a bit bigger. It seems


like the, you can see the detail gets added as you go up. So this here is kind of a scene


from food. Here you have a city that's even might even be, I don't know what city that


is, sorry. And the bottom one, a bunch of cows, this could be Switzerland. We know.


So yeah, as you would expect, it first tries to model kind of the general framework, general


setting, it seems to match the colors even a little bit. It's like the colors, general


setting, what is it? So that's really interesting, right? That means the sort of highest level


information that it has really refers to kind of concepts. It does refer to color, which


and things like this, but also the concepts, right? So there's no view of a cow here on


the left or the specific city on in the middle right here. It's just like a city landscape,


which is really interesting because a lot of models, I think they would, they would


much more try to model everything at once, but in less detail, right? And then if you


go down with the principal components, they would try to just have color blobs everywhere


where they're relevant. And you can see as you go up, as you add principal components,


what's also interesting is that as you add principal components, the scene can radically


change, right? So that's also quite interesting. So here we can, on the top row, you can see


that we now add, we add tomatoes into the mix. So where it went from a general, are


these gnocchi maybe? So from a general Italian kitchen setting, it goes more and more into


this direction and it ends up in something that's fairly similar to what we're seeing,


right? And the same here on the bottom with the cows even. You can even see that a little


bit, they have even the same, same positioning, but I'm not sure if I'm over interpreting


this right here. You can also see that you have the general mountain landscape down here


and then more and more, it kind of refines to the maybe a bit more correct colors, a


bit, also the background gets a bit more accurate. Yeah, this is, it's just quite, quite cool.


And as I said, this is just a lot of evidence for the fact that these things have some sort


of abstractive representations of what's happening. Samples using different conditioning signals


for the same decoder. We passed, in the first row, we passed the text caption to the decoder


and pass a zero vector for the clip embedding. In the second row, we pass both text caption


and the clip text embedding of the caption. In the third row, we pass the text and the


clip image embedding created by an autoregressive prior for the given caption. Note that this


decoder is only trained to do the text to image generation task without the clip image


representation. So they have a lot of inputs, it seems, into their decoder. So what is the


first one? We pass the text caption to the decoder. So there's no text embedding from


the clip decoder or anything. There's only text caption, which it also conditions on.


A group and an oil painting of a corgi wearing a party hat. Hedgehog using a calculator,


motorcycle parked in a parking space next to another. Yeah, so this, as you can see,


the top row, it picks up kind of on like one word or so out of the, is this intelligent?


It's like we should make the meme with the butterfly. But as you can see, it kind of


picks up on like one word or so out of the out of the text, and then it just creates


kind of a realistic image that contains that word. It doesn't have the capability of fusing


different things and so on. And I'm not sure what that is, because they say it's the same


encoder. And that has been trained with various inputs. And they just pass a zero vector for


the clip embedding. So they kind of stifle the input. It's not, it's not a true comparison


in the sense that they have actually trained the decoder to only work with this type of


data. So I'm not sure what this means right here. It could mean that clip embeddings are


really important. It could also just mean that you have to actually train the decoder


with this because you just stifle it from some inputs that it doesn't have. Now, if


you add the text embedding from clip, you can see that it already gets a lot more capable.


So a hedgehog using a calculator. Yeah, there is a hedgehog and there is someone I guess


using a calculator. Motorcycle or this wire metal rack holds several pairs of shoes and


sandals. I guess, I guess not really. And then as soon as you add the image embedding


as well. So that's the image embedding of, created by an autoregressive prior for the


given caption. Okay, so the prior will create the image embedding. So this is the full system


at the bottom except having a clip image embedding that comes from some sort of image instead


of from the prior. Yeah, it's difficult to say what this means, honestly, because they


just take away stuff and obviously it's not gonna, it's not gonna go as well. So the classifier


free guidance is, I don't actually have it in my head exactly what it is, but it's like


you, maybe you have a classifier, I don't know what they use as a classifier here, maybe


clip itself and then you can mask some stuff and then you have the direction of where you


want to go and then you just go more into that direction, but I don't have it on top


of my head what it is. Okay, glide, I've done a video on glide, so I don't remember myself


too accurately what it does, but I love how the different papers continue this work, right?


So Dali, glide, make a scene and then unclip, green train coming down the tracks, crazy.


Yeah this goes into the direction of GANs which just have become so good at this point


that the optimizations are micro, we're still a few ways away from there, but it's getting


there. Samples from unclip and glide for the prompt, a red cube on top of a blue cube.


Yeah pretty good. Reconstructions from the decoder for difficult binding problems, we


find that the reconstructions mix up objects and attributes and the first two examples,


the model mixes up the color of two objects, in the rightmost example the model is not


reliably reconstructed relative size of the two objects. Yeah so probably, you know, given


this might also be a property right of the text descriptions, maybe not the binding problem


as much, but the relative sizes you rarely have a text description that describes the


relative sizes of things and therefore the encodings might just end up with sort of what's


contained in the image and then that's what the models learn that okay what's mostly important


is what's contained. A sign that says deep learning. Deep deep. This is just deep deep.


Deep deep. Come on. High quality photo of a dog playing in a green field next to a lake.


Unclip samples showing low levels of detail for some complex scenes. Okay yeah so, Dolly


2, oh Dolly 2 and Unclip seem to be the same model right. So that just because I was wondering,


I was wondering sort of because they kept talking about Unclip but I do believe that


Dolly 2 is Unclip. Correct me if I'm wrong right here but here they describe it yes this


is Unclip that's Dolly 2. This thing right here. So yeah I don't know what to what to


think of this. It's certainly super impressive what it can do right if you if you look at


oh there's join waitlist okay okay we'll do that. It's certainly super impressive like


the applications of this are I believe quite far reaching because this allows more and


more yeah it decouples I think the mechanical I've already said this I think in the past


it decouples the mechanical skill of making these things happen like the photoshop skill


or the painting skill or or nunchuck skills, fighting skills. It decouples the mechanical


skills from or it decouples them from the creativity part. So usually you have like


okay I want to do something and then the second thing is okay I have to have to actually implement


it and this thing decouples it it kind of takes that mechanical aspect away from you


and therefore you're left to focus on what should happen and I think with these editing


capabilities that it can do and the variation capabilities this could be very powerful if


built into if built into GIMP and then in GIMP you can just say okay enter text a piece


of text create an image okay edit here enter piece of text make some variations on this


thing right here select a bunch of stuff very very powerful and I guess a lot of people


would pay a lot of money for it which might be the end goal here. All right I hope you


enjoyed this kind of live reaction live reaction to this type of research I'm excited to see


if this actually ever gets out here but yeah let me know how you like this type of stuff


if we do live reactions in the future again and yeah I'll I guess I'll see you around


Backpropagation and the brain

Hi there! Today we're looking at Back Propagation and the Brain by Timothy


Backpropagation and the brain

Lillikrub, Adam Santoro, Luke Morris, Colin Ackerman and Jeffrey Hinton. So this is a


Backpropagation and the brain

bit of an unusual paper for the machine learning community but nevertheless it's


Backpropagation and the brain

interesting and let's be honest at least half of our interest comes from the fact


Backpropagation and the brain

that Jeffrey Hinton is one of the authors of this paper. So this is a paper


Backpropagation and the brain

that basically proposes a hypothesis on how the algorithm of back propagation


Backpropagation and the brain

works in the brain because previously there has been a lot of evidence against


Backpropagation and the brain

there being something like back propagation in the brain. So the question


Backpropagation and the brain

is how do neural networks in the brain learn? And they say there can be


Backpropagation and the brain

many different ways that neural networks learn and they list them up in this


Backpropagation and the brain

kind of diagram where you have a network and it maps from input to output by


Backpropagation and the brain

having these weighted connections between neurons. So the input is


Backpropagation and the brain

two-dimensional and then it maps using these weights to a three-dimensional


Backpropagation and the brain

hidden layer. Usually there is a nonlinear function somewhere at the


Backpropagation and the brain

output here of these so they do a weighted sum of the inputs and then they


Backpropagation and the brain

do a nonlinear function and then they propagate that signal to the


Backpropagation and the brain

next layer and to then to finally to the output. Alright so how do these


Backpropagation and the brain

networks learn? The one way of learning is called Hebbian learning. The


Backpropagation and the brain

interesting thing here is that it requires no feedback from the outside


Backpropagation and the brain

world. Basically what you want to do in Hebbian learning is you want to update


Backpropagation and the brain

the connections such that they kind of match their own previous outputs or even


Backpropagation and the brain

increase their own previous outputs. So you propagate a signal and then maybe


Backpropagation and the brain

this neuron spikes really hard and this neuron spikes really low then if you


Backpropagation and the brain

propagate the signal again right then you want to match that those


Backpropagation and the brain

activations or if you if you propagate similar signals. No feedback required so


Backpropagation and the brain

basically it's a self amplifying or self dampening process. Ultimately though


Backpropagation and the brain

you want to learn something about the world then that means you have to have


Backpropagation and the brain

some some feedback from outside right so with feedback what we mean is usually


Backpropagation and the brain

that the output here let's put this away the output here is goes into the world


Backpropagation and the brain

let's say this is a motor neuron right you do something with your arm like you


Backpropagation and the brain

hammer on a nail and then you either hit the nail or you don't let's say you


Backpropagation and the brain

don't hit the nail so after it looks like crooked there you have feedback


Backpropagation and the brain

right so feedback usually in the form of some sort of error signal right so


Backpropagation and the brain

feedback you can be like this was good or this was bad or it can be this was a


Backpropagation and the brain

bit too much to the left or so on the important part is you get kind of one


Backpropagation and the brain

number of feedback right and how bad you were and now your goal is to adjust all


Backpropagation and the brain

of the individual neurons or weights between neurons such that the error will


Backpropagation and the brain

be lower so in Hebbian learning there is no feedback it's just simply a


Backpropagation and the brain

self-reinforcing pattern activation machine in the first in these kind of


Backpropagation and the brain

first instances of perturbation learning what you'll have is you'll have one


Backpropagation and the brain

single feedback and that you can see this is a diffuse cloud here what you're


Backpropagation and the brain

basically saying is that every single neuron is kind of punished let's say the


Backpropagation and the brain

the feedback here was negative one that means every single neuron is is punished


Backpropagation and the brain

for that so how you can imagine something if you have your input X and


Backpropagation and the brain

you map it through through your function F and the function F has a weight w1 and


Backpropagation and the brain

so on right so you map X through it right and then you get a feedback of


Backpropagation and the brain

negative one and then you map X with a little bit of noise plus M right


Backpropagation and the brain

dadada and you get a feedback of negative two right then you that means


Backpropagation and the brain

that the direction of this noise was probably a bad direction so ultimately


Backpropagation and the brain

you want to update X into the direction of negative that noise by modulated of


Backpropagation and the brain

course by by some some factor here that's that kind of tells you how bad it


Backpropagation and the brain

was so this could be the negative two minus negative one yeah that makes big


Backpropagation and the brain

sense no yes that would be no it would be negative one minus negative never


Backpropagation and the brain

mind so basically with a scalar feedback you simply tell each neuron what it did


Backpropagation and the brain

right or sorry if if the entire network right the entire network did right or


Backpropagation and the brain

wrong so the entire network will lead to this feedback you don't have


Backpropagation and the brain

accountability of the individual neurons all you can say is that whatever I'm


Backpropagation and the brain

doing here is wrong and whatever I'm doing here is right so I'm gonna do more


Backpropagation and the brain

of the right things now in back propagation it is very different right


Backpropagation and the brain

in back propagation what you'll do is you'll have your feedback here let's say


Backpropagation and the brain

that's negative one and then you do a reverse computation so the forward


Backpropagation and the brain

computation in this case was these weighted sum of this layer now you do a


Backpropagation and the brain

little layer wise reverse computation which means that you know how this


Backpropagation and the brain

function here this output came to be out of the out of the inputs and that means


Backpropagation and the brain

you can inverse and you can do an inverse propagation of the error signal


Backpropagation and the brain

which is of course the gradient so this would be your your you you would derive


Backpropagation and the brain

your error by the inputs to the layer right so this basically tells in the


Backpropagation and the brain

back propagation algorithm you can exactly determine if you are this node


Backpropagation and the brain

how do I have to adjust my input weights how do I have to adjust them in order to


Backpropagation and the brain

make this number here go down right and then because you always propagate the


Backpropagation and the brain

error according to that what you'll have in each in each layer is basically a


Backpropagation and the brain

vector target so it's no longer just one number but each layer now has a target


Backpropagation and the brain

of vectors and it says okay these are the outputs that would be beneficial


Backpropagation and the brain

please this layer please change your outputs in the direction of negative 2


Backpropagation and the brain

negative 3 plus 4 so you see this is so the negative 2 would be this unit the


Backpropagation and the brain

negative 3 would be this unit and the plus 4 would be this unit so each unit


Backpropagation and the brain

is instructed individually to say please this is the direction that each unit


Backpropagation and the brain

should change in in order to make this number go lower you see how this is much


Backpropagation and the brain

more information than the perturbation learning in the perturbation learning


Backpropagation and the brain

all the units simply know well before was bad and now is better so let's you


Backpropagation and the brain

know change a bit and here you have detailed instructions for each unit


Backpropagation and the brain

because of the back propagation algorithm so ultimately people have kind


Backpropagation and the brain

of thought that since back propagation wasn't really possible with biological


Backpropagation and the brain

neurons that the brain might be doing something like perturbation learning but


Backpropagation and the brain

this paper argues that something like back propagation is not only possible


Backpropagation and the brain

but likely in the brain and they propose this kind of back prop like learning


Backpropagation and the brain

with the feedback network so they basically concern all the they


Backpropagation and the brain

differentiate hard between these two regimes here in this hand you have the


Backpropagation and the brain

scalar feedback which means that the entire network gets one number as a


Backpropagation and the brain

feedback and each neuron just gets that number and here you have vector


Backpropagation and the brain

feedback where each neuron gets an individual instruction of how to update


Backpropagation and the brain

and they achieve this not by back propagation because still the original


Backpropagation and the brain

formulation of backprop as we use it in neural networks is not biologically


Backpropagation and the brain

plausible but they achieve this with this back prop like learning with the


Backpropagation and the brain

feedback network and we'll see how this does but in in essence this feedback


Backpropagation and the brain

network is constructed such that it can give each neuron in the forward pass


Backpropagation and the brain

here detailed instructions on how to update itself all right so yeah they


Backpropagation and the brain

have a little bit of a diagram here of if you do heavy in if this if this is an


Backpropagation and the brain

error landscape if you do heavy in learning you basically you don't care


Backpropagation and the brain

about the error you're just reinforcing yourself if you do perturbation learning


Backpropagation and the brain

then you it's very slow because you don't have a detailed signal you just


Backpropagation and the brain

you're just relying on this one number it's kind of if you were to update every


Backpropagation and the brain

single neuron in your neural network with reinforcement learning considering


Backpropagation and the brain

the output the of the neural networks or the error considering that the


Backpropagation and the brain

reward not using backprop and then with backprop you have a much smoother much


Backpropagation and the brain

faster optimization trajectory so they look at this and they they come to some


Backpropagation and the brain

some conclusions first of all so here's here's backprop basically so in


Backpropagation and the brain

backprop as we said you have a forward pass and there you simply compute these


Backpropagation and the brain

weighted averages and you you also pass them usually through some sort of


Backpropagation and the brain

nonlinear activation right and the cool thing about this is in artificial neural


Backpropagation and the brain

networks is that once the error comes in you can exactly reverse that so you can


Backpropagation and the brain

do a backward pass of errors where you can propagate these errors through


Backpropagation and the brain

because you know it's kind of invertible the function doesn't have to be


Backpropagation and the brain

invertible but that the gradients will flow backwards if you know how the


Backpropagation and the brain

forward pass was computed so first of all they go into a discussion of


Backpropagation and the brain

backprop in the brain how can we even expect that and one cool piece of


Backpropagation and the brain

evidence is where I find is that they cite several examples where they use


Backpropagation and the brain

artificial neural networks to learn the same task as humans right and or as as


Backpropagation and the brain

animal brains and then I have no clue how how they measure any of this but


Backpropagation and the brain

then they compare the hidden representations of the living neural


Backpropagation and the brain

networks and the artificial neural networks and it turns out that the these


Backpropagation and the brain

the networks that were trained with backprop can clear up much more of the


Backpropagation and the brain

variance of these hidden activations then networks that were not trained with


Backpropagation and the brain

backprop so basically that means if you train a network with backprop it matches


Backpropagation and the brain

the biological networks much closer in how they form their hidden


Backpropagation and the brain

representations and they they do a number they cite a number of experiments


Backpropagation and the brain

here that show this so this gives you very good evidence that if the hidden


Backpropagation and the brain

representations they look as if they had been computed by backprop and not by any


Backpropagation and the brain

of these scalar updating algorithms so it is conceivable that we find


Backpropagation and the brain

backprop in the brain that's why they go here next they go into problems with


Backpropagation and the brain

backprop so basically why why would we why so far have we believed that


Backpropagation and the brain

backprop isn't happening in the brain so now let's I want to highlight two


Backpropagation and the brain

factors here that I think are suffice they have more but first of all


Backpropagation and the brain

backprop demands synaptic symmetry in the forward and backward paths so


Backpropagation and the brain

basically if you have a neuron and it has output to another neuron what you


Backpropagation and the brain

need to be able to do is to pass back information along that neuron so it kind


Backpropagation and the brain

of has to be a symmetric connection right you have the forward and the


Backpropagation and the brain

backward paths and these need to be exact right and this is just not if you


Backpropagation and the brain

know how neurons are structured they have kind of input dendrites and then


Backpropagation and the brain

there's this accent action potential and along the axon the signal travels and


Backpropagation and the brain

the back traveling of the signal just I think is very is very very very slow if


Backpropagation and the brain

even possible and so it's generally not invertible or inverse compute capable so


Backpropagation and the brain

this is one reason why backprop seems unlikely and then the second reason here


Backpropagation and the brain

is error signals are signed and potentially extreme valued and I want to


Backpropagation and the brain

add to that they also let's talk about this somewhere that error signals are of


Backpropagation and the brain

a different type right that's a different type so first let's see what


Backpropagation and the brain

signed error signals are signed yes we need to be able to adjust neurons in a


Backpropagation and the brain

specific directions right if you look at again what we've drawn before here we


Backpropagation and the brain

said here this is how these neurons must update so the first neuron must must


Backpropagation and the brain

decrease by two this must decrease by three and this must increase by four now


Backpropagation and the brain

in backprop we need this but in if if we assume that there is something like a


Backpropagation and the brain

reverse computation or signaling here happening then we still have the problem


Backpropagation and the brain

that usually these output signals are in the form of spiking rates which means


Backpropagation and the brain

that over time right so if a neuron wants to if a neuron has zero activation


Backpropagation and the brain

there's just no signal but if a neuron has a high activation it spikes a lot if


Backpropagation and the brain

has a low activation it kind of spikes sometimes what what he can't do is


Backpropagation and the brain

negative spike right like zero is as low as it goes so the thought that there are


Backpropagation and the brain

signed information in in the backward pass is inconceivable even if you have


Backpropagation and the brain

something like a second so you can imagine here instead of this backward


Backpropagation and the brain

connection because of the symmetry problem that we have some kind of second


Backpropagation and the brain

neural network that goes in this direction still you'd have the problem


Backpropagation and the brain

that here you can only have positive signal or zero and they might be extreme


Backpropagation and the brain

valued which okay it can't be really encoded with these spiking because they


Backpropagation and the brain

are they're limited in the range they can assume but they are also of a


Backpropagation and the brain

different type and I'm what I mean by that is basically if you think of this


Backpropagation and the brain

as a programming problem then the forward passes here are our activations


Backpropagation and the brain

right and the backward passes here they are deltas so in the backward passes you


Backpropagation and the brain

either propagate deltas or you propagate kind of directions so the activations


Backpropagation and the brain

are sort of impulses whereas the backward signals are this isn't you how


Backpropagation and the brain

you need to change their their gradients ultimately so it's fundamentally a


Backpropagation and the brain

different type of data that is propagated along would be propagated


Backpropagation and the brain

along these directions and that makes it very unlikely because we are not aware


Backpropagation and the brain

as this paper says that the that neural networks get neurons can kind of switch


Backpropagation and the brain

the data type that they're they're transmitting all right so then the paper


Backpropagation and the brain

goes into their n grad hypothesis and what this is the hypothesis basically


Backpropagation and the brain

states that the brain could implement something like neural networks by using


Backpropagation and the brain

by using an approximate backprop like algorithm based on auto encoders and I


Backpropagation and the brain

want to jump straight into the algorithm no actually first they do talk about


Backpropagation and the brain

auto encoders which which I find very interesting so if you think of auto


Backpropagation and the brain

encoders what is an auto encoder an auto encoder is a network that basically


Backpropagation and the brain

starts out with an input layer and then has a bunch of hidden layers and at the


Backpropagation and the brain

end it tries to reconstruct its own input right so you feed a data in here


Backpropagation and the brain

you get data out here and then your error the error signal it will be your


Backpropagation and the brain

difference to your original input now the usually when we train auto encoders


Backpropagation and the brain

in deep learning we also train this by backprop right we feed and this error


Backpropagation and the brain

here and this goes back but if you just think of single layer auto encoders so


Backpropagation and the brain

let's let's go over here single layer auto encoder with let's say the same


Backpropagation and the brain

number of the same number of units in this in this layer what you'll have is


Backpropagation and the brain

so this this is input this is output and this is the hidden layer right you'll


Backpropagation and the brain

have a weight matrix here and you'll probably have some sort of nonlinear


Backpropagation and the brain

function and then you have another weight matrix here and they call them W


Backpropagation and the brain

and B another way to draw this is I have weight matrix going up then I have a


Backpropagation and the brain

nonlinear function going transforming this into this signal and then I have


Backpropagation and the brain

the B going back right so I'm drawing I'm drawing it in two different ways up


Backpropagation and the brain

here or over here and with the second way you can see that it is kind of a


Backpropagation and the brain

forward-backward algorithm where now the error if you look at what is the error


Backpropagation and the brain

here the error is the difference between this and this and the difference


Backpropagation and the brain

between this and this and the difference between this and this right and you can


Backpropagation and the brain

train an auto encoder simply by saying W please make sure that the that the


Backpropagation and the brain

the the input here gets mapped closer to the output and the B the same thing this


Backpropagation and the brain

will become clear in a second so but basically sorry this I mean the the


Backpropagation and the brain

hidden representations you'll see basically the idea is that you can train


Backpropagation and the brain

an auto encoder only by using local update rules you don't have to do back


Backpropagation and the brain

prop and that's what this algorithm is proposing namely if you think of a stack


Backpropagation and the brain

of auto encoders this this this transforming one hidden representation


Backpropagation and the brain

into the next right this is the feed forward function right what you can do


Backpropagation and the brain

is you first of all you can assume that for each of these functions here you


Backpropagation and the brain

have a perfect inverse right you can you can perfectly compute the inverse


Backpropagation and the brain

function that's this this G here of course this doesn't exist but assume you


Backpropagation and the brain

have it what you then could do is you could if if you knew in one layer and on


Backpropagation and the brain

the top layer of course you know if you knew that okay I got this from my


Backpropagation and the brain

forward pass but I would like to have this this is my desired output right so


Backpropagation and the brain

in the output layer you get this this is your error signal if you knew you you


Backpropagation and the brain

you could compute an error right here this is what you do in the output right


Backpropagation and the brain

now in back prop we would back propagate this error along the layers but now we


Backpropagation and the brain

don't do this instead of what we do is we use this G function to invert the F


Backpropagation and the brain

function right and by that what we'll say is what hidden representation in


Backpropagation and the brain

layer 2 what should the hidden representation have been in order for us


Backpropagation and the brain

to obtain this thing right so the the claim here is if in layer 2 we had had


Backpropagation and the brain

h2 as a hidden representation then we would have landed exactly where we want


Backpropagation and the brain

it right that's what this G function does because here we use F so had we had


Backpropagation and the brain

F h2 and used F on it we would be exactly where we want instead we had h2


Backpropagation and the brain

here and used F on it and then we landed here where we don't want so this is


Backpropagation and the brain

where we want we would want to be in layer 2 and this is where we were so


Backpropagation and the brain

again we can compute an error here again instead of back propagating that error


Backpropagation and the brain

what we'll do is we'll use the inverse of the forward function in order to


Backpropagation and the brain

back propagate our desired hidden representation and you can see there is


Backpropagation and the brain

of course a relationship to the true back prop here but the the important


Backpropagation and the brain

distinction is we are not trying to back propagate the error signal we're trying


Backpropagation and the brain

to invert the desired hidden states of the network and then in each layer we


Backpropagation and the brain

can compute from the forward pass we can compute the difference to the desired


Backpropagation and the brain

hidden state and thereby compute an error signal and now we have achieved


Backpropagation and the brain

what we wanted we want an algorithm that doesn't do back prop that only uses


Backpropagation and the brain

local information in order to compute the error signal that it needs to adjust


Backpropagation and the brain

and by local I mean information in the same layer and also the data type that


Backpropagation and the brain

is propagated by F is activations right of hidden representations and by G is


Backpropagation and the brain

also activations of hidden representations both of them are always


Backpropagation and the brain

positive can be encoded by spiking neurons and so on so this algorithm


Backpropagation and the brain

achieves what we want they go a bit into detail how the actual error update here


Backpropagation and the brain

can be achieved and apparently neurons can achieve you know in the same layer


Backpropagation and the brain

to to adjust themselves to a given desired activation so this algorithm


Backpropagation and the brain

achieves it of course we don't have this G we don't have it and therefore we need


Backpropagation and the brain

to go a bit more complicated what they introduce is the this following


Backpropagation and the brain

algorithm the goals are the same but now we assume we do not have a perfect


Backpropagation and the brain

inverse but we have something that is a bit like an inverse so we have an


Backpropagation and the brain

approximate inverse and they basically suggest if we have an approximate


Backpropagation and the brain

inverse we can do the following so G G is now an approximate inverse to F what


Backpropagation and the brain

we can do is this is our input signal right we use F to map it forward to this


Backpropagation and the brain

and so on all the way up until we get our true our error right here this is


Backpropagation and the brain

our error from the environment right this is the nail being wrong and then we


Backpropagation and the brain

do two applications of G right so this is an application of F we do two


Backpropagation and the brain

applications of G one we apply G to this to what we got in the forward pass right


Backpropagation and the brain

and this now gives us a measure of how bad our inverse is right so if G is now


Backpropagation and the brain

an approximate inverse and this now we see here oh okay we we had H 2 in the


Backpropagation and the brain

forward pass and we basically forward passed and then went through our inverse


Backpropagation and the brain

and we didn't land quite exactly where we started but we know that okay this


Backpropagation and the brain

this is basically the difference between our our inverse our forward inverse H


Backpropagation and the brain

and our true H and then we also back project using G again the desired


Backpropagation and the brain

outcome so we invert the desired outcome here now before we have adjusted


Backpropagation and the brain

directly these two right because we said this is what we got this is what we want


Backpropagation and the brain

but now we include for the fact that G isn't the perfect inverse and our


Backpropagation and the brain

assumption is that G here probably makes about the same mistakes as G here so


Backpropagation and the brain

what we'll do is we'll take this vector right here and apply it here in order to


Backpropagation and the brain

achieve this thing and this thing is now the corrected thing are corrected


Backpropagation and the brain

desired hidden representation corrected for the fact that we don't have a


Backpropagation and the brain

perfect inverse and now again we have our error here that we can locally


Backpropagation and the brain

adjust again all the signals propagated here here and here are just neural


Backpropagation and the brain

activations and all the information required to update a layer of neurons is


Backpropagation and the brain

now contained within that layer of neurons right and and this goes back


Backpropagation and the brain

through the network so this is how they achieve how they achieve this this is a


Backpropagation and the brain

bit of a of a close-up look and here are the computations to do this so basically


Backpropagation and the brain

for the forward updates you want to adjust W into the direction of the H


Backpropagation and the brain

minus the H tilde and the H tilde in this case would be this the the hidden


Backpropagation and the brain

representation that you would like to have so you would update your forward


Backpropagation and the brain

forward weights into direction such that your hidden representations are closer


Backpropagation and the brain

sorry that your forward hidden representation is closer to your


Backpropagation and the brain

backward hidden representation and the backward updates now your goal is to get


Backpropagation and the brain

a more better to make G so sorry W here these are W are the weight of F and


Backpropagation and the brain

B are the weights of G so in the backward updates your goal is to make G


Backpropagation and the brain

a better inverse right so what you'll do is again you'll take the difference


Backpropagation and the brain

between now you see the difference here here here right not the same error so


Backpropagation and the brain

here you in the W update use what we labeled error here in the G update you


Backpropagation and the brain

use this error here this is the error of G so when you update the function G you


Backpropagation and the brain

want to make these two closer together such that G becomes a better inverse


Backpropagation and the brain

right because you're dealing with an approximate inverse you still need to


Backpropagation and the brain

obtain that approximate inverse and and this here is how you learn it this


Backpropagation and the brain

algorithm now achieves what we wanted right local updates data types check


Backpropagation and the brain

assigned check and so on I hope this was enough clear in essence is pretty simple


Backpropagation and the brain

but it's pretty cool how they work around this they call this a difference


Backpropagation and the brain

target propagation and I'm not these these kind of papers I don't think they


Backpropagation and the brain

invented this maybe I'm not sure maybe they did and maybe they didn't and this


Backpropagation and the brain

paper just kind of frames it in this hypothesis it is unclear to me I I'm not


Backpropagation and the brain

familiar with this kind of papers so sorry if I misattribute something here


Backpropagation and the brain

all right then they go into into how could these things be implemented


Backpropagation and the brain

biologically and they go for some evidence and they also state that we


Backpropagation and the brain

used to look at neurons basically in this way where you had input and feedback


Backpropagation and the brain

here very simple simplistic view of neurons whereas nowadays even the


Backpropagation and the brain

the company computational community views neurons in a more differentiated


Backpropagation and the brain

way where you have for example different regions here on the Soma that can be


Backpropagation and the brain

separated from each other and you have interneuron interference and so on I'm


Backpropagation and the brain

not qualified to much to comment on this stuff but I invite you to read it for


Backpropagation and the brain

yourself if you want all right so this was my take on this paper I find the


Backpropagation and the brain

algorithm they propose pretty cool if you I hope you liked it and check it out


Backpropagation and the brain


Planning to Explore via Self-Supervised World Models (Paper Explained)

Hi there, today we're looking at planning to explore via self supervised world models


by Ramana Sekar, Ole Ribkin, Kostas Danilidis, Pieter Abil, Dani Jarrahovner and Deepak


Patak.


So this is a paper that concerns reinforcement learning and specifically sort of self supervised


reinforcement learning.


So what do they mean?


Here's a graphic right here.


In reinforcement learning, usually, you have an environment and an agent, right?


So you have this environment.


And let's ignore the without rewards for now.


And you have the agent.


And the agent needs to interact with the environment in order to achieve a maximum reward.


So the reward is given by a certain task, right, you have to do something in this environment.


In this case, they consider these these types of tasks where I think the top task might


be called run forward.


So your reward is more the further you go with this walker.


And how you can influence the walker is you can sort of give give a bit of force onto


its joints right here.


And you have a bunch of sensors.


So the main task is actually to keep it to balance it on its feet and then sort of walk


forward such that it never falls over, right?


And otherwise you get negative reward you lose.


So in this case, what they want to do is they want to say, wait, if we just train a reinforcement


learning agent for each of these tasks individually, that will use a lot of data.


And basically, we can't reuse the learned reinforcement learning agent for each of these


individual tasks.


It's sort of like if you have many, you know, image tasks or NLP tasks, you don't want to


learn one model for each one individually.


But you might do something like a common joint pre training.


And this is exactly this for reinforcement learning.


And it's even called self supervised, right?


Like we are used to in the in the classification setting, self supervised learning.


What does it mean?


So it means that at first you're in an environment without rewards.


So basically, the agent is just dropped in an environment, and there's no rewards, it


can just do actions and observes of the states from this environment.


And after that, after a while of that, then the tasks come in.


So task A, task B, and task C are three different tasks, all in the same environment, but all


requiring the agent to do different things like running forward, or running backwards,


or do a front flip or things like this.


So the how fast the agent can adapt to these individual tasks, very much depends on what


it has learned during this phase where there were no rewards, right?


So the agent is tasked to just explore the world via what they call here task agnostic


exploration to explore the world to learn something about the world in order then to


generalize to these tasks.


And in their case, they learn this global world model.


So the agent is supposed to learn somehow how the world works, right?


And this is, this is the way that this agent is then able to adapt really quickly.


So in essence, what does this agent do?


The agent works as follows.


It gets an input observation.


And it runs that through an encoder, which is usually something like a convolutional


neural network.


And that will give you a set of features.


So this is sort of an embedding of the state that you're in.


And that you can incorporate into a latent state at time t.


Now, usually in these RL algorithms, or what can happen is that you incorporate the last


latent latent state, so the latent state from the step here also goes into the latent state


of the next step, right?


So here was the sorry, the last observation, observation comes in features, latent state


and so on.


Ultimately, and then and so on comes in from here.


And there's usually like an RNN going over the time steps.


But ultimately, here, the agent has to decide on an action using this policy network.


Now, how is this trained to this policy network, it has to come up with an action, but there


are no rewards.


So usually, we would train this policy network with like an actor critic method.


So we would also train some sort of a value function, and then the policy would try to


maximize the value function.


And if we don't have rewards, how are we going to do that?


So people have thought about this for a bit.


And people have come up with things like intrinsic motivation.


Intrinsic motivation is a term where you're trying to say something like, if you're in


a room right here, like this, your agent is right here, then you just you know, you do


something.


And maybe your agent goes down here.


If your agent were to go down there, again, it would sort of not really learn anything


because it has now already gone there and is already learned from those states.


So you might want to explore some different space, right, like here.


And in the next episode, you might want to explore this room right here.


So this this notion of intrinsic motivation to explore, it has a bunch of different formulations


of how exactly you can formulate it.


But just imagine basically, the entire state is filled with a bunch of coins.


And I'm going to draw this as green dots, sort of like Pac Man.


And everything is filled with these green dots, right?


And what the agent wants to do if it has no rewards, it will simply collect those green


dots.


And once one is collected, so if I go here, I'll collect all these green dots, these are


now no longer there.


So that area doesn't give me any reward anymore.


So you can imagine sort of like this.


So as an intrinsic reward, you simply reward the agent every time it finds itself in a


new state that it hasn't seen before.


So you train it to seek out novel states that usually when you just have like an actor critic


method, and that's what this paper here criticizes, if you have it's called retrospective novelty.


That means if you train a model free algorithm, which is an actor critic, right, if we just


plug in into here, something like a three c, that will simply have a policy and a value


function.


And in this case, if we train it on intrinsic reward, the policy will simply tell you where


to go to find more green stuff.


But you can only train it.


So you use this to run an episode.


And then you observe how many green things you found in that episode, right, if your


episode goes here, and then you put that back into your buffer to learn from.


But at that point, you've already collected the green things, right.


So the reward signal is actually a bit off because you want to train your agent that


there that it should seek out novel things.


But as soon as you've explored them, they're not really novel anymore, because you have


now explored them.


But still, you're going to train your agent, telling your agent that this area right here


has lots of has given me lots of rewards.


So the agent is going to be encouraged to repeat that.


They say this right here, the retrospective novelty, model free exploration methods not


only require large amounts of experience to adapt to downstream tasks, they can also be


inefficient during exploration.


These agent usually first act in the environment, collect trajectories, and then calculate an


intrinsic reward as the agent's current estimate of novelty.


This approach misses out on efficiency by operating retrospectively.


That is, the novelty of inputs is computed after the agent has already reached them.


Hence, it seeks out previously novel inputs that have already been visited and would not


be novel anymore.


Instead, one should directly seek out future inputs that are expected to be novel.


Now, so what this paper is doing, it's basically saying, can we build a model that estimates


the future novelty of a state that we maybe haven't seen so far.


And here's where that goes.


So what do they do in this policy?


So the policy isn't just trained to maximize the novelty in the world.


Instead, sorry, it uses planning, it uses planning in latent space.


So what this model does is it learns a world model in latent space, the world model takes


as input these features that you saw right here that the encoder gives you, and it predicts


the future hidden latent states.


These are the things you saw here.


So these things that are always made by incorporating the new features with the old state, it tries


to predict it.


So technically, these things here should have like some sort of, no, this is actually exact


here, but these here should have some sort of a tick or something to indicate these are


estimated, these are estimated future states.


And this model right here, this is an estimate, this is a world model.


They use dreamer for this, and I have made a video about dreamer.


Dreamer tries exactly that it tries to estimate what is the future, but not in actual world


space, but in latent space.


And yeah, so it tries to estimate its own future.


And the cool thing here is that this is probabilistic, or you can make it probabilistic.


So you can technically from this one age that you have here, you can run out many futures


in your imagination.


And since you don't need the observations, you only need the latent space, you can simply


forward roll your RNN and sample from it and you have many trajectories in the future.


Now, the fact that you have many trajectories leads to even a different thing.


So what you can do for each of these hidden states, they have a head here that predicts


the so called latent disagreement.


What does this do?


This consists of a whole bunch of models.


These are ensemble models.


They're the same for each time step.


But what they take in, they take in the latent state of the model and the action that you're


about to do the action that you imagine you would do, right?


So this is the imagined state and this is the imagined action in that state.


And then it will compute the next features.


So the next whatever in the next step would be the age.


So right, right, where do we put it?


Whatever in the next so if I have this age, and I have this state, it tells me if I do


action a one, and if I were to execute this in the real world, what would be my next age


that I would get?


So basically, by performing an action, I will get the next observation.


And I will encode that to get the next features.


And this small model would try to predict what are the features of the next state if


I were to execute this action in this state.


Okay, so it's kind of a future predictor.


But also not in observation space, but in latent space.


So it tries to predict the latent features of the next observation.


And the the split here, you might think that there is bit of a, it's like almost the same


this latent state here and the features.


But as we discussed before, the latent state can incorporate the last or the history of


latent states, while the features simply are only a function of the current observation.


And that's why they predict the features.


They really want to predict the observation.


But history has sort of shown that if you try to predict, for example, the pixels of


the observation that won't serve you really well.


And therefore, what you need to do is you need to predict the latent features of the


observation that works much better.


So they have a bunch of these models right here.


They have a bunch of models with different parameterizations, they instantiate k different


models of that, and they all run the same.


So these are all the same inputs through these different models.


Now these different models have been initialized at different points.


So they will make slightly different predictions.


And the crucial part is, so if, if it's really deterministic, what the next state is going


to be right, so say you're in this state, you perform this action.


And so if you have a ball in your hand, and you drop the ball, then the ball is going


to fall down, really deterministic.


That means these next these estimated next features, if the models are any good, they


all agree.


And this variance here between the estimates is very, very small.


And now if, if the uncertainty over the next state is very high, and this can be due to


two facts, either, it is actually uncertain what's going to happen.


So maybe you have a really a piece of paper and you drop that and due to the wind, you


can't know what's happening, or because your model has simply not learned yet what's going


to happen.


In either of those cases, you don't know what's going to happen.


Therefore these these predictions here, sorry, are going to be very different from each other.


And because of that, this variance will be high, and this variance you take as the intrinsic


reward.


So in each step, you basically try to predict over the next actions you can do, which ones


leads me to a leads me to a situation where I don't know what's going to happen, where


I cannot really predict the variance in my prediction is high.


So I really don't know what's going to happen.


And that is going to be the states you seek out.


Okay, so this is the core of the paper.


Basically, you do this planning in latent space in order to find the states or action


that leads you to a state where you don't know what's going to happen.


And you measure that by trying to predict it using slightly different models.


And if they disagree a whole bunch, then you can use you sort of you, you say, I don't


know what's going to happen.


And therefore, I want to go there because I want to learn about that state.


Now this, this is the this is the entire thing.


It has a bunch of problems, as you can imagine.


So this is the reasoning behind it, right?


Now they try to make a a deal out of basically their latent disagreement here agrees with


minima, sorry, maximizing the expected information gain.


They go into the theory right here and say, okay, if I have a state and an action, and


I had and this w are the dynamics parameters of the world.


So the w characterizes how the world works.


And the h here is the next state, or sort of the features of the next observation.


And the I is the mutual information between h and w.


So this right here measures how much information of the next state is contained in the dynamics


of the world.


If this is really low, and I have a good world model, then I should be able to predict the


next state really well.


And this, this, they say, okay, selecting the most promising data during exploration,


we want to select the action that maximizes this information gain.


So the the, the more the mutual information here, we want to select the action that maximizes


that.


They decompose this mutual information into two things.


They decompose it into this thing right here, which is simply the entropy of the next state,


given the current state and action.


This is simply the total uncertainty, including the fact that it could actually be stochastic


like dropping a paper, and the fact that you haven't learned yet what happens like if you


drop a ball, but you haven't learned that yet, that is also uncertain.


So this is that part.


This is the total uncertainty minus this right here.


And this is the uncertainty if you know the dynamics, right?


So this is the, the wind, basically, in the paper example.


So you want something where the total uncertainty is high, but the the kind of uncertainty of


the of the stochasticity of the world is low.


If you maximize this quantity here, this total quantity, sorry, if you maximize this entire


quantity, because I called one of them total, that means you are going to seek out actions,