Those are the non-important weights whereas on the outside those are the important weights.


Movement Pruning: Adaptive Sparsity by Fine-Tuning (Paper Explained)

This is called magnitude pruning because it goes by the magnitude of the weight the absolute


value of the weight.


So you don't actually need so there's not one threshold here you don't need a threshold


you simply need a method to order the weights right and then you keep removing them until


you're satisfied with the size.


So this is magnitude pruning.


Now what's the problem with the magnitude pruning in these kinds of tasks?


They say however it is less effective in the transfer learning regime that has become standard


for state-of-the-art natural language processing applications.


So what do you do in these transfer learning regimes?


In the transfer learning regime and actually let's go with the image example right here


even though it's mostly used in NLP we can do the same thing.


So let's say we have a classifier here for cats and dogs our classifier and we had a


big big database of cats and dogs images right so we were able to train that fairly well


and we don't prune it yet we have this full network.


Now we want to adapt this to a task where we want to recognize whether or not the animal


is sick.


So we developed this app for a veterinarian and it's like a short screening for a particular


disease that a cat might have and we already have this cats and dogs classifier so it's


reasonable to assume that this classifier has some good features to work with cats and


dog images.


So what we can do instead of because let's assume for this other task we just have this


tiny little data set which is not enough to train a neural network of this size right


but so in a first step we'll train this big neural network on the cats versus dogs and


then what we do is we transfer learn it so we transfer all the weights right here and


here we have a different task now sick or not sick right this is cat this is dog and


here it's sick or not sick not sick and of course we can't transfer these particular


weights but we hope that the features here will sort of be the same so we transfer them


and then we train these weights including the the head right here this part we train


it on this little data set and we hope that we already have this good starting point we


only need to you know learn the basically the specifics of what makes these two data


sets different and we won't have to learn entire task of dealing with cat and dog images


from the get-go okay so this is called transfer learning now in this case we combine the two


so first we want to transfer learn like if we build this app for vets and then we might


say oh this is not you know this is not only for vets this is actually for anyone you know


who has a cat or a dog at home so what we could do is build an app where anyone at home


could scan their cat and it would output like a probability of the cat having that disease


so we want this neural network is still the same size as this neural network so now we


want to do the pruning we want this neural network to become sparse to only have a couple


of connections left such that it's a few kilobytes large but retain performance now they say


when you do this step you can't just do the magnitude pruning like you did right here


and why not because this is not this model right here is not the result of a training


step like of a regular training process but is the result of a transfer learning process


where first you do the big training and then second you adapt it and why is that the case


well ultimately what you want to do is you want to prove the non-important weights now


there could be a weight right here this one that is very important for the cat versus


dog task but that is not important for the sick versus non-sick task and we also we know


that in these transfer learning settings the weights they don't tend to move that much


in general the research shows that once you've trained a neural network basically the beginning


is important but then once you did it like if you adapt it or transfer learn it and so


on the weights they won't move that much so in essence this weight maybe starts out right


here and it will sort of stay around this place it will maybe go a little bit down because


it's not important but it won't move much during transfer learning that's just a property


of transfer learning so this paper here says we can't just use magnitude pruning when we


transfer learn because what we'll basically go by what we'll basically say is the will


assign the importance based on based on the original neural network task on the cat versus


dog we we will misspecify the importance of the weights what we should do is actually


measure the importance with respect to this task and how do they achieve it so on a high


level they're basically saying okay if we start out well this was fatal if we start


out with a point over here let's make that red red I want the color red well it's blue


now so if we ah if we start out with a point over here what we should do what we should


do is we should observe how it moves during transfer learning if it moves towards zero


then it's probably not that important for the new task and if it moves to the to be


even larger then it's probably important for that new task okay so that's that's a high


level now how do you measure how it moves and what exact how exactly do you do all of


this during training such that you don't make mistakes that's the point of this paper they


say we propose movement pruning a simple deterministic first order weight pruning method


that is more adaptive to pre-trained model fine tuning we give mathematical foundations


to the method and compare it to existing zero and first order pruning methods okay so yeah


we said so that's basically on a high level that's that now how do they actually do it


what they do right here is the following they say what we can define we can define each


network layer basically as a matrix multiplication by a weight now you can express pretty much


any neural network as such a multiplication with a weight so you have X the in the signal


in each layer and you multiply that by the weight matrix at W now if you prune the neural


network you can see that right here what you're saying is I basically in here I have the matrix


M which is a mask so the mask is either 0 or 1 for if a weight is active or if a weight


is not active now this is not a matrix multiply actually this is like a Hadamard product but


you have this mask matrix and what decides on this mask this mask is decided as you can


see right here by this S so S S is a matrix that for each entry in W it will decide how


important it is now in the classic sense in the magnitude pruning you already saw that


this is just going to be the absolute value of W IJ okay and then the top V simply means


that you take the whoever are the most important the most magnitude that those are going to


be one in the mask and everything else is going to be zero in the mask okay that's how


this S this the W determines the S and the S determines the M okay so what you ultimately


use is the M right here but in now what we want to do is we want to actually make the


S based on the movement and the movement is not really a defined concept because it goes


over steps and so on so how do you do the movement in in a kind of dynamic way and this


paper says you should do it by gradient so you you should observe the gradient of your


loss function with respect to this S matrix to this importance matrix right what does


it mean so the what does it mean it means let's consider this quantity right here if


S is the importance of a particular connection and if the gradient is large that means this


this connection moves a lot right it like the loss pulls it into a particular direction


okay so we're not talking about yet which direction actually the gradient has a sign


it's either positive or negative right so by this quantity you can decide how much does


this new task want this particular importance score to move so if this is a direct direct


measure of how much basically the loss function pulls on that importance score how much and


it now you can simply decide if and they have these they have I think I have a diagram yes


so but I don't like that let's go so we have right here we have what's the value of this


gradient of L with respect to S and here is W so if the gradient is positive and W is


already positive that means the gradient goes into the positive direction so you increase


the loss function in that let's put the negative gradient here because you do gradient descent


right so so if the negative gradient is positive and the weight is already positive in in this


case that means you're all the weight is already high but now the loss function wants to push


it even higher so that must be a very very important weight right like it's like very


good the same goes if the gradient the negative gradient is negative and the weight is already


negative the weight being negative already means the weight you know it has a negative


sign and then the gradient wants it to go even more negative the the optimization procedure


says this thing should become even more negative and also we say that's probably a good weight


now the other two cases means basically the weights already positive but the gradient


wants it to go negative which means it's pulled towards zero now it's entirely possible that


it's going across zero and going like if you're here going from over here going to here cross


zero and become like super large but that violates our basic assumptions that the transfer


learning doesn't move the weights too much right what you're caring for is basically


this local neighborhood right here okay so you can make the fair assumption that these


weights are not that important in the case where the negative gradient goes against the


sign of the weight so this this is of course discrete right now but we can actually assign


a number by how large the gradient is and by how large the weight already is and therefore


we can make a score so the important score right here as you can see is the weight multiplied


by the gradient of the weight and they can actually show mathematically that if you do


this over multiple steps so you optimize while you do this pruning and they do some sort


of a soft pruning so you can kind of correct your mistakes later on I mean they have hard


and soft pruning but in any case they can correct their mistakes later on this will


actually result in these important scores being an accumulation over the training over


the entire training of this quantity and that's pretty cool because that means eventually


you sort of have a consistent estimator of these important scores across your training


procedure okay because the main fear with something like this of course is that it's


very brittle and very much depends on the training dynamics and who knows if in step


one something bad happens and so on but the the math behind this here gives sort of more


evidence that this it can be like a self-correcting mechanism and is actually not too dependent


on the particular training dynamics so they do this experimental setup now they have some


have some quirks here actually let's first go to the the actual different methods they


compare different methods right here where they say okay there's magnitude pruning it's


a zero with order which basically just means you just look at the weight magnitude that


that's it it's top V which means you just pick the top whatever and the objective is


just the loss and the scores are just the absolute value we've seen this now movement


pruning on the other hand is first order which means you look at the movement in our case


of the gradient as you can see here those are the important scores and you use this


straight through estimator which is basically just a way of saying that even though you're


masking some things in the forward step you shouldn't mask them in the in the gradient


backward step because you still want gradient signal to reach so if you have layers and


you have a weight right here at least that's how I understand it I have not read that paper


but if you mask this one here you still want the gradient to sort of flow backwards because


you still need the actual important scores for the weights that here below connect to


this weight I think that's what is meant not entirely sure in this though so but you can


see that the objective function is also the actual loss function now this is contrasted


to a baseline called l0 regularization which is quite similar but is also first order but


has this sort of regularizer right here and uses the Gumbel softmax in order to determine


the scores and as you can see it also has a different score function and it has this


continuous hard concrete importance in masking function sorry masking function and they have


a variant of movement pruning that tends to perform a little bit better which is soft


movement pruning where instead of just going by the loss function optimizing the loss function


they optimize the loss function plus something now they have as you can see here they have


a thresholding masking function and the threshold is actually dynamic or it's determined by


the important scores and they have then a regularizer that make the important scores


sparse so instead of saying we just want the top 5% of weights they now just put a lot


of mass on this lambda right here which will cause the S to be sparse and you know they


if they are not happy with how many weights they from the simply increase or decrease


this lambda such that such that they get to their desired sparsity of course you see there's


the direct trade-off with the loss function right so the more you put weight on this lambda


the the less weight is you put basically on the loss function itself so you have the the


trade-off here is very explicit whereas in basic movement pruning it's just given by


you masking away completely the bottom 1 minus V of percent of the weights but you see the


the score function is the same oh well the score function here the score function is


the same okay now there there are quite a number of tricks here like there's this a


sparsity scheduling function and so on so as always in NLP and with any big models there


are a bunch and bunch of engineering tricks that make everything work better and you can


never exactly tell how much is due to that and how much is due to the the actual technique


but if you know you can sort of assess whether or not these it's done well and this here


actually the rationale makes sense and that's why I tend to think that it is actually a


a better method and the experiments are very very convincing let's say okay this is just


a pictorial comparison where you can see movement pruning sorry magnitude pruning all it does


is it looks at after you fine-tune what are the weights and it just cuts away everything


in the middle doesn't care about how the weights were when before however movement pruning


looks at the combination of what were the weights before and what are the weights now


and it cuts away everything where the weights moved towards zero which are these quadrants


right here and it leaves in everything where it moved away from zero or that's that's the


ordering let's say that how much it moved okay experiments now as you might already


have figured out by now in the machine learning and especially the NLP community the methods


presented always outperform the previous methods in this case it's pretty special so they test


this on these number of tasks quad and NLMI and QQP and these are quite hard tasks from


an NLP perspective like squad is question answering and analyze like language inference


so these would be on the I would I would guess these are on the foreign NLP system on the


harder side of tasks that's it's fairly cool and as you can see here now just focus on


first of all focus on this MAP which is the magnitude pruning so that's the baseline if


you will and the purple one the SMVP which which is the soft movement pruning okay now


you can also focus on the MVP right here but they're approximately the same now the RPP


you can maybe see in the graph is performing fairly well even compared to the full model


it's another baseline basically but we just want to compare those two and you can see


that in this regime that the magnitude pruning outperforms the movement pruning but however


in this regime the movement pruning is much better and that's the percent where the percent


of remaining weights is very very low so this is kind of the extreme sparse case where you


only have 10% or you only have 3% of the weights left and you can see that the movement pruning


is outperforming the magnitude pruning by a lot okay now they do discover that so this


this happens in all in all of these tasks as you can see right here and they do they


do discover that if you then distill the model further so in distillation this is yet another


technique that you can use to boost the performance of the transfer learned model so in distillation


you would not only train the model on you so you have you now you have your this model


that you transfer learn and you have the prune version and in the prune version what you


would do is you would simply also train it on this data set but what you can also do


is you can distill this model right here the one that you train on the same task right


that's that's presumably better because it still has all the weights okay you can run


a data point through both so the same data point goes through this and you get an output


which are logits which is like representing a distribution saying yeah it's about this


high now instead of assigning the hard labels so here we also get the label right it's a


supervised learning task like one and zero you also put the data point through this model


right here obtain whatever that model would have said and let's say it's about this and


now presumably this model is better already so you say well the label here says that it's


this class but the model that's really good says you shouldn't be too sure about it so


you can sort of mix the two losses and this this process of transferring the knowledge


of this model to here is called distillation with the lower model being the teacher model


if you do distillation you can actually improve your performance even more and they show that


in the experiments here especially again in the low in the low parameter regime but you


can see for example in squad here that the distilled movement pruned method now catches


up with the magnitude pruned method in the also in the high in the not so sparse regime


okay and they analyze these weights and as you can see that expected the magnitude pruned


method it will simply cut out anything right here that's not not a surprise whereas the


movement pruned method it will leave a lot of these weights alive so as you can as you


can see basically it's it's very much the case since you can outperform the red the


yellow one can outperform the red one it is almost warranted to say that the this magnitude


pruning wasn't the best choice it's actually a better choice to leave some of those weights


in and actually cut some of the weights that are large out just based on their movement


now the v here in the middle is of course due to the fact that if a weight is here it


was probably not super important in the first place and since since this thing removes anything


that moves towards zero any point starting let's say around here moving towards zero


would end up here so all the points that end up in this region probably moved towards zero


during training and therefore are cut away so there's there's not just like for there


to be points there would have been points that started even more in the middle and then


moved out to here right and there's just not as many so that's why you have the v shape


is very natural to expect right here so they they analyze this then in terms of the where


the model cuts the weights out now they experiment on the BERT base thing that which is a transformer


with 12 layers and if you don't know what BERT is you can go look at my video on BERT


but you can see that the magnitude pruning will sort of cut all the weights on the layers


equally so it will sort of go through the layers and take away let's say 90% of each


here you can see 10% of weights remaining whereas the movement pruning especially the


soft movement pruning will actually make a large difference it will remove much much


more of the later layers weights and keep the lower layer weights which I think if you


do transfer learning from these language models it tends to be that the lower layers maybe


pick up if you if you think of a CNN the lower layers might pick up you know on these essential


image features like corners and so on and the higher layers will pick up on the task


specific things now if you do like a big pre-training tasks you might have a lot of information


that you need there but then if you distill it and transfer it down to like a small set


small task where only a single thing is important like in squad it's only important what's the


answer to the question then you can probably remove a lot of that superfluous information


that was there like high-level features from the pre-training task I mean that's my my


guess here but they also have explained that so yeah that was this paper if you're still


here and you enjoyed it leave a like tell me in the comments what you think and I'll


BERTology Meets Biology: Interpreting Attention in Protein Language Models (Paper Explained)

Hi there. Today we'll look at Bertology meets Biology, Interpreting Attention in


Protein Language Models by Jesse Wigg, Ali Madani, Lav R Varshney, Kaiming Hsiang,


Richard Zucker and Nazneen Fatima Rajani. This paper is a investigative paper into


models that are trained on biological data, specifically into BERT models,


actually into a one specific BERT model that is trained on protein sequences. Now


it is trained to simply perform language modeling on these protein sequences, but


out of this language model you can then inspect this BERT model and read


important biological data of these proteins, higher order data from the


attention heads of the BERT model, which is pretty interesting. Basically means


that the information of these higher order functions is at some point encoded


in the structure of the language of the protein sequence. So we're going to go


through what this means and how this comes about and what they did in order


to investigate. I think this is a pretty cool investigative work and probably


very promising for future research. Yeah, as always if you like content like


this, consider sharing it out and leaving a like. Also tell me what you think in


the comments. So biology. Really quick for people who maybe have never heard


this. In your every cell you have this thing called DNA, which basically is a


an encoding of all of your biological functions. Now usually biological


functions are realized through proteins. So DNA is basically a building plan for


all of your proteins. This happens in the following two steps. First there is


this transcription step where RNA is built. This is basically a copy of your


DNA, but is only single strand as you can see right here. And then there is a


translation step that finally translates the RNA into the protein. What will end


up is just a sequence of these beads right here. Now these beads are what are


called amino acids. So a protein is simply a chain of these amino acids.


There are 20 different amino acids and the order of these amino acids in the


chain makes the function of the protein. Now specifically we know about these


proteins that it seems to be very important how their three-dimensional


shape is. So a lot of these different amino acids have different chemical


properties. Some are sort of I think negatively charged, some are neutral, some


are acids and so on. So they have very different chemical properties. So once


you build this protein and you kind of release it into the cell it will curl up


into a three-dimensional structure. So this one might be you know doing


something like this and sort of form a circle or something like this.


Just because these proteins here they kind of attract each other maybe


electrically and thus the protein forms a circle and the function of the protein


is very much related to its shape. So if it is a circle it could maybe you know


trap something else in here. So you really have to think of these things


like kind of tools. There are proteins that cut other proteins and they are


really shaped sort of like a scissor that exactly fits these other proteins


such that you can effectively cut them. So sometimes you can substitute an


amino acid for a different amino acid like this here. If it doesn't change the


shape very often you're fine. The protein function isn't changed.


But if you change a different amino acid that is sort of vital to the shape and


the shape changes then your protein very often loses function. So mutations in DNA


sometimes lead to mutations in protein. Not always because there is some


redundancy in this translation step from RNA. But if they do lead to a


different amino acid it doesn't actually mean that the function changes. So there


is sort of value in analyzing the sequence of the structure of proteins


rather than the structure of DNA. Of course it's also important to analyze


the structure of DNA but it is also it is equally important to analyze the


structure of proteins because they not all the information is in the sequence.


Not all the obvious information is in the sequence. So what does this paper do?


This paper goes and takes a model that has been trained on protein data. So if


you look at this protein it is simply a sequence of amino acids and these amino


acids they all have names. I think I have a table somewhere here. Yes. So these are


the different amino acids that exist and you can see a protein is simply a


sequence of these names. So usually they're abbreviated by like a three


letter abbreviation or just a one letter abbreviation. So a protein might be A, V,


M, M, V, A, G and so on. And this is just a string of text right? So what I can do


is I can train a language model on this. A language model is simply a model that


takes a piece of text and tells you what's the next piece of text. So what's


the next letter? What's the next word? In this case what's the next amino acid?


And we can use tools from NLP for that. Specifically we can train a BERT model.


Now BERT works a bit differently than a standard language model. BERT does


what is called masked language modeling. So you would take this string you would


feed it into a BERT model right here. And I've made an entire video on BERT if


you want to check that out. And what you'll do by inputting that you'll mask


out some of the tokens. So you'll maybe mask out this one, mask out this one and


then you ask the model to reconstruct those. You say that here's an M and here


is an A without seeing them. So the model somehow has to learn from the


surrounding amino acids what this amino acid could be right? So it has to


reconstruct this sequence. So the hope here is in natural language is that BERT


somehow learns something about language itself. By being able to reconstruct


these things it has learned something about language about which words appear


together and when. It might even learn very long distance relationships between


words just because it has to predict those. And the idea carries over to


biology. So we might hope that a BERT trained on an amino acid sequence will


learn something about the language of proteins about the amino


acid sequence. And our goal here is to be to ask can we somehow infer the 3D


shape of a protein which is the important part from its sequence right


here. So given its sequence can we infer the 3D shape? Now as I understand it


usually this has to be done in like a simulation. So you would you would build


this in a simulator and then you do like some sort of a molecule simulation to


see how this ends up in a 3D shape. You could train a model to just predict the


3D shape but in this case we're just interested what does the BERT model learn


about the 3D shape while only ever having been trained on predicting the


next or predicting the sequence of amino acids. So it's never been trained to to


look at the 3D shape. And that's our goal here. So specifically we'll look at two


different things. So here you can see examples of proteins and their high


level structure. So in in these proteins what you call the primary structure is


this sequence of amino acid. This is simply which amino acids are in in which


order. There is a thing called the secondary structures and we often


observe that spans of these amino acids like substrings form these what are


called these helixes as you can see here or these sheets. I don't know how they're


strands in English. We call them sheets or I think these are the alpha helixes


and these are the beta sheets and there is also a turns. I think this here might


be a turn. So there are these kind of secondary structures and then the


tertiary structure is how these this is still one this is one protein. This is


one unbroken chain of amino acid and you can see this here kind of forms this


double ring which would be its tertiary structure. Very important for


predicting the tertiary structure is to predict when two amino acids are close


to each other. So if we have a chain right here and the chain as we saw


before kind of turns and bends on itself then these two amino acids here are very


close in close contact and to predict which amino acids are in close contact


to each other helps you determine the the tertiary structure. So that's a


consequence of it. So we wonder does BERT know intrinsically which of these


amino acids are going to end up being in contact with each other without ever


having been trained to do it. The second thing we're interested in are binding


sites. So here, you might not be able to see, but we made this example before


where this sort of forms a loop and then I say can trap something here right like


another molecule and this is what what we would call a binding site. A binding


site is a one amino acid that maybe through the structure of the surrounding


amino acid as well but also through its properties and how it is exposed in 3D


shape acts as sort of a receptor for other molecules. It binds to other things.


It can, so think of your hemoglobin that that traps the the oxygen in your blood


or something like this. It is where a chemical reaction or a reaction with


something else will happen. That's a binding site and we are interested does


BERT, the BERT that is only trained on a language modeling objective know which


ones are the binding sites because you know that would be very interesting and


not something BERT was trained on. By the way, I particularly liked Richard


Soccer's tweet on this. I think he tweeted out, oh BERT trained only on


language model can predict binding sites and biological properties and formulated


it like it was you know like GPT-3 was formulated like if we train on Wikipedia


our model can do math. I thought it was kind of a satire headline. If we train on


Wikipedia our model can predict biology and also it can tie your shoes and cook


your dinner. Yeah but so it's trained on language modeling on biological data and


now that makes sense. So they're going to look at two different things or actually


more than two different things but they formulate this in an abstract way right


here. So what they'll look at is the so-called properties. The property F and


this property F can be for example that a amino acid is a binding site. The


property F can also be that two amino acids are in contact with each other. So


F always takes I and J. If in the case for example where this is the contact


property then it simply is the indicator function for when I and J are in


contact. And if it is a just a binding site then I think we are looking at J. At


the token level property we define to be an indicator that returns 1 if the


property is present in token J. Okay so whenever J is a binding site then that


holds. So what we're looking at are these attention heads in BERT. If you don't


know BERT has an attention mechanism which basically means from layer to


layer each token can attend to all other tokens. So here the amino acid sequence


I've drawn it twice and the next layer representation of this amino acid will


be able to gather information from all of the other amino acid through an


attention mechanism through a dynamic routing algorithm. I've made a video on


attention is all you need if you want to find out more how this works. Now what


we're interested in is the strength of these connections. So the hypothesis is


if molecule here is 1, 2, 3, 4, 5, and 6. If molecule 1 and 3 are contact sites


then maybe we will find a layer where this connection between 1 and 3 is very


strong. That would indicate that there is a connection site or that would


indicate that BERT has learned something about the connection sites. Okay if we


find this repeatedly so if we look at many many proteins and whenever we know


that there is a contact between two things and then we observe that the


corresponding attention is very high then we can be pretty sure that BERT has


learned something about contact between amino acids. Okay the same goes for


binding sites. So if 4 here is a binding site and then all the


connections all the attention that the higher layer gets from 4 so all the


information routed away from 4 is very strong that means all these other tokens


are paying special attention to the token number 4 to this amino acid. And


if we find that there is a big correlation with this being a binding site then we


can reasonably conclude that BERT has learned something about binding sites.


Alright so we're going to do a correlative analysis for proteins where


we know the binding sites where we know the contacts right we can analyze them


we can run simulations therefore we can know them. So we're going to look at this


quantity right here which is simply a normalized quantity. So we're going to


look at the attention in a given attention head so as you know BERT has


many layers with many attention heads and we're going to look at whether or


not this property is active and just normalize it by the total attention in


that head so that we get some kind of a percentage number. That's the first task


we're basically going to look at how does the attention correlate with these


properties. And the second task we're going to do is this probing task. So a


probing task is like a linear probe in like a classifier. So what we're going to


do is we're going to take a layer right here and even though it's an


intermediate layer we're simply going to run it through a linear classifier and


then decide is this a binding site for example or not. Is a given


amino acid a binding site or not? Is a given pair a contact or not? So this is


kind of a linear probe but this sort of takes a backseat in this paper. The


analysis is really on the attention heads and what the attention heads learn.


And that's already it. They take a pre-trained BERT model so there are


these BERT models that are already trained on these protein databases and


first they look simply can we find attention heads that correlate with a


given amino acid. So here you see the attention to the amino acid. This is


proline I believe and this is phenylalanine. Is that the same in


English? Yes, phenylalanine and proline right here. So you can see that


the plots here are there's almost no attention pretty much throughout the


network that pays special attention to the amino acid proline except this head


right here seems to have if you look at the scale over like a 70% of attention


always goes to proline in this particular head. So this is layer 1 head


number 11 focuses 78% of its attention on proline. Now this is not that special


if you think about it because in language models as well in natural


language models you might want to think that you have some mechanism in your


neural network that's especially specialized on like a very particular


word in the language because that might just be a often occurring very


particular word. For example in English maybe the is very important or the word


what these are like very indicative very often occurring words so it is


reasonable to expect to find an attention head that pays a lot of


attention to these things especially here where our vocabulary size is 20


instead of like 30,000 in natural language. And the same goes for this


phenylalanine where you can see that in the last layer and in the


first layer you have attention and also in the proline you have in the last


layer. So why does this make sense? Because what we would expect from like


single tokens these are not interactions yet these are not biological functions


yet. So we know that in the lower layers of a neural network we have these kind


of basic features basic feature extractors and here these basic feature


extractors appear to be simply paying attention to one specific token in the


vocabulary a lot. So they kind of these heads sort of specialize for single


for single amino acids. And the same in the last layer so in the very last layer


the task of the very last layer is to prepare for the classification task. So


if you remember the BERT model you have layer layer layer layer and at the


end you'll have to predict which ones are masked down here. So at the end you


have to predict single amino acids again so if there's a proline masked here


you'll have to predict the proline. So it also makes sense that the last layers


would very much specialize to single tokens. So this does make sense. Now our


question is going to be do we find the biological function where would you


expect them? We would expect the let's say the tertiary, sorry the secondary


structures which are sort of one level higher than the primary structures we


would expect to find them maybe here and then we would expect to find the


tertiary structures maybe somewhere here okay because these are most highest


level. And then it goes it goes back again or maybe it's like we find the


tertiary structures rather here and here again and then in the middle we'll find


the the most high level the tertiary structures or yeah blue secondary. This


drawing is getting too too too weird but there could be multiple scenarios but


that could fit here but until now it sort of makes sense. So they do an


additional investigation where as I told you sometimes you can substitute an


amino acid and nothing really happens right and in fact that this probably


happens in you right now you probably might have some mutation that changed


some amino acid and you don't even realize because it's just it's fine. No


notice. So the biologists can build these matrices of how much you can substitute


proteins with each other. So here you see this BLOSUM62 substitution scores which


are very I guess very high if you can substitute two protein, two amino acids


with each other and the effect is negligible. And it's very low if it's the


the other way around. Now this is interesting so far but you compare this


to this matrix right here. This is the attention similarity. So what we'll do is


for each two amino acids we take those two attention things, those two attention


matrices and we'll calculate the correlation between the attention


matrices. And our hypothesis is that the more correlated the attention patterns


are between the two amino acids the more likely we are to substitute them


because as a direct result of our language model our language model is


it's reconstructing right these things. So our language model is going to treat


if in natural language is like a a synonym right is our language model is


going to treat synonyms very similar to each other because they're synonyms they


can be exchanged. So a good language model should learn that they are almost


the same and therefore the attention pattern is going to be almost the same.


So a high correlation we hypothesize is a means that the function of the amino


acid is similar and therefore we can substitute it easily. So this here is the


matrix of the correlations between each two attention patterns of these amino


acid. And if you compare the two right here they look extremely similar. Just


have a look for a little while and you'll see that the patterns they do not


match perfectly but they are very very similar. The dark spots are in the same


places the light spots are in the same places. So this already makes a good case


that the language model here has learned something about biology. Now what we want


to do is we want to investigate higher order higher order functions. So here


we're interested in these contact maps right. So how likely is it that two


amino acids are in contact and we'll look at it through the lens of attention


as we did before. So here is the percentage of each head of each head's


attention that is aligned with contact maps averaged over a data set suggesting


that head 12 4 is uniquely specialized for contact prediction. So look at this


this this head here is just spiking. So remember before we said our analysis is


whenever whenever we're basically measuring the correlation of two things


being in contact because we know it from our simulator or from our data set the


correlation of that with an attention connection being particularly strong. And


we find it in this attention head right here. So this layer 12 head number four


will always peek out whenever two things are in contact. Now you can see that it's


it's not like always it's like 25 percent of its attention but significantly more


than anything else right here. In fact if you group the things by this attention


you can build the following plot. So you can see right here probability two


amino acids are in contact as a function of attention between the amino acids in


head 12 4 showing attention approximates perfectly calibrated estimator which


would be the green line. So here we simply for each for each pairs two


amino acids for each pair of amino acids we plot we make a histogram right here


of what they're sorry not a histogram we plot the probability if they have the


attention weight point nine we plot how likely is it that they are in contact. So


this is this if we just look at the data and we simply take this attention weight


as a measure as a predictor of being in contact we get the blue curve and the


green curve would be if we could perfectly predict from this attention


head what the probability of contact would be and you can see that the fit is


fairly good you can't predict with super high accuracy but the fit is fairly good


and you can see the general trend that as the attention in this head rises the


probability of the two amino acids being in contact with each other also rises. So


we can sort of confidently say that BERT has learned something about a higher


level biological structure just from the language modeling


objective. How can we interpret this? This must somehow mean that it is


possible it is vital for reconstructing the sequence from its


surrounding so if we delete this right here if these two are in


contact in the 3D structure that makes probably probably means that this thing


right here is a very good predictor of what was here. If we mask this out


and we're asked to reconstruct which amino acid was there then it probably


helps to look at its neighbors right it probably always helps to look at one's


neighbors especially also in natural language but if these two are in contact


they have very special connection to each other it's very


you can basically read out from this one which one this was. This is sort of like


if you have a sentence and you say


I don't know I can't come up with one right now but if it's like


and then there is a name like Mark and then and then there is him


right and you would expect if I drop out okay let's do it the other way around if


I drop out him then from the text right here you can probably determine that it


is some sort of pronoun but then you go back and you see it's Mark okay so it's


not like it's not like it or some or she it's probably he or him this is sort of


the analogous structure right here in biology. The second thing we're looking


at is these binding sites. Now these are single properties of different


amino acids and we're simply looking at all the incoming or sorry all the


other tokens that focuses their attention. Why is this important? Because


these binding sites are central to the structure of the or to the function of


the protein right if this here is a binding site then that's a very central


important point of the protein. So a lot of these other things are going to be


determined by what this binding site is. This binding site needs to have a very


particular function and therefore probably needs to be a very


particular amino acid and the other things here are sort of supporting this


binding site because they form the 3d structure around it and so on. So you


would expect a lot of attention to be put on this binding site and what do we


find? We find that it's a bit more murky than before so you can see that the


attention is kind of spread out. Percentage of each head's attention that


focuses on binding sites especially in the deeper layers binding sites are


targeted at much higher frequency than would occur by chance. Head 71 has the


highest percentage with 34%. So also here you can see that it is spread out but


this is because multiple heads are now focusing on these binding sites because


probably binding sites come in different variations so you'll have lots of heads


specializing on attending to binding sites and they say it is much higher


frequency than would occur by chance and you can see here this head is the


highest with 34% of its attention focused on binding sites. You can also


see the general trend of the attention being rather in the later layers which


we would expect from a tertiary structure. Now yeah it would be


interesting here. Here you also see that actually most of the things are in the


in the last layer which points to points to rather maybe lower lower level


information because we reasoned before about the last layer or I was just wrong


but also in a general trend you can see that the attention is rather shifted


towards the later layers because this is sort of a higher order function. Okay if


you look at the same calibration experiment you can see that the


picture is not as clear. There is the general trend at the beginning but then


it sort of flattens out so you can sort of differentiate the very probably not


a binding site from the somewhat probably a binding site but it's not a


perfectly calibrated classifier and that might just be because there are many


things specializing in different types of binding sites so you can't just go to


this one head so this is just for this one head. You can't just go to that one


and expect that to classify all the binding sites because you might want to


be you might want to combine all of the high ranking ones here to form a


classifier. The last experiment they do is these linear probes which where they


just go and they just build classifiers from different parts of the network and


you can see right here that what is predicted and how well they work. So each


bar here is going to be the difference of performance so this is differential


performance of diagnostic classifier by layer sorted by task order in figure 8.


Each plot shows the change in performance between the given layer and


the previous layer. So a bar up shows it's performing better than the


previous layer bar down shows it's performing worse than the previous layer.


So you see right here that these are the secondary structures right here and


you can see that there is a lot of performance in the earlier layers right


here and sort of not that high performance in the later layers whereas


for the tertiary structures the binding site and the contact you can see that


there is a bit of performance in places but it sort of tends to be more


towards the middle certainly more at towards the middle of the end of the


network than the secondary structures which sort of makes sense with our


hypothesis. You can also see this here where they show the percent of attention


focused as a function of layer and the red is the center of mass and you can


see that as the secondary structures this their center of mass is at a lower


layer in general than the tertiary functions. All of this is not perfect of


course but it's still an open question I guess whether or not it's not perfect


because we haven't built a strong enough language model yet. Do I want to say GPT-4


is now for biology and not for language or is it because there is really


you need you really can't very well predict these things just from a


language model. I mean you should technically all the information is there


but maybe the language model objective as such isn't able to capture that


information. So yeah this was the paper it's pretty simple they have in the


appendix they have a lot of these additional experiments or full


experiments I believe for all the amino acids and so on and I invite you to


check that out. In general I like this kind of work because it's very applied


it's and it can you know tell us something about the nature of both these


language models and the biological things that we that we care about in


biology. Okay I'm just talking crap right now thanks for being here I hope you


[News] Soccer AI FAILS and mixes up ball and referee's bald head.

So, there is this recording of the soccer match which is quite interesting because


the camera of the match is AI controlled which just means that it's programmed to


track the ball. Now it tracks the ball by visual features and what's funny about


this particular one is that the AI switches constantly between the ball and


the bald head of one of the referees which if you look at it looks exactly


alike especially in low resolution at which I guess the camera would operate


on. Yeah, if you haven't seen it go look at it is quite funny but it highlights a


more interesting point. Technology fails. Now this particular system it's probably


not very much AI, it's not very smart. I can guess that it's very standard kind


of feature extractor maybe something like a Huff transform with a few sift or


surf features here and there to look at the color things and kind of low level


information to track the ball is usually enough and it's probably more robust


than deep learning. Let's be honest here. But while this instance is funny a lot


of times when these systems fail they have bad or even catastrophic


consequences. Let's say a self driving car mixes up a head of a child


consequences can be quite grave. So I would like to put this to the sort of


people who advocate for having things like broader impact statements in papers


and saying that the entire AI research process should be filled with


considerations of ethics to the end application. We all agree that these


things can fail but let's take this particular instance right here. If this


system is trained at all, it's probably not trained on too many bald heads and


therefore simply mixes up the ball and the bald head because it looks almost


the same. Interestingly enough, this is one of the situations where the system


disproportionately often fails for white men but let's leave that out of the


picture for now. Where in this process exactly should someone step in and say


wait this is ethically concerning should the inventor of the Huff transform? I


don't know who that was maybe Alfred Huff? Paul Huff. Say ha you know if if my


system detects circles and images then obviously the negative consequences


could be that it mixes up a head with a ball. Interestingly enough the Wikipedia


page of the circle Huff transform says that it can be used to detect people's


heads. I just thought that was funny. Where in the process except at the end


when someone actually takes the technology and puts it into a camera that


person should consider the failure modes knowing what the technology is about to


go to the inventor of a circle detector and expect from them to predict kind of


these negative outcomes is ludicrous. I'm sorry try to write the broader impact


statement for the Huff transform doubt you would have come up with this failure


mode or anything similar to it if it hadn't actually happened. And you


shouldn't like circle detectors are useful and they sometimes fail and when


they fail we'll deal with it. After all even with the best broader impact


statement this wouldn't have been prevented. That was just my two cents go


The Future of AI is Self-Organizing and Self-Assembling (w/ Prof. Sebastian Risi)

Hey there, today I'm talking to Sebastian Risi, who is the director of the Creative AI Lab and


a co director of the robotics, evolution and art lab at the IT University of Copenhagen. He's also


the co founder of a company called model AI that uses AI for various aspects of game development.


Specifically, today we're going to talk about a blog post that Sebastian wrote that's called


the future of artificial intelligence is self organizing and self assembling. We're going to


talk about systems that have no supervised instance controlling everything but contain


little elements that all need to somehow communicate locally with their neighbors to


come to an agreement about the whole thing. Think of something like an anthill just organizing in


tiny parts to achieve a bigger goal. Now we've had massive success with these big supervised model,


essentially a central instance controlling everything and that works wonders for the


problems that we're currently solving. However, if you think of things like the most complex


organisms that ever existed, which is probably human society, at least as far as we know,


that is not supervised that has no central instance except the Illuminati but you know,


so essentially human society is self organizing and self assembling lots of little parts,


making decisions on their own communicating locally. And what emerges is this absolutely


beautiful thing. Now, as you can imagine, this is not mainstream self organizing and self assembling


systems and related things like open ended and lifelong learning. These are not the current hype


topics, but I believe strongly that they will be in the future. Things like this will play a big


role when we push beyond the limits that we are definitely going to hit when using supervised and


centrally controlled systems. Applications of this are numerous. I already mentioned things like


game development. In fact, a lot of Sebastian's experiments are in things like Minecraft and other


games just for visual, you know, oomph in their research. However, the applications are possibly


unbounded and could touch every area of AI and the greater field of technology. So join me this


interview was absolutely awesome. You should follow Sebastian and follow his research and


the research of his collaborators. Very, very interesting. I like it. It's out of the box.


It's new. It's creative. It pushes beyond what I know. That is it for me. We'll dive into the


interview. I'll see you around. Bye bye. Hello, everyone. Today, I have Sebastian Riese with me


who is a professor at in Copenhagen working in the general field of self organizing and self


assembling systems, which is, I think, an entire different world than the current paradigm that


we're used to. We're used to having our deep networks training them really top down with


supervised signal, sometimes self supervised. But I guess that's still kind of like a top down


supervision. There's gradient descent, there's all these things where essentially an outsider,


outside us human or some, some constraint is globally enforced. And there's an entirely


different world that goes much more along the lines of nature. And that tries to come up with


structure from from the bottom up. And that I find this really cool, and is really promising.


And I think it's sort of can solve problems that are really hard to tackle with these classical


algorithms. And I think the field is upcoming, even though it has existed for a long time. But I


believe that is definitely worth to look at. So today, we'll talk about a first and foremost,


this blog post, the future of artificial intelligence is self organizing and self


assembling, but also a bunch of other things in this field. So Sebastian, welcome. And thank you


so much for being here. Thanks a lot for the invitation. Very happy to be here. So why aren't


you working on just scaling deep learning more and more to bigger and bigger models? What's the


appeal of going like really small, really, really modular? Right. Yeah, I think there I mean, one


reason is there a lot of people working on or in this field. So I like to work on things where


they're, you know, there's, there's maybe not so many people working on it. And I find this field


particularly exciting. And we have seen that we can scale up deep learning, and it can do like


amazing things. But we have also seen that these systems still tend to be quite brittle. So we have


reinforcement learning agents that, that perform beyond human capabilities in some domains, but


then you add a single pixel in this kind of in the sock in this Atari breakout, and the system


completely fell down. And there are a lot of other examples, like image recognition examples, where


you slightly change an image or you rotate slightly. And instead of detecting a fire


bus, it's detecting something else. You have examples of Tesla driving into like an airplane


because it mistakes it for something else. So these systems are amazing at a lot of things,


but they're still very, very brittle in other tasks. And so that's why I'm particularly


interested in this kind of idea of collective systems and self-organization, because these


systems have this inherent kind of robustness. You can take away parts, you can add parts,


and the system will not completely break down because there is no central leader. It's like a


self-organizing process, a collective system, and that's what kind of fascinates me. And that's why


I'm more recently, we're going a lot in this direction. And it seems to be very fruitful


direction where there's a lot of interesting things to discover that we haven't really looked


at yet. I think as a motivating example, we can show this thing right here, which is a collection


of what are called swarm robots, or here it's called a robot swarm. Could you describe what


is happening right here? What are we looking at? Right. This is great work from Radhika Nakpal's


group, where basically they have these kilobots, a thousand of them, and they follow a specific


algorithm. And that allows these thousands of kilobots to assemble into a certain shape,


like those shapes we see here, like a star, a K, and I think this wrench. And this system shows


basically they only have very limited information, these kilobots. They can only basically see their


surroundings. But just by having this kind of local communication, these kilobots are able to,


over time, to assemble into different shapes. And so this was one of the seminal papers that


showed that you can run actually these kind of algorithms inspired by nature on a large scale,


on a large swarm of robots. And this is basically one great example of this.


What limited it is that those rules that those robots follow, like they have a specific plan,


they needed to be designed by humans. So it's a human-made algorithm, they follow it, and


you can compile it into making different shapes. But what we are more interested in is can we do


similar things, but can we instead learn these rules with recent deep learning, machine learning


methods basically, combining this deep learning with ideas from collective intelligence to


create even more complex structures, growing more complex structures.


This, I think, reminds a lot of people probably of something like ant colonies. Also maybe,


not necessarily evolution, but the development of just cellular organisms in general, where there's


not really, well, I'm going to step on some toes here, but an intelligent designer directing every


step of the process up there. Is it fair to say that these things you said inspired by nature,


is it fair to say that something like an ant colony implements one of these algorithms?


Yeah, exactly. So it's inspired by what you see in swarms of animals, of insects doing, like


ants, they are amazingly robust and they have this kind of collective intelligence that is bigger.


They are made out of simple units, but together they do these amazing things and termites,


they build these amazing structures. And so I think for this work, it's actually, I think it


was termites that was the main inspiration for this. And then you also have the same thing in,


the same kind of collective thing happens through morphogenesis. Like when we are grown basically


from one cell by division and local communication, it's growing these amazingly complex structures.


And both processes show that by very simple rules, you can get amazing things. And there are many


other examples. And one thing that these systems have in common is that you can remove parts and


it still kind of works, which is very different to our current like neural networks where you


change something slightly and oftentimes they will just break down.


I think, yeah, you demonstrate this later by training robots and then removing limbs of them


and they can still kind of adjust to it. And I think the arch example of these local rules you


have also in your blog post, which is this game of life, which is obviously, as you said, these are


hand designed rules still give rise to like a really complex set of phenomenon, which is,


I believe, even like undecidable to really decide from a starting point. I'm not sure about the


the lore behind Game of Life. Yeah, exactly. I mean, they're basically you can build any,


I mean, with this, it's a universal computer. Basically, you can build any kind of program


that you would want with the cellular automata. Of course, it would be like a super massive


cellular automata, but they, as you said, they show that even these kind of simple rules,


they give rise to things that replicate things that move across the screen. And so people have


found like all kinds of amazing structures by basically not changing the rules, but changing


the starting configuration of these kind of cellular automata. When we think about combining


this with deep learning, we quickly get to these neural, what are called neural cellular automata.


You have some examples right here. And I think I have the website open somewhere. This is work


that appeared in DistillPub, which is obviously cool interactive journal. So this, I think this


was one of the first even articles to appear out of Google. And so this here, I can maybe interact


with it, you can destroy parts of it, and it'll kind of regrow it. And all of this is happening


just by local interaction. So there is no, there's no kind of global organizing system that tells


these things what to do. But every single pixel in here essentially has a feature vector and


communicates with the neighbors. And how they communicate is, am I correct to say that the way


they communicate with each other, that is the part that is learned through deep learning? Exactly.


Yeah, you can imagine like you have basically a copy of the same neural network like running in


each cell. And that and that network takes into account like information from the neighbors,


the neighbor state, and then it decides what should the next state of that pixel basically be.


And you have these like RGB values, that's one thing it decides on. But then it also has these


additional channels, like hidden channels that it can basically, it can decide what kind of


information would be good to communicate to my neighbors. And so this work was not like the first


that used neural networks to learn rules for cell automata, but it really came


of revived the field. And what it did is that it showed that you can actually, you can make the


whole system differentiable. So we tried similar things before where we used evolution to optimize


neural networks, which is this field neuroevolution. But it's quite difficult for evolution if you have


a specific target in mind, like you want to grow the cell Amanda or you want to grow a certain


other structure. It's quite hard for evolution to learn these kind of supervised tasks. And then


basically this paper showed then if you have a target, you can just use recent tools like do


autodiff, differentiate to the whole system, and you can actually efficiently learn how to grow a


certain structure that is only grown through this local communication of cells. And that was one of


the, that I think revived like the whole field. And there's a lot more papers now using neural


networks for cell automata to grow all kinds of things, game levels, robots,


is is how do you train such a thing? You said the full thing is differentiable. And there is


a target in this case, right? Is it is it the is it the fact that you are in some starting state?


Do you let it evolve for a couple of steps and then kind of measure the loss and then do something


like back propagation through time? Yeah, exactly. Yeah. So you let it grow and it's


like, is it how close is it to the to the final output? And then it gives you the error to correct


it. And then they do all kinds of tricks like that. You want the system to be, of course,


robust that if I let it grow for 50 steps instead of like 20, I still want it to look like


like a salamander. So they do some kind of a few tricks that like doing the same thing


and letting grow for different amounts of time to get the system to be that it grows. And it also


kind of knows when to stop growing because that's an important part. Also, nature, like if I if


like if through morphogenesis, there's it grows an organ, it should it should know when to stop


growing that organ and like not grow forever. So that's one important ability of the system.


Just to learn kind of when to stop. If you were to, let's say, criticize this particular work,


what would what would your criticism be? What's still missing from this? Or where is it weak?


Yeah, so this what this showed is that it's basically it doesn't, if you would critique it,


you would you could say, well, it's not that bad. It's not that bad. It's not that bad.


You could say that it does not. But that was also not the goal. It doesn't discover the structure


itself. It has a target. So it has some kind of human design target, like the salamander that is


drawn by a human. And so in that case, that's one limitation. So actually one follow up work


that we will be published soon, we actually combined evolution and this system where


evolution we let evolution come up with like these soft robot in that case. And evolution is good at


discovering like variety of different morphologies. And then we use basically this method to make the


structure very robust. So we let evolution discover the structure and then we cut off all kinds of


limbs and let it regrow. So combining kind of the creativity of evolution with this kind of making


things robust through this gradient descent based training. That is the yeah, the work on soft robots.


I've seen that it just looks really cool. So this would be one thing that is that is discovered this


sort of kind of hopping tripod. And obviously this, I think soft robotics in general are


rather new field and combining them with like evolving system seems quite appropriate. So here's


one with a cut off limb and you can learn to regrow it. Right. How in general, how do you teach


a self organizing system to regrow things? Do you have to explicitly program like you have to


explicitly train it to regrow things? Or is this just a natural consequence out of how the system


was trained in the first place? Yeah. So sometimes it can often it already has some inherent


robustness, but it will without explicit training, it will probably not be able to do this like


perfectly. And it will be that it sometimes works and sometimes doesn't. So in these cases, we


explicitly and also in the case of the work by Google, like they explicitly like you explicitly


remove stuff during the training process so that you confront the system with, you know, this kind


of this damage that it has to recover from. So it is it makes the system more robust if you


specifically train for it. And I guess in nature, that's probably one reason that the system had to


work for all these different environments. So there is a lot of like they were in your aunt


colony. Sometimes you had more, sometimes you had less. And so these systems are because of the way


they were, they are evolved. They also show this kind of similar level of or like superior level


of robustness. At this point, are we already at the point where you would say that this surpasses


or this is very advantageous to classical deep learning? Or are we still in the realm where


let's say, everything would be fairly possible with classic supervised top down deep learning?


I think like this, it would be possible to have it grow and recover. But I think that the secret


here is that it only uses local communication basically. You could of course have a network that


would, I don't know, a network that you query that would could like similarly to earlier work like


compositional pattern producing networks, CPPNs, where you query basically each location in space


and you ask it what should the voxel be. And of course, these systems could then, if there's


damage, they could, you could ask them again and they could recover. But the trick here is that


it's only based on local communication. So if we ever want these things to work in the real world,


then it's really advantageous to have things that only require local communication basically.


And so that's one goal is that ultimately we want to take those systems from also the simulation


later on. And, you know, we have some initial work and we want to really create complex things also


in the physical world. If you say in the physical world, because if I think of, there was,


oh no, this was the paper, the physical cellular automata is at least a thing that is doable


in the real world. But if I think of something like, I don't know, a Tesla car, for example,


or something like this, that is in the real world, yet it is still, you know, a central controller


that controls the whole car, and there is still top down and so on. And it is also trained in


that way. What are the types of physical situations where these would really, the local communication


would really come in handy? Yeah, like I could imagine, like, let's say you have a building or


something that could automatically detect if it's damaged. And then, you know, it could, like our,


you know, our skin, it, you know, it's damaged and it's regrowing, it's self-healing. So you


could ultimately, I mean, this is like science fiction, but imagine a building and then you,


it gets damaged and then automatically it recognizes it's damaged and then it, you know,


automatically recovers from this damage. More other like science sci-fi, if you have imagine


you have a swarm of nanobots, they only can communicate locally, right? But they have to


figure out their shape, they have to figure out their, what they can do in an environment. So in


those situations, this local communication would be very advantageous. I don't know if it would


necessarily be useful for this kind of, you know, Tesla, this car example, but I could imagine a


lot of other like application areas or drones that have to coordinate somehow together only being


able to sense each other locally. So more these kind of in that areas. One thing I'm quite excited


about is just getting this from like this 2D version to a 3D version. And then you can imagine


building all kinds of things and it would automatically know you're building, you know,


a table or you're building a chair or you're building this and this, which I think it's quite.


So this is one example also of, so yeah, the self-classifying amnesty just


where basically the system cannot only be used to grow something, but it can also be used to


self infer its own shape. So you build something out of small components or you draw like a digit,


and then by having the cells communicate with each other, they figure out, oh, I'm part of an eight


or I'm part of a one. And so basically this is then what we replicated in this physical,


where you can put them together, make digits, and then each of these cells would tell, would figure


out what part, what shape am I part of. So this is a physical instantiation of the demo I have


here online. This is another Distill article where, as you exactly said, these things, they


figure out themselves what they're part of. And you made this, this is your paper, into a physical


instantiation, which I find really cool. And now you're taking it to 3D, right? Yeah, that's the


plan. And of course, currently these systems, like this kind of self classifying amnesty digits,


it does not work as well as like using like state of the art deep convolutional neural network or


transformer or what you have. But I think ultimately, these systems, maybe we can integrate


some ideas also for things like object detection to make these systems kind of more robust by having


a more kind of distributed object detection where you have this system where the components,


maybe it could be a combination of something convolutional. But then you have the system on top


where you have this local communication and they figure out together kind of what shape am I looking


at. And maybe that could make these systems also more robust in the future and maybe less thrown


to kind of this adversarial attacks that we currently see in the future.


Has anyone tried with like, maybe this would be interesting, like to take something like this,


and try to like make an adversarial? I don't even know how that would look like, but something that


a human would clearly classify as like a seven, but there's like a slight twist. Yeah. Yeah,


I'm not sure people have actually studied it so much on this. Trying to see how what


kind of adversarial attacks these systems could, I mean, fool like you could fool them. I'm sure there are also some,


but maybe the combination of kind of both this and more classic deep image recognition techniques


could make them more robust. So you've taken also this idea of this 2D


cellular automata and you've applied this in 3D here in Minecraft, which so this is a


morphogenesis. How do you how would you define morphogenesis? Just quickly. Yeah, I would define


morphogenesis as like growing a complex structure based also on this kind of local communication.


So how our like bodies are grown is morphogenesis. How our bodies are grown is morphogenesis.


How our like organs are grown, how our nervous systems is grown basically from like a single starting cell.


And so this is what we do here. And again, the structures are not found by the system itself.


Like we took like an existing apartment building and then we trained the system in the same supervised way


to regrow it basically. And we were surprised that it could also grow like these kind of functional


machines. We actually had it growing like this temple and then we found that the trap in this temple


still worked. So because it had all the components, like there was not one single mistake and that


allowed these kind of functional things to still work like this kind of like caterpillar you see there.


And can you also said you can destroy part of it and it will regrow?


Which, yeah, is this have you made this playable somewhere in Minecraft itself?


Or is this just purely your demos?


Yeah, currently it's not. I mean, you can download the code and stuff, but it's not that we have a server


where you can play with those things. But it would be very interesting. We actually, we organized this


Minecraft open endedness competition where we like a related field like can you have an


algorithm that can like natural evolution create all kinds of novel things without limits?


And that's also where we use this Minecraft framework. But it would be real fun.


Like one thing that I want to try to also pursue in the future. Imagine you don't have it grow like


caterpillars, but you have it grow like cities. And then depending on the environment that you as the


human decide, like the mountains or like the desert, it would grow a different type of


city. So like that's one thing we're looking at now. How can you incorporate also feedback back


into the algorithm? Because this caterpillar will always grow the same caterpillar. But if I put this


caterpillar in a small box, it should maybe grow a small caterpillar. And if it's a large box,


it should grow a large caterpillar. So how can you kind of incorporate this environmental feedback?


That's another thing that I'm very curious about.


Yeah, do you see beyond gaming maybe, which I can definitely see applications of this,


do you see applications that are not in the physical world as we talked before, but maybe


in the still in the realm of the digital world? Are there applications? I don't know what all


you're thinking of, but distributed applications, network applications,


any sort of things that you're very excited about that maybe aren't super obvious if you just see


the Minecraft examples. Right. I mean, one thing that we are basically, I think, like two things.


One is like just this Minecraft, I think, could also ultimately teach us something about


biology itself. So if we could, because we don't know everything yet about how this exact


morphogenesis process works in nature. I mean, we know a lot of things, but we don't know how


is it so accurate. So there are certain things that we don't know yet. And so by simulating


this process, like a very simplified model, but maybe there are things we can learn from these


kind of very simple models. So that's one area I'm also very excited about. And so taking these


systems as a very simplified models biology to learn something. The other application is


what I'm excited about is using those things. But instead of growing Minecraft structures, you can


grow actually artificial neural networks. So you're basically kind of replicating our brains are not


like designed and fixed. They are grown through this developmental process. So what we did with


this recent work, this hyper NCA is taken basically instead of having growing and


caterpillar, we grow a pattern and then we then we with a neural cell automata and then we convert


that pattern into a policy network. And that policy network then is we can use this for our RL


task, for example. So that's one area I'm very excited about and making this systems more


performant because currently we apply to quite a lot of different systems. So we can use this


systems more performant because currently we apply to quite simple problems. But I think


ultimately this kind of idea of this growing neural networks can be very powerful because


that's how our brains are created. So we're trying to replicate that process,


hoping to create also more adaptive neural networks.


What do I gain out of so in this here I have these developmental steps on the left I do,


I essentially start with some configuration of weights essentially, and then I let the


cellular automata run for a number of steps self organizing here, then I take it into a network


and then I execute the network and presumably, I have to learn this somehow in this paper,


what you are doing is you're using if I recall correctly, a variant of evolutionary


search. I could also like in whatever way I learn it, I somehow have to learn how the cellular


automata here reacts. What do I gain out of this instead of just training my policy network?


So far, I would say it's the you don't gain so much directly. So far, this method is not that


they outperform like current deep RL methods. But ultimately, basically, there is this


hypothesis also popularized more recently by Tony Zador, this kind of genomic bottleneck


hypothesis that means that we only have 20,000 genes and they guide the growth and self organization


of our brains with trillions of connections. And so it's a much smaller genotype that encodes a


much larger structure. And so this kind of compression is hypothesized to also allow us to


and animals to deal with situation they haven't seen, like to basically that the robustness that


animals show is part because they have to go through this bottleneck, this compression. And


this is the information you give to the next generation. So there's some limit on the information


you can get. So that might bias the system towards learning rules that generalize well,


like learning rules that generalize well. And so this is the hypothesis here that at the end of


the day, we can have a very small neural cell automata, which is basically like the genome and


that encodes a much larger network. And that hopefully would then be more robust. But that's


something we have. That's basically what we're working on, which we haven't really shown yet.


But that's the hypothesis and the hope. One other thing that's kind of funny that it can do, like


you can basically let the growth continue and not just have one network grown, but multiple networks.


So like we applied this to this quadruped domain. So we had it grow for 10 steps to grow one brain,


like one neural network. Then we put it into this quadruped. Then we have a slightly larger quadruped.


So we let it grow for a little longer and then put it in the middle. And then we have a little


more. So basically one NCA can grow multiple different neural networks.


And that's also one thing that I'm pretty excited about that we want to apply also for like more


complex domains. And again, here you had an experiment with where you damaged these quadrupeds


and the system is able to adjust. Can you explain how that works?


This system is able to adjust to a damaged morphology, like a cut off a limb or something.


So here it was basically trained to on these, like on all these different morphologies.


And then we had it basically by continuing the growth, you can get a controller that was trained


for one morphology and then you continue it and you get a controller that works for M2 and you


let it grow a little longer and then you can actually put it into the next domain.


And you let it grow a little longer and it has a morphology for M3.


So in this case, those were basically seen during some other experiments.


We have results where it has damage that was not seen during training. Here it basically was trained


to being able to deal with this particular type. So if we would damage it in another way,


it probably wouldn't work anymore with these metamorphosis networks.


But yeah, so the hope is also that if you know how to control one quadruped,


then there should be that you don't have to start basically from scratch. There should be some


information there that allows you to also grow something that is related and not having to start


like all over again, basically. This flows, I think, into a lot of ideas from, as you said,


the open-ended community and the sort of don't have explicit goals community. I think parts of


your blog posts and papers mentioned algorithms like quality, diversity, map elites and things


like this, which are obviously very exciting and very different from how we do deep learning today.


So far, we've always looked at things that have either an explicit goal, like here is the


salamander I want to build, or here is the Minecraft structure I want to build, or have some sort of,


I want to say, goal in a more abstract sense, like the reinforcement learning goal of maximizing the


height in this case, right, for these robots that stand on top of one another. Yet, how do we go


away from this? Is there a natural progression in these self-organizing systems to go away from


having explicit goals that would be more difficult to pursue with the classic deep learning systems?


Right, I think in general, so I think that like two things, like one is the


representation, which I think these neural cell automata are like a great representation for a lot


of like growing structures, growing neural networks. And then, yeah, the other thing, as you


mentioned, is like the search. How do we actually get to systems that show


interesting, these interesting properties? And so there seems to be a recent trend, I mean, not just


in these self-organizing systems, but also in deep RL in general, to not train on one thing,


basically, but train on a variety of different things. So there was also this more recent paper


by, I think it was DeepMind, this XLN that they showed, like basically, if you train agents in a


lot of different changing environments, they develop more robust skills, basically.


So I think, basically, here it's, we also, what I think it makes these self-organizing


systems quite difficult to train is that these landscapes, the fitness landscapes, basically,


they are probably very kind of, not very smooth, because changing like something small in these


self-organizing systems can have like this cascading effect. So that's why these traditional


objective-based rewards, they work, but then they don't, it's still difficult to optimize.


So that's why we're more looking into this kind of open-ended, like what you mentioned, quality


diversity methods, basically, where we're not trying to optimize for one particular outcome,


but we're trying to find things that differ in some interesting ways, basically. And I think


those methods, particularly for this kind of self-organization, they are very powerful,


basically. They are better at navigating like these kind of very complex landscapes with many local


optima, but they're also slightly more expensive because they're looking at the larger space


of the search space, basically.


What, maybe these two questions in one, given these outlooks, what field that deep learning


is good at right now? Do you expect these methods to be better if, you know, let's say if we invest


the resources and figure out, you know, the tricks of the trade enough, what parts that deep learning


is good at now, could these methods overtake deep learning? And then on the other hand, what's kind


of the, for you, the most exciting area that we haven't even unlocked yet with deep learning that


are accessible with this? Like, it's two different things, but I'm wondering about what you think


about both of these directions.


Right. So I think it's also, I wouldn't say like overtake deep learning. I mean, we use basically


deep learning as a tool for basically like kind of training systems.


Sorry, I mean deep learning and like that, just the thing we do right now, right?


We have objective, loss, supervised training, single neural network.


So I would assume that these systems would be able to have a lot of different domains.


I think the one kind of probably the closest, I think what we would see is that they would


make our IELTS agents more, you know, like more robust, more adaptive. And that's also already in


this work that we have there is like where we have basically, in this case, we trained not only the,


we had completely random weights and we only trained local update rules, basically the habit rules.


And then we show that through this system, we can actually during the lifetime cut off a leg.


Again, we are always somehow mutilating these robots. We're not very nice to them.


But basically, this is an example, I think, where we already show that this is more adaptive than the current RL design.


So in the current basically deep RL, I think the one main drawback is that we train a system


and then we freeze the neural network and then let it do its task. And this seems like kind of very unnatural


that like you have a frozen brain. Okay, maybe you have like some recurrent connection that allow you to learn something.


But basically, we have this training period, then we freeze everything in the system and we apply it to domains.


So that's not like lifetime learning in normally these systems. But the idea here is,


in general self-organization, that we never wanted to stop learning. We never wanted to stop adapting.


We want the self-organizing process to happen the whole time. So I think in any domain


where there are things that you might not have anticipated during test time, these systems could be beneficial.


Like might it be there's a pixel edit, you're losing a leg or you wanted to do something else.


I think that they already show that there's some... they can be superior in those domains.


And that's one thing that I'm pretty excited about. To apply them to more complicated domains,


not just these like quadruped locomotion tasks basically. But anything where you have something unanticipated happening,


I think there can be a benefit of it. And then was the second question...


A new area that we haven't even... like we have no chance currently of tackling with our tools.


Yeah, that's a great question. I mean, I think this new area is this kind of rapid lifetime adaptation basically.


I think these systems are great for if you know what you would expect. But things like basically


having things that work in unknown environments. I think that's a really exciting area.


I mean, you have like animals in nature and you can put a dog into a new environment and


it will not completely break down. It will still know kind of what to do and to interact with the environment.


And we don't have that yet for our agents. We can put them in environments they're trained for.


You put them too far out, they don't know what to do. And I think that too...


So this working out on environments and also having this kind of common sense,


I think is maybe also an area I think in the future that these systems could be applied to.


Although I don't know exactly how. But that these systems have more common sense and don't directly


break down. Like kind of giving them this kind of innate abilities that we humans are born with,


some animals are born with, that allows them to do a little bit more common sense things


than current deep learning systems that don't have that property basically.


And this, I think you say it even here at some point, this in addition to the fact that there is this genomic bottleneck, right?


You already said this, the genes encode or only have the capacity to encode very little information.


And what we're doing here is we're learning essentially the rules to learn the rules,


which can be compressed in a much better way than the rules themselves.


And there is a reason to assume that this will result in that kind of common sense,


that if you have to essentially learn the meta rule, then that will make you generalize better.


I mean, it's an argument. I'm not super convinced yet. But if you do then some parameter sharing,


you show in some experiments, you can compress this even further. So that might be a way to tackle that.


And also this in Tony Zador's paper, he actually points out that this bottleneck,


like there's some organisms in nature that have many more genes, for example.


So maybe it is a feature that we have that number of genes that it's compressed.


And so that gives us some hope that also having a similar feature in our artificial systems should be beneficial.


But we only showed that for very, very simple tasks so far.


And deep learning goes into the exact opposite directions, right?


Where the more parameters, the better. We have the double descent phenomenon and we can go essentially infinite.


And it always gets better, which is weird, right?


Which is also giving amazing results, I think, recently with the whole language models and so on.


So it's definitely it could it would be cool if in the near future people discover like a fundamental connection between,


you know, the good results we get by scaling up and the actual principle from biology,


which seems to be more like compressing and scaling down. It would be nice if those were to join together somehow.


And hopefully we can be part of that and contribute it to some extent.


But I agree, it's really interesting that like that you scale up networks and then your local optima disappear,


like everything just works better. And here we basically we want to go the opposite direction.


But it's not necessarily that we of course, we still want the final models to have trillions of connections.


But what we basically want is we want the trainable parameters to be low.


And I think that that's the fundamental difference that we have a small number of relatively small number of trainable parameters.


But they give rise to much more complicated system, exploiting things like self-organization growth over time.


This is, I think, because you said before, you're not you're not an opponent of deep learning.


In fact, deep learning is used inside of the cellular automata to sort of learn these rules.


I find it interesting if you look in nature, that there are cells and they self-organize in some way, right?


By whatever means that is learned. But these cells then make up brains, right?


And brains are naturally very top-down planners. They're in the moment, they look ahead.


And then the brains somehow organize into societies and the societies again are very distributed,


very local, very interaction on a person-to-person level.


What do you make of this? Do you think there is like an optimal switch from local to global to local to global


that we could sort of stack on top of one another? Or is this just a happenstance of the universe?


Yeah, that's a great question.


And even more, the humans in the societies, they organize themselves into hierarchies, right?


There's top-down control and somehow it gets even crazier.


It's a good question. Do we need all of this in our artificial systems?


Maybe we need all of this to get to real, more general artificial intelligence.


Because also one thing that is really crucial is our culture, right?


I was reading this great book recently. If you just put humans somewhere by themselves,


they are not very good at surviving. But we are good at surviving because we have all this cultural information,


all this knowledge that other people made that we can build on. And that allows us to do all these amazing things.


So maybe to get our AIs to do really amazing things, it's not enough to having single agents in complex environments.


But it needs to be multiple agents. They need to be simulated maybe over multiple generations.


So there can be some cultural knowledge transferred from some agents to other agents, similarly to how it happens for us.


But of course, that also makes the simulations much more complex and expensive.


When you have to simulate cultures multiple generations, then we need some more better compute, especially at the university level.


I think that's one advantage that nature has. It has lots of distributed compute available.


That said, there is an interesting part in your blog post where you describe how to train these things,


or how to steer the development of these swarm systems or distributed systems.


One quote here you have is, guiding a swarm system can only be done as a shepherd would drive a herd by applying force at crucial leverage points.


By subverting the natural tendencies of the system.


And then another one is, the self-assembling brain knows no shortcuts.


In which I believe your argument was a little bit that it's very hard to predict what a change does until you observe it.


Because the interactions can be kind of nonlinear, very dynamic, very hard to predict.


That was basically the argument that Hissinger made in his great book, Self-Assembling Brain.


Basically, the system needs this process of growth and you have to put energy into it to observe the outcome.


You cannot predict, and that's also things they showed, Wolfram what he showed with simple 1D cell automata.


You cannot predict the state of the system. You have to actually run the system, even if it's a simple 1D cell automata.


And that is also apparently, the question is, do we also need to do that to growing our neural networks?


Instead of designing them, maybe we need to go through this kind of process of growth with learned rules to really unlock what these systems can do.


There is recent work in using, for example, GANs or so to predict things like fluid dynamics.


They're not extremely accurate, but they can give a pretty good estimate of a given starting state and then a highly dynamic nonlinear system.


And then they can predict some steps into the future. I've seen the same galaxy development and so on.


Is there any happening like this where you can say, well, I don't have enough computer to run all these swarms.


But I can sort of train a surrogate model that will give me the end in sort of a one step fashion.


And then these forces that I poke at the swarm at, I could determine those using the surrogate model.


Yeah, I think that that would be really interesting. I think it could work for some limited steps in the future.


But I think you would still need to, at some point, you need to basically run this model.


I mean, maybe in the first generations you could help to have a surrogate model that somehow helps you to sort out the things that are really bad.