Like this will not grow into anything. So I think you could use it there.


The Future of AI is Self-Organizing and Self-Assembling (w/ Prof. Sebastian Risi)

Later, I guess you would probably have to run the system when things get more complex.


But I think there's also another role for these surrogate models, which is something I always wanted to try to predict.


Basically, the learning abilities of these systems. So you have an agent and an environment.


So maybe you don't need to simulate the whole lifetime. But you can have some kind of some test that would test how capable is this agent.


So having some kind of surrogate that could look at certain parts of the neural network and already predict, will this be a good learner or not?


Basically, but yeah.


It is in one part, you also, it has very, very, remember, like I got into machine learning and graphical models were the hot thing at that point.


It was just before deep learning. And this reminds me all this self-organizing systems with the local communication.


They remind me a lot of belief propagation, things like this. Graph neural networks obviously are right now up and coming, let's say.


Do you see connections between all of those things or is that just kind of a superficial connection?


Yeah, I definitely see there's a big connection to these also these graph neural networks, basically.


Like, I mean, they're very close to like a more generalized form, basically, of like a cellular automata where you have different basically neighborhoods.


Depending on your topology of the graph. And they also seem to be, I think they're super interesting.


Also, actually, how I got into neural networks is the first lecture I had as an undergrad was actually on neural networks and about these self-organizing maps.


Which these co-honen self-organizing maps that basically can do clustering based on some are like kind of like K-means.


But on a much more, they can do it better. And you have to get these like nice visualizations out of them.


And apparently, there's also some process in our brain. I mean, we have these topographic maps also in our brains.


I was always fascinated somehow by these self-organizing maps. And even though I did a lot of like some other things during my PhD, somehow now I'm coming back to this kind of self-organization.


And using these recent deep learning tools, I think we can really unlock like the power behind them.


There was a, do you know the ARC challenge? The Abstract Reasoning Corpus by Francois Chauvet?


Yeah, yeah.


There is, I'm not sure if they have an example right here. So for everyone who doesn't know this, this is a task where you get, so the left ones are demonstration examples.


There is always like an input grid and an output grid. And then you get a test example where you only get the input.


So here, the rule is, I've looked at that before. So the rule is kind of there is the gray in the middle and you kind of fold the right hand side onto the left hand side.


And then you, the solution here on the right hand side is kind of the sum of the two.


And this is, these are things that humans are surprisingly good at, but are very difficult for a machine to learn.


And this is a data set and the training examples, there are not many training examples.


So there is not really a way to learn this through brute force training.


There is a little game that people can play. I think I've reported on this before, but there is a game for anyone who's interested where this is the Ark game.


You can find it on the GitHub page of Alexei Borsky.


And you can just choose one here. They're divided into different levels and yeah, you can try them for yourself.


So, well, this looks even familiar like cellular automata.


Do you think that it like self-organizing systems in one way or another in the way we've looked at them today or in the way you've seen them could be useful in solving challenges like these?


Because challenges like these are related very much to, let's say, something that we would call intelligence.


Yeah, I think the hope would be that if we can get this kind of bottleneck algorithms to work where we exploit.


So I'm not sure we could apply self-organization directly, but what I could imagine is that we develop these kind of genomic bottleneck algorithms that can guide this self-organization growth of a very complex node network.


And that network then could maybe be used for this kind of task.


And the hope would be that because it has this compression, it would maybe develop an algorithm that would allow it to solve these kind of tasks that require more like high-level cognitive skills.


But of course, that's still, yeah, we're still a little far away from that, I think.


And I guess I don't know what the current state of the art in this task is.


I think it's still largely unsolved.


So this could be a great test domain, I think.


But yeah, I think I'm not sure I have high hopes that it would already.


I think we're still probably missing some other ingredients that we don't have yet to kind of make progress there.


Yeah, but by the way, I just clicked on one randomly, but I think here the rule is, I think if people get it, they can see that you always kind of select the smallest of the shapes that is there and kind of replicate it.


At least that's my hypothesis, right?


Yeah, maybe.


Oh, I think maybe you take the one that fits in the box.


Oh, yeah, yeah, yeah, right.


But it's like this kind of like you need to understand what shapes are and so on.


So it is very much that this is very high level.


This is very bottlenecky.


It has a bottlenecky feel to it.


Like you're probably not going to get very far with like a CNN trained on these pixels directly.


So that's I can see something like this very much be in the domain of like first open endedness, but then also self-organizing things made up like simple rules, making up something very complicated.


There's two other domains that I think also are exciting.


Like one is this animal AI benchmark where basically they it's like an animal AI Olympics where you apply AIs to tasks that animals normally are good at.


Like and like, for example, trying to figure out which one is the tool and then you use that tool to get a reward.


And so this is also where current methods basically they've pretty much failed on more complicated tasks.


And then they also had a mid-term experiments where they had children perform these tasks and they are still much better than like any of our deep RL methods.


So in the simple task deep RL performs pretty well.


Once it gets to more complicated things, then the system basically these systems fail.


So this is one task that like in the recent grant proposal that I proposed that there would be a good test domain for these methods basically.


Because the whole point is to act in an environment that you haven't seen during training.


Even though the environment is made out of the same building blocks, like there's rewards, there's like barriers.


But how they are composed, all of this is new basically and never seen before.


And the other one is this also by I think was DeepMind this alchemy task where you have to learn to kind of.


It's a task that we have to learn basically about the structure of the domain, what things you can put together.


And then you have to use that knowledge to like building on that knowledge basically.


And this is also a very difficult task for all of our current methods.


So I think this could also be very good task to basically as the North Star to drive these the progress in this kind of area.


And the hope is that these kind of self-organizing system, they should be, hopefully would be better at this.


Where can people, if someone wants to get started in diving into the world of self-organizing systems, swarm intelligence, maybe a bit of open-endedness.


Is there a good place for people to get started, like get their feet wet?


Yeah, I would say I was recently rereading this great book from Melanie Mitchell, this complexity.


I think this is a great starting book on kind of these ideas of complex system self-organization.


There's something about cellular automata in there.


So I think this is a good kind of good point to get a broader overview of that kind of whole field of basically complex systems of organization.


And hopefully also the blog post hopefully can be helpful to some people and also plan to write more on that as well.


But this I would suggest this is a this is definitely a good place to start.


And is there some, you know, in deep learning, it's usually Keras, I train a CNN on MNIST or CIFAR-10.


Is there like some standard thing that every one of your students goes through?


Now I send a lot of them to this great Distill article basically and looking at this growing NCA's because they also have a great like this Colab notebook where you can play with the system.


So I think this is a great starting point to where you both have neural, like you have cellular automata and you have like how recent tools can be used to grow them.


So I think this is a good place to play around with basically.


OK, yeah, I've I've spent more than more than more time than I want on these things because they're quite great.


It's great that it's also so interactive and fun to play with.


Yes, definitely. Yeah, I think is there anything else that you would like to get out there to people about this field?


Yeah, I just hope that people would be not only everybody running basically in the same direction, just doing like what everybody else is doing.


So hopefully this will be also get a few more people into this field of complex systems and self-organizing systems and combining the ideas of deep learning.


Because I think there's a lot of things, interesting things to discover basically here and a little bit less people working on it than than the like.


Like working on foundation models and language models and all those other things.


Yeah, it's certainly I think I think is certainly an interesting area.


And I guess especially if you're at a university without the super duper clusters, probably just strategically a PhD in this field would maybe be more of a advantageous position for new newcomers to the field.


Actually, like Hinton had this great quote recently on this other podcast, like it's always a good idea to figure out what huge numbers of very smart people are working on and to work on something else.


Because you don't want to do maybe what everybody else is doing.


And I think so. I would suggest this is a great field where a lot of I think interesting discoveries basically waiting to happen.


I agree. All right. So, Sebastian, thank you very much for being here today. This was very cool.


Yeah, thanks a lot for the invite.


I hope to see. Yeah, I hope to see a sprawling future for your field.


Kivy with Python tutorial Part 5 - Dynamic Resizable Placement

What is going on everybody welcome to the fifth KV basics with Python tutorial video in this video?


we're going to just be slowly adding on the last video, so


What I'm going to do is I'm going to call this now simple KV 3


3 save that and my referenced KV file will be simple KV 3


3 here


Okay, so let's just run that make sure I did everything correctly I did fancy that okay, so


like I was saying before


What if for example you've got?


Let's do


Stick button here, so we've got widgets button, and then you've got we see here. We have a


Font size 40 font size 40 well what if we did this like font size 40 okay?


That's fine, so I'm just going to cut this and now we'll come here and let's say


Button as the parent colon one two three four paste font size 40 whoops


Save that let's run that now


And you can see now we've got size 40 on both buttons, but we didn't have to specify that


Because we're using now a parent button so we could do the same thing again


Let's say


Size will cut that


whoops


Paste that position needs to be you know custom and then you know color could be the same as well I


Like green better, so we're going to do green


And we'll get rid of color and we'll get rid of size so now we just have a position in text for example


So save that and we can run this


They've run into each other. I guess it would be the position


Let's go back over here. I don't know what I was thinking yeah, okay, so so position this one needs to be 170


save there we go so Keevy tutorials fancy okay, so we can kind of see now hopefully they


Were the power of these Keevy?


The key using the Keevy language can come in is where we have kind of like these global variables


And of course you can you can kind of do this within the python script as well


So it would be a little messier than doing it this way, but you can you can do that?


And so a lot of it is just going to depend how much dynamic code


Do you want how much kind of structure do you want and all this kind of stuff so anyways?


That's that the only other thing. I'd like to do now is let's talk about position, and we're going to talk about a little bit of


relative positioning so for example what we could do is


We've seen a size. That's totally fine


Size though size could be a bit of a function as well, but what we're going to do position for now


So let's say we wanted to position something


near the top


So how might we do that well let's do with Keevy and let's say


Let's say we want it


Position will do root dot x


comma


roots dot top


And this is then we have to do minus self


dot


height


Okay, let's save that run that make sure that works, and then I'll explain what's happening


okay, so we've got Keevy now right at the top and


Keevy will or should at least remain at the top as we resize the window and sure enough it does and the window has changed


Obviously we can also see tutorials


Moves as well, but anyways closing out of this


Let's explain what's happened here


Okay, so what's happening here is you've got root dot x that's referencing the x of the root right so the root widget


Which would or the root app rather which would be simple Keevy you know app right? That's our root class


and then root dot top is referencing the top of that app and then


self dot height is referencing the self of


Widgets right self is that specific widget and we're referencing the height like I was saying before


So 0 0 is the bottom left so if you positioned a button at 0


0 or let's say at let's say the window is a hundred pixels tall and you position the element at 0


100 that is the top left corner of that window


But then the button is drawn above that point so the button would be off the screen so we use minus self dot height


To move that buttons you know bottom left down


The height of the button which is actually referencing here the 75 so anyway


That's how that's what's going on there, so that's a little more


Dynamic of code and also kind of shows what I was


Telling you before that you can put some Python logic on the other side of these colons


So anyways that's going to conclude this this tutorial video if you guys have any questions or comments


Words as Features for Learning - Natural Language Processing With Python and NLTK p.12

What is going on everybody welcome to part 12 of our Python with NLTK for Natural Language


Processing tutorial video series.


In this video we're going to be building on the last video which is kind of starting up


the processing for our Naive Bayes algorithm for classifying text as being either positive


or negative using the movie reviews data set.


So moving along we've got all the way up to this point where we basically get all of the


words in the movie reviews data set and then we've also got some documents that we can


use for training.


So let's go ahead and I've commented out these two things.


I'm actually just going to remove them and I'm going to remove this too.


And now we're going to go ahead and get started down here.


So first of all we need, we want to have some sort of limit on the amount of words.


So right now we have this frequency distribution of all words and the frequency distribution


is ordered from the most common word to the least common word and the amount.


It's like a keys and values basically and so you've got, it's a huge, huge amount of


words and we don't actually, we could train against all the words but it would probably


be relatively useless.


So we want to kind of limit this a little bit.


So we're going to say word underscore features equals the list of all underscore words dot


keys, we're not really focused too much on the values, like how many there are and we're


just going to say we want to use up to the top 3000 words that we're going to check against.


So if you remember in the last video we did the top 15, well the top 15 words were kind


of meh because a lot of them were empty words like the, a, and a lot of stuff was like punctuation,


like periods, commas, and dashes.


So we obviously don't really want that stuff included but this will be good enough to encompass


a lot of words that are commonly used and then we can train against these top 3000 words,


finding out which word is most common in negative and which word is most common in positive


reviews.


So that'll be the word features.


Now we actually want to have some sort of quick function that is going to find those


features within the document that we're using.


So let's come on down here and now we're going to say so define find features and we're going


to find these features by searching in a document and that document is going to be part of these


documents soon and we'll pass them through.


So within the document it's going to find features and we're going to say words equals


the set of document.


So the document if you recall is the just going to be the first part of the tuple is


just a list of words so when we convert something a list to a set we get only one basically


iteration of any unique element so we don't have many iterations we just this is basically


all the words not the amount of those words but every single word will be included in


this set of words.


So we've got that now we're going to say features equals and this will be just an empty dictionary


for now and we'll say for w in word underscore features so that's for the word in these top


three thousand words we're going to say in this features dictionary the key that is the


word in these three thousand most popular words is going to be equal to the boolean


value for w in words.


So this is this creates a boolean of either true or false so if the word if one of these


top three thousand words is within this document that this will ring true if it's not this


will ring false.


So that's how we find those features and then at the very very end of this after we've run


that we're just going to return features.


So now let's test this new function and we're going to do something like this like print


and then we can do we want to print the result of font not fund find features of and we want


to do this for movie underscore reviews dot words of the following let's search in the


negative data set and the first one is CV 000 underscore 29416 dot txt.


So then we'll do that and then we'll call something we'll say feature feature underscore


I'll call it feature sets like this equals and this will be a list and then it will be


find features for the roof for the various review whatever we're putting through and


the category category for the review and category in documents.


So this one liner makes a little bit more sense to use for me but basically the feature


set is going to be the find features and the categories we're kind of we're going to end


up converting and in fact let me run this one talking since this kind of takes a while


sometimes so we're going to end up doing is converting the data so each status set of


again each document is going to be the review on just the words and then the category and


what this is going to do for us is it's going to convert that to not a review but find features


so it'll convert before it was just the words now it'll be the words with true or false


so it'll be a dictionary of the top 3000 words and whether or not they're contained in each


review and then the categories so now what we can do is using these top 3000 words this


is starting to build what we're going to be able to test against and train against is


the existence of words and their impact on whether or not something is positive or negative.


So now that we have that let's kind of pop over here so you can see what I mean so feature


set so first we printed the find features so find features movie reviews you that this


is doing what this little bit right here will do followed by category so this has a contain


you know contains basically all the words let's see if we can find a true most of these


are false like it doesn't contain any of these words let's just search for a true see if


it has one okay so it does have one up here and it's the word is before so before is one


of the top 3000 most common words this contains it and this is a negative review so you might


actually find that before is far more common to appear in negative reviews as we go on


but we'll see it may or may not be but anyway we can close this out now and we'll go ahead


and cut off the review or the other review the tutorial at this point and we'll continue


building on the algorithm in the next review we'll actually probably be able to build the


tech or the training and the testing maybe just the training but we'll probably be able


to get to that point in the next tutorial so stay tuned for that if you have any questions


or comments up to this point we kind of ran through a few things pretty quick but hopefully


I explained it pretty well if you have any questions or comments up to this point please


feel free to leave them below otherwise as always thanks for watching thanks for all


Sparse is Enough in Scaling Transformers (aka Terraformer) | ML Research Paper Explained

Hello there, today we'll look at sparse is enough in scaling transformers by researchers of the


University of Warsaw, Google Research and OpenAI. This paper on a high level proposes a set of


building blocks to introduce sparsity into transformers. And this results in an architecture


called the scaling transformer. In the second half of the paper, they then introduce additional


features to the scaling transformer to make it into the terraformer. Both the scaling transformer


and the terraformer, they're really fast at what they call unbatched decoding. Decoding is


essentially inference in such a transformer model. And unbatched means that they can do this for


single sample. Of course, they're also faster in batch decoding, but I guess the effects are not


as pronounced. And we're going to see why because the sparsity really shines through if you have


single examples and can only activate very small parts of the network at the same time.


So the effect of all of this, at least for the scaling transformer is right here. If you have


a model with 800 million parameters, I guess today that be called a small model. The baseline


transformer has a decoding time of about point one, six seconds. Whereas if you add all the tricks to


the scaling transformer, you speed that up by a factor of about 2.6x. That's not that pronounced


yet. Yet the effect really shines if you go to bigger models. So if you go to 17 billion parameter


models, the baseline transformer takes about 3.6 seconds on this particular hardware to decode


the Terra, no, sorry, the scaling transformer with all the tricks activated takes about point


one, eight seconds, giving a speed up of 20x. And so in different settings on different


configurations, the speed ups can in fact get even higher, I've seen up to like 37x or something


like this, which is quite quite fast. And this all while the performance doesn't degrade, and that is


surprising. So they say surprisingly, the sparse layers are enough to obtain the same perplexity


as the standard transformer with the same number of parameters. So they have the same number of


parameters. It's just that they activate them sparsely when forward propagating, which is much


faster and needs much less memory. And this results in the same perplexity when language modeling. So


essentially means that the performance is on par. And also they say, if they if they integrate with


prior sparsity approaches, that's where they achieve the terraformer. They can do fast inference


on long sequence even with limited memory. This results in performance competitive to the state


of the art on long text summarization, which is another thing where their model is state of the


art or equivalent to state of the art, while being much more sparse, much more memory efficient and


much faster. So yeah, we'll dive into this the architecture it's quite is quite a mess. Like


there are engineering tricks, engineering tricks, engineering tricks. And, you know, the the,


you have to wonder a little bit, you know, what came first, like which trick came first,


and which trick necessitated which other trick, but we'll go through the architecture, through


all the different pieces. And you'll see what this is all about and where the savings are done.


All right, if you enjoy content like this, you know, don't hesitate to subscribe. I don't want


to do the other YouTube to show the graph. I'll do like, I'll do this. Here's the graph.


Here's the graph. So many of you are not subscribed. I mean, look at that. Excellent.


All right. So the point with the the sparsity gains is that if you implement them somewhere,


then that part is fine. But then another part is still dense and is still the bottleneck. So you


kind of have to introduce them everywhere. So if we look at a classic transformer model, and they


specifically, I think, refer to like the stack of attention is all you need, and so on. So what they


have, basically is they have two attention modules. So there's attention one, I think there's


attention two, and then there is this feed forward layer. Okay, so we're going to take care of all of


those right here. Attention one is called self attention. So if I have a sequence coming in here,


the self attention would be essentially attention in between the elements of the sequence.


The second attention block is I think encoder decoder attention or something like this,


the variants vary a little bit right here, but I would have sort of a second stack of this right


here, I would have a input sequence right here. So this would be the input, this would be the target


sequence that I'm about to decode. Maybe this has some causal attention, who knows. The second layer


of attention here is specifically attention that goes to the encoder sequence right here. So it's


attention in between the encoder and the decoder and the feed forward. So this essentially these


two mix all the information of the different tokens together. And the feed forward layer


simply takes a single embedding of a single token and feeds it through a feed forward function. So


all the tokens are handled by the same feed forward function. The first thing this paper


does is essentially eliminates the distinguishing between the self attention and the attention


between encoder and decoder. And I think that makes sense. That's also what a lot of other


models do. So famously BERT is an encoder only model, GPT is the coder only model. And if I


understand them correctly, they're as well, they're simply taking the encodings from the source and


then just pre pending them to the target or something like this, you know, safe to say,


there are lots of things that one could do right here. But what I wanted to say is that we now need


to replace each of those things with a sparse version. So we need a sparse feed forward.


And we also need a sparse attention block. So how are we going to achieve this? First,


we're going to the sparse feed forward layer. Remember, a feed forward layer is I have a


sequence of embedding. So that's these are all vectors. And these are all embedding vectors.


This is a sequence of embedding vectors that came out of the attention module, right. And


the feed forward layer essentially is a matrix. And I simply pass each of these through a matrix. In


fact, it's not one matrix, I think it is usually two matrices, one matrix that sort of, well,


that's not how you draw a matrix like this, and then like this. So you kind of blow up the dimension


in the middle. And then here, there is a relu non linearity in between. And the point is what I


already said, you'd feed every single token by itself through this function. So this becomes


like a large token, then there's a relu. And then this would become sort of a token of the input


dimension again, and you feed this token through as well individually, which gives you this one,


and so on. So, in essence, we have a vector, right, a token, all the tokens are independent,


we have a token, and somehow we need to make this sparse. Right now, it's a dense multiplication


twice. So there's two matrices right here. And if dense multiplication, right, so what do we do?


The first thing they say is that, well, given that there is a relu non linearity right here,


right, there's a relu. A lot of the things here essentially are going to end up being zero,


right? So it makes sense, it makes sense to do sparsity here. Now, I don't, I don't follow that


entirely. You know, I guess half of the stuff will end up being zero, yet the sparsity goes


much further. So but maybe, maybe they, maybe they justify why they can set some things to zero.


Not entirely sure, but I found that reasoning a bit shaky. But here is essentially, you know,


you don't need a reason to introduce sparsity if it works, it's good. So here's how it works.


First, and this is what I found a bit confusing. So it essentially starts on the right,


then it goes to the left, but it I guess it's easier to start on the left. So what we want to


do, I see here is that input vector, right? And here is that first matrix. So the first matrix


is of dimension d model, which is the same as this dimension, and d ff, which is the feed forward


dimension. And usually, I just multiply that together, which would give me a vector in the


dimension of the feed forward layer, right, which I then send through my relu. However, however,


what I want to do, I want to compartmentalize, I want only certain columns here to be activated.


Right, so essentially say, I already accept that a lot of my things in my result are going to be


zero, because you know, they will go to a relu anyway. So I'm going to accept that some of the


things will already be zero. So let's say all of these, I already accept they're going to be zero,


I don't even need to calculate the matrix multiplication between the vector here. And let's


say, this column right here, don't need to do it, because after that, they will become zero anyway,


so who cares. So I'm simply going to decide that some of the things are just going to end up being


zero. And they justify this by saying, well, there's a relu. So some of the things are going


to be zero, but more, more here is like, you know, six out of eight are going to be zero. And now I


only need to calculate the remaining columns. And that is the sparsity right here. Effectively,


they subdivide all of the they subdivide the whole matrix into these compartments. So we'd have two


different compartments right here. And of in each compartment, only one column can be activated at


the same time, right? I think, yeah, yeah, there's one, one of them, it's decided on one of them,


one of them can be activated, and only that one needs to be loaded from memory. Only that one


needs to be calculated as an inner product with the vector. And so the cells here where an actual


value is going to be are sparse. Now, the question is, how do we decide which ones we're going to


activate? By the way, if you can see then for the second matrix, you know, the same thing applies.


In fact, I can use that same mask from here. And I can again say, well, in the first module,


column number three was activated here, right? So row number three of this matrix needs to be


activated. The other ones don't matter because they're zero anyway. So there's a zero coming in


right here, being multiplied with this row, you know, who cares what the result is the the input


is zero, actually, well, people care, it's zero, right? But it means you don't even need to need to


do it. You can simply just load the rows that you are that you know, are potentially non zero.


So yeah, how do how do you decide? How do you decide which ones you should load from memory?


Essentially, you're simulating you're already pre committing to a relu pattern, right? So this is


how you do it. Essentially, you build, you build, you take your input vector right here, and you're


trying to somehow see how that works, we somehow come up with a new input vector. And you're


going to do that with a vector of with a binary vector with numbers between like zero and one. So


everything right here is like a point 1.5 point 3.8. So every single entry has a value, every


single entry will output like the probability that that particular element should be non zero.


Simply sample from that distribution, and use a straight through Gumbel softmax in order to


back propagate. So they also do a lot of tricks right here, I think they mentioned that in the


forward propagation, they even sometimes need to do a actually to pass just the softmax output


instead of the actual sampling. So there's a lot of engineering tricks to actually get this to work.


But safe to say, that's during training, we are we care about inference during inference, you sample


exactly one per module that is non zero. Okay, so you have two different workflows, the workflow one


goes here decides what needs to be non zero, right? And then given that information, you can do


this feed forward layer in a sparse way. But that is all useless if this right here is,


is not sparse. So this is actually not sparse, but it is low rank. So they say, well,


in order to figure out which things need to be non zero, we technically don't need as much


information as you know, actually propagating information. So what we can do is we can have


a low rank, essentially, it's another feed forward layer, again, doing this blowing up the dimension


to the feed forward dimension. But we make it low rank. So instead of instead of, wait,


yeah, instead of blowing up the dimension in between, we shrink it down, right, you can see


right here, we shrink it down to a low dimension. And then we go to the dimension of the feed forward


layer to decide which things are one and zero. And that's a thing you're going to see often in this


model is that they make use of low rank combined with sparsity. And it's also a bit of a of a


trouble that I have, because for some things, a low rank approximation is fine. But you know,


there's a reason we have dense multiplications everywhere, because sometimes it's not because


with a low rank multiplication, you essentially restrict your function space to a very, very small


subspace. Yeah, but it seems to work. So the trade off here is that you get to do this sparse,


which means that the time it takes decreases and the memory, but you have to this here,


over this, this is new, right? You didn't have to do this before, you could simply do the


multiplication. So this is going to add to your compute, while this here is going to be faster.


And now it's about whether whether or not you can make this side sufficiently low rank,


such that the gains over here are more than the time that you have to invest to compute this max,


this mask at the first place over here. Again, for these particular problems that they look at,


it seems to be working, right? But these kinds of trade offs, it's not guaranteed, like it's not


so clear to me that it would, you know, just work. Like it's not it's not straightforward,


that that trade off would be positive right here. There might very well be problems where


this rank right here is just too small to carry meaningful information, you need to make it bigger,


and that would sort of vanish all the savings you make over here. Because these savings are,


I mean, essentially linear in the sparsity. And this these gain, sorry, these, these,


this right here is essentially linear in the in the low rank dimension. So there's the trade off


right there. They here is how you how you can express this, you can essentially express this


as the original multiplication with the first matrix relu through the relu, then times the


controller output. And all of that then goes into the second multiplication. That's how you can


represent it mathematically. That's not actually what you do, right? Because here you still have


the full multiplications with the weight matrices, but it will result in the same thing as this


formula. Alright, so that is the sparse feed forward layer. And they do show that it it does


show that it decreases the coding time quite a bit. And interestingly, it also doesn't degrade


performance too much. In fact, you can see right here, this blue line is the average of the baseline


models. And if you if you don't go too sparse, you still have quite good performance. So this


is quite close. Only if you go more sparse, does your perplexity here start to suffer. I think that


that is one of the surprising things that there is a level of sparsity you can go at where you're


actually considerably faster, while your performance doesn't degrade yet, again, can very


well be because for the problems we look at the sort of the they're not difficult enough to really


make use of the capacities of the dense models. Okay, so feed forward is done. Now we go to the


attention layer. And the attention layer, again, is split up into two parts. In fact, they don't


even they don't even really deal with the attention mechanism itself. What they actually care about


is in order to do attention, attention is something like, I have my queries and my keys,


and I do an outer product and I normalize by something that I can't remember. And then I


multiply by my values. This is the attention formula. And what they care about is how do I get


the queries, the keys and the values. They in order to make attention itself, the sparse or


long range or efficient, they rely on on different techniques that from other papers. So for example,


they will later include the performer and the reformer architectures, which make attention


itself sparse or efficient or low dimensional. However, in this particular paper, they care about


how do we even get these matrices and usually, you get q by multiplying your input by a weight


matrix like wq, you get key by multiplying your input by a key weight matrix, and you get v by x.


So all of these are dense multiplications. And obviously, they now become the bottleneck once


we have the sparse feed forward layers, the dense layers in in the attention layers become the


bottleneck. The question is, can we use the same trick here as we did before? And the answer they


say is no, because the structure of the feed forward layer here was such that it had the relu


in between, right? So, and that's why they argue. So naturally, a lot of things are going to end up


being zero, which we can exploit by making, you know, just just a few more things zero, I guess,


but they don't, they don't want to do this right here, because here, like none of the things


necessarily are going to be zero in the output of these calculations. So the q or the k or the v,


they don't have many zero entries, so might not be justified to go sparse and just say, well,


make stuff zero. So what do we do instead? Instead, we look at this diagram here. So on the top,


you have what the current attention mechanism looks like, as I said, there is a there is a dense


layer essentially in front of each of these three matrices, which is that's how you that's exactly


how you get the matrix in the first place, right? We're going to look at a thing, which they call a


multiplicative layer. So which is this malt right here, and the multiplicative layer potentially


could replace the dense layer. However, they go a step further. And they say, they end up with this


architecture right here, where they have a multiplicative layer, then it's a one multiplicative


layer for all three matrices that is shared, and then one convolutional layer for each of the


different matrices, which is going to make stuff even faster. And then they also they drop kind of


this, this dense mechanism right here, and they simply add right here. Again, I like, I'm pretty


sure this works right now for these particular problems. Hope like maybe because the problems


don't make use of, of the parameters, or the original models were just poorly engineered,


they didn't, they never actually needed all of these, you know, parameters like this one,


and we're all fine. This could also be the case. So we have two things to look at inside of the


attention model, the multiplicative layer, and the conv layers. And these kind of go together.


And it also goes together with what's usually done in the attention mechanism, which is multi head


attention. So I'll draw a diagram of an attention mechanism for the about 500th time, but you have


some sort of a sequence, right? And every sequence, I'll replicate the sequence over here. So


every sequence emits what's called a like a query, which is a vector, some vector,


which are the queries, and also every element in the sequence emits a key. So the keys are also


some vectors. And the keys are also some vectors. And then routing is done via inner product overlap.


So probably these go would be routed together. These two would be routed together. This would


probably be routed here, it can also be routed to multiple stuff, but you route essentially via inner


product. So that's how you construct the weight matrix or the query key matrix for then multiplying


by the values. The idea behind multi headed attention, which is what's usually on is that


let's not only have one such block, let's actually have many such blocks in parallel, right? And


instead of using the entire vectors that are output right here by for example, that are in Q,


Q are these queries, right Q, or is a matrix and every row or column don't exactly remember is one


of these vectors right here. They say, hey, let's instead of so Q is a matrix, let's say every row


but for for let's just say every row, if I'm wrong, then you know, just reimagine.


So instead of taking the entire vectors here, like the entire vectors as queries,


we split the vectors into in this case into three parts. And this first part right here,


that becomes the query for this attention mechanism, the second part becomes the query


for that attention mechanism. And the third one becomes the query for yet another attention


mechanism. That's multi headed attention, same with the keys, same with the values. And yeah,


so now, now we're prepared. So what we want to do right here is we want to take a token. And


remember, we now need to make a query. Let's say we want to produce the queries, right. So from this


token, we need to produce a query vector, not only one, but number of heads, many query vectors from


this token using some sort of some sort of a linear layer, some sort of a linear function.


So that's how we do it. They say, we have this matrix right here, the weight matrix D. And what


the weight matrix D the weight matrix D is, there's the same dimension here as the input,


and has as many as many rows as we have different attention heads, right. So what we're going to do


is we're going to element wise multiply. And I would also add right here broadcast, right,


broadcast. So if you've used NumPy or TensorFlow or Pytorch, you know the broadcasting operation.


So the broadcasting is done. This is of dimension one right here, the broadcasting is done between


this one and this s right here, this is going to be broadcast into this form right here. And you can


see now, I mean, it's just an element wise multiplication. So all that is, is like differently


scaled versions of x in each dimension, right. So each row is essentially x, a little bit shaky. So


let's double shake x for the bottom row. Okay. But this already is now a vector,


one vector for each of the attention heads. Now, since element wise multiply is probably not going


to get us very far. We also multiply this by an actual matrix. But instead of multiplying it by a


D model times D model matrix, again, we go into a low rank, low rank regime, and simply say, okay,


we have this number m, and that's going to be a reduction on reduction on our dimensionality.


So this isn't D model by a D model matrix, which will probably be expensive. It's a D model by


m matrix. And out comes this. So this is going to be the query vector for the first attention


mechanism. Sorry, no, this is going to be the query vector for the first attention mechanism. And


this is going to be the query vector for the second attention head, head, I meant to say head.


There is a thing like, they don't just choose m arbitrarily, they in fact choose, I believe,


s times m equals to D model, right? That is, that is their their formula. So they, if they


split into s different heads, like let's, in this case, you see s is two, then m is three. And that


has a very particular reason. Namely, they say with this particular construction of the element


wise multiply followed by the multiplication by this weight matrix E. If if we do it like this,


then they can have a theorem, where is the theorem? There's the theorem. The theorem essentially says


that they can they can represent an arbitrary permutation. So they say, the minimum thing,


the minimum thing that we have to be able to do is to take x and kind of permute it. So to place


every single element of x in the output wherever we want. Essentially, they say, every part of x


should be able to be forward propagated to all the attention heads or to any of the attention heads.


And if a theorem that says that if they constructed like this, any permutation is within the realm is


within possibilities for some matrices for some weight matrices D and E. So that's kind of their


justification of, well, we can represent all permutations, so it can't be too bad.


Right? Yeah, I found a little bit of another way of, you know, seeing this, if you look at this with


the element wise multiply and so on, it is easier to understand this as let me try to draw this up


maybe over, oops, the boops over here. So if you think about it a little bit, it is like so you


have, and you also look at the formula, this formula right here. You can clearly see that this


is in fact a matrix multiplication again, so you have, I would say you have, if you look at this as


D times x times E, where x here is a matrix that has zeros, but x on so on the diagonal, it's x,


right? Which would give you, it would give you sort of a so D is kind of this shape, then x is


that shape, but only the diagonal is filled with x and then E is like that shape. So, and D and E are


fixed matrices. So you can see that what the, what this multiplicative layer is doing essentially is


it defines outputs, it defines outputs. So these are the number of outputs and this is the


dimensionality of the output. And what you're able to do is, is in some higher dimensional space,


you're able to manipulate the coordinate system scaling a little bit, a little bit arbitrarily,


but you cannot mix the individual dimension freely. You can simply in that high dimensional space for


a given mixing of dimensions. That's what these matrices here do for a given mixing of dimensions


for given linear projections from the low dimensional to the high dimensional space,


you're able to manipulate the coordinate system. So if you learn, you need to be able to find


matrices D and E such that for arbitrary samples, the manipulation of the coordinate systems there


makes sense. It's a little bit like, you know, like doing a PCA or something on a on a data set,


right, but it's just like during training right here. So, yeah, I'm not sure. Again, this is quite,


this is quite a loss. This is quite a trade off with an actual dense layer right here. So,


but it's interesting to see that it works, right? And again, this is only conceptual right here.


If you were to actually do this, you would lose all the benefits that you would lose all the


benefits that you had. And again, you can see a little bit that the trick here isn't necessarily


sparsity, but mostly low rank, this is mostly like a low rank function. Yeah. Okay, so we have


the multiplicative layer, we end up with the queries and the keys and the values for each


attention head. And now we're going to, they essentially say, okay, we could do this for every


one of the three things or or we simply do it once, which would give us this property of,


which would give us this property of the permutation being able. And then we can do


something even cheaper if we want to get the individual matrices, right. And so the trade off


here is while here, still every permutation was possible for the different matrices. So the Q


could have different permutations than K, then V, or different functions here, we're simply going to


resort to one function one mixing or shuffling around of the dimension. And then we're going to


do something even cheaper, which is this convolutional module. And this convolutional


module is also fairly simple to see. So this output y right here, and draw it again over here,


you have two vectors right here. And they say it somewhere, they say that dimensionality somewhere.


So you have two vectors, one per attention head, this is the output of the multiplicative layer.


And presumably, you would have those per token, right, we just looked at one token,


but the next token, let me draw it in this color, the next token would also have them. And then the


next token would also have two of those. Right, let's do this. So what you'd get is


a tensor that has the sequence length l, it has the number of heads, what's s, I guess,


or number of modules. And it has m, which is that, that essentially that low rank dimensionality


that the keys and queries and values live in. And they simply treat this as an image, and then they


run a convolution across it. So the convolution is going to be let me see if I can draw this properly,


the convolution is going to be across these two, so the filter is going to be like this, and then


in all the dimensions. So like this, I'm terrible at drawing, but the filter essentially is going to


be f in the dimension of s, f in the dimension of l, and m deep, and you have m filters of those.


So you, you have an s by l by m tensor here, and you transform it also to an s by l by m tensor.


Essentially, you can just think of this as a regular convolutional layer. So you can see


this as a convolutional layer. And what, again, what does the convolution go over? Remember that


the multiplicative layer is simply works on a single token, it mixes, it kind of is able to


shuffle around the tokens dimensionality is a little bit to permute them a little bit in the


best case. And in all other cases, it essentially manipulates the scaling in a high dimensional


way. And now with the convolutional layer, what we can do is we can bridge a little bit of


information already between the tokens even before we go into the attention module. So given that the


convolution is across the l and the s dimension, it means that for the s dimension, information is


able to be passed between neighboring attention heads. And for the l dimension, it means information


can be passed between neighboring tokens in the sequence. So that potentially gives some sort of


a positionality to tokens, because now that there's a notion of being close together. And also it


gives maybe a little bit of a meaning to different attention heads, because the attention heads up


until this point, they've just been kind of unordered independent things. And now they hang


it. This all of this is sort of one of the things why the the exact conclusions of this paper are


going to be hard to assess, even if they do ablations, right? They at the same time where


they introduce efficiency, they also introduce entirely new ways of, of sort of doing things,


they introduce new paths when it where information can be passed from between things. And so it's


very hard to point down exactly where things go right and wrong. So this was the sparse or


rather low dimensional attention module. Again, this is first one of these multiplicative layers,


which is element wise multiply followed by matrix multiplication to a lower dimension. And then


that is followed by these by these convolutions, but these convolutional layers right here.


So they call this whole thing a multconv. Right, if they combine all of this together,


you can see right here, the blue with the shade is the average of the baselines. This is perplexity,


so lower is presumably better. And you can see up to some noise, all of these things are fairly


consistent. Right? They follow the trajectory of the baselines quite neatly. Some are even kind of


bit lower this one right here, though I'm not sure if there is a there is exactly confusion


because so the F right here is the filter size, right? And the S is the the sparsity in the


multiplicative layer. So essentially how many attention heads it splits stuff into. And you


can see right here there is a conv is just a conv. And there's just a mult but the F is with the mult


which confuses me because the F is the filter size. So technically that should be with the conv,


I guess. If the authors are watching, please, please leave a comment. If I'm wrong right here,


I'm confused. In any case, they show that the baseline transformer don't particularly do that


much better in these NLP tasks or even do worse sometimes, as you can see right here,


though everything is pretty much within like a standard deviation than these scaling transformers.


So this architecture that we've discussed right now is this scaling transformer. The last thing


to do would be to add a sparse loss layer. So they can replace the dense layer with a multiplicative


layer similar to previous sections. This speeds up the coding time, say, sorry, they say, but


may degrade perplexity results are in the appendix. So the loss layer might not might be


the last refuge of really dense things to do. But remember, due to the fact that in the feed


forward layers, we sample from this distribution to really be sparse, or in fact, we might do argmax


right during inference. That's where the speed up comes from. During training, we actually have to


forward propagate the softmax from time to time so that the training works. And that means that


the benefits of sparsity are lost. Because if we don't hard sample ones and zeros, if we soft


sample them, then all the rows are still activated, and we need to track everything. And the same goes,


I think a little bit for batch inference. So if I have batch inference, even if I hard sample,


right, different samples are going to have different activation patterns. And therefore,


you know, with enough samples, all the things are going to be one somewhere. And therefore,


I probably need to load the entire matrix right here from memory, I need to do the multiplication


with the entire matrix, possibly not for all the vectors, but also possibly something like a GPU


probably wouldn't care that some stuff is zero, it's going to be as fast just to do all the things


at the same time. But that might be a hardware limitation. Okay, so that was the scaling


transformer. And now we're going to supercharge the scaling transformer, which makes it into a


terraformer. I don't think there's any relation to the tool terraform. But, you know, we're running


out of names of formers. So yeah, this was the last refuge, I guess. So what they do is they use


essentially, they use essentially the architecture from the attention from reformer. So yes,


we focus on the locality sensitive hashing attention from reformer. Was that reformer? I


thought that was perform I am confused by my by my own stuff. reformer Yes.


So they do two things, right? They have an architecture for long sequences. While integrating


sparse attention layer into a scaling transformer, we noticed architecture is suboptimal. That's what


I said at the beginning. separating decoder self attention and encoder decoder attention is not


necessary anymore. From the perspective of efficiency, we remove the encoder decoder


attention that I said that at the very beginning. But just concatenate the encoder representation


before the decoder tokens. So they replace the encoder decoder attention by essentially two


attention blocks. That is that, okay, I guess there's no performer in here, just the reformer.


So the LSH I've done a video on this locality sensitive hashing instead of full attention. So


if you have really long sequences, you, as I said, you need to compute inner products between


all pairs between all pairs of, of nodes right here of tokens. And this is cumbersome. There are


various techniques to speed that up. One is LSH locality sensitive hashing, where you essentially


create hash buckets, and then you hash all the vectors, all the vectors inside of it or all the


inner products become hashes. And you look for essentially hash collisions that indicate where


you want to calculate and check and a whole everything that's not a hash collision, you don't


need to check. So locality sensitive hashing has been long standing technique to make inner product


search in high dimensions or inner product computations, and looking for the most close inner


product in in among very many elements have very fast. So they borrow that from there. And then


also they include the recurrent blocks. So recurrent blocks is no, that's later. First,


it's the reversibility. All of this is just so similar. reversibility is


also apparently in reformer. And what reversibility means it's kind of this architecture right here.


So again, we have two attention, and then one feed forward, right? The second attention replaces the


encoder decoder attention. And reversible means that instead of having one strand, like one flow


of forward propagating information, right, one flow of information, we have two. So we have


two information flows forward, and then every function that's applied is applied to one flow


and added to the other flow, right, this gives you this and this one right here is simply forward


propagated as a residual connection, essentially. And then x two is taken. So this the first


flow of the actual function would be this right here, right, you can see, this is the flow of


hitting all the functions. And you can also see that we always have a signal for each of the


functions, we always have a signal that travels without being touched by the function right here.


Okay, so that signal right here, and this is the signal right here. And that makes the blocks


reversible. And that means that I can, I don't have to keep activations in mind. This limits,


this limits the capabilities a lot. So non reverse, an example for non reversible would be,


well, this here is non reversible, because because of the fact that the signal is not


reversible, because because unless I do like a linear function that goes from exactly the same


dimension to the same dimension that is non degenerate, unless I do that, I cannot possibly


reconstruct the input right here, like the signal right here x from the output y, not even for a


one of those blocks, right, it's not possible for me, essentially, to do this, or Yeah, so the


reversibility changes that essentially means I can always reconstruct from the from the signals,


I can reconstruct the intermediate activations. And therefore, I don't need to store them. Because in


a normal neural network, as I forward propagate, I need to store a lot of intermediate stuff,


like right here and right here, in order to then during back propagation, I need those things.


Because otherwise, I couldn't calculate the gradient, so I need to store the activation


somewhere, reversible networks, reversible blocks do not have this property, they do not need to


store because they're reversible. And they're made reversible not by changing the individual modules


like this, or this, but by simply having this construction of the two strands of information,


and the modules simply apply between the two. That's it's pretty smart architecture. But one


has to say, it has very often significant trade offs, because these things being reversible,


also brings some some properties, like there are a lot of functions you cannot express anymore,


because you need to keep everything reversible. So again, I think, for the problems they


particularly look at here, it might work, it might not work for all problems. I think that's a bit of


a general thing in this, in this paper right here. It's more like we're gonna have to test for every


new task we tackle or new challenges, new modalities, whether these things still hold. The


last thing they build in is recurrence. And they say it's for generalization. And that is, if I


understand it correctly, it is they use simple recurrent units, not like an LSTM because they say


that would be too slow. So simple recurrent units, they're still fairly complicated. Like I've looked


them up, I didn't know what they were, they're still, they're still okay, complicated. So it's


not just like a recurrent layer, it's actually you know, it has gates and so on, like bit like GRUs


or LSTM cells. And if I understand correctly, this goes between so as I said before, in the feed


forward layer that every single token goes independently through that, if I understand this


correctly, if I understand this correctly, this introduces a recurrent connection in between these


data, well, did I understand it correctly? Okay, um, we also add recurrence to the feed forward


block of terraformer recurrent layers allow information to propagate in time, even a even in


a single decoder block. Okay, I think I understood that correctly. So within the feed forward block


right here, there is a recurrent connection between the different tokens. Every token goes


independently through that. But now we introduce actually a sort of dependence or a function that


goes from the first token to the second to the third, and so on a recurrent, small recurrent


neural network. And again, they, one can only speculate why they have this in here. I mean,


they say that this the results on C4 are minimal, which is their language modeling task.


And they say, the biggest benefits are when they do like these, these toy tasks where you need to


copy a decimal digit, and then you can train at on 128 digits, but then you can test on 256. So it's


over two times longer than seen in training. So they really make this point is for generalization,


though it is very, very odd. Like, this is a very odd addition, I can I could get them until like,


you know, here, it says, yeah, okay, you go for long sequences, you know, that's cool. Long


sequences are cool. It's cool if your model can, you know, also do long sequences fine,


then memory efficiency, okay, you know, so given that is all sparse and low rank, and so on, you


also might want to use less memory cool, but then returns for this is this is quite an odd choice,


I feel. And it could be that it simply didn't work like, so they also say that the terraformer here,


in sort of these tasks, like summarization, that it sort of beats or matches state of the art,


matches much, much larger models, and so on, it could, I can imagine that their numbers were


were slightly smaller, like slightly worse than kind of the baselines. And they were just looking


for something to add to pump up those numbers. And this worked, if this is the case, if that's a big


if, again, it's very dangerous, because it might work for these particular problems and not for


others. If not, if this was really just like an idea they had and said, Well, it'd be cool if


that's in there, then you know, good, like, I'm willing to I'm willing to accept that as well.


Alright, so that was the terraformer. And here you see, so the terraformer now has over a 37x


speed up on it's a considerably large model. But for this large model, it requires less than 100


milliseconds per token of decoding time, while not degrading in performance too much. So that is,


that is, I think, quite an achievement, even if it's only for particular types of tasks like these


here, it is quite an achievement. And it's a bit of a shame that the speed ups are only for like,


they're only so huge for the really huge models, I guess it makes sense, because these effects are


often compounding. You know, so it for you and me with like, our regular old computers, laptops,


it maybe won't make that much a difference. In terms of speed, it might make a difference in


terms of memory because of the reversibility. But other than that, yeah, but it's it's good for,


like, if you work if you want to work with larger models, but you don't necessarily have to compute


and you do inference, this might be something for you. They specifically say that not everything


has been tried yet, they still don't do quantization, which could yet deliver another


speed up. And there's also lots of things to do to actually speed up training. Maybe there's a way


to get around this Gumbel softmax need to forward propagate the true softmax from time to time and


so on. So lots of engineering, lots of kind of choices that are interleaved, very hard to say


where gain comes from. But undeniable gain has been made in huge form. And that's cool. All right,


BLEURT: Learning Robust Metrics for Text Generation (Paper Explained)

Hello there, today we'll look at Blurt learning robust metrics for text generation by Tibalt


Selam, Tipanjan Das and Ankur P. Parikh. So this paper on a high level proposes a new metric for


text generation tasks such as machine translation by leveraging a BERT model to produce like an


automated metric, an automated quality metric. And they make this BERT model robust by pre training


it on a very wide array of tasks that they can use synthetic data to train it. And therefore,


the model and this resulting score is very robust to shifts in distribution. And they advocate that


this could be used in the future to assess text generation systems. Alright, as always, if you


like content like this, consider subscribing and sharing it out and leaving a like tell YouTube


that it's good content. Of course, only if you agree. So what's the problem with with evaluation


for text generation? So if you know, the machine translation community, basically, what they do is


they have these data sets where they translate from one language into another, let's say, English to


French. And they have a training data set that is fairly okayishly large. And then they somehow need


to evaluate this, okay, so you have like a test data set, but all you can really do is sort of


calculate the perplexity of a language model that you produce or of a translation model that you


produce. There's not really a metric for translation. So the gold standard is to get it to


humans. So you train on this data set, you produce a program, let's this is your machine translation


program that you produce from the data. And you you let this run on your evaluation data set. And


you give the results to a bunch of human raters. These could be regular people, these could be


linguists that are experts in translation in both languages. And they will score the each of the


outputs of the machine translation systems. And at the end, you will get like a number like eight,


your your system is eight good. The problem, of course, is this process is very, very slow. So the


machine translation community does this every year. And it's, it's quite slow. And it's quite


expensive, as you know, it requires these humans here to assess all of these systems output. And


you want a sort of a sizable output, right, because you want sort of a good sample of the machine


translation system. So this is not really satisfactory. But like an automated score, like


perplexity is also not satisfactory. What people have done is they've come up with proxy scores for


the humans. And two of those scores are called rouge and blue. And specifically blue is one of


these metrics that people use. And it, it kind of it takes n grams in the sentences. So n grams would


be like snippets of like, let's say four words after one another. And there would be these


snippets and that the machine translation system produces, and then it would go into the validation


data set, and look at the gold standard translation that was also produced by humans. And it would


also look at these snippets of size four. And it would just kind of assess how many of the snippets


overlap. Of course, the machine translation system has never seen the label like the gold standard


for that particular set. Otherwise, it wouldn't, you know, be fair. But you basically compare n


grams of output and gold and some gold standard, you can have multiple gold standards and so on. So


this blue metric is more of like a heuristic. And it has been found to correlate fairly well with


the humans up until recently, of course, with the explosion of neural machine translation, and


especially transformer based machine translation, I guess and but also the their their system that


use LSTMs with attention. These systems have become extremely, extremely good. I don't know if


you've noticed, but Google Translate has been getting better and better really fast. I remember


the the first years of Google Translate when people still made fun of it. I don't think many


people make fun of it now. At least it's not a meme anymore. So this, the more the better and


better these systems were, the more these metrics like blue and rouge have diverged from the humans.


And they're not really reliable anymore, especially if you compare really high skill systems to each


other. Blue tends to not correlate well with humans. And therefore, we're looking for a new


metric, a new metric that correlates well with humans, but can be evaluated automatically. And


this, this paper here proposes this blurt. Can we just stop with the variants on BERT? We get it,


use BERT for everything, but you know, yeah, so they say it's a learned evaluation metric based on


BERT that can model human judgments with a few 1000, possibly biased training examples. Okay, so


what what you what you would do, in these cases, is now the the creation of a metric becomes a


machine learning task itself. So what you'll have is, you'll have a data set of things that are gold


standard translations by humans, you will have the output of the machine translation system, okay,


you put them together. So you have the gold standard sentence, this is this would be the optimal


translation, you'll have whatever the machine translation produced, and then you'll have a human


look at it and create a score, like this eight right here, it says this, these two sentences, they


match eight good. Okay, so eight, maybe it's out of 10. So this, this bottom thing is a very good


translation for the for the top thing, like to match the top thing or the human assesses the


quality of the sample. And now we have a training data set, right, you have a z they call I think


this z and z tilde or something or y. Yes, this is zip, zip. They call this y, which is the gold


standard label. This is y tilde, whatever the machine produced, and or x, x, x tilde, and then y


is the label. So your task is now given x and x tilde, predict whatever the human would say about


this. Okay, so if you can collect a few of these samples right here, of different machine translation


systems, then you can formulate you can make a data set out of this, right and formulate a machine


learning task. And that's exactly what these competitions have done. So it's like a meta meta


competition now can it's a competition for designing the best metrics of the other competitions,


basically. And of course, the difficulty here is that the data set isn't static. Because if you come


up with a metric such as blue, let's say you come up with a better blue, you would want these, these


these other tasks to use it in the next years as well, because the thing about metrics is you need


to be able to compare to like previous years and so on. So you would want a metric that is still


valid for other years and other sort of other slightly different tasks and also for other


machine translation systems. So if you just learn on data from now from this year's this year's


competitions, and in five years, all of these models will have become so much better and they'll


produce different output. And that's the difficulty your metric should still be valid at that point.


And this paper basically deals with the deals with the fact that can you learn such a metric from


data that exists at one point in time, that will be robust to shifts in distribution. So in five


years, the machine translation systems, they're better, they may be used different language


constructs to translate certain things, because that's they assess that better, can you still make


a good judgment about which one, which of these systems is better than the other system? Can you


still assess how humans would rate the systems. And they're saying that they found the method to


do this. This blurt as they said, not only have they found the method, but their method only uses


a few 1000 possibly biased training examples. And they do this via a new pre training scheme. So


they say a key aspect of our approach is a novel pre training scheme that uses millions of synthetic


examples to help the model generalize. Now, why is it important that it only uses a few 1000


training examples, because these are generated by humans, right, and humans are expensive. So and,


and it's not like like ImageNet, you do it once you have it for 20 years, this is done year after


year, and you need real experts like translation experts. So this is expensive. So the fewer of


these actual training examples that the thing can can be efficient on the better. So they


circumvent this by using, they say millions of synthetic examples to help the model generalize,


they do this in a pre training step. They stay. Blurt provides state of the art results on the


last three years of the WMT metrics shared tasks. Now this is this, this is this meta task, where


you're asked to come up with a metric for the other tasks, and the web NLG competition data


set, in contrast to a vanilla bird based approach yield superior results even when the training data


scares and out of distribution. Right? So let's have a look at what they do. Okay, so they say,


what what do we need to do to fine tune BERT for quality evaluation? Now, if you don't know what


BERT is, BERT is this, it's basically a model that takes in a bunch of text. So you have a bunch of


text, and then it's a model that is a transformer, I've made a video on it, if you if you want to


check that out. And then you get as outputs, you get like a sequence of vectors, but important,


most of the time is only the first one, which you then use in a subsequent task. For example,


if you want to do classification, you would output you would put a classification layer on top to


classify it into certain classes. If you want to do regression, you do that. You can do other things


with these outputs right here. But for this particular paper, only this CLS output is


relevant. Okay, so you would input this pair of gold standard and output of the machine. And the


output would be a output y, which is either a number or a class label. And in this case, it's a


number, okay, so you input right here, you input these, these two things, and outcomes this whole


sequence and you only you take the CLS output vector, and you put it through a linear layer


right here, weights and bias, and that would output the number y and the number y you train to


be as close as possible to the human to what the human would say about this pair x, the gold


standard and x tilde, the output of the system, okay. So this is what you would do. If you were


simply going about it, just take BERT, take the the model. So BERT is really good at language,


right? Take the model and train it on this data set. They say, however, fine tuning BERT requires


a sizable amount of IID data, okay, we don't have that in these tasks, which is less than ideal for


a metric that should generalize to a variety of tasks and model drift. So the problem with just


applying BERT here is that way, they say it, you don't have enough data, and it's it won't be a


robust solution, it will only work for this particular data set that you train it on. The


solution they say is the pre train on synthetic data. So what does that mean? They say, the key


aspect of our approach is a pre training technique that we use to warm up BERT before fine tuning on


the rating data. And you might know BERT, BERT training, which is where you do this masked


language model pre training, right? So if you are given a piece of text, let's say you're given this


piece of text right here, what you would do is you would drop out a couple of words like this one


and this one, and ask BERT to reconstruct it like a denoising auto encoder. And that way, BERT


learns sort of about language in this particular way. Now, they they're not saying you should


replace that what they're saying is first, first, you should do this masked language model pre


training. Second, you should do their synthetic pre training. And third, you should do the fine


tuning. Now, if in the in the naive approach, you would skip this step too, right? So their claim is


that by introduction of this step two, that you could be a lot better and a lot more robust,


because you've already had like, so you're already exposed to information in this step that will


make you more robust to distribution shifts in this fine tuning data later. Okay, now, I've short


inter interlude right here, I've advocated for this step to be called priming, priming, because


otherwise, you always have to say like, okay, pre I want to pre train BERT, but I don't mean like


pre pre training, like I don't mean this, this, this is already called pre training. I want to pre


train after pre train. So I just vote for this to be called priming. I have no idea. Like if you come


up with stuff like this, probably you've heard it somewhere. So I'm, I guess I'm, I might not be the


inventor of this. But it is a good sounding word. And it sort of fits, right? Okay. So they say we


generate a large number of synthetic reference candidate pairs. So what they're going to do is,


they're going to take a bunch of text, and in their case, I think it's Wikipedia. And for each


Wikipedia article, they're going to, so they, they take Wikipedia, they're going to draw sentences


or samples or paragraphs from Wikipedia. And these are going to be your Z. And then they're going to


kind of muddle with them a bit, they're going to disturb them a bit to make them a bit different,


to make them go Z tilde. And this simulates that the difference between what the machine


translation outputs and the gold standard sentence, they're usually not exactly the same, right? If


you translate a sentence, there are many ways you can do it. And their goal is to sort of produce a


data set that has sentences and sort of perturbed versions of the sentence, but not perturbed, like


randomly, but perturbed in a sort of language knowledgeable way. Okay. So how do they how do they


do this? They have three different ways. First of all, mask filling with BERT. So what they're doing


is they take a BERT that can do language modeling, right, a pre trained BERT. And let's again say we


have this text right here. And they simply drop out two words or so and fill them in again with


BERT. Now BERT might produce the same words, or it might produce slightly different words, depending


on how many you drop out, you can choose the kind of amount that you perturb the sentences. Okay. So


the second is they back translate. So what they do with back translation is they use a machine


translation model. Now it doesn't matter which one you take out of they use any machine translation


model to take a sentence Z and then they map it to another language, say, from English to French.


So this is Z French, and then they map it back again, with the Z tilde is now the French to


English translation. So you need to translation model first you translate it to French, and then


you translate it back again. And that would sometimes give you the same sentence. But often


it will give you sort of a paraphrase of the sentence that you had at the beginning. Okay,


so that would be the second version that you could make pairs of sentences that are sort of similar.


And the third way is just to drop out words. And they just found this to help. Okay, so now they


have a giant data set of sentences and perturbed versions of sentences. So what are they going to


do with that giant data set? And the answer is they're going to take this Z and Z tilde and


you're going to put that into BERT into their thing that they prime now this is the priming


stage, right? This was pre trained on mask language modeling. Now they want to prime it,


what are they going to do, they're going to take this CLS vector. And now of course, this is not


the final task. And we don't have final labels for these two things. So we need to somehow come


up with our own late tasks and labels for them. And they decide to go a whole bunch of tasks. So


they go like, they go like, I think, I don't even know they go eight or so or five or so different


tasks, they construct different tasks to perform with these two things. And this could be metrics


like blue or rouge or this BERT score right here. So you simply calculate the n gram overlap between


Z and Z prime, that would be one of the scores. It could be the back translation likelihood, which is


how likely does a back translation model assess this sentence? Oh, here's all the things, six,


six different tasks, okay, so six different tasks. And that the catch here is. So what would happen,


for example, with blue is, you would take a model and you would calculate the blue score between


those two things. But you wouldn't input that into BERT, you would ask BERT to predict the blue


score. Okay, so BERT would be outputting B hat, and B would be the actual blue score. So you would


train BERT to predict the blue score of this particular pair of input. And so one you one you


take as like the input and the other one you take as the reference. Okay, and you would ask BERT to


predict the blue score of this to predict the rouge score, you would ask it all of these signals,


you ask this one the same model, you ask to predict all of these scores for these two things, okay,


so you can calculate all of these scores by either blow is like a script you run, or you have some


other model, like a pre trained translation model that you use to assess the that you ask how good


is this in terms of this particular task back translation. And then you try to predict that


score, where is important, you're not training the model to perform these tasks, these tasks,


you already have another model that's specialized to these particular tasks, and you simply ask them


to score the input, okay, you have an entailment model that outputs how much by how much does the


second sentence entail the first that basically means does the second sentence follow from the


first. And of course, this is not, you know, it's not actually proper input data for that task,


but you can still ask the model. And if these are good translations of each other, if these sentences


match, then the second one should probably follow fairly well for the first. But at least you can,


if you make BERT predict that it will learn something useful about the relation between the


two sentences. So the entire game name of the game here is to come up with tasks that if BERT


learns to predict the score of these tasks on those inputs, sorry, on pretending one is the


input and the other one is the output or on the two inputs, and then trying to predict the score,


then BERT would learn something useful. Okay, so that's, that's the, the, the trick here is to come


up with these pre training tasks, and you train them all at the same time. And by doing it all at


the same time, and by doing it on many, many different perturbations on these different tasks,


you hope that your model learns something. Some, it's kind of becoming a tune to the variations


that language can have and what it needs to pay attention to. And then you hope that if you then


have done this, then take this model, and then do step three, which is fine tuning on the actual


data you have, you would guess that it becomes very good at that data, but also it retains all


of these abilities, and and generalizes better to other sort of distribution shifts. All right. So


that is that is the thing here and they on this task on this metric learning tasks, they do out


perform all other models right here. And what I find interesting is down here where they now test


for the for the distribution shift. So what they're saying is, okay, this has all on data,


basically, where, you know, we train on training data and evaluate on testing data, and they're


sort of the same, they come from the same year from the same machine turn translation models.


And we don't really know how, you know, next year, the machine translation models might be different,


does our score still hold. So they try to simulate this by splitting the data. And they introduce


this skew factor. So what they'll do is they'll split the data. So usually, as you can see right


here, the training date, the ratings, these are the human ratings, the training data is sort of


distributed like this would be the test data. And the training data would almost be overlapping that


if you can see like the dotted lines right here or so. So you can see the overlap between the test


and the trained out of the human ratings is very close. Now they say we can we can skew that we


can sort of filter the data, such that in the training data, only very bad sentences are and in


the test data, there are only very good sentences, okay. And this simulates the fact that, you know,


we this might be the previous year's data that we train our metric on. And then we evaluated on the


next year's data where all the systems have become better. And what this does is it simulates


the fact that the human ratings are better than the test data. And what this does is, you can see


right here, the bottom axis is the test skew. And the color here is the training skew. Okay, so what


interests what we're interested in is to the right and down the colors. So as these skew increases,


the human ratings decreases, but it, it still remains fairly well. But especially the training


skew, if you update the train, so if you make the training examples really bad, so to say, it the


score just drops down. And they can show pretty well here that if you add this pre training, then


the score, except in this extreme case, so the score for all of these, it remains relatively high


and especially remains above the blue score, which is always sort of worse. Right. So this is is


pretty, as pretty neat and shows this power of this pre training. Basically, that's that's the


the robustness to quality drift metric, and they have a bunch of other metrics right here where


we can see, for example, how much of this is going to translate and so on. But I don't want to go


too much into that. I more want to make some comments on on this work. So what what I think,


so first of all, in a paper like this, what what I would like to see is like the extrapolation right


versus the the blue score, right? Because I mean, okay, it seems like yeah, this skew of three is a big number.


But who knows if three is a big number, right? Like, we can't assess that. What we need to see is really


the where the crossover point between the models to to assess where does it where is it no longer


valid and so on. The second part here is that my problem with this method of splitting the data,


I mean, yes, okay, you split the bad from the good, but it's not it's not only that these things are


getting better. So right now, everyone's using transformers for everyone's using BERT for


everything, right. And BERT is a specific architecture that is going to be good at specific things,


at specific grammatical constructs in specific languages, right. So it's the mistakes it makes


are very systematic. Now, if in one year or two years, all of a sudden, the new model pops up,


I don't know, like, someone discovers that graph neural networks are really good at machine


translation, these models are going to be attuned to a very, very different set of construct, they


might be better overall, but they're going to make a different sort of mistake. And so I think just


assessing the skew via just dividing up the data in into bad and good ratings, I don't think that


covers these distribution shifts that they set out to cover, right. What I would have expected


is something like them, because these tasks are repeated year after year. And I would have


expected them to, for example, train on 2017, and then evaluate on 2019, or something like, or show,


like evaluate on 2017, 2018, and 2019. And there, we would have a much better assessment of a


distribution shift over the years, right. So so it is not super convincing to me. And what is most


worrisome is if you look at their pre training tasks, I mean, okay, there is a there's blue and


rouge. But there is BERT score, right, there is entailment, which is also a BERT model. And the


back translation, I mean, who knows, that's probably either going to be a transformer or a


LSTM with an attention mechanism, which is the attention mechanism is the basis for transformers.


So all of these things are basically going to make the same sort of bias mistakes, right,


they're going to, to, it's not it's not like there is Gaussian noise on top of these things,


all of these things are going to be weak in the same sort of assessments, and not agree with,


like, they're going to have systematic errors in them with respect to them predicting the


human scores. And if we evaluate our systems that some are also using exactly that thing, right,


so these the systems we evaluate, they are using the same type of models as here, they're going to


fall prey to the same type of mistakes. And then if we switch over to systems that use some different


right, so next year, we have some systems that use different techniques, they're going to be like,


exactly, maybe not bad in these particular things, but bad in other things. And then this


thing will output a systematically biased assessment. So it's sort of a house of like,


if you've seen these images of plugging in the power strip into itself, and you have infinite


power, it's like it's very, to me, it's seems very dangerous to have a such an overlap of


architectures and methods to evaluate systems as you have in the systems themselves. But I hope


this will be regularly checked with human scores and assessed as to how much these systems are out


of sync or in sync with humans. Alright, this was it for me for Blurt. Check out they have the code


Author Interview - Improving Intrinsic Exploration with Language Abstractions

Hello, this is an interview with Jesse Mu, who is the first author of the paper improving


intrinsic exploration with language abstractions. This paper is really cool because it combines


the knowledge that is inherent in language with the problem of exploration in reinforcement


learning. I've made a comprehensive review of this paper in the last video, so be sure


to check that out. My guest today, Jesse has seen the video and we're able to dive right


into the questions, criticisms and anything that came up during the video. The interview


was super valuable to me. I learned a lot. I hope you do too. If you like then please


leave a like on the video. Tell me what you think in the comments. Tell me how I can make


these videos better. Above all else, and I'll see you around. Bye bye.


Hi everyone. Today I'm here with Jesse Mu, who is the first author of the paper improving


intrinsic exploration with language abstractions, which is really cool paper. I've enjoyed reading


it. I like the bringing language into the reinforcement learning domain. I think it


makes a lot of sense and I was very happy to see this paper. Yeah, Jesse, welcome to


the channel.


Yeah, thanks for having me.


So I've, I've presumably the viewers here have already seen my, my little review of


the paper. What would be your, maybe for people who haven't seen that or just in your words,


you're like short elevator pitch of the paper itself. What would that be?


Yeah. So the way that I would pitch the paper is that reinforcement learning for a while


now has wrestled with perhaps the central problem, which is how do we encourage exploration


in these environments with more complex tasks and longer time horizons where the extrinsic


reward that you get from the environment is very sparse. So in the absence of extrinsic


rewards, how do we encourage agents to explore? And typically the way we do so is we assume,


and this is a very cognitively appealing intuition that we should motivate an agent to achieve


novelty in the environment, right? We should make it do things that it hasn't done before


encounter states that it hasn't seen before, et cetera. And then hopefully we'll enable


the agent to acquire the skills that we actually want the agent to acquire in the environment.


But the problem with this of course is how we define novelty. In a lot of scenarios,


you know, there are environments that can look very different, but they have the same


underlying semantics. So the example I have in the paper is like, you know, a kitchen


and the appliances might be differently branded and differently colored, but ultimately every


kitchen is a kitchen. And you know, the way that you approach kitchens and the way that


you operate in them is the same. And so the idea of this paper is we should be using natural


language as the measure for how we describe states and how we describe actions within


states and use kind of traditional approaches to exploration, reinforcement learning, but


simply parameterize them with language rather than with state abstractions, which is usually


the way in which exploration is done in these kinds of environments. And so what we do is


we take existing state of the art exploration methods and then kind of see what happens


when you swap in language as a component. And do you get better performance? And we


showed that, you know, in a variety of settings, at least in the kinds of RL environments that


people have been looking at in recent work, we do see a gain in using language to parameterize


exploration rather than states.


Yeah, that is, I think it's very apt to describe it as you, it's not suggesting like a new


exploration algorithm, but it's simply the re-parameterization in terms of language.


And coincidentally, these environments, they do come with this kind of language annotations,


which we do focus on. I like the, so I think what I really liked about this paper is just


the research mindset, in that any other paper or a lot of other papers they would have done,


they would have tried doing like three things at the same time, like, you know, we, we have


a language generator, and we do this, and we do that. And what you're I think, doing


correctly from a standpoint of research is you keep pretty much everything constant,


the algorithms constant, right? Even the environments, you assume that you have a perfect language


oracle, and you just add the language, which I really appreciate as like a reviewer, let's


say. So I think this gets us right into our, or my biggest, essentially criticism of the


paper, or what I what I called in that you add language to these algorithms, but you


just said we swap in language. And to me, it felt more like it's not really a swapping


in, it's more like you add language on top of what these algorithms are doing. And therefore,


can't I just see your method as adding more data, essentially, there is features that


are available from the simulator, right, which the other methods just don't use, they just


discard this part, and you just add this part. Do you have an indication in how much of your


effect is really due to language? And how much of the effect is just due to the fact


that you have more data available?


Yeah, that's a that's a great question. And it's definitely a point that I think a lot


of people will fairly make against the paper is, yeah, we're using extra data, right? And


yeah, I think my verb swap was maybe only accurate in half of this paper, which is that,


you know, in amigo, which is the first method that we look at, it really is a swap, right?


So if you read the paper, the traditional kind of amigo teacher network proposes coordinates,


x y positions as goals, and here, we're just completely, you know, eliminating that kind


of goal specification, and we're moving towards language. So so that can be seen as more of


a swap. Although, of course, in novelty, which is the second method that we look at, that


is definitely more of kind of an addition, as you say, because we keep the extrinsic


bonus. And we do have experiments that measure what happens if you don't have novelty by


itself, you only have the kind of language novelty bonus, and it doesn't do as well.


So you're right that, you know, I would say that we explore like this idea of swapping


in language and in a bit of the paper, but there are points where it's more of kind of