And that means that these first two dimensions right here, really encode the force in the


Discovering Symbolic Models from Deep Learning with Inductive Biases (Paper Explained)

way that we know the equation is.


So that's when we know the equation, right?


When we know the equation, we can simply say, okay, does this fit?


And when we don't know the equation, we can use this symbolic regression.


And what turns out is exactly this thing right here.


Now, you might object that this isn't really that force right here.


But as I said, there are many, many symmetries.


So for example, this, this R hat right here, I believe, and this is I've, I'm not a big


physics person.


This R hat, I think this is the vector of the delta x delta y, right?


So delta x delta y is in this R hat.


So we already see that delta x and delta y here, this already looks like some sort of,


this already looks okay.


No, actually, if we go down, it gets even clearer.


So here they have the outputs of that.


Yes.


All right.


So in this first case, this is the same example right here.


So they say, you in this spring example, so this is a system where the particles are connected


by springs, and we do L1 regularization, what we expect is this equation, this is we know


that this equation holds in this spring system.


And what the neural network combined with the symbolic regression gives us is this equation.


So right here, you can see there's this delta vector, and it's a product.


It's an inner product, a dot product with this a, which is a numerical constants.


And you can see that there is this form of product with numerical constants.


What you can also see, so for example, here, the delta y here is 1.36 and 1.37.


That's, you know, the same number, and here, it's 0.6, 0.6.


Okay.


So here you see, for example, r minus 1, and here it's something like this minus something


divided by r.


Doesn't seem the same, but again, due to the symmetries, you can, if you take this and


you simply divide everything by r, you'll end up with this vector right here, a times


delta x delta y times 1 over r, no, times 1 minus 1 over r plus b.


Right.


And now you can see it already looks very much similar.


And it's only off by like, it's only a transformation away from what you want.


So that's why I said you can describe these equations in many different sort of equivalent


ways.


We can't ask the neural network to really figure out, you know, the exact one we want.


As long as it figures out a one that is equivalent, we're happy.


And we're, I guess we're pretty happy here.


So also, in this case, right here, you can see that it correctly predicts this relationship


that it should be divided by r to the third power.


And there is a delta x delta y delta z, if you simply consider, so delta z here, I guess,


is simply a factor of zero.


And it even has this discontinuous problem where the force breaks after a certain while,


it can even parse out this if condition right here.


So that's, that's fairly cool, right?


But it to me that that is pretty, pretty cool result that you can actually parse out these


equations with just these graph networks, and then the symbolic regression.


So they do the same thing for this cosmology example, where they have these simulators


of the universe, and they let them run.


And these kind of distribute this dark matter.


And I guess your task is, if I give you a bunch of these points right here, tell me


where the other dark matter is something like, I don't understand this, but in essence, it


is the same kind of problem, right?


You want to figure out the dark matter properties from the surrounding dark matter or properties


of other things.


And again, here, you can see pretty well that this is the equation they get out.


So the equation they get out is going to be a sum right here, over.


So here, the output for node i is going to be a sum over all the other nodes j.


And then some function of that sum.


So this right here is the equation that came out of our edge model of our edge neural network.


And this here, that includes this one, it was the equation that came out of our vertex


model.


As, you know, the same here in this spring law, this came out of our edge model, this


came out of our vertex model.


Again, this rests on the fact that physical systems can actually be described often as


these sums of independent interactions.


And that's why all of this works.


So they do give very, very detailed instructions on how they did everything.


I think the most unclear things in this paper are the physics things that are assumed sort


of that you know, I don't I didn't.


Yeah, but other than that, it's pretty straightforward.


Their appendix is also pretty detailed in how they do all the representations and so


on.


They have different formulations other than this l one regularization.


As I said, they have bottleneck, they have like a KL formulation, they really describe


how the graph neural network works here, and so on.


So all in all, I enjoyed reading this paper, here is a bunch of examples of these particle


systems.


And yeah, and here is a bunch of examples of where you'd have a linear relationship


that where you can say, Oh, look, this really describes that force, or a nonlinear relationship


where you can make the claim this doesn't really describe the force.


Well, because it's not linear relationship indicates that what the network found is a


rotation of what you really want.


And that's good, because it's equivalent, not nonlinear basically means that you can't


really it doesn't really describe what you want really well.


Yeah, and I'm going to leave you with that, I absolutely invite you to check out the code


Parti - Scaling Autoregressive Models for Content-Rich Text-to-Image Generation (Paper Explained)

Not a day goes by in AI research in which we don't get a new image generation model these days.


So take a look at the top row right here and listen to the prompt that generated them.


Oil on canvas painting of a blue night sky with roiling energy.


A fuzzy and bright yellow crescent moon shining at the top.


Below the exploding yellow stars and radiating swirls of blue, a distant village sits quietly on the right.


Connecting earth and sky is a flame-like cypress tree with curling and swaying branches on the left.


A church spire rises as a beacon over rolling blue hills.


That is a 67 word description of Starry Night by Vincent van Gogh.


And it is also the prompt that generated the top row of images.


And the paper does this to show that image generation models, specifically this one,


they have become super duper capable of incorporating not only wild concepts, as you can see here,


co-locating the Eiffel Tower with the Sydney skyline and fireworks and whatnot,


but also, you know, minute details about things in the image and where things are and how things look.


So we've gone from essentially conditional GANs where we could create one of 10 classes


to something where we can input like a little essay about what we want to see and get it out.


So this is by a group of researchers out of Google Research,


and they are a parallel work to the Imagen model that you might have seen.


So this model or the paper is called Scaling Autoregressive Models for Content Rich Text to Image Generation.


But the model is called, let me grab, if I can, let me grab pen.


The model is called P-A-R-T-I.


And I have no clue how to pronounce this.


This could be party.


Maybe the pronunciation is on the art or on the part because it's pathways like it's, huh?


Or part-tie or I have no idea. Let's call it party.


And party is a model that generates images from text as we have so many models.


However, it doesn't do this in the same style as like Imagen, which is a diffusion model.


It is an autoregressive model. So here you can see a bunch of other outputs like this.


This is insane. Look at the left side right here.


A photo of a frog reading the newspaper named Toad Day.


The newspaper is named Toad Day. Like, how crazy is that?


That in itself is pretty funny.


But we know that these image to, sorry, these text to image models are pretty bad at spelling stuff in images.


Well, not this model, as you can see right here. It gets it completely right.


It doesn't always get it right, but it gets it right often enough.


Or this one, portrait of a statue of the Egyptian god Anubis wearing aviator goggles.


Like another connoisseur of fine eyewear, I see.


White T-shirt and the leather jacket. The city of Los Angeles is in the background.


High res DSLR photograph. That's literally that's the academic version of the Unreal Engine trick right here.


And you can see the images spot on.


So this requires a lot of knowledge, not only of, you know, what a DSLR photograph is,


but also how the skyline of Los Angeles looks, how the Egyptian god Anubis looks.


Right. And the composition of things together like these.


This god was never in a leather jacket depicted, I guess maybe on the on the Internet you'll find anything.


But you can see a bunch of more examples right here.


I specifically love the thing on the left side here.


You can see that they generated images.


So the prompt is three quarters front view of a XYZ coming around a curve in a mountain road looking over a green valley on a cloudy day.


So X here is any of the colors blue, red and yellow.


Y is any of the numbers.


1977, 1997 and 2017.


And Z is any of these car types.


And now look that the model can essentially track the historical evolution of these cars.


So not only does it know what a Porsche is, it also knows how a Porsche in 77 looked like.


Maybe it's not exactly the correct year, but this is pretty crazy.


You can see a bunch more examples right here.


They do a lot of examples with animals.


I specifically like the raccoon here in the style of cubism.


So this is going to be very, very powerful technology.


We can immediately see that, you know, the quality of these models gets fast, gets quickly, sorry, gets well, gets better so quickly that in the foreseeable future,


we're going to have super powerful tools to just create and edit images from text.


Look at the left side here, a giant cobra snake made from salad.


You know, I'm sure they even say these are cherry picked, but still this is insane.


Now, I would love to tell you that behind all of this cool development is a really cool idea, like is a smart architecture and something like this.


But I'm afraid it is not. It is simply scale and not simply scale.


I mean, you have to have the sort of correct base architecture.


There's nothing like particularly there's no cool invention in architecture or a neat trick involved or anything like this.


It's really just plug basic things together, make them really big, train them for long on a lot of data and you'll get quality.


So this is the model overview right here, the overview of this party or part time model.


This is, as I already said, in contrast to Imagen, it is an autoregressive model, so not a diffusion model.


What happens is that on this side here, you have this VQGAN image encoder and decoder.


Well, they don't call them encoder and decoder. They call them tokenizer and de tokenizer.


So if you are not aware, autoregressive models, they work on tokens.


Now, tokens in usually in natural language processing are words or part of words.


So these would be tokens, token one, token two and so on until token N.


And then what you would try to do is you would try always to predict the next token.


That's what makes it autoregressive.


You feed in parts of a token sequence, like parts of a sentence. You try to predict the next one.


That's exactly what you see right here in the architecture.


So you pass in the start of sentence token. You try to predict the first token and you pass in the first token.


And then from these two, you try to predict the second token and then put that here from these three.


You try to predict the third token and so on. That's the autoregressivity. In text, that works well.


However, in images, it's not quite obvious how to do that.


That's why you first need to get from the image space to the token space.


So we need a way for any given image that we get out a sequence of tokens.


And it can't be the pixels themselves.


We would like to have tokens that are kind of latent and have sort of a bit of meaning, not just individual pixels, because that, first of all, is too many pixels.


And second of all, there's not too much, let's say, information in the single pixel.


So what we do is we have these image tokenizer and de-tokenizer.


This is a VQGAN that is powered by a vision transformer.


So essentially, this is a model that takes this image, it ships it through a bunch of layers.


And at the end, so let's say the image at the beginning has a bunch of rows, a bunch of columns with its pixels.


This goes through a series of maybe downscalings and so on.


No, actually, it's because it's a vision transformer, it probably even tokenizes like it patches the image at the very beginning.


So these would be image patches.


Then these are transformed by a transformer to a latent space.


Maybe they are compressed.


And then you get tokens.


So at the end, you can take these things right here or the things that correspond to them in the latent representation.


You can take those as image tokens and you can unroll essentially this image and then feed it into this model.


Hey, just a short interjection here from Janek from the future.


The idea, I forgot, the idea behind the whole setup here is behind the whole VQGAN is obviously that these things here.


Are tokens, which means that they come from a set vocabulary.


So the way you train a VQGAN isn't just to give you this latent representation of like token like things, but then you also quantize them.


So there is also a vocabulary somewhere where you have a set defined set of tokens.


I believe in their case, they have like eight eight thousand tokens or so.


And your image tokens must be of these eight thousand.


So the image has a bunch of tokens, but they all must be one of the things in the vocabulary here.


Now, the vocabulary is also learned.


There are some techniques by which to learn the vocabulary.


But this quantization is actually what then enables you to treat essentially to treat it as a sequence of language tokens, which also come from a vocabulary.


All right. Back to Janek in the past. The image tokenizer is trained as an as it says here as a VQGAN, which means that you encode and then you decode again and you try to get out the same image.


And at the end, this representation here in the middle is really valuable because it's a tokenized representation of an image.


So you put that into the transformer right here.


And this is, as we said, an autoregressive model.


So it gets as an input, obviously the sequence so far, it tries to predict the next image token, but also gets as an input the text.


So this is the prompt that the user puts in.


So the prompt is encoded in a transformer encoder and is then fed in as a side input as a target for attention.


So whenever in the layer here you have queries, keys and values, I'm going to guess the query can also look at the transformer encoder.


The query can also look at the keys right here.


So over here, you'd only have keys and values.


If you don't know what all of this means, I have a video on attention is all you need, where you can learn how attention mechanisms work.


So essentially the way this is trained is the following.


You attach a sentence here or a description of an image and you attach an image right here.


The image is then patched, it is fed through the VQGAN encoder, its latent representation is obtained.


That latent representation is put here.


And then you essentially train a decoder language model that has cross attention into the text representation of the prompt.


So you simply train this thing right here like you would train a GPT model or any other model.


And this thing right here is trained, as I said, as an image reconstruction model.


And this thing right here is trained, I guess, jointly with this.


Actually, don't know. This could this could not be true, but I think it is true.


I think it is trained jointly. So that's the model.


As I said, it's very basic. I wish I could tell you something more interesting right here, but I can't.


It's a standard, you know, bunch of transformers in sequence.


Essentially, every single component right here is a transformer.


And because every single thing is a transformer, you can scale this thing by a lot.


By the way, here you can see a bunch of the I'm not going to go into the architectural details quite quite as much, but they do also train an upsampler.


So they have images of resolution 256 by 256.


Ultimately, they do train an upsampler as well, where.


So here this is the upsampler super resolution upsampler where they can go from their pipeline, which does 256 by 256 to a 1024 by 1024 picture, essentially.


But this is just upsampling, right? So there is, I mean, technically no extra information right here.


This doesn't get to look at the prompt or anything like this.


It simply gets to look at this image and then make a four times larger image out of that.


So where did we leave off? Oh, yeah.


I also wanted to say if you now want to get an image out of this thing, so not training, but inference.


What you do is you attach only the prompt right here, right?


You encode the prompt. You put the start of sentence token right here.


You let the model generate one, then you put that here, two, then you put that here, three and so on.


You let the model generate the image tokens here.


You take those image tokens, you feed, you arrange it into the latent representation of the VQGAN and you use the decoder right here in order to generate the final image.


So that's the whole flow.


And then you put it through the super resolution if you want that.


Here you can see the basics, the basic architectural layouts.


So there is the smallest model has 350 million parameter.


You can see it has 12 encoder and 12 decoder layer.


It's pretty standard transformer scaling laws right here.


I mean, scaling laws, pretty standard transformer architectural laws.


They go through a 750 million parameter model, three billion.


And the last one here has 20 billion parameters.


So that's a decently sized model. It's not as large as the large language models.


And they do use things like sparse conv attention and things like this.


But it is, you know, it's pretty large, I would say.


You could not run that at home very easily.


So where does that get us?


They have a big description right here, how they solve this architecturally, how they chart the model, how they use parallelism, which is very interesting.


I'm just not an expert at it.


So if you're interested, I'll leave you to read this part.


I found the at least the drawings here pretty cool.


So apparently this the signal is routed like, you know, like so, like so and so, so like in like a snake type of arrangement so that always you can pipeline so that always one thing is essentially busy as you send data to the next thing and so on.


But as I said, I'm not the expert in this and I'd rather want to get to the other things which are the data sets that they use.


So they have three data sets, three main data sets right here.


One is MS Coco. Now MS Coco, as they show right here for the image on the right hand side, it simply says a bowl of broccoli and apples with a utensil.


So it just kind of is a high level description of what's in the image.


It's like an image, simple image caption right for this image right here.


Whereas the localized narratives data set, you can see that its description is way longer.


It's more linguistically prosaic, but it is also much more descriptive of the actual image.


Like so the top is if you want to tell someone what's in an image and the bottom is more like if you want to like really paint the picture like pun intended or if you want to describe the picture to someone so that they could maybe recreate it in some way.


And it turns out that we are now at the point with these image generation models where they are so good that we need data sets like the bottom one to really push them to their limits.


And not only that, but the authors here find that there are even problems with that because these image data sets, they're always created in a way that an image is given and then the humans are asked to write a description, which is really good because then you have image and description together.


However, the authors here know that this prevents, for example, fantasy pictures like we saw before the raccoon in cubism that it doesn't exist.


So it can't be in any data set or a noob is in a leather jacket doesn't exist.


So it can't be in any data set.


Now, while we rely on generalization during training for the model to learn these things, we actually need data sets like that to evaluate whether they can really do these things right.


Otherwise, we're left with sort of subjective evaluation.


So they come up with their own data set, which is called party prompts.


That's actually also the thing they release as far as I understand it, obviously, as all of the recent works in big models, this thing isn't released.


There's no code. There's no, I mean, the code would be trivial. There's no weights. There's no training recipe. There's no, some of the data sets are proprietary if I understand correctly.


So the paper is more open about what they do, but still that there is no way of accessing this.


So party prompts. This is a data set that essentially only consists of prompts.


So there is no images in this data set. And I believe the only way you can really assess thing is you can let the model generate stuff and then you can let humans rate it.


That's essentially it.


The party prompts, it is pretty interesting because they create these prompts by letting the prompt engineers sort of, they choose, for example, a challenge.


So the challenge might be perspective, right, which could be, you know, I need a prompt that asks for some object in some specific perspective that is unusual.


Or, yeah, quantity, like I need a prompt that asks for a given number of things because we know that these models, they're not super good at counting, right?


I mean, we also thought the models aren't super good at spelling and now it turns out, well, if we just make them bigger, they are.


So, you know, I'm fairly confident they're going to be good at counting in a short while.


That's the challenge. There's also, if I recall correctly, this is this upper table right here, like categories.


So there are categories, animals, there are categories, illustrations and so on.


So you can see this is a diverse set of category challenge combinations and they make a bunch of prompts for each one.


I think they have about 1600 prompts in total in this party prompt eval set, which is a pretty neat thing to have, even if it comes without images.


So now they train the thing with their whole architectural shabangs with the parallelism and the pipelining and the yada, yada, yada on TPU v4, I think.


So this is a huge operation. So what does that give us?


I want to just jump the evals here on the metrics because yes, yes, yes, they're very good, very good.


They're also very good as rated by humans, humans, very good.


Which is what's interesting is they have, for example, a retrieval baseline which simply retrieves images from the training data set.


And even if the obviously image text match, the party model wins because you can actually create an image and not retrieve one.


But even in image realism, you can see the retrieval is only slightly higher in realism, right?


Every single image is real that the retrieval retrieves.


And still the humans rate the realism of party almost the same, which is quite speaking for the model.


The loss curves are also pretty interesting, especially interesting that the 20 billion model here, it takes quite a time to come down here, right?


It kind of has to get surpassed by the three billion model initially and then overtakes it.


Which maybe means that we haven't exactly found the right training recipes yet for these largest of models.


So this now is the cool part where they put the model, the models next to one another.


So this is the same prompt with all of these different models.


And you can just see where scale gets you.


This is a portrait photo of a kangaroo wearing an orange hoodie and blue sunglasses standing on the grass in front of the Sydney Opera House, holding a sign on the chest that says Welcome Friends.


And you can see my this, these these things right here, this and this.


There may be like dolly mini kind of style pictures.


And there are also that scale, right?


And then we go to the three B model, and this is something that would be familiar maybe from something like Dali or Dali, maybe between Dali and Dali to write these things.


You can see they're bad at spelling.


But as soon as you go bigger, all of a sudden, welcome friends.


But a boom, there it is.


Not bad at spelling anymore.


All you need to scale.


That's crazy.


The sign very deep learning.


Look, as the model learns to spell, initially it can only do Russian or whatever.


And just eventually, it would actually be funny if that was like actual Russian and it said very deep learning.


Can you imagine how crazy that would be?


Well, in any case, and also the Grand Canyon, right?


So there's kind of structure here and so on.


But this very, very deep learning.


Perfect.


A blue Porsche parked in front of a yellow brick wall.


You can see it doesn't always work, but it works better and better and better with scale.


Crazy.


And here, this is like maybe like is this a direct shot of Gary Marcus?


Because the challenge is like an astronaut riding a horse.


So astronaut riding a horse in the forest, even the three billion model.


Oh, no, it's going to be a horse riding an astronaut, which is going to come up later.


And I promise it's going to be funny.


But yeah, an astronaut riding a horse in the water in front of them, water lilies and so on.


A map of the United States made out of sushi.


So as you can see, these these results are fairly insane. Infinity, the back of a violin, four cats surrounding a dog.


So now they're really testing these individual categories.


Infinity is an abstract concept. Back of violin is perspective. Four cats surrounding a dog is this quantity metric.


You can you can see there are four cats, right?


So, yeah, I'm pretty confident that with scale, these types of problems are going to be solved.


Scroll gives an apple to a bird.


Yeah, so. What's interesting is they have this narrative of what they call growing a cherry tree.


So obviously these samples here are cherry picked, which means that they take out whatever they think are good samples to present in the paper.


However, they detail fairly extensively how they arrive at this thing.


So what they do is they don't just come up with these long prompts by themselves.


Well, these aren't long. OK, but you know that these long prompts with Anubis in front in a leather jacket in front of Los Angeles skyline,


they don't just come up with them on the spot. They have a process of coming up with them.


And the process is detailed here.


So, for example, they have this idea of combining like a sloth with a van.


Right. So they start by just exploring the model and entering things like a smiling sloth, like what comes out.


Right. And a van parked on grass.


There are always good images and bad images that turn out.


And they sort of learn how they have to tweak the prompt to get what they want.


Once they're happy, they go on. So they modify the prompt a bit.


So here is the smiling sloth wearing a leather jacket, a cowboy hat and a kilt or wearing a bow tie and holding a quarterstaff.


So they kind of explore. They go more and more, as you can see, as you go down this tree, this cherry tree, as they call it.


They go down and down. They detail well. Sometimes there's problems.


This one, I believe, has two arms on this side and so on.


So, but still, they refine and refine and refine. They finally try to combine them. Right.


Yeah, here is a combination. They refine again. They try to combine the two prompts again.


And at the end, they get to something that they might be happy with.


For example, the thing here on the left, like this one right here.


But I found this pretty interesting, like this process of arriving at these things.


So you can't just enter any old long sentence and expect the model to do well.


But what turns, what might, what will work often better, at least as they describe it, is to go through this process right here.


Which also means that full artistic freedom is a bit away.


So it is almost like, yes, you are guiding the model with your inputs, but also the model is kind of guiding you by what it does well and what it doesn't do well if you go via this process.


And if you don't go via this process, then I guess you can expect that you can expect that it might not work as well.


So they also have some failure cases, which is pretty cool.


For example, they failure cases like color bleeding, where you describe the color of one of the things in the image and sort of the other take on that color.


There's also counting failures and so on, localization failures.


For example, here the prompt is, the prompt is, oh yeah, the Great Pyramid of Giza situated in front of Mount Everest.


That's the bottom two pictures should be that.


You can see this. OK, I mean, this isn't, this isn't too bad.


But this here is just like the pyramid with sort of a Mount Everest cover, right?


You can see these models, they sometimes if they can't fulfill the prompt directly, they'll kind of mix, they'll just try to get it done somehow and get it really close in text embedding space.


That's exactly what you can see right here.


There's a bunch, a bunch of examples and this one, I told you, it's the horse riding on an astronaut.


So they have to actually specify the horse is sitting on an astronaut because the riding is just, is just riding indicates too much that the horse is on the bottom.


But I just found the horse riding on the astronaut to be absolutely hilarious.


Especially this one.


Yeah, but all in all, I guess what I wanted to say is that this is complaining on a on a very, very high level, right?


The paper itself is like moving the goal posts already by sort of criticizing itself for, oh, well, I specified like nine apples in a perfect arrangement.


I don't have, or right, ten red apples and it's only eight red apples.


Like what a what a loser model. Look at that.


I mean, this is it is crazy good how these models are and the failure cases here are, you know, yes, they're failure cases.


But I don't think that if you told me three, four years ago that this is the type of error that we're at solving that I would have said, yeah, I believe that.


I would have way guessed we're still at the point where, you know, we we have mode collapses, we can't create most of the text stuff, we have artifacts and all kinds of things.


And I think this is, yeah, it's it's kind of mind blowing how fast the progress here is.


Obviously, half a year ago or so. Yeah, I would have expected something like this.


But I believe, yeah, a lot of people must be very surprised and including me.


Yeah, like spelling mistakes, like complaining that, you know, sometimes text is still not spelled right.


Like, nah. Even though, right, Dali couldn't do it at all.


And now this thing is doing it almost perfectly, as you can see right here, combining abstract concepts. Look at the thing on top.


It's it's insane. Or here, like, oh, this leg is in behind the race car.


Come on. This is better than I guess anyone had expected.


So, yeah, I don't want to waste your time too much more.


I just thought this was absolutely cool and I'm very excited to see where this is going next.


Of course, huge bummer that we don't get access to this.


I hope this finds its way into some products that we can use.


As you know, I'm all for these companies making money with their inventions.


I mean, I think it's cool that they are inventing and, you know, if they want to make some cash off of it, you know, good for them.


But I do hope that we actually get to use it.


And I it's going to be a fun future where for every presentation or anything, if you need like an illustration, you just you just type it right.


You don't go to the Internet to search an appropriate stock photo.


You just type it. It's so cool. Or you want to change something in a picture. You just erase it.


You just say, well, ever here, change that part to something else. So cool.


No Photoshop skills anymore. No drawing skills anymore. Just you and your mind and your creativity.


All right. That was it. As I said, the paper presented in this new system is fairly simple.


All it does is scale a bunch of transformers in sequence. Essentially, I presented a evaluation benchmark, these party prompts, and it presented.


Yeah, they're their model, which is ridiculously insane.


Divide-and-Conquer Monte Carlo Tree Search For Goal-Directed Planning (Paper Explained)

Hi there! What you're seeing here is a divide and conquer Monte Carlo Tree Search in action.


This is a planning algorithm that plans in a kind of an unconventional fashion.


So we're going to explore this today in this paper.


Divide and conquer Monte Carlo Tree Search for Goal-Directed Planning


by Gian Battista Parascondolo and Lars Pyhsing and a list of other authors.


And I believe this is from Deep Mind and Max Planck and Eta and... yeah, that's it.


Alright, so what does this thing do? It is a planning algorithm.


And planning might be not really familiar for you.


So let's say you are in this room and or this set of rooms right here.


There is a bunch of walls and you are up here and you want to reach the goal down here.


So first of all, this is a goal-directed problem, right?


Here it says goal-directed. Goal-directed means that you give the algorithm a goal to reach.


And this could be a different goal each time you're on the algorithm, right?


So you give it a goal to reach. Then the second thing here we see is planning.


So it is a planning algorithm. What does planning mean?


Planning means if you're traditionally reinforcement learning, you would think,


ah, I'm just going to go ahead and run my agent here.


And maybe you can move in the four different directions, run my agent here, do some things, right?


Blah, dah, dah, dah. Maybe I hit a wall, I get a negative reward, I try again.


In planning, you don't have to move initially.


What you can do is think about moving and you can think ahead of what's going to happen.


And that's usually because you have some sort of model what happens.


This is very famous applied in, for example, AlphaGo or AlphaZero, where you have,


you know, if I move to the right here, I'm going to be here, right?


So you will know once you've reached the goal, right?


If I'm here, I go down, I reach the goal. Cool.


So you can think all of this without actually moving in the environment.


You can think yourself ahead. What would happen if I did certain things?


And that in turn also means that you can think ahead multiple different paths.


So you can think ahead what would happen if I move right, what would happen if I move down?


And then you can think in the next layer, if I move right, what would happen if I moved right again?


What would happen if I moved down instead?


So you can easily see the planning problem becomes a tree search problem.


In this case, we've done a breadth first search, right?


And eventually you'll see that this will get you to the goal.


So this breadth first search, or maybe you want to employ depth first search, will ultimately get you to the goal.


We can represent this as a search tree.


So you're here in a particular state and you have a bunch of actions, in this case four, to move left, up, down or right.


And you can choose any of them and you will get into a new state.


And from each of those, you could choose any again.


And you can think ahead all of this, you can construct this entire tree.


And one of these branches will lead you to the goal.


Promise.


What is the problem?


The problem is if the path to the goal here is, let's say, D steps long, then this tree here is going to be D layers deep.


And in our case, that means we'll have four to the D nodes in that tree before we even reach the goal.


And that is just a long, long or a big tree.


And we can't construct all of it.


So algorithms have come along.


For example, the A star algorithm where you can incorporate something like a heuristic.


And the heuristic would be, well, it's generally good if I'm close to the goal in L2 distance.


So you would not build the entire tree here, but you would prefer nodes that will bring you towards this heuristic.


So this node down here is closer to the green node in terms of L2 distance.


And this node down here is even closer.


But then you're kind of stuck.


So A star will explore a bit probably along this wall here.


And once you're here, you have a clear path again, right?


So you can simply take the actions that minimize this L2 distance.


So this will already get you to a real good point.


Monte Carlo tree search, as it was employed in AlphaGo or AlphaStar, has a similar structure where it, after a certain while, it stops and evaluates a heuristic to say what's the value here and so on.


So it is in for some problems, an even better method of constructing the search tree in a way where you don't get overblown by the number.


So the Monte Carlo search tree in this algorithm refers to the fact that we are generalizing Monte Carlo tree search from the let's say the AlphaGo paper.


So what's the idea here?


So far, we've known everything.


The idea is the following.


If I had an oracle, if I am the master here, and I can tell the agent, agent, look, I guarantee you that this state right here in the middle, you will pass that state.


If you want to reach the goal, you will pass this for sure.


If I tell this to the agent, now what can the agent do?


The agent can say, oh, okay, if I know that, I can simply,


I don't have to search for a way to the goal.


I'd much rather search for a way from my start point to that point where I know that I'm guaranteed to be at some place.


And then I can search also from a way from there to the goal.


Right.


So this now, remember our long our path was D steps long for the original problem.


This is now let's say D half long.


Each of these paths and that means we just construct two trees.


Each one of them is going to be four to the D half.


And the other one is also four to the D half.


And if we add them, that is much smaller than the original four to the D tree that we build.


So right there, we have subdivided our problem into two sub problems.


And each of them are much easier than the original problem.


This paper basically does this recursively.


So it will subdivide the problem by proposing some middle state where you are going to be at some point for sure.


And that for sure, we're going to take a look at.


And then for each of those problems, again, it will try to subdivided into sub problems and therefore recursively solve the sub problems until they're small enough that they can be basically solved in one step.


So that's the big idea here.


And this is illustrated in this point right here.


So you are in this S zero, the start state, and you want to go to this S infinity, the goal state in your case.


What this paper does is it proposes to split the problem here in the middle and then simply solve the two problems recursively.


Now what is a bit confusing right here is that it is the planning already is a tree search, right?


So a plan is like a tree.


But we are searching over plan.


So we're searching for the best plan, which means that we are searching over trees.


And that search itself is a tree search.


So the search itself is a problem where we go down one route, and then oh no, and then we maybe go down another route.


And here and then here.


So the search is a tree.


So we're now tree searching over trees.


That's the kind of tricky thing to remember.


So each of these plans, even if it's half, if it's only half done, like this is only half a plan, we don't know what what's going to happen in here.


This half a plan is a node in the tree that we're searching over.


And then it splits it, it splits as you can see here into two sub problems.


The two sub problems also are nodes in that search tree.


So you see that this top thing here would correspond to this node, even though in itself, it is a plan, a tree, a string in this case.


And each of the two sub problems would become these particular nodes in the search tree.


So keep that in mind as we go through this paper.


It is very easy to get confused in this respect.


The algorithm is pretty simple in this case.


So this algorithm rests on this traverse procedure.


So we're going to we're going to traverse this, what they call these or nodes.


So they they divide the problem into and nodes and or nodes.


I don't believe that's particularly necessary for us to think about.


But here's how this works.


So they traverse the or node S to S prime.


This is simply again, this is a node, but the node is a path from S to S prime, where we don't know yet what happens in between.


Right. So what we'll do is we'll run this procedure, select and select it.


You can see it outputs an S prime and the S prime is going to be a node here somewhere in the middle,


where the model says this is your subdivision point.


And then it will recursively traverse the left and the right branch of this tree.


So it will subdivide the problem into two problems and then recursively call this traverse.


You see that's the function that we're defining.


Call this traverse function on these.


So it will subdivide this problem into these problems and for the next for the next step.


Again, it will eat for each of the problems, propose a middle node and subdivided further and so on until you have a full plan.


Right. At some point, you're going to have a full plan.


Now, here again is the important thing to remember.


This is just one branch of the search.


This is just one possible plan.


And we are going to do a tree search over these plans.


So this select function here, it has returned this S prime, but it could have returned any point between S and S double prime.


So let in it. This is just one branch.


I'm going to I don't have space to draw here, but I'm going to draw it down here again.


So it could have also returned this particular node here.


It's a different S prime and then subdivided the problem into different problems.


And then, of course, those problems are now different.


So they would be subdivided differently again and so on.


So this top part here is just if you consider this thing here, your root node.


This is where you search from.


This top part is just one node, one branch in the tree.


But we could also subdivide like this.


And then that would be another branch in your tree.


And this tree here is the thing that you're searching over.


Important to keep this in mind.


We're searching over these different possibilities.


Now, the rest of this algorithm here is basically the carryover from Monte Carlo tree search.


And I don't want to go into that in this video.


But if you're interested in how to actually implement this, you'll have to go look at MCTS.


And then all of this just carries over from that algorithm because you have to keep estimates of value and visit counts in this tree and so on.


And also you have some sort of a value estimator.


But yeah, I'm mainly concerned with how the tree is constructed right here.


So basically, here's the here's the difference between a between the Monte Carlo tree search and the divide and conquer Monte Carlo tree search in Monte Carlo tree search.


Ignore the yellow one for now.


You're in the green position and you want to go to the blue position in Monte Carlo tree search.


What you're searching over is the next action to take.


In this case, you have four possible actions to take.


That's what you're searching over.


And that's what you build your search tree from.


Your search tree is going to be which action to take right up, left, down or right.


That's why you have four actions in month in divide and conquer Monte Carlo tree search.


You're not searching over actions.


You are searching over the best way to subdivide this problem, right?


You're searching over which of these all the black squares should I use to subdivide my problem into sub problems.


So naturally, you you can already see what kind of possibilities do we have here to subdivide this problem.


I drew one white square, but any of the black squares are candidates to subdivide this problem, right?


Any of the of the black squares could be potential subdivisions.


And this is what we search over.


So in in Monte Carlo tree search, we search over the actions, which gives us this four to the D tree.


But in divide and conquer, we're searching over all the ways to subdivide the problem.


As you can see there, that are many, many more possibilities.


So from this first starting node, we have like, like 100 possibilities to subdivide this problem into two problems, right?


And each of those again, if you now you've decided on a subdivision.


Let's say you decided on this one right here, you say I want to pass through that point on my way to the goal.


Now, you have to subdivide that in this sub problem into two problems.


Again, every possible black square, I'm not saying which one is good, a good thing to subdivide the problem.


I'm just asking what is a possible candidate.


Every single black square here is a possible candidate for for a path from here to here.


Right. And again, for this particular sub problem, you have to do the same thing.


So the the search tree here, even though we said before, it is this one is very deep.


And this one is probably only log D, log 2D deep.


It with is going to be enormous. And that is the catch, right?


The catch, this is not a method that is like a magic pill.


The catch is, even though your tree is not as deep, it is much, much wider.


And it is intuitive, right? Because you still have to have the ability to construct any possible plan.


So your tree is going to have as many nodes as the original Monte Carlo tree search tree if you were to fully expand it.


Right. So it's your trading of depth for width here.


I hope I hope that's a bit clear.


So your entire your entire promise of this method is going to be can you from all of these possibilities.


So from all of these, you don't even you don't even want to go and search even one layer deep through all of these.


You don't even want to consider all of them. Right. You want to search in this tree.


You want to limit your search to very particular ways of subdivision here.


If you can do that efficiently, if you can pick efficiently candidates to subdivide,


then this could be a successful thing because your deep is now not as your tree is not as deep as the original search tree would have been.


And you can limit the width effectively to only very few candidates.


So here we could, for example, make a heuristic that will always only pick squares that are kind of on this straight path to the goal.


So everything rests on how you do this select action.


This thing here. The entire algorithm relies on the fact that you can select effectively select in between states where you're pretty sure that the algorithm will have to pass through there.


Because the worse you make these predictions, the worse your tree search is going to work.


And what they do, of course, is they use deep learning as you might have guessed to do that.


So they have they will have a model that for a particular start and end goal will give them a probability distribution across candidates.


Now, everything that's black here also has probability mass, but it's just so small you can't see.


And these blue ones are the lighter blue, the more probable this model thinks that this is going to be an in between state.


Now, the tree search can now limit itself effectively to only the ones here with the highest probability.


Right. So we select the ones with the highest probability and will only search plans that have these as the first possible subdivisions.


Again, we're searching over plans. So we're searching over ways to subdivide the problem into smaller problems.


That is our search space. So once we've decided on one of them, let's say here the yellow one.


Again, we have to evaluate that model for each of the sub problems. And this this is kind of a step that's missing here.


So in between here, there would be a model evaluation that would again tell you which of these in between states were probable subdivision candidates.


And then you would select again one of those in that particular search branch and in a different search branch.


Right. You're searching over these things in a different search branch. You would select a different one and see, is this possibly a better way to subdivide the problem?


And so on. So the question, of course, is how do you train this model? How do you train a model that gives you good candidates for subdivision?


And the answer here is it comes from the idea of hindsight experience replay.


So let's say again, you are here and you want to go here and you're not very good at it initially.


So they train this model, as I understand, along with letting their agent act in this environment.


So the agent uses the model to plan, but initially it's not very good. So maybe the agent will fail a lot of times.


So instead of going to the blue square, it will reach this white square right here.


It will go here, here and here. We'll reach the white square. Instead of saying I failed, what you can do, and this is the idea of hindsight experience replay,


is to say, well, I did fail, but I did reach something, right? I have reached a thing.


And it's actually possible that that thing could have been the goal. But this particular episode, this was the goal.


Remember, the goal changes every time. It's a goal-directed policy. So it says, well, this could have been the goal, possibly.


So if I just pretend this was the goal, then I have a training example for a successful run.


So the hindsight experience replay basically pretends that what you have achieved, even if you failed, was your actual goal.


And that gives you a positive training example for an episode with that as a goal.


And it could have been the goal because the goal is chosen at random.


So this gives you a good training example. Now this paper just generalizes the hindsight experience replay or applies to their particular framework.


And they say, well, if I reach this thing, that means any point on this path is a good candidate for subdividing the path, because I did actually reach the point.


Remember, the goal is to propose a point where you're for sure are going to pass through.


Now, since I've taken this path to this goal, I have passed through any of the squares in between.


And so these are my possible sub candidates and all other black squares. I don't want that.


So now I have a classifier I can train. I can say any of these squares on my path are good squares to subdivide and any not on my path are bad ones.


They go a step further, I believe. And they actually say, we're so if this was M steps, we're actually going to take the particular square that is reached after M half steps.


So the exact middle point of that path is going to be our training example for subdivision.


So you have a classifier that has exactly one one target to train.


So this you train along with acting in the environment. And of course, your model for proposing subdivisions is going to be better and better and better and better.


And that makes your planning algorithm better and better. And that makes you collect better episodes.


And so you can sort of sort of get bootstrap yourself up this thing.


Now, this is the basic experiment of the paper. They also do this in a 3d manner, where they move this little spider here around.


So the spider was trained to just move from one block to the next block. And the planner basically has to tell it where to go.


And they show that they outperform the traditional Monte Carlo tree search.


Now I have to say this is cool, but you have to remember this is only advantageous in very, very specific types of problems.


So first of all, it has to be this goal directed nature. Otherwise, you probably couldn't train this predictor here super well.


Then, given that you have such a good predictor, the problem needs to be such that if you have a start state, there could be many ways to go about reaching the end.


And if you have an end state, there could be many ways from where you could come from.


But but there is like some bottleneck state in the middle, where you're pretty sure that you're going to have to pass through it.


So if your problem is of that nature, right, if it has these bottleneck states, where you can predict with reasonable accuracy that you're going to have to pass through,


then this is a good algorithm to consider and is obviously I mean, it's intuitively outperforming the original Monte Carlo tree search,


because you have much less deep search tree, and you can effectively limit its width by using that model.


They also have made this website where they kind of show videos of their spider.


And I haven't seen it in a while, but it is it is like next to the mouse, if you can see it.


So. So you see, this is kind of a continuous control problem that also requires planning.


And they also have these kind of gifs of how they're there, what order their plans are constructed in.


So I invite you to check this out. Read the paper. If you like this, subscribe, leave a like, leave a comment.


How much memory does Longformer use?

So I wanted to come back to this paper here about the longformer. I have done a video on this. If you


How much memory does Longformer use?

haven't seen it then this video is probably not going to make much sense to you. But in the video


How much memory does Longformer use?

I go over what the longformer is, what it does, how it compares and so on. And the gist of the


How much memory does Longformer use?

longformer is that it can now do a transformer model on a long document as you can read here.


How much memory does Longformer use?

So I've gotten a lot of questions of like does that mean we can now have much longer documents,


How much memory does Longformer use?

right? The BERT model doesn't fit into my memory. Can this solve my problem? And I just kind of


How much memory does Longformer use?

want to go into the math of the longformer memory requirements here because I think I've alluded


How much memory does Longformer use?

to it but it is quite a... I think the graphics here are just a bit misleading from the way they


How much memory does Longformer use?

implement it. Now I've already gone over something like this in the last thing. So Roberta, let's


How much memory does Longformer use?

spell this correctly, Roberta that is their baseline has a size, let's call that n0 of 512.


How much memory does Longformer use?

So they can have 512 tokens at the same time. So if you have a sequence that is way longer than 512


How much memory does Longformer use?

you need to chunk it up into pieces of 512 and usually do something like overlapping pieces or


How much memory does Longformer use?

something like this, right? And now the promise of the longformer as it is in the paper is that


How much memory does Longformer use?

you can put all of this into the longformer, right? And it will do this sliding window attention


How much memory does Longformer use?

thing where it basically slides a window here, this window across this input sequence and only


How much memory does Longformer use?

does this local attention right within the window and then it also has some global attention that


How much memory does Longformer use?

it constantly has. Now what I find interesting is that in their experiments their window size here,


How much memory does Longformer use?

so the longformer window size is 512, right? So within that window you have the classic n squared


How much memory does Longformer use?

full attention, alright? So let's just go in into that. How much memory does the longformer really


How much memory does Longformer use?

do? We've already calculated it here a bit but I want to take this still apart a bit. So as you can


How much memory does Longformer use?

see on the left here you have n times w that you have for this middle band, right? So this middle


How much memory does Longformer use?

band is n times w. Then you want to add the global attention, right? So the global attention, you can


How much memory does Longformer use?

already see it right here. If you have one, two, three, four locations of global attention you have


How much memory does Longformer use?

four times two because you also have them in this direction, right? You have them in both directions


How much memory does Longformer use?

times your full sequence length. So plus two times full sequence length times the number of global


How much memory does Longformer use?

attention. I call this s over here. So as we saw up here the window size here was n0 in their


How much memory does Longformer use?

experiments. So let's replace this window size by n0 and actually let's factor out the n so we'll


How much memory does Longformer use?

get to n times n0 plus 2s. Alright so you can already see that Roberta originally had n0


How much memory does Longformer use?

squared. Now if n is larger than n0 that means you already use more here. The kind of trick,


How much memory does Longformer use?

let's, it's not really a trick, it is true that this is order of n, right? If n is your input


How much memory does Longformer use?

sequence length. But in this here is technically order of n squared if n, if this is n. But the


How much memory does Longformer use?

sequence length in Roberta was the was the window size of the long former. So this is n0 squared.


How much memory does Longformer use?

And here technically you'd have to say this is n times n0. So if n is larger than n0 you can see


How much memory does Longformer use?

that this uses more memory given that. So in their experiments they use a model that on paper uses


How much memory does Longformer use?

more memory than the baseline model. And saying that it it scales linearly with sequence length is


How much memory does Longformer use?

because I mean of course it scales linearly because they can now input these long sequences,


How much memory does Longformer use?

right? And the attention, sorry the memory requirements scales basically linear and also


How much memory does Longformer use?

linear with the window size. Now the window size still needs to be apparently large-ish in order


How much memory does Longformer use?

to achieve the performance. So the fact that the performance is equal or better is not really a


How much memory does Longformer use?

secret because it uses more memory, right? It's not like this model uses less memory but outperforms


How much memory does Longformer use?

the old one. It uses more. If you want to look at it you have to ask okay I have Roberta and right


How much memory does Longformer use?

now I can do n squared. So this is n, this is n, so there's n0, n0. This is my sequence length that


How much memory does Longformer use?

I can put into Roberta. You have to ask yourself what kind of sequence do I want to put in? And if


How much memory does Longformer use?

you say I want to put in a sequence that's twice as long, right? I want to put in this long of a


How much memory does Longformer use?

sequence. So n here would be twice n0. Then you have to take this put it here, put it here and


How much memory does Longformer use?

then you realize that your window size of the long former can only be half, right? So if you have the


How much memory does Longformer use?

same amount of memory you can double your sequence length at the cost of halving your window size.


How much memory does Longformer use?

But that doesn't yet include the cost of the global attention. So any global attention you do


How much memory does Longformer use?

will come basically on top of the window size. You see this here, right? So you decide on...


How much memory does Longformer use?

Let's do it like this. You decide on how long you want your thing, your input sequence length to be.


How much memory does Longformer use?

Then you decide, and that means that's this rectangle here, then you decide how many global


How much memory does Longformer use?

attentions do I want. And here I say I want one global attention and you have to cross out as many


How much memory does Longformer use?

rows here as you want global attention. And what remains is your window. Actually you have to cross


How much memory does Longformer use?

out twice. But we don't have... We only have one left. But you get the point. You have to cross out


How much memory does Longformer use?

two times s rows of how many global attentions you want and what remains will be your window size.


How much memory does Longformer use?

In this case it's just a window size of one. So that's how you would construct a long former that


How much memory does Longformer use?

takes in the same amount of memory as your classic model but can take a full n sequence length.


How much memory does Longformer use?

Alright? So I just wanted to kind of make that clear, go through the calculation myself.


How much memory does Longformer use?


Python 3 Programming Tutorial - Function Parameters

Hello everybody and welcome to another Python 3 basics video. In this video we're going to carry on with functions. In the last video you were shown just a very basic function and we didn't use any parameters. In this video we're actually going to include parameters and give this function more functionality.


So, let's define a function and we're just going to call this function simple underscore addition and we're going to have num1 and num2. And basically when you write this stuff out, you're just making it up like a variable name. You can pick anything you want, just write it in.


It's going to act like a variable within your function. So as usual, colon, enter, and now we will write out this function. Now what we want this function to do is just some simple addition for us. So, we're going to say answer equals num1 plus num2.


So that's going to give us our answer variable and just for the sake of printing out something else, let's also go ahead and print num1 is, and actually we don't need this here, num1 is and then num1.


Now what we want to do is actually print our answer too. So let's go ahead and print the answer. So print answer and now it will print out our answer. So num1 is going to be whatever we specify in the num1 parameter and the answer is going to be whatever num1 plus num2 is.


Obviously if you put letters or string or whatever in here, it's not going to work. So anyway, enter, enter, and we'll run simple underscore addition five three. So we'll save that and we'll run it.


And we get num1 is five and then the answer five plus three equals eight. Cool, easy enough. And there are no limit to the amount of variables that you can have or the amount of parameters that you can have in your function.


The only thing you really want to look out for at this point is the order of the variables as well as the quantity of variables. Now the order is as follows. So if you really wanted num1 to be five but instead you did three five, obviously num1 is going to be three, not five.


Now there's a couple of things that you can do here. You can one, protect yourself somewhat by doing something like this. So you could say num1 equals five and then num2 equals three like so. And you can hit, you can run that.


And it's going to work just like it always has. But you're not going to really see people do this too often. But what you could do is you could say num2 num1. And so even though they're in a different order than we specified that we'd receive them, because we're saying it explicitly here, we're allowed to do that.


So num1 is three and sure enough that did work. So that's going to work for us. Now quantity is also going to be an issue. So let's just revert back to what we were doing before, three five. We run that. That works out for us.


But if we have only one variable called, we get an error. Simple addition, missing one required positional argument. By that same token, if we have three five three or four, let's do that, and we run it, we're going to get another error. It takes two but three were given.


So that's why quantity is a big deal. Now that's going to be it for this specific video. In the next video I'm going to be covering the default variable assignments because default variables can also kind of change up quantity.


TAPAS: Weakly Supervised Table Parsing via Pre-training (Paper Explained)

Hi there, have a look at this table on the left. So in this table in each row you can see the


following things. The name of a wrestler, the number of times that wrestler has been a champion,


and the combined number of days where that wrestler has been a champion or like the sum of


the length of all their championships. Along with the column that is the rank and this is ranked by


the combined days attribute. So this table is very interesting by itself. But if you look at the


right we have a couple of questions. And let's try to answer them. Which wrestler had the most


number of rains? So for that you need to go to number of rains column, right? And you need to


mentally sort them and you'll find out that eight is the highest number. And therefore Ric Flair is


the wrestler you're looking for. Second question, the average time as a champion for top two wrestlers.


Now we need to go the top two wrestlers we can guess that pertains to the rank. And then so we


want the average of these two numbers. You even have questions such as which of the following


wrestlers were ranked in the bottom three? Which the answers would be all of those? And then after


that out of these who had more than one rain? And you can see that's Dan Severn. So the paper that


we're having here is trying to answer questions like this if given a table. As you can see,


this is a pretty, pretty hard task. And so pretty excited to read this. The paper is called tapas


weekly supervised table parsing via pre training by Jonathan Herzig, Pavel, Christoph Novak,


Thomas Miller, Francesco Piccino, and Julian Martin Eisenschloss. Full disclaimer, I know


these people, so I might be slightly biased. Alright, so you've already seen the task. The


task is you are given a table and a question and you're trying to answer that. Now, there,


it's not it's not as easy as that. But the table questions come in different forms. As you have


seen, sometimes you just need to select a cell from a table like we have here. The first question,


I simply most number of rains, I simply select whatever that is. So the answer here is already


in the table, Ric Flair. And this they call a cell selection task. This is wherever you need to


select a cell. The same for these bottom here. So which of the following wrestlers are ranked in the


bottom three and out of these which one of more than one rain, all of these answers are in the


table somewhere. The second thing is what they call a scalar answer. That is when the answer to


be computed is a number that is not in the table. So these average time here, which turns out to be


3426 is nowhere to be found in the table. So there actually needs to be a computation performed by


the model. And lastly, you have these things called ambiguous answers. Now the ambiguous answers


refer to a thing where it is a number that you're looking for. So how many, but the number here is


in the table. So you can think of this in terms of training data. If you have a task like this, and


you have training data, and you just have the question, you just have this question, and you're


given the answer to right, you can teach your model either to select this number here, or you can


teach your model. That will be wrong, right? Because how many world champions are there with


only one rain? To simply select the cell here is not correct, because that cell, even though the


number is two, it doesn't mean the same thing. It's not counting, right? So the correct program


here would be to count the number to count the cells where there is a one here, which is also


two. And they call this situation ambiguous answer. So you might have already guessed that a single


model that does all of this needs to sort of have multiple modes. That's exactly what they propose.


So they propose a model that takes in the table and the question. And then in the first step,


it selects its mode. So the mode is either the cell selection, or it is to compute something.


And then whenever it's a cell selection, it simply has a component to select cells. But


when it's a compute, it needs to decide in the second step what to compute. And then also select


the appropriate cells. So this is the model. Now this stuff like this has existed for a long time


in these table answering things. But the way we want to do it here is end to end with a single


deep learning model, of course, because we want to be better than anything else. And the trend in


deep learning is to put more and more into one model and to have it end to end differentiable.


Alright, so you see, we need multiple components, right, we need some sort of a mode selector,


we need some sort of a cell collector, and we need a thing that decides if we are in the compute


mode, what computation to be done. Now, let me present the model that this paper proposes. So


this paper proposes to embed the question here. So you can see here, that's the question into a


BERT input. So this is a transformer right here, this is BERT or any variant of BERT that you can


think of. So the question is embedded as natural language. And then interestingly enough, the table


right here is also embedded as language. We'll get to that in a second. But the question and


the table are in the input. And then the model is asked to do two things. First of all, it's asked


to do an aggregation prediction. So this can either be one of these programs called count,


sum, average, or it can be as you can see here, none. So no aggregation. So this handles our first


two components, it can decide to perform a calculation or none. And if it is performing


a calculation, it can decide to do a count, a sum or an average. Now the of course, the the model


here is not limited to those computations. You can think of extending this to any further computation.


The important thing is that they have a number as an output. Second of all, there is a cell selector.


So depending on this aggregation prediction, you need some cells like if you want to compute an


average, you need the cells to compute an average over. So the cell selector here will select cells


from the table is specifically it goes by row and column, sorry, column and row. Since these tables,


usually they have a header, right, this is the table header, where the attributes are listed,


it makes sense to first in the first step, select which column you want to select from. And then if


once you have a column, let's say this column here, in the second step, you say which of the cells you


want to select. Now these can be multiple. But the way the system is set up, it's first a column


selector, and then a cell selector within that column. So you can only ever get columns from the


same cell in this thing. Let's remember that for later. Alright, so this is what the model does. Now


let's look at the input. The input to the model is this here. Now this if you refer from this before,


this was in this blue box. And then here, you'd have the computation selection. And here you have


the cell selection. Right. So this is, this is how you can relate that. So usually, if you input


something into a transformer, what you want to do is you want to embed this into into a token


embeddings. So first, you want to split everything you put in into what are called tokens. Now tokens


are either things like words or word pieces. The important thing is to have a dictionary for it,


and each one gets mapped to a to a vector. So this here is your query, you take your query as a


string, and you tokenize it, and you get the embeddings from the embedding table. And that's


your input, right. So it's a sequence of token embeddings. And then you also embed the table.


And this I find pretty cool here in this model and somewhat special is that the table is actually


presented as just natural language. So you can see here, the table is one string. It's just a


single string that goes from left to right. It's just the serialized table. So this table right


here, you can see these are word pieces. So this table if I reconstruct it, if I can attempt to


reconstruct it, it is going to be a table that has as the headers, call one, call two, these are the


names. So in days before here would be name of the wrestler. And this would be number of days.


Right. And then here, zero, one, two, three. So this this table right here, corresponds to this


string right here. I hope you can you can make sense of that. So the table is just put there as


one long string. And then in order to make the model realize, you know what the table is, you


have these special embeddings. So usually invert, you have what they're called position embeddings


to indicate where in the sequence that is. So in a simpler in the simplest case, these are embeddings


for the numbers 01234, and so on. So wherever the position is, this, you can all look up in the


attention is all you need video, I've made that if you are unfamiliar with transformer inputs,


then also the segment embeddings simply indicate where what a token is part of. So for every token


that's part of the query, you see you have segment zero embedding. And for every token that's part of


the table, if a segment one embedding, this is simply to tell the model, hey, this particular


token is part of the question or part of the table, then you have the new things. So this


paper newly introduces the following embeddings, column and row embeddings. Now these for the


question, of course, they don't make any sense, but you have to put something here. So you just


put column zero. But for the table, you see, there is a column one, and column two. And the this


exactly so we've seen that this here is the header of column one, and this is the header of column two,


and then it goes back column one, column two, column one, column two. And you can see here,


this zero is in column one, and this one is in column two, and this in column one again,


and the same for the rows. So you have row zero for the headers, and then row one for the first


two numbers and row two for the second two numbers. So this is all of this. So you see,


these two are in the first row, and these two are in the second row, all of this is to tell the


model all of this information down here is to tell the model how this table looks. So if it wants to


select the second column from the third row, it would look in this information to see which cell


to select. And then the last thing they introduce is this so called rank embeddings. Now, as we've


seen before, if this first column here is maybe the sorry, the number of days of something. So


this is the number of days. And this second one is the number of rains, so how many championships,


the table can only be sorted at maximum by one of them. So you want to sort of for each cell,


you want to tell the model, let's extend that table by two numbers, four and one. So for each


column, you want to tell the model the ranking of the numbers. So here, it's pretty easy. This is


rank one, this is rank two, this is rank three. Further on the left side, this is rank one,


this is rank two down here, and this is rank three. So the model has an will have an if you


give this information, the model will have an easier time to detect, like, give me the top two,


or something like this, give me the worst, give me the best, give me the highest, and so on. The


model will have an easier time doing that. So that's why the rank here, as you can see, the


zero, and also the number one are embedded rank one, and the other two rank two, because they're


just lower. Now, I don't feel I feel they could could have given a better example than this table,


I feel you could actually put real names here to make clearer, not call one and call two. And I


feel you could give a somewhat smarter content. Because if you just look at the picture here, you


cannot see the correspondence of these rank tokens, because in essence, they are exactly equal


as the row tokens. But fortunately, we can read the text. Oh, there's the table. Ha. So I have


actually not seen that, but I've discerned it correctly for this particular for this particular


input. Alright, I think that's the half of the magic is how you encode the input in such a thing.


And this seems to be, first of all, a pretty cool idea. But second of all, it exactly is what this


kind of new regime of NLP is about is that you basically put everything as a string, you annotate


it in a smart way. And that lets the model figure out a lot of stuff about the input. People used


to people used to do the very different things. So people if given a query, and a table like this,


what people would do is they would somehow, first of all, get the table headers and, and kind of


guess the data types of the attributes. And then they would formulate reformulate the query, maybe


also with a neural network, maybe with something else into something like SQL in order to actually


have an SQL statement to select the correct cells or perform the correct aggregations. And that is


somewhat brittle. And it's just much less deep learning than this model. So I like this part of


the model. Now, the problem, of course, is, as we've seen in this multi step process. So how do


we first of all, if you build if we want to build a cell selector, that's pretty easy, right? We've


seen this so we the cell selector is first column, column selection, and then second row selection.


And this can be multiple rows. So that's fairly easy. Selecting cells either for just returning


or for aggregation pretty easy. But how do we do the actually the aggregation selection is also


pretty easy, because we can just do a multi class classifier, right? So the classifier will simply


tell us a give give us a distribution. And then we see, okay, the sum aggregation is probably here,


the the what the model wants. The real question is, how do we train this? And how this is trained


is what I find really interesting. So as we've seen, they have training data, the training data


comes in the form of tables, questions and answers. As we've seen before, we don't know how to get to


those answers. So when the question is, which wrestler had the most number of rings, we just know


the answer is Ric Flair. Now they they do, again, a two step process for their training data that


mimics the two step process of the model. So the first step is, is the answer a number? Is the answer


a number? If no, then it is definitely a cell selection task. So they if it's not a number,


they just restrict themselves to selecting cells. If the answer is not in the table, then that just


means that the correct thing is to select no cells and just say I can't answer this question. If it


is a number, then again, you have two options. So is it in the table? If yes, we are in a weird


situation. If no, not in table, then it is an aggregation. So if it is a number that is not in


the table, that means that the answer is a number that is not in the table. That means the answer


must be computed via one of these aggregations. And if the answer is a number but is in the table,


then we are in this ambiguous answer setting where the it could be that we need to select the cell,


but it could also be that the same number by accident is in the table but actually needs to


be computed from other numbers. And they do this in the most deep learning way possible is that


they do basically a soft decision here. So they let the model when they let it select what to


compute, they let it make a soft decision. What do I mean by that? So let's say you have these


three operations count, some and average. And you have the cell selection. So the cell selector


will basically tell you I will select three cells, the three cells contain the number seven,


the number eight, and the number three. Alright, so and the question was, I don't even know what


the question was, but the cell selector tells you these three cells are to be selected, you do this


by simply selecting the cells where the cell selector has a higher probability than the cell


one half. Now, your your aggregation selection module gives you a softmax distribution over


over the actions. So it's not very much count here, maybe that's 0.1. This here is maybe 0.3.


And this is the 0.6. What you do is you simply compute all of them. So you want to compute the


count here, which is three, you want to compute the sum here, which is 18. And then you want to


compute the average, which is six, huh? I made a good example by accident. And then you simply weigh


the outputs here by their probabilities. So you say since the model wants point one,


puts what point one probability on the count, I'm going to have 0.1 times three,


plus it wants point three times this. So 0.3 times 18 plus 0.6 times six. Now, I'm not gonna


So this is six plus 0.3 plus 3.6 9.9. So that that's how the model computes things, it simply


puts probability on these operations here. And then you simply take a weighted output with respect


to the computation of all those things. Now, I'm pretty sure that's completely invalid,


because for the same numbers, for example, the sum is going to have a much larger


like variance than the average. And, and that's somewhat going the count maybe somewhere in


between depending on the numbers. So this, just to take the weighted average here, and then,


of course, right, so what they do is they do have this, this is the model output and you have the


correct answer. Let's say the correct answer was actually was to compute the the average. So the


correct answer is six. So what they do is simply they take the squared error. And that's their


loss. Actually, they don't take the squared error, they take a approximation to the squared error,


which is square until some delta. And then it's linear. And then they take the squared error,


linear. And this is simply to be a bit more outlier of us than they do other things to be


more outlier of us. But this, so this is the model output. And this is the correct answer.


And they simply count on the fact that this will, this will back propagate. So if you


want to make these two things closer, if you're the model, right, you have the option of simply


putting more weight from the from the other ones on to the average operation. And that will decrease


the 9.9. Because you as you can see, both of these numbers will get smaller. And no wait.


This isn't the yes, sorry. So you will you will decrease these numbers. So this is the output we


got from the weighted average, right? So if we decrease these weights, you will put weight from


here to here, that will bring the number 9.9 down. And that will get you closer to the answer you're


looking for. But you can also achieve this by you can achieve this even more, right? So this 9.9 is


too high. If we want to bring the 9.9 down, we're much better off by taking some of that output and


actually putting on this here because three is the lowest number, right? The only agreement here is


that we want to take weight away from the 18 from the large one. So I'm extremely surprised that this


works. Given that it is so super ambiguous what the model should do with these operations. And I


I highly doubt that you can extend this set. So it's of course agnostic of what these aggregations


are. But the to be able to extend this to many more aggregations is will I think lead to much


more of these situations where the model is entirely unsure of where to put the mass of where


to put the weight. And I would be interested to see what happens if you have a data set with like


20 or 50 of these aggregations and not just three. So this is the this is the let's say the the


interesting part here the other if you go the other way, when you have this cell selection task,


it is just to select a cell right and then you simply have the cell selector that part here that


does the selection that you also you train every time simply to give each cell a weight, right? So


this this is simply the softmax over column and then the softmax over rows. And you can train that


using the cross entropy. Now training the cell selector from data is pretty easy when it's a


cell selection task, right? Because the answer is in the table and or is not in the table and then


you know to select no cell. So you do have the training data that a particular cell is the


correct cell and you can train the model to select that cell. But it is actually a pretty hard


task if it is for example, you're looking for an average operation. Because not only do you are


you not really sure that it's an average operation, you just know that that kind of gives


you the correct answer. You also don't really know which cells to select for this average operation,


right? Because depending on which cells you select, and of course, that's going to be a


soft selection as well. The the average answer the average will be different depending on which


cells you select. So they're basically counting on this loss here to back propagate not only through


the the selection of the aggregation to perform, but also to the cell selector to set which cells


to to select. So from this weak signal, it's almost like the reinforcement learning problem where you


have the weak signal, and you have like a billion ways to get your number closer to that signal,


and not not really accurate understanding of what you need to do is you're just relying on the model


through lots and lots and lots and lots of data to kind of figure out which natural language


questions to map to which cell selection and aggregation. So this is it's a it seems like


impossible, but it works. The last thing we need to talk about is this ambiguous answer setting.


And as you can imagine, it's pretty simple that they also let the model do an a soft selection


between the cell selection tasks, so no aggregation and the aggregations to be performed and basically


let the model figure out itself, which one is better to do an aggregation or to do no aggregation.


Suffice to say this, this only works for pretty, I think it only works for pretty limited amount


of tasks pretty limited amount of questions. And you might have spotted there even these questions


that are follow up questions, which are another thing they build into the model and I don't I'm


not really going to talk about this, but they do have this concept as well, which I find maybe a


bit out of place, but maybe it's just part of their data set somewhere. Maybe it's just these


companies want to get into this conversational mode. So everything needs to be context dependent.


The interesting part here is really the computation of the aggregates and specifically


the question of which of these aggregations to choose. And this, again, this is so surprising


that it works and fairly, fairly cool. I think that is the gist of the paper, they do extremely


thorough evaluations here on these data sets and ablations to see what really counts and what


doesn't. I don't really want to go into that safe to say their results are better than


anything else before. I believe they I believe they're actually on par with another model, but


in one data set, but they beat them on every other data set. So that's, you know, that's cool.


I don't think there was a bar diagram. Never mind. I invite you to check out this paper,


look for yourself. They have the code online if you want to train a model like this yourself.


Other than that, thanks for listening. If you like this content, please subscribe,


I'm taking a break

I'll go on a bit of a summer break. You might have noticed that the frequency of videos,


I'm taking a break

especially paper discussion videos has been going down a little bit. That's because I've


I'm taking a break

been preparing to summer up a bit. And we're really close to 100k subscribers. Thank you


I'm taking a break

everyone who's already here. If you're not subscribed, subscribe. I hope we can do a


I'm taking a break

sort of proper channel recap review celebration once this happens. So yeah, I'm gonna make


I'm taking a break

this really short. I'll be gone for a bit few videos in the pipeline. Not too much though,


I'm taking a break

we'll see if there's any any surprise or something like this. So this means I won't be checking


I'm taking a break

Twitter, LinkedIn, etc. As much if you really need to catch me during this time, you'll


I'm taking a break

probably find me still every now and then checking the discord community if you're not


I'm taking a break

a member yet. It's a really nice community. I absolutely suggest you become a member.


I'm taking a break


Assessing Game Balance with AlphaZero: Exploring Alternative Rule Sets in Chess (Paper Explained)

Hi there! If you play chess you'll probably recognize the following moves as illegal.


In the top row pawns move two squares at a time while they are not on their home row.


In the bottom row you'll see a pawn moving backwards and another one moving sidewards even.


So in classical chess these moves are illegal but there are variants of chess where these moves aren't illegal,


where they are actually explicitly part of the rules. These are alternate chess rules and this paper is about exploring those rules.


What happens if you implement those rules? How does the gameplay change? And what can we learn for general games?


So the paper here is called Assessing Game Balance with AlphaZero Exploring Alternative Rulesets in Chess by Nenad Tomashev, Ulrich Paquet,


Demis Hassabis and Vladimir Kramnik, the former three of DeepMind and the latter was the World Chess Champion for these eight years depicted.


So the paper tries to bring together two different worlds. First it is the chess world.


So a lot of this paper is explicitly about the game of chess. If you don't play chess or if you occasionally play chess like myself,


this might not be the most interesting paper though it contains some really interesting kind of bits.


The other world is the reinforcement learning world which you'll see in the AlphaZero name right here.


So the reasoning behind this is the following. Chess is a really really old game and rules have evolved over time


and have sort of consolidated on the rules we have today. But also strategy has evolved over time and lots and lots of thinking and theory has gone into the strategy of chess.


And to change the rules around, you can change the rules of chess. However, you can't really assess how the game would be played by humans if the rules were changed.


Because you don't have a thousand years of the entire humanity studying these new rulesets and therefore you're kind of stuck with assessing the games from the perspective of someone who has learned the old rules.


But reinforcement learning to the rescue. So consider the following rule changes. No castling. This is a really simple rule change. No castling. Castling is disallowed throughout the game.


If you don't know what castling is, castling is like a special move where there is this rook and the king is right here. I don't know how to do the king.


And if there's nothing in between, they can sort of swap positions. It's called castling. It's a special move that you can do. And it allows you to bring the king to the outside where the king is safe and to bring the rook to the inside where it can potentially cause a lot of damage.


So it's a very, very favored move by a lot of players. And no castling, the rule change probably alters the game a lot. Because if you think of the chessboard, kings start about here, they can only move one square at a time.


So to get them to safety will require like four or five steps for them, while you have to move everything else out of the way, including the rook that stands here.


So players might elect to just leave their kings where they are, but then they can't really open up in the middle as much because that would leave their kings exposed.


So it is fair to assume that just introducing this one rule might change the games around quite a bit, how the game is played.


But as we said, we don't know. This is from someone who has learned classic chess, and all the grandmasters that we have have played and learned classic chess. So how do we assess this?


This paper says that AlphaZero can be used to assess these new rules.


So AlphaZero is a reinforcement learning algorithm that can learn these board games very, very quickly, in within one day or so. And it can learn them so well, it can beat humans at the game easily.


In fact, modern, modern grandmasters and so on use these algorithms in order to learn and to better their play in order to expand their theory, their knowledge of the game to play better against other humans.


So AlphaZero, imagine AlphaZero can solve a game to perfection.


What we could do is we could simply give this rule to AlphaZero together with the all the other chess rules, and then let AlphaZero solve the game, give it a day and 50 billion GPUs, solve the game to perfection, and then look at what AlphaZero came up with, kind of look at the games, how they turn out, and whether or not they are more interesting, less interesting, longer, shorter, and so on.


So that's, that's what this paper does. So there's the implicit assumption, which you need to believe in order to believe anything in this paper is that AlphaZero actually has this ability.


There is pretty good evidence that it does because AlphaZero can solve classical chess and Go and Shogi and a bunch of other board games, all with the same hyper parameters, it can solve them such that it is easily at superhuman power.


So, but you need to recognize that this is an assumption. So what is AlphaZero? If you don't know what AlphaZero is, AlphaZero is a reinforcement learning algorithm, but not in the kind of basic reinforcement learning sense, it is a reinforcement algorithm that has a planner included.


What do I mean by this? So if you are in a, let's consider the game tic-tac-toe. So AlphaZero for tic-tac-toe. In tic-tac-toe, you have this board, and you have a situation where, let's say you play, your opponent plays this, and now you're tasked of playing something.


You wonder, should I play maybe here or here or here? Where should I play?


So what you can do is you can train a reinforcement learning algorithm, you can do Q learning, what not, okay, that will maybe work.


What's better to do is you can plan. So in planning, what you want to do is you want to build a tree of possibilities.


So we're going to consider all your possibilities, and in this case, you have eight possibilities. So we want to consider all the eight possibilities, and I'm going to draw just some of them. So up here, you're going to consider the possibility that you place here.


And here, you're going to consider the possibility that you place in a different spot right here. Okay, and you can see how this goes. So if you want to plan, and here you have your opponent has seven possibilities, and here your opponent also has seven possibilities and so on.


So you get this entire tree of play. But if you could do that, and if you could do that to the end, then you could easily simply choose the path here where you win, okay, where no matter what your opponent does, you win.


You can find such a path if it is possible at all to win, which it is not in tic-tac-toe, right? If everyone plays optimally, it results in a draw.


But let's say you could win, you could choose the path that gives you the best result. And that's it, there's no learning involved. Okay, so alpha zero works with a planner and planners usually construct a tree.


So in an abstract way, you're in a situation, and you consider all your options. And with all your options, you consider again, all your options, and so on. And you do a tree search. Now this tree in tic-tac-toe, it's already huge, as you can see, in something like chess, it is way, way huger.


Okay, and therefore, it's not possible to actually search the entire tree, because you need to consider every single possible future situation from the board position where you're in, right?


This here is the board position where you're in. And this is the future, the entire future of the game. So every single possibility.


So alpha zero uses this thing called a Monte Carlo tree search. It has several components. So its first component, and they right here, they have a description, and it's very short.


Alpha zero, this is alpha zero. This is what it does. It's like this is almost comically short. So what you do is you put your state, so s is your state, okay? s is, it's the board as you have it right now.


Okay, this here, that's, this is s. Okay, you put this into a neural network, and the neural network gives you two things. First of all, it gives you p and v.


So that's the second thing. So v will simply give you a number v will tell you that this thing right here is about a plus point five, maybe. So it says, so plus one is winning and minus one is losing.