I COOKED A RECIPE MADE BY A.I. | Cooking with GPT-3 (Don't try this at home)

Jonas is just looking up adjectives for bad food.


I think I'm gonna need them. Look at this stuff.


We're gonna go to the store, buy some random stuff, put it all into an AI that generates


recipes and we're committing right now to cook...


Can you just move your hands in a kind of random manner?


And eat whatever it outputs.


All right everyone, this is Jonas.


He is an expert in non-convex optimization and also a very, very good cook.


Mamma mia! It's going to be extra spicy for him today


when he has to follow instructions by not-so-good cook, which is the GPT-3 language model.


Yeah, let's do it. Awesome.


So here's the plan. We're gonna go to the store and each of us is just gonna buy some random items.


We don't know what the other person is buying. All right, what's real, really weird.


And we'll come back and whatever we have, we'll put into GPT-3 and ask us to generate a recipe for it.


And we'll try to follow that recipe as closely as possible.


As closely as possible.


As close as possible.


And then whatever comes out, Janik's gonna eat it.


And if it turns out great, I'm gonna give it a try as well.


No, just kidding. We're both gonna eat it. We're committing now. We're doing this.


Absolutely. So there's a couple of rules.


Rule number one, Jonas is a vegan, which means that today we're going


full CO2 neutral, absolutely organic, healthy, 100% cow-friendly, ethically perfect, vegan.


Yeah.


Just yeah. Rule number two, we're gonna follow the recipe as closely as possible.


If it suggests an ingredient that we happen to have, we're going to put it in.


If we need to wait for a couple of hours, come on, who's got time.


But other than that, we'll do whatever it says.


There's lots of videos on how to do biking.


Probably haven't done it yet on Minced Meat.


And rule number three, we must finish our porridges.


Are you ready?


Totally. Let's do it.


Let's do it. To the kitchen.


To the kitchen.


All right, we are back from the store and we got ourselves a whole bunch of food.


It's way too much. Jonas, how was the experience?


It was lovely. So we went shopping and we found lots of tasty, healthy, vegan food items.


I am very sorry about that, but that was my restriction.


I'm sorry, Janik. So today it's going to be a vegan day.


All right. We have pretty normal stuff.


This is an avocado.


It's not just an avocado, it's organic avocado.


Well, I have to check the imprint.


Nice, nice. It's actually imprinted.


I've never seen that.


You should start doing that.


We got some vegan plant-based butter.


How ugly is that? Have you tried this before?


Yeah, it's pretty good actually.


God.


Tofu, the classic. The staple.


We also have vegan plant-based...


What is this made from?


It's mince meat made of no cows and no pork.


It's made of peas.


Peas. Yeah.


Probably other good stuff.


Probably tastes like peas too.


All right. What else we got?


We got pomegranate, chocolate, garlic, sweet potatoes, mushrooms.


What else, man?


Kale. We have the superfood here.


Jesus Christ.


It's all in here.


If we're not going to be so hipster after this, we got kale.


Kale.


We have these tasty Gwurzburgruten.


How is this ever...


How is the chocolate ever going to be?


It's not any chocolate. It's a cooking chocolate.


Of course.


And we have soy...


Soy...


Whipped cream?


Soy whipped cream.


Okay.


It's beautiful.


All right.


Orange soy cream.


We're going to put all this into GPT-3 and whatever it spits out...


We're going to cook it.


And we're going to eat it.


He's going to eat it.


GPT-3, trained at OpenAI, is a giant neural network called a transformer


with over 175 billion parameters.


It is trained as a language model, which means that if you give it a piece of text,


it can predict what the text will look like that follows it.


It can do so with remarkable accuracy, and just like a human would, can do it in multiple ways.


So you can sample many times given the same starting text,


and you will receive many different answers.


GPT-3 can do this because it has been trained on a scrape of the entire internet.


In a way, it is the collective knowledge of humankind,


at least what has been written down in the internet.


So let's see if we can make that collective knowledge work to generate one recipe.


Now remember that I said that you can sample from the model and get multiple answers.


We were a bit disingenuous here in that we sampled a few times to make sure that the recipe


was reasonably long and contained at least some funny parts,


though we genuinely were ready to accept whatever came out as long as we could do it in a few hours.


So what we did here is we input our list of ingredients and then let the model generate the recipe.


The model is usually pretty consistent and outputs actually regular recipes,


though I think the fact that we sampled a few times plus the fact that we gave it


such a weird combination of ingredients made it a little bit thrown off.


Okay, reduce the size of your prompt.


Damn.


You have too many ingredients, man. This must be like dirty.


We don't have salt and pepper.


This is way too little.


Wait, that's it?


This is too little.


The other instructions are not long enough, I guess.


Yeah, serve the bread with mustard and pomegranate on top.


Shred the carrot and grate the cheese. What cheese?


Still not as good.


Not as good.


So at the end, we got a recipe that we were reasonably satisfied with and we went ahead and cooked.


The recipe started out with us boiling the potatoes and carrots,


which was definitely a good surprise for me because I was worried as unboiled potatoes


aren't really something nice to consume.


So at least GPT-3 had the foresight to boil potatoes.


Then step two.


In the meantime, prepare the vegan minced meat or use precooked soy meat.


Jonas also enhanced our meat with some very skilled shamanistic procedures.


No viking, no hipster, man.


The recipe went on, asked us to fry the butter, add the garlic.


Computer science people, here's how you do garlic.


How do you do garlic?


Smash.


That's it.


You can just peel off the...


Add the mushrooms.


Oh, it's totally gonna kill us.


And stir for two minutes.


So far, so good.


We're gonna add soy cream, stir and cook for three minutes.


Okay.


This is the soy cream.


Add it, add it, add it, come on.


All the way, yeah.


Three minutes, cool.


Next time you're set.


Tell all your vegan friends to subscribe to Janek's channel.


This is coming along nicely.


Step five.


Add the pickles, tomatoes, and beans.


Stir and simmer for another five minutes.


So the pickles are in there and it's looking tasty.


This recipe wasn't so bad until now.


We actually, we don't have pepper.


This is already burning.


It's going absolutely great.


Next comes the bread.


Cut the bread in small squares and fry in the vegan butter until golden brown.


A chunk of butter that we're gonna put into the pan.


We decided to take a new pan for this instead of adding the bread to


whatever we had already.


See this?


This is the last thing your arteries see before they go.


Okay, we have to put the bread now.


You ready?


Sure.


Let's put the bread.


No!


Next, cut the limes into cubes and squeeze the juice into the bean mixture.


Easier said than done.


Step eight.


Add the soy sauce, parsley,


salt, pepper, cumin, cilantro.


Where did it come up with that?


All right, we're going to leave that away as per our rules if we don't have it.


Do you have cumin?


No, I don't know.


Good.


And dried figs.


In the meantime, the bread's doing great.


Also the potatoes.


It's looking super healthy.


And the carrots.


Should we ever stop boiling the potatoes though?


It doesn't say so.


I think at some point we should stop.


Maybe later.


We didn't exactly have all of that, but we made some substitutions.


I have ketchup.


I mean, we can totally get ketchup.


We're just going to replace the cumin and the cilantro with the coriander.


Yeah.


It's looking better and better actually.


We totally need to figure out a name for this recipe.


The GPT toast or something like that.


Add the kale.


Kale cannot be unhealthy.


Step nine.


Pour the bean mix into a blender.


The blender!


It's blender time!


This is where the recipe started to turn a bit.


Blending the bean mix was definitely a first for me, but it was a lot of fun, I have to admit.


One, spit!


And whatever, it's going to come together all in your stomach anyway.


So who cares?


Step 10.


Bake for five minutes in the oven at 180 degrees Celsius.


Celsius.


That's Celsius for you Americans.


Oh, you're beautiful.


I think three blue, one brown had a nice mnemonic where you distribute 100 degrees Celsius onto


like a semi-circle.


So here you have this, you have a semi-circle and then here is like 50 degrees Celsius and


here is 100 degrees Celsius and here is zero.


And so if I want to like 60 degrees Celsius, then this angle right here, I'll just take


this, which is like 110 degrees.


So this is like 110 degrees, I add 32 and that gives me like the 142.


So 60 degrees Celsius is like 142 Fahrenheit.


Is that correct?


I don't know.


It doesn't fit.


Maybe you should first take it out, but ChibiDrew didn't say so.


It seemed a bit pointless to bake something for five minutes, but we trusted the recipe.


Step 11, cut the sweet potatoes in cubes and add to a pot with the remaining butter.


What more butter?


Come on.


I'm going to have to do 100 workouts to compensate for this.


What am I supposed to do with the carrot?


Doesn't say.


Oh, the carrot.


So the carrot never ever enters the recipe.


With the remaining butter, add the red beans mixture.


Yeah.


So the carrots just out of the game now.


Add the red beans.


The most surprising part about this is that this was probably the exact point when the


potatoes were cooked the best.


So props to GPT-3 for timing us so perfectly.


We then had to cut the bell pepper into cubes, add to the pot and add the vegan minced meat.


You can actually eat this raw, right?


You can, but let's not do it.


Right, this is kind of sticky.


Minced meat is there.


What is this?


This is the rest of the minced meat.


Yeah, we didn't have enough butter because you put all the butter in the pot.


Look, the carrots are still alive.


Come on, carrot, you're part of the game.


You're part of the team.


We need you.


And cook everything in the oven at 180 degrees for 10 minutes more.


Once that came out, we added the avocado, chickpeas.


Okay, let's skip the chickpeas.


Let's skip the chickpeas.


The chocolate and served one bread with mustard and pomegranate on top.


It might not be the most obvious choice, but this was the ingredients that we gave to GPT3,


so we had to do something with them.


And kudos to the model that it waited until the very last second until it added the ingredients


that it really didn't want to add and I really didn't want to eat together.


At the end, we got a nice warm meal and we were absolutely thrilled to see what it would


taste like.


Are you ready?


What part are you going to start with?


We committed.


The sandwich with the chocolate and the mustard on top?


I think I'll get myself a nice piece of chocolate, bean, lime, avocado, carrot.


Wait.


I'm definitely, definitely make sure to add some of the pickles.


Fatty buttery brat.


Nice.


Mustard and pomegranate.


Uncooked kale?


No, not yet.


I need some of the minced meat.


Okay.


Minced meat and the chocolate.


You have the chocolate piece too?


I have the chocolate.


Let's do the chocolate.


Come on, chocolate.


Formidable.


Chin chin my friend.


Thank you.


Yeah.


Enjoy.


I like the chocolate part.


It's all together.


It's sweet and salty and bitter and sour and buttery.


Oh my God.


The sweet potatoes.


I don't like the sour part of it.


There must be the lemon.


We have way too much lemon in there.


Like two entire lemons.


Well, they told us to.


And the pickle.


I mean, come on.


Have you ever cooked like fried a pickle before?


It's just.


I'm actually surprised the sweet potatoes are cooked through.


We had them in the pot for like an hour almost.


Yeah.


So why not for that?


I'm almost done, Janik.


The carrot.


It wouldn't be the same without the.


Did this grow?


No, I don't know.


All right.


This is the last piece of not fully chopped garlic.


How do you like it?


Excellent.


So this is just the bread.


I'm going to eat some, but I feel.


I think it's more like a low carb guy.


I feel we've fulfilled our duty.


It's just the bread remaining.


The rest is done.


Awesome.


Well, thanks everyone for watching.


If you have recipe ideas, please don't send them to us.


Yeah.


Subscribe.


Check out Jonas's Google Scholar.


Review his papers.


Accept them.


Strong accept.


Smash accept.


And yeah.


Bye bye.


Stay healthy.


Resolution-robust Large Mask Inpainting with Fourier Convolutions (w/ Author Interview)

Hello there, today we're looking at resolution robust large mask in painting with Fourier


convolutions also called Lama by the Samsung AI Center, Samsung Research, EPFL and the Skolkovo


Institute of Science and Technology. This is a special paper review because I'm only going to


introduce the paper briefly maybe 15-20 minutes or so, and then we're going to talk to the first


author of the paper and go a lot more in depth. So if you like if you like conversations with first


authors, and the ability for me to ask dumb questions to them, then stay tuned for that it's


going to be in the second half of the video. For the first half though, I first want to demonstrate


to you what this model can do. Hey there, this video is sponsored by clear ml clear ml is an


ML op stack that is fully open source, it can do experiment tracking orchestration deployment


model and features stores and much more. The self hosted tier is a first class citizen in clear ml.


As I said, it's fully open source, you can look at it, you can audit it, you can extend it, you can


run it on your servers. And if you ever come to the point where you need the extra features, you


can totally upgrade anytime they'll gladly take your money. They have a free tier in the cloud,


which gets you going pretty far. Now we talked about experiments tracking last time clear ml


with two lines of code will track any experiment that you do track the metrics, the outputs,


the environments, the dependencies and make everything super duper reproducible. But this time,


I want to talk about a second part, which is the orchestration engine. So the orchestration engine


is responsible for packaging up your experiments, including all dependencies, and then distributing


them on your hardware. So that means you can simply submit an experiment to a particular queue,


and clear ml takes care of running this wherever it's needed. So this is super cool, because it


means I can get going on my laptop, run a few experiments there. And as soon as I'm ready, boom,


I ship it to the cloud. So here's an example. Look at this experiment that has already been run,


I got some output. But now I would like to do something different with it. So I click here,


I say clone, I give it a meaningful name, like two. And now I've cloned this experiment. And


this is kind of a draft experiment right now, it has no results yet. But what I can do, I can go


into my configuration into my hyper parameters, and I can change around the hyper parameters. So


I wasn't really happy with the last experiment, I feel a bigger batch size might be needed. So


from 128, let's go to 129. Now I'm pretty sure that's going to make all the difference right


here. So I save and then I simply click on in queue, I submit it. And now clear ml simply takes


care of running that experiment for me. As you might guess, you can have different cues, some for


GPU load, some for long running tasks, some high priority, as you're used to from any scheduler.


This can also be used in an automated fashion, meaning that you can use this for automated


hyper parameter search. And you can even do things such as scheduled or triggered tasks. For example,


if you want to trigger a training run every day on new incoming data, that's totally doable. Now,


orchestration is just one part of clear ml I've shown you experiment tracking last time. And


there are many more features to their product. If this sounds interesting to you, if you're an


open source fan, go check them out. And thanks so much to clear ml for sponsoring this video.


Let's get into it. You can already see it a little bit in figure one right here, the model is able to


take a picture, you draw a mask on it. So this is the blue area right here, and the model would


auto complete the picture. So the model doesn't see the mask, the model simply sees what is


unmasked, then the model is asked to complete that missing area. As you can see, it fills that area in


you know, very, very cleanly. And especially if you look right here, this irregular structure


of these door holes, or whatever that is, is preserved, even across very large areas. This is


very, very cool. This is very difficult to do with these in painting systems. In fact, there is a


project website right here, all the code is available. They give this in a little bit more


of an animated flare. So you can really see the effect that these models are having. And it's


pretty, pretty cool, especially take a look at these repeated structures that are often in the


pictures. So these meshes or the lines right here, these tend to be extremely, these tend to be


especially difficult for in painting models, because in painting models are usually built on


convolutional neural networks, and convolutions, notably take into account very local context.


Whereas for these patterns, you need to take into account kind of a global context, that's exactly


going to be the message right here. There is an app, there are actually a bunch of apps based on


this model. This is a third party app. So this is not by the author, but it is an app built from


these models. There are also as I said, code is available, there's like a hugging face space,


there is a collab by the author. But this particular app, let's just take a picture right


here. It works best on natural images, of course, but we'll just take the channel logo right here.


And we'll say we want to erase the pie sign right here. Look how cool that works. What about the paw?


Okay, that that is that is kind of disturbing. How about the nose? No, no, no, I don't like that.


But it should be able to Yeah, see, so it kind of completes lines if you cross them out. So this


should complete the table but remove the leg. You can see it's fairly robust, even to you sort of


miss specifying a bunch of things. So here I draw over the headline if you saw that, and it remains


the head headline remains. So I removed this part, but I crossed into here a bit, you can see the


line kind of remains now it's got a bit of hair. Yes, kill it with fire. In any case, this is


available for you to use if you have more sensible pictures, I'm sure that that will work a little


bit better maybe. There are also different versions of the model. So keep that in mind. And they works


also on different resolutions. That's why it's called resolution robust, large mask in painting,


which is also very cool. So what is the core idea of this paper, the core idea is going to be these


Fourier convolutions right here. And these Fourier convolutions are going to be enabling the model


to take into account global context from the very beginning, what is the problem with a


convolutional neural network? The problem usually is that in a convolution, if I have a picture,


a convolution on a particular point will take into account its local neighborhood, right, and then I


sort of slide this over the image right here. And that will give me my representation in the next


layer, maybe that's going to be even of the same size. So for a given point right here, I will have


a receptive field of the point in the original image, plus some neighborhood. Usually we work


with three by three convolutions. So all I'm going to do really is I'm going to look one pixel to the


top and one pixel to the bottom, one pixel to the left, and one pixel to the right. And that's about


it, I'm not going to do any more looking around. So how does a convolutional neural network


integrate information across the whole image? And the answer to that is by going for multiple layers.


If I simply represent the picture as a set of pixels in one dimension, imagine that the one


dimension here is, is the picture, and I'm going to need a bit more space for that. So as you can


see in the first layer, from the first to the second layer, let's say we look at this particular


point right here is going to have a receptive field of three. So it's going to look at these


pictures, sorry, at these pixels right here. In the next layer, if you can see that the same


location is also having a receptive field of three right here. However, since for example, this


particular pixel right here also had a receptive field of three. And this particular one also has


one also, as you can see, and from layer two on their total receptive field of that, so that all


the information in flow is going to be from a receptive field of five. Therefore, the more


layers we have, the more of information, the more spatial information can be included for a given


particular location in the output. But as I said, that takes a lot of layers that takes depth,


and especially for these in painting applications, what you want is kind of global information.


These masks right here, like these masks, they're pretty big for an in painting application. So


they're pretty, pretty wide. And if you can imagine a convolutional layer that looks at a three by


three pixel neighborhood, that might be something right here, you know, so you're going to have a


whole lot of convolutional kernels that just see the masked pixels, they see nothing of the outside,


they simply see a bunch of masked pixels for a whole bunch of layers, right, layer two, layer


three, until like layer four, there's like, there's nothing no information at all at this position


about the outside world about the world beyond the mask. And even then, it's only like this area,


we need to go many more layers before we get access to information that is way outside of here.


And at that point, it may already be too late. So the Fourier convolutions, they solve that they


have the ability at every single layer to look at a global context. And how are they doing this?


It's not super expensive. In fact, they're doing this by using of course, Fourier transformations,


a Fourier transformation will map a signal to its corresponding frequency domain signal,


it is essentially a different way of representing a signal. So if you have a signal, let's say,


you have like a pure sine wave, do a Fourier transformation of that entire thing, you can


represent that as the components in the Fourier spectrum. And that would simply have like one


component at the particular frequency at which this sine wave at which this sine wave is operating,


that's the that's not the frequency, that's like one over the frequency right here. But in a more


in a more general sense, a Fourier transform will decouple the spatial resolution and give it a


transform it into frequency resolution. So if you have a Fourier spectrum, maybe you have a very


complicated signal right here, a complicated signal that will give you also a complicated


Fourier spectrum, like you have a lot of this, you have like negative this frequency, a lot of this


frequency, not too much of this frequency, and so on. If you do a convolution in this domain,


you simply convolve across neighbors of the signal. However, if you do a convolution in Fourier


domain, you can see you convolve across frequencies, you can evolve across neighboring


frequencies, which means that these three things represent three particular sine waves frequencies,


maybe the lowest one is like a super long sine wave, the second one is like a bit of a faster


sine wave, the third one is even faster sine wave. But what is important is that every single


component in the Fourier spectrum represents information about the entirety of the signal.


And that is exactly what we want. Whereas the regular convolution is localized in in pixel


space, the Fourier convolution is going to be localized in frequency space, but global in pixel


space. That is very cool. And of course, Fourier transforms are also one of the things that are


extremely fast. It's essentially a linear algebra operation. And there are very fast implementations


of discrete Fourier transforms called fast Fourier transforms. And that's exactly what they do right


here. The whole architecture is going to look like this, there is going to be the image, the input


image x, there's going to be a mask during training that is produced by a mask generation algorithm,


x is then masked out. And the model is tasked to predict the missing pixels that are hidden by the


mask. As I said, the model has no access to what's below the mask, I guess that will be kind of


pointless, right? Yeah, so what we do first, but also this is a fully convolutional architecture


that makes it able to essentially transfer to different resolutions, which is another advantage


here being a fully convolutional. So what we do is first we downscale a little bit as far as I can


tell these images are something like 256 by 256 during training, or it works on crops of 256 by 256


somewhere in that range. But the cool thing is it can generalize to high definition images like


1920 by 1080 or something like this, the same network. So the train the network that's trained


on this low, low, quote unquote, low resolution can generate can generalize to very, very high


resolution. And it won't lose performance. But we'll see that in the experiments. So first,


there's down sampling. And then the model is just this, it's just nine layers. They also have a


variant with 18 layers. But the base model is nine layers of this fast Fourier convolution residual


block. As you can see, it has a residual connection right here, like a normal resnet, whereas a normal


resnet would have two convolutional layers right here, we opt for these fast Fourier convolutional


layers. Now, they look a bit complicated. But essentially, what we do is we carry two different


signals across the entire network, one signal contains local localized information. So one


signal is going to operate in the original domain of pixel space and has all that those properties.


So it looks at its neighbors and so on. And one, a signal is going to operate in in more of the


global domain. And then in each layer, those two strands of information get to exchange information


with each other. So the whole signal is represented as this block here with the two components. But


it's essentially just we have like two strands of signal. And then every now and then they get to


exchange a bit of information, right? One is the local, the local branch, and one is the global


branch of information. So what do we do with the local branch, we have different operations right


here. So we have a little conv layer that is in pixel space, actually, we have two of them,


right, two conv layers. So we pass this the local signal, this is really just if you just consider


this path right here through this one, then ignore ignore this here, if you just go here,


this is just like a normal conv net, right? This path here gets information from this side here.


It receives it and then there is an addition. So what is that? That is simply this global signal,


the global signal, also doing a localized convolution in pixel space. So far, there is


nothing special, if we were to just do this, this would be it would be pointless to have two strands


of information, right. But the important thing is that the global strand comes to be in a very


special way. So for that, we have to look what information arrives at the global branch right


here, because that's the information that's going to be passed in here for the next layer. For that,


we see from the local branch, there's a three by three convolution going over here. So let me draw


that in greenish over here. And that is going to be mixed with this global strand of information.


And the global strand of information is going through this spectral transform block. The


spectral transform block is essentially pretty easy. There is a there's a batch norm, sorry,


a convolution batch norm relu block, this is a one by one convolution, this is simply simply a linear


operator pixel wise, essentially, there is a batch norm, there is a relu for the non linearity. And


then what we do is we do a fast Fourier transform in 2d. And at the end of the block, we're going to


invert that. So fast Fourier transform to operate in Fourier space, and then invert the fast Fourier


transform at the end. And inside of it, we're going to do a convolution batch norm relu block


right here. So the convolution again, that's a one by one convolution, I believe, followed by batch


and followed by relu. So actually, even forget what I said about localized convolutions right here,


if they just do one by one convolutions, they really operate just on the individual elements


of the spectrum by itself, not even they don't even consider localized sorry, neighborhoods of


frequencies, they just operate on the individual frequencies, one by one, which is is an option,


like one by one convolutions are are a thing. So you know, pretty cool. This by itself also has a


residual connection right here, I'm going to guess to make signal flow better or more more stable or


some something like this, the observant people might object and say, hey, this thing right here


actually outputs complex numbers. So this is in the space of complex numbers. So you'll get vectors


with entries like a plus, plus IB. But what we do is simply we take those, and we stack them. So we


just make like vectors out of them, A and B. So if there is a bunch of numbers, it will just be like


a one b one, a one b one, b one, a two, b two, and so on. And we just consider this to be a real


vector of double dimensionality, or a real 2d signal of double the dimensionality as before.


And that is how we do it. I mean, it's not, it's not entirely correct, right. But the model in this


way has access to all the relevant information, it can do what it wants with it. Yeah, it can,


it can learn that half of the dimensions correspond to two phases, or, or whatever,


whatever the complex part of this is. It's been a while since, since been a while, since Fourier


transforms. Okay, so these are the exactly so here, that's done, we have sorry, go back up here to


start it, there is first a real FFT, as you can see, that gets you to complex space, then there


is complex to real in which we transform the C channels into two C channels, but now we're in the


real numbers. Then there is this ReLU batch norm conv, which retains the signal. And there is real


to complex where we go back into complex space. So from reals to the two C channels into complex,


just C channels, and then we reverse the Fourier transform. And that is a Fourier convolution,


as they define it, if we integrate, no, that is the spectral transform block right here,


the Fourier transfer, the Fourier convolution is this entire construct right here, as you can see,


the spectral transform information then flows in here is combined with some local information that


really should be green. And that then goes into this global output, and obviously will become the


global input to the next layer. So that is how they fuse localized information with global


information in every single layer. And that turns out to be pretty, pretty powerful. They do have


other improvements right here. And it's it's crazy to see that just how much engineering and how many


tricks go into these models to really get them to work. So they also stress that loss function is a


really, really important topic right here, because you can't simply reconstruct the original image


right here, if you simply re tell the model to reconstruct the original image from here, it's


going to be bad because if your mask is pretty big, pretty wide, there can be many possible


fillings of the mask that makes sense. And since there are many possible ones, if you don't account,


if you don't reward the model for getting one of the possible ones without punishing it that it


didn't get all the other ones, the model is going to be very confused and is simply going to output


the average of all the possible ones, which we don't want, we want one of the possible ones. So


what we do is we apply a perceptive loss, they call that a perceptive loss. And they explain that


over here, what you do is you feed the image, the original image, this is the real one, and the fake


one, and you can already see there's going to be like a discriminator later, right? But you feed


them both through a pre trained neural network. And then you compare at intermediate points, or


even like at the last latent layer, you compare the two feature maps. So depending on how this


network is trained, if that outputs very perceptually salient features, you'll get like


a nice loss that doesn't punish you for getting any pixels wrong. But that encourages you to get


something that is perceptually similar to what was there in the original image. They also stress that


it's really important on how you train this network right here. They suggest to make this


network also include global context using either also Fourier convolutions or dilated convolutions.


And here you can see, that's essentially the formula. That means we take the features from the


original image and the features from the fake image, and we calculate their distance. And that's


going to be the high receptive field perceptual loss. This is not the only thing they do. They


also have as you can see an adversarial loss. There is also a regularizer on the gradients. So


yeah, the final loss you're going to end up with is like a mix of all of these different losses.


There is also a discriminator based perceptual loss. And this part right here is by itself,


again, a conjunction of two losses. So rest assured, there's the loss architecture right here


is very, very intricate. And I'm going to guess it's taken a lot of experimentation, not only by


this paper, but by the whole field here to really come up with with nice losses that make your


outputs nice. Obviously, there's going to be a bunch of hyper parameters here to tune, which is


always fun, but they've, they seem to have done a pretty, pretty good job. The last thing they


stress, which is important is how you generate masks during training. So during training, you


can't just you know, take your finger and draw on pictures. Like I did, you have to have some


heuristic way of generating masks, and not going to go into the detail of how they do it. You can


see here compared to this is one of the one of the baselines. And this is one of their heuristics,


they have a mix of these large masks, and the box masks. So sorry, both are large, but one is called


wide masks, which are kind of polygons that they round off the corners, I think, and box masks,


which are sort of heuristically generated boxes right here or stacks of these boxes. And that's,


and they mix those two together in order to get the final masking for their images. You can see


these are fairly large, like this one here covers more than more than half the image. So these are


challenging, challenging tasks. But it is through training with such large masks that you get the


models to really learn to fill in it consistently. So what you can see is that in their results, and


we're not going to go into all the tape, like they have a lot of tables, a lot of ablations, but red


essentially means that it's worse than their model, you can see almost all of the table is red,


except some models in some of the benchmarks, for example, in the narrow masks, you will find


situations where other models might outperform their model. But as soon as you go to like wide


masks, it is no longer, it's no longer really a competition at all. Yeah, so their model seems to


be really good. Those white masks, they do a lot of ablations where they switch out different, for


example, different convolutions right here, they show what if we switch the Fourier by a dilated


by a dilated convolution, which is also a way to increase the receptive field rapidly, or by regular


convolution. And again, while there might be some improvements sometime on narrow masks, as soon as


you go to wide masks, the other models degrade pretty quickly, the dilated convolution actually


holds up fairly well right here. But one disadvantage of that is that it's very hard to go


to higher resolutions, because the higher resolution you go, the dilated convolutions,


their receptive fields will also shrink, while the Fourier convolutions receptive fields will always


remain essentially global. So here we have some comparison to baselines. You can see of course,


they chose these pictures well with kind of the regular structure in the background. But check


this out. Like this is even this is even their model. But with regular convolutions, and even if


they go deeper, doesn't really help. But like this, this is just insane, right? I get it, they


picked this picture, but it is like is really good. You can also see this building how it's completed


over here with different methods. And then with their method, and the mask was, you know, fairly,


fairly big, as you can see, also the bottom list, the mask is huge. Yeah, here they show what


happens if you go to higher resolution. So on this rather simpler problem, you can see that


a lot of the models do well in the top row, if you just have the kind of a lower resolution.


But if you go to really high resolution, a lot of the models struggle while the Lama model here


still does a big, a good job in their larger model seems to be even better.


Yeah, again, lots of ablations, but I'm going to stop right here. And we'll go over to chatting


with the first author about this. So I'll see you in a bit. Hello, everyone. I'm here with Roman


silver off and Elisaveta logachova, the authors of the Lama paper and Lama system as well, I guess


I think this is as much a paper as it is an engineering effort. And just because I'm here


and just because looking at the paper, it already dawns on just how many things are important in


this system. And then trying this out myself, it really works like it's snappy. It's really cool.


And the results are pretty great, I have to say for a for a learned system. So first, like welcome


both of you and big props on big props on the system is very cool. So you've seen you've seen


the video, what did strike you? What where did I get it wrong? Yeah, first of all, I think that


you did a great job in describing the overall paper. And I have almost no, you know, I have


almost nothing to no complaints. Yeah, no complaints regarding that. And maybe one point


regarding the overall overall point of the paper. And yeah, as it's seen from the title,


Fourier convolution might be stand out a little bit more than other components. But the actually


the paper is about that all three components like, like we generate data and how we process


images with a neural network and how we optimize this, how what losses do we choose,


all these three components are important. And yes, sometimes they can be relatively easily tuned from


existing methods. And allow to such easy tuning can help to significantly improve the results. So


that's that's what's the overall point of the paper. Yeah, I had this I had the feeling to you


again and again, stress that a lot of these things are important, especially the three main


components. And you did a lot of ablations to also show that all of these are important. That's


why I find it so impressive, right? Because people usually just put which one did you start with


first? Did you first have the idea of the Fourier convolutions? Is was that the motivation?


Yeah, no, initially, we started when we when we started overall project on the in painting,


we just started with a classic picks to picks. So just get clone and predict an existing code base


from piece to piece. And then we tried to step iteratively identify the most weak points and


try to understand what is the reason behind that weakness. And at some stage, we understood that


most architectures we tried really lots of different architectures and we tried existing


blocks from other in painting papers. And we found that almost none of them can handle repetitive


patterns. Well, and yes, we started it. When we think about repetitions, the one of the most


obvious thing that came in mind is Fourier transform, because it is very natural thing to


handle periodic signals. And first, we started composing a layer on our own. And then we just


googled and found that FFC, which was proposed for recognition task. And we thought that it is a


great thing to start with and took it and modified it and tuned for that particular task. And yeah,


it worked pretty well. So these would be the the Fourier convolutions. Was it already in the form


that we see in the paper with the two strands of information like the global and the local? Or did


you have to shake things up? No, this, this, the right part of this feature is reflect the original


form of this fast Fourier convolution as it was proposed by the authors. Cool. And did it work


out of the box? Yes. But when we tuned that for in painting, we figured out that the local branch


is not really important. And we can handle almost everything with just global branches that spectral


transform. Yeah. So but you still kept the local branch in? Yeah, because it helps for stability,


especially in not such large images and large masks. So if we try to push the generalization to


high resolution to extreme and to train on very low resolutions and then infer in very high, high


resolutions, then using only global branch will pay more. But in the real world, some combinations,


some combination of these two is more practical. Yeah. So this is it's something I found interesting


because you have this point of these large, large masks, or very wide masks, and so on. And you


stress the importance of your algorithm that produces these different masks. Now, when I look


at these pictures, it doesn't seem that different, right? If I look at the top row, you know, there's


also like some parts of the picture are also occluded relatively big parts, there are kind of


some squiggles. They're even relatively wide, right? Why do you have an intuition? Why is the


mask generation algorithm so important? Is it? Is it important that it's close to what humans do


later? Or is it important that it is of a certain shape because of the architecture of the network?


Or what's the deal with that? Yeah, as with the architecture, we started with an existing


heuristic to draw that masks. And we actually we follow the same algorithm as the one used in Deep


Field version two, the first row in that figure. Why masks should be wide? Yeah, because it is


important because the width of masks forces the generator to pass the information more far within


itself. So if we can cover almost all input image with very thin lines, for example, we can mask out


every second row and every second column in the input image, and that would be very something very


similar to a super resolution problem. And the percent of the image will be covered by such mask,


but the network wouldn't need to pass information far. Yeah, that's why


wide masks are important and they are more important for fully convolutional architectures,


but for Fourier based, they always help as well. And we have a couple of histograms in


our supplementary material, which compare actually the first row of that figure with the mask generated


by our algorithm. And the difference is pretty huge, actually. It is cool to see that the difference


is so big. I think that it was mask that it was point from which we started actually, because we


aimed to invent real world examples. And in that examples, masks actually are huge. So we started


with big masks in our validation set, and we saw that all other algorithms have failed to


fail to fill this large holes. And then we started to think on how we need to change our model,


that it can incorporate global information. Yeah. Is your algorithm deterministic? Yeah.


If I give it the same input and the same mask. So if I, and this is, is this correct that the cleanup.pictures


app that is really your small model that runs here? No, this is the large model. Oh, this is the big


model already. Okay. So here, you know, I've taken this, but what happens? Have you ever


tried just masking the whole picture? What's kind of like the default output? That's an interesting,


I don't know what will happen. Something average, a constant color maybe.


Let's see. Yeah. All right. Pretty unspectacular, but I guess it's very gray is very high probability,


right? Okay. Yeah. Cool. And then there's the third component is the loss. And I have to say


the loss is a monstrosity. There are like 50. So first of all, you have, now this is the adversarial


part of the loss. And then on top of that, you have like the discriminator perceptive loss. I'm


going to guess that's the same as the perceptual loss, but in the features of the discriminator.


Yeah. So the features which are used to calculate a discriminator based perceptual loss are updated


throughout the training. This is a pretty commonly used. This is a commonly used loss in image to


image tasks. It helps to stabilize training. So the idea is that the discriminator bases its


decisions on features which are perceptually meaningful. So very similar to the perceptive


loss that you have up here, right? I think that feature matching or discriminator based


perceptual loss helps mostly because it provides a clear signal to the generator. And if in adversarial


training, we have to balance a discriminator and generator. And if one part is more powerful,


the whole thing collapses. And discriminator based perceptual loss helps the generator to catch up


when discriminator becomes too powerful. Yeah, that makes sense. For all of these losses,


right? And then you have a regularizer on the gradients, and you have this wide field perceptive


loss and so on. Did you plan this from the beginning? Did you say, you know, here are all


the good losses that I know of? Or do you have more losses that you ended up not including?


My question is how, if I'm a, if I'm a researcher or an engineer trying to come up with such a system,


how do I decide which seven losses go into my final loss, right of the 50 possible losses that


I could do? Do I try them all? Or are there some some guidelines? Actually, I think all of these


losses, except for high perceptual loss are pretty common. And they all


really often used in image to image tasks. Like we need something to force our model to


to create a realistic picture. So we need discriminator and its loss. We need to


reconstruct something that we can reconstruct, so we need some loss for reconstruction.


And editing the losses to restrict it, so we need something that works on features.


But we worked a lot on, we made the paper parameter search, of course. And we changed our,


we changed our, we worked on our perceptual loss form. Because we started with this common


perceptual loss based on BGG model. But we had a feeling that it can be not perfect, because it's,


it was the model that learned on classification task and models that was, were trained on


classification. They seem to concentrate on texture and not global structure. So


we decided to try something else. And then we find this model that learned on segmentation tasks.


And on the set that is more similar to ours, as I said. And we tried it and it worked.


So the segmentation task as a training task for the perceptual loss model is sort of a better


preconditioner than the classification task. Yeah, yeah, because it is natural for the


segmentation model to focus more on boundaries of objects instead of their textures. And in case of


in painting, good texture can be learned using only discriminator. Because there is a lot of


freedom in how we can generate fine grained textures and there is no need to put some,


any supervision on that part of the image. But it's also important that models that are used for


segmentation are different. So we compared in our ablation, we compared the same model that was


that was on the train on classification and it works was the same model.


Yeah, you have not only do you have a different task with the segmentation, you also include also


high receptive field layers into that model. So the sense that the logic is that if that model


also includes global information more, it's, its signal to your model will also be more


sensitive to that global information. It's a little bit like, you know, in reinforcement


learning, people do reward shaping, it seems like you do reward shaping by how you train the


different the different discriminator models that that then give your model the sort of the signal


to learn. Yeah, I like the sort of the meta idea here. That's pretty cool.


Unfortunately, I'm not familiar with word for shaping from reinforcement learning. But our idea


here was that basically, we have two losses here. The first one is discriminator or the serial and


which focus more on fine grained details. And the second is perceptual loss, which focus more on


global text, global structures.


For the Fourier convolutions, maybe a little bit more conceptual, right? We have this local


information in one strand, we have this global information in the other strand. And it's it's


clear that for these large masks, as you show the system works pretty well. What kind of data does


your system work not well on? Like what's what would be sort of the worst input that I could


give to your system like this, this up here is really beautiful, right? Well, what picture could


I take such that it is absolute garbage? Yeah, actually, lots of images will be processed bad


with our model. I mean, part of course, I can give it a picture that is, you know, very dissimilar to


the training data set. But let's say I actually had a training data set, what, what would be the


worst domain or the worst kind of picture? Yeah. It's kind of creates half of human or something.


Yeah, our model focuses mostly on the ground due to how it was trained. And yeah, it cannot recover


foreground objects really well. It cannot do something that requires it to actually know


everything about what works and not just take it from picture it sees. Yeah. So is it is it? Do you


feel that the model mostly learns how to sort of copy elements from the part it sees to the parts


that are masked? Do you think that the learning is mostly teaching the model how to do that? Because


it seems the model is very sophisticated in, you know, in Photoshop, you take this stamp tool,


right? You say, I'll take a little bit from over here, put it here. Do you think your model is


just like a really, really good user of that tool, in a sense? Yeah, it seems yes, yes. And in order


to be able to create big parts of images from scratch, we need a different kind of model. And we


most probably need a kind of testity within the generator. Because without it, it is not possible


to create something from nothing. Yeah. Also, our model is quite small. So it's kind of really


remember everything. Yeah, that is something that I left completely out of my review, I think the


fact that your model is compared to the the baselines you compare to is a lot smaller,


right? It's it has way less parameters. That is something that's I think, very cool and enables it


to run inside web applications and so on, like on or maybe on a mobile device or Yeah, I have I have


another question and to the Fourier convolution. So here we have global information and and local


information, right as sort of two different things. You mentioned in the paper that other


models that have more global information or access to wider information could also work


such as a vision transformer or something like this. My question is, is there an in between


between local convolutions and Fourier convolutions? Okay, I mean, there's dilated


convolutions. But if I think of a Fourier transform, you transform into a space where


locality no longer matters, but frequency matters. And in the original domain, frequency is just kind


of doesn't matter. But locality really matters. Is there a transform? Are there transforms that we


could do that put us in between where, you know, the as I go in the x coordinate, it's a little bit


of frequency and a little bit of locality? Like, is there hope that instead of having multiple


columns of information, we could sort of choose our space wisely to trade off local and global?


Or do you think this is already, you know, local, like a mix with two channels is a good way to go?


That's, that's a very good question. Yeah, and I don't know the answer to it. And


one thing that comes to my mind is, there is a short time for you to transform, which is


often used for music processing, sound processing. And yet, kind of combines local convolutions with


Fourier transform over it is roughly can be described as processing the whole signal with


a sliding window and transform each each, each sliding window with a with Fourier transform.


Yeah. So it is most obvious combination. If you had to give your intuition, why the Fourier


convolutions make such a big difference here, of course, like that we've already discussed Fourier


transform kind of loses the locality of the signal and gets global information. But why


Fourier transforms? What's what's kind of good about this particular function that you chose and


space that you chose? Surprisingly, if we throw the local branch away, it will still generate


something meaningful, meaningful. So spectrum transform doesn't loses that local local local


correlations completely. And I think that this is due to the fact that the generator has spectral


transforms and spatial transforms interleaving each other because here we can see that we have


cone one by one between between two FFT and we have two more convolutions before and after


the spectrum transform. They are as well one by one, so they don't capture local content directly,


but then can combine channels on that particular locations and yeah, that maybe that can somehow


replace traditional convolutions. The fact that this this spatial and spectral transforms are


interleaved. Yeah. And when we think about generalization to higher resolution, I think


spectral transform helps because of the fact that low frequency part of spectrum does not that


strong, doesn't depend on the resolution on the resolution that strong. And it is almost the same,


no matter if we have to 2056 or sorry, 256 or 2000. Yeah, yeah, that by itself is one of the


cool properties again of your paper, the fact that it can scale up to sort of very high


resolutions, there are artifacts appearing, but they are not nearly as much as in other other


models looks pretty cool. Yeah, it doesn't scale up perfectly, but yeah, it's better than fully


convolution architectures. Cool. Yeah. So where do you think? I mean, maybe you don't want to


disclose necessarily, but but what what do you what is the plan for the future?


We don't know where we get throughout research. But yeah, the most obvious thing here is that we


can try to improve the way generalized to higher resolutions. And the second point is that we are


trying to understand why actually it works that because it Yeah, it has lots of components. And


we conducted an ablation study regarding if validating if each of these components matter,


but this is just a surface. And we can go more in depth in that. And we are not satisfied with


our laws because it's that huge. There are many components that we need to balance. And we want


better laws with just one. Just one but but make everything work. And so yeah, I mean, I was almost


I was expecting you to say we're not happy with our loss. We want more. We want like more


components to make it but it's is I think it's pretty cool that the goal is also to make a


system that's kind of as good but simpler. I think they'll make it also much more accessible. Cool.


Yeah. Roman, Elisa, sorry, Lisa. Is that correct? Yes. Okay. Lisa and Roman, thank you so much for


being here. Was it was a pleasure. Do you have any last criticisms to the video? Or shout outs?


No. Thank you very much for the discussion. It was really fun. And thank you for your channel,


because you make a real good job in helping others to be in time and to catch with this


Corner Detection - OpenCV with Python for Image and Video Analysis 13

What's going on everybody welcome to another python with openCV tutorial in this tutorial what we're going to be talking about is corner


detection. Corner detection can actually be used for quite a few things mainly


we're going to see it like three-dimensional recreation you can see


it in motion tracking and also it gets used in character recognition fairly


frequently so let's go ahead and dive in. We're going to be using the following


image feel free to use this image or create your own as usual the image for


this tutorial is available at pythonprogramming.net for this tutorial


you can also click the link in the description for this specific tutorial


so that's the image we're going to use and ideally we're going to try to find all of the corners now that said it might be kind of hard to tell in this picture I can tell actually you should be able to tell that little thing on the left that I drew there


drew in Microsoft Paint and as you can see those slanted lines have quite a bit of aliasing issues going on so what we're going to be doing is detecting corners and what it's going to do is find all the corners so with all that jaggedy edginess going on there a lot of corners are probably going to be detected if you let them be detected.


so anyway and rightly so those are corners so keep that in mind so now what we're going to do is you'll need CV2 and NumPy imported and then what we're going to go ahead and do is image and that image that we're just looking at I've got it saved so CV2.emeread to bring in the image the image name that I called it was open CV-corner-detection-sample.jpg


and then we're going to gray it out we're going to say CV2.convert-color image CV2.color underscore BGR2 gray then finally again gray now is going to equal np float 32 we're just converting this to a float 32 of gray and that's mostly to satisfy the algorithm that we're going to use to actually find the corners


so now corners is equal to CV2.good features to track and then against gray we're going to find up to a hundred of them 0.01 here and basically that's quality and then finally the minimum distance between corners so when you're running this algorithm it's on what


how many are we willing to find up to image quality and then the minimum distance between every time it finds a corner so in the case of slanty lines like we've got if you put that number high enough it's not going to catch those alias slides as often anyway it'll still find them it'll just make them few and far between


so that's corners and then finally we're going to convert that to numpy.int 0 of corners at any point when we use numpy and we do these conversions feel free to print out the results if you're not sure what it's going to look like print it out generally it's just data type conversions


I wouldn't spend too much time worrying about these conversions it could just as easily be happening in the back end and you really wouldn't have known these even existed it just so happens that to work with CV we got to do this stuff so anyway now we're going to go ahead and do is we're going to say for corner in corners


what do we want to do x y equals corner dot rabble and then we're going to create a circle now so CV to die circle and we're going to do this circle at on the image in the location of x y for that the center of the radius will be three color to be five and then negative one fill that bad boy in finally CV to


die in show will show the corners on the image and I think that's good let's run it okay sure enough we find pretty much all the corners possible I don't really see any other corners besides maybe in the inside of that E that would have been nice and the bottom of that why should have been a corner but we probably ran out of corners


so for example if we did 200 we might actually get all those filled in well we did get down on the why we did not get on the inside of eat I'm guessing it's this this aliasing over here that's really causing us trouble


except up to 20,000 corners nope still don't get it in there so maybe that's not it's just not a corner y'all it doesn't exist well let's say you track 10 corners race gonna find 10 corners so bam bam bam bam bam and then some over here okay so you just want to make sure you're going to let as many corners show up as possible it doesn't appear to actually


give like weights to any of the corners like obviously the corners of the squares should be like immediately picked but not necessarily are they actually picked so anyway so that's corner detection and again that would be used you could use corner detection to track objects motions or with three rendering like sometimes you want to take a picture of something and then maybe 3D print that something if you can detect the corners and you can draw lines between the corners you can help to


kind of render that object in 3D so that's corner detection and the other thing that people use it for is actually character recognition should be detect the corners of characters and then draw lines between those corners a lot of times the character actually comes to life so so it's using character recognition as well especially if if they're fat characters like we've got going on here if you kind of connected all these these are maybe too fat but if you connected all these dots you would actually be able to see the character


and if you connected all these dots you would actually kind of see like oh yeah those are definitely there's a shape going on there the E would be kind of hard but anyway in the next tutorial we're going to be talking about is actually feature matching between images so so far as far as like using object recognition of any type we've done template recognition we're going to be talking about in the next tutorial is a little bit more advanced so templates going to match almost identically based on template whereas what we're about to do is feature match


GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (Paper Explained)

OpenAI has a 175 billion parameter model.


You thought that was large. That's cute.


Check out Google's 600 billion parameter model, 600 billion floating point numbers doing things at the same time.


This has absolutely become a body part measuring competitions between companies.


Google be like, Oh, GPT-3, I spit on you. I spit on you and your little tiny 175 billion.


Okay, let's stop kidding. This is a giant model that Google has trained right here.


The paper we're going to look at today is called G shard scaling giant models with conditional computation and automatic sharding by Dimitri Lepikin et al of Google.


And this paper basically tells the story of how they built this 600 billion parameter model,


how actually they attempted to build a model that had a trillion parameters, but just didn't manage to quite train it.


And this is all using this system called G shard.


So I haven't actually seen the code out for G shard yet, but I'm going to maybe assume that this is something that they're going to release at some point, who knows?


Or maybe I just haven't seen it yet.


So this is basically describing a system on how to train these giant models.


So if you if you have watched my video on GPT-3, which of course was this 175 billion parameter model of OpenAI, which already was record breaking.


You this paper, the paper was very much like, oh, we built a model and look at what things it can do.


So that was the OpenAI paper. This paper here is like the complete opposite.


It basically says, Oh, yeah, we do language model. But But here is how we built the model, which is, you know, equally cool.


So OpenAI basically just made everything bigger. And here they say to make everything even bigger, you need some tricks and how to build models.


And they've basically, they've developed this entire framework to build these giant models.


And this paper mainly describes that framework. And the actual task here, which is machine translation, is almost sort of a side thing in the paper.


It's it's all it's just a task to showcase what this system can do.


So this is very much an engineering paper, rather than that much than a machine learning paper.


And that's how you have to look at it right here. That being said, the machine learning results are of course, quite impressive.


If you look at this graph, here, you have a quality gain. It's a difference in Blu score.


And this is a quality score for machine translation over the previous state of the art.


So over their baseline, which, as you can see, here, you have 37 billion weights, 150 billion weights and 600 billion weights, which they only train.


They train for you know, 2000 and on 2048 TPUs for just four days. That's they stress this is very efficient because they just have to train it for four days on 2000 TPUs.


Absolutely crazy. So let's have a look at what this paper does.


If you enjoy this, if you enjoyed this at the end, consider, you know, sharing the video out if you like it and tell me what you think about this stuff in the comments.


All right. So we'll go through the abstract and then we'll go through highlighted sections of the paper because the paper is 23 pages long, so I won't be able to cover everything.


Just kind of give you the high level ideas and highlight a few things.


Actually, let's not go into the abstract. Let's go into Yeah, these results first.


So as you can see, they managed to continue the trend. The trend in NLP has always been and at least since, you know, transformers were invented, the bigger the better, like larger model, larger data, more compute means better performance.


And this is sort of unbroken here. As you can see, if you increase the number of parameters in these models, you do get a very, very big gain in these blur score, though it sort of seems to be kind of a logarithmic scaling like you have to you have to keep doubling and doubling and doubling the number of weights sort of like Moore's law in computation.


You can see that at the same time, the training wall time is going down and the computational cost, the computational cost of these models, it doesn't scale quadratically, like you would expect it scales linearly.


And that's the big difference here in how these authors scale their model, rather than how the open AI authors scale their model.


So in a traditional model in traditional transformer looks like this. So it has these blocks of attention. If you don't know what this is, I have a video called attention is all you need. I explain how the attention blocks in transformers work.


So this is nothing different. These are just transformers standard transformers, there is an encoder, and the decoder, everything works as you know, so you have these blocks, you have n blocks, these are the number of layers that you have.


And in these blocks, you always have an attention layer, and then a feed forward layer that acts on the tokens. So without repeating too much what an attention mechanism does basically in you have inputs tokens. So this is a sequence.


It's technically a set processing unit, but we use it for sequences of text. So here you have six tokens, a sentence of maybe six words. And then you transform it with the attention layer by having this attention mechanism that routes information from tokens to from positions to other positions, maybe like this route is here route this here.


And then you have a feed forward network that is applied on a per token basis. So each of these tokens now goes through this feed forward network, and is kind of transformed. So the embedding of that token is transformed by that feed forward network.


Now every token does this. And it's always the same feed forward network. So this network here is the same as this network. Now, usually, when we talk about scaling transformers, we talk about this part right here, we talk about the attention mechanism.


And also we talk about this part, the number of layers. So, you know, we talk about scaling the number of transformer layers, more layers, more layers, more layers. And if we want to scale the attention mechanism, what that basically means is, we have we increased the context size of the text we can input.


So transformers are very limited by the size of this context right here, that they can take the like the original transformer started with something like 512 tokens that they were able to take because this attention mechanism has quadratic complexity.


This went up and the open AI GPT-3, I believe, had a context size of 2048 tokens, which if it scales quadratically, that's quite an achievement. And also it stacked the layers very, very deep.


Now, in this paper, they scale the transformers differently, they basically leave the context size. And I believe that their context size is 1024. So significantly smaller than the open AI context size, and they don't scale the layers.


So their largest transformer is 36 layers, whereas I believe GPT-3 was maybe correct me, but I think it was like 90 or 100 layers or something like this, at least significantly larger than this. Instead, what they scale is this part right here, the feet forward layers.


This might seem counterintuitive. But they basically, they basically say, what if we didn't only have one feet forward network right here, but we had many, right, we don't always have the same, we have many, many feet forward networks, different ones that can do different things.


And that's what they call experts. Each one of these feet forward layers is an expert. And then you have yet another routing mechanism, kind of like in attention, you have a routing mechanism that decides which tokens go where, okay, so this token here, this token here, this token here, and the sort of the implication being that different tokens, different parts of the input you want to transform require a different kind of transformations here.


And these different experts can sort of specialize in how they transform the input. Now, their task here is going to be machine translation as a multi task setup.


So what you'll have is you'll have all kinds of languages like French and German, what's the, and maybe like a lot of languages, I don't know any other languages. And you want to translate all of them to English, and you want to do it using the same model.


Right. So these experts here, they might specialize in, you know, the individual languages, like maybe you will have to handle a pronoun differently, if it comes from German than if it comes from French, you want to do it with the same model at the same time.


That means you maybe want to have the one expert specialize in German pronouns and one expert specialize in French pronouns. Also, you can think of the experts as maybe one specializes in question words, it doesn't matter which language they're from, and the other one specializes in some sort of other kind of linguistic feature.


In any case, this number of experts here is if you want to scale that up, then that becomes the bottleneck of the transformer, they go up to 2000 2048 experts in parallel.


So that doesn't fit into a single accelerator anymore. And that's why the entire system has to be sharded. And that's what they call G shard. So G shard, the main application here is going to be how, how can we build this giant model in on many, many distributed computers, where the attention mechanism isn't the problem, the attention mechanism, we just distribute, like we do data parallelism, the attention, it lives on all the computers.


It lives on all of the accelerators, it synchronizes, and so on. But the experts here, there's only so this expert lives on machine a this expert lives on machine B, this expert lives on machine C. And then we do a hard routing. So we don't do a soft routing, like an attention, we do a hard routing where one token goes to one or at maximum two experts.


So this is sent to these machines. And then after the machines, you kind of gather all the results back right here. So G shard is the system that enables this sharding of these experts, and the everything in between everything that is necessary, but it can also be applied to shard any computation. And that's why it's so cool.


So here you see what, what they do, they always they take these transformers. And they always consider a block of two transformer layers. So this is a block of two transformer layers, you can see there is twice the attention. And there's twice this feed forward. So in one point, this feed forward is just a regular everything, all the tokens go through the same network.


So that's like a classic transformer. But here, you have a lot of these different experts, and the tokens are routed to these experts. It's important that the tokens are hard routed, right? If the tokens were soft routed, you don't you don't gain anything, because every token has to go through every expert.


But here, the tokens are hard routed to the expert, which means that you can, if you if I have an input size of 1024 tokens, maybe only 10 go to this one, and maybe only 10 of that those go to this one. Now you also have a batch size, of course, I haven't actually looked at what the batch size here is, but you usually have quite a large batch size in these things like maybe a batch size of 1000 as well.


So ultimately, what you'll end up is, you know, 1000 times 10 tokens going to the first expert and so on. But still, you can significantly parallelize this computation. Okay.


So this this, if you use G shard, this is going to result in the following in the thing on the right, where you have two machines, this is machine one, and this is machine two, you can see that the machines will what happened here. Oh, someone made the PowerPoint mistake.


So you can see that the the attention, everything is shared between the machine. So this here and this here, these are synchronized, the weights are synchronized, right, you simply do a data sharing.


But here, you can see that you have model parallelism, model parallel mixture of experts, where on the first machine, you have the first expert, and then you have E devices. And on the last one, you have the last expert.


And then it's all routed out and routed in again. And then you can continue your transformer. And this is layer after layer.


So what's the problem here? The problem is that an operation like this is going to come to incur significant sort of overhead in terms of communication and so on, if you were to do it naively, and it's going to be a real pain to program this.


And that's why G shard is made to do all of this automatically. And you don't, you don't incur much of a cost, because you distribute.


So what's the difference to the old scaling? Why don't they just make transformers larger in number of layers? And that's because this this is, I guess, what open AI ran into as well.


If you make transformers simply larger in number of layers, sorry, if you make it transformers larger in the attention mechanism, it just won't fit into memory at some point, and you'll you'll have to share that somehow.


And you can do this with G shard. If you scale it in number of layers, that incurs significant cost where you have to wait, because you have to forward propagate, and then you have to backward propagate in your training sequence.


And if you have just too many layers, then a lot of the a lot of the frameworks get at their limit, where at some point, they say, Well, I still have to wait for the signal to come back in order to continue.


And they explore this in this benchmark right here.


You can see they say, the largest model, the 600 billion parameter model that achieved the best translation quality was trained with 2000 TPU v3 course for three days, a total cost of 22 TPU couriers.


In contrast, training all 100 bilingual baseline models would have required 29 couriers. So the model here is faster than if you train them individually. But if you want to train a single transformer that is just very deep, and achieves reasonable performance, you have to invest a lot more.


Our best quality dense single transformer model 2.3 billion parameters. So it's also significantly smaller. Achieving this was trained with g pipe, which is a previous framework. So g pipe is kind of a task runner that also distributes computation was trained with g pipe on 2048 TPU course for six weeks, or a total of 235 TPU couriers.


By the way, for if you if you have $1 per TPU hour, that'll only cause a lonely, I guess, set you back about 2 million or so, easy peasy.


Or even 200,000 just, you know, a tiny, tiny bit of, of money.


But you can see that this transformer model that is dense, which means that is a classic transformer where you stack the transformer layers, you stack them, you stack them, you stack them.


It in fact, it has 96 layers, their baseline 96 layer transformer model, that's sort of what opening I did, they just kept stacking the transformer layers, you get a model that has less parameters and trains for much longer, and its performance is only about this good.


Whereas here, if you scale not into depth, but into width of these experts, and it's not dense, but it's shorted, which means it calculates this in a in a kind of sparsified way, because it has this hard routing, you can scale up to a lot more parameters.


So 600 billion parameters over 200 times more parameters than the deep model, and you can get a much better performance.


Okay, so this is what is different here, it scales into these experts, rather than scaling into depth or, or size of the attention mechanism itself.


Alright, the question, I guess that you come up with if you're a machine learner is how do you back propagate if you route if here you route to these different experts, and you do a hard routing, like here, how do you back propagate the signal because it seems like you need a soft routing, but this has been handled.


In fact, these mixture of experts has been introduced previously in a paper I think called outrageously large language models or something like this. And, and so they've introduced that, you know, it, it still works.


So back prop still works through. So basically, you have a back prop path through here.


And because you put a little bit of noise in this routing, every path gets explored a few times, and therefore you have enough back prop signal to make it work, it can it could technically fail, but they do observe generally that it does work if you do this kind of hard routing with a bit of noise.


Alright, so where do we go from here? As I said, this is an engineering paper, and it's a long engineering paper. So they, they set up their, they set up a lot of a lot of the details of engineering directly in the paper, which we're not used to in the machine learning world,


they really detail how they shard things and so on.


Which is pretty cool. But, um, I invite you to look at the paper yourself, if you really want to know what's going on right here.


Suffice to say, they, as you can see, right here, what they do is, this is the input right here.


And then they have this weight matrix, which is a this routing. This is learned routing weights. Okay.


So you have trainable weights that decide how to route the input, and that's dependent on the input. So you have a bunch of inputs that comes from the lower layer.


And this matrix right here determines where to route them, basically says, okay, the input is a vector like this, I know that must probably go to the expert number three.


Okay, and you have a softmax across that. So it's a really, it's an assignment to it's a soft assignment to the experts.


So once you've done the soft assignment to the expert, you do a hard assignment by collecting the top two.


For each token, you say, you collect the top two experts, and you only send it to the top two experts, and you ignore all else, which is not a lot right there.


At times, there are 2000 experts in the system. And, yeah, you distribute and you have some noise.


So with a random probability, you actually don't even send it to the second expert, you just leave it at the first one.


And with some noise, you send it also to the second one. And I think that that noise is part of what it of what makes the system work a bit.


And then you also have this auxiliary loss right here that you add on top, which just makes sure that you distribute evenly.


So this encourages the system to distribute the tokens evenly, because sorry, what it penalizes is a this here is the mean assignment to each expert.


So it penalizes whenever the mean assignment is out of out of line, basically.


So a distribution assignment to the expert where one expert gets a lot of tokens, because I don't know, it happens to be really good at something.


So all the tokens are routed to it, and the other expert don't get a lot that's penalized.


So you encourage the system to distribute tokens evenly between those experts.


And then there are also like upper limits where you drop tokens and so on. They really built a system that is out for performance, rather than machine learning correctness.


So they demonstrate how to do this in in sort of code with their system.


And the cool thing about their system is that you don't have to do much.


What you'll have to do is just specify which tensors are sharded at along which dimensions and the system does the rest.


So this is pretty cool. So this here is this mixture of experts, mixture of experts as you would write it in code.


And they make use a lot of this Einstein sum notation.


If you don't know what the Einstein sum notation is, it's a general notation to describe matrix or tensor multiplications.


So a for example, if you were to multiply two matrices, you could have a string there.


You describe it as a string and it comes from how Einstein wrote up the kind of tensor contractions in his work.


So if you want to multiply two matrices, you can't you could put the string a b, b c goes to a c.


So this and then you put two matrices right here. This would tell it, OK, I have a one matrix.


I'm going to call the axis a and b. I have another matrix or tensor where I'm going to call the axis b and c.


Now I have the resulting tensor and I want the first axis to be a and the a is this one.


And I want the last axis to be c and the c is this one. And b is nowhere.


B is not in the output, which means it should contract over b. So it should sum along b.


Sorry, it should multiply along b and then add. So it should contract over b.


So this here describes a regular matrix matrix multiplication.


Now, if I could do something else, I could do something like a just a element wise product.


An element wise product would be something like this.


a b comma a b goes to a b, which means here I have a in the first input and here I have a again.


And I'm so you already see that you can even though these are different tensors, you can call the axis the same,


which means that they're going to somehow be multiplied together.


Now, if you leave it away here, it means that it's going to be contracted and therefore the axis no longer exists.


But here we don't leave it away, which simply means that these axes are going to be multiplied together.


And the same for b right here. So this describes an element wise product.


You can go really funky with this. So this here would be a row wise dot product where a is


for all the a's its element wise, but then over b, it's contracted.


So, you know, you can go you can go wild with the Einstein sum notation.


You can describe a lot of things with it.


So here is this algorithm to distribute the computation among these different experts.


So you have the inputs and the weight matrix for the they call this the gates function.


That's the routing function to these experts. So what do we do?


We first of all, we have these tensors. They have these grouping, this grouping dimension right here.


So they come along to along groups, which in our case, we could maybe say these are batches or the batch dimension.


So they come across groups and there is the sequence length and there is this M right here.


That's going to be the feature dimension, the M. And you can see the M is contracted.


So the M is no longer here. So the gating function is going to route each input token right here to one of the experts for each thing in the group.


So you can see you can express this with an Einstein sum notation.


Then you have a top two gating, which selects the top two from each of the last from each of the entries.


And that gives you this dispatch mask and the sorry and the weights that you have to use at the end to combine.


You can use the dispatch mask in order to distribute the inputs. So you have reshaped inputs and so on.


So I'm not going to go through all of this right here, but you can express all of this in terms of the Einstein sum notation.


And you can express pretty much any sort of computation that is along the line.


You can express the attention mechanism and so on.


You can express the feed forward layers in terms of these Einstein sum notations.


And the underlying the underlined dimensions here are the dimensions where we want to shard the computation.


So here, because we have this g underlined, that means that we are interested in sharding the computation along this axis.


So this I said, this is the batch dimension. This is your classic data parallelism,


which means that the first machine gets the first couple of data points, the second machine gets the second couple of data points and so on.


And you can see in the weight matrix, there is no sharding, which means that the weight matrix lives on every machine as a copy of one another.


This is different from from here, where you can see that what we're now going to do is here, it's still sharded according to the batch,


but we now are going to shard this according to the different experts.


So we're going to route whatever the inputs are in to these experts.


And then we're going to execute the computations on the experts.


So this is now sharded according to the experts. And at the end right here, you can see this is still sharded according to the experts.


We're going to put it back together. And now it's sharded according to the groups again.


That's what we said. We have the input right here, the inputs.


And the inputs are maybe distributed according to the according to machines, right?


We have these go through the first machine, these the second, these the third and so on.


This is your classic data parallelism. But then we have all of these experts.


And now all of a sudden, we're going to route these things to the individual experts and we're going to execute the computation in parallel on the experts.


And then after that, we're going to put back together from wherever we got them now.


So this goes here again. And so this is just the reverse of what we did before.


So right like that. So you get all of the outputs again.


I hope you kind of can imagine how this happens.


So the first difference is that's sharded according to a different dimension.


And the second difference is, is that when we shard in data parallelism, we execute the same computation on all the machines, which means that we have the same weight matrix.


If we do X times W in a feed forward layer and we shard this thing here in data parallelism, what we do is we send the X to different machines.


We split the X, we send it to different machines. This is X1, X2, X3, X4.


But we always multiply it with the same weight matrix. That weight matrix lives on all of the machines and is regularly synchronized.


It's kept synchronous in some way. Whereas if we shard X to the experts, then the experts have individual functions.


So the expert one is different from the expert two is different from the expert three, and so on.


Which means that before it wasn't important where X was routed, because we would execute the same computation.


So we can just, you know, shard it according to you know, the first 10 go there, the next 10 go there.


But here, it's now crucially important where they are routed to, to which expert.


And that's why we learned the function that is going to route them.


So this is learned. This is this first line here. These are the weights that we learn to route.


Then we route right here. And we calculate your, your, we calculate the feed forward layers on the expert.


You see that this WI and WO, they are the weight matrices of the feed forward layer.


The feed forward layers are, you have your input, you multiply it by WI, you have a ReLU, ReLU, and then you multiply it by WO.


So it's kind of a two layer feed forward network.


So this two layer feed forward network, as you can see, this is sharded according to the experts.


And then, and the important part is, of course, that here, the weight is also sharded according to the experts.


And that's what makes each expert different. And then it's combined again down here.


So I hope you kind of get the idea of what this algorithm does.


And the fact that we shard according to these experts is in fact different than your regular sharding where you shard the data like the batch, the batches.


But keep the model in parallel, keep the model synchronized.


With their system right now, this is how easy this is.


So before we simply stated our algorithm in Einstein sum notations, there is no way to underline code and that magically happened something that was simply for us to visualize.


Now we want to apply their system in order to make this actually sharded.


And with the G shard system, and as I said, I don't know if the code is out or it will be out.


But with the G shard system, this is basically all that you have to do.


So you have these functions, they're called split and replicate.


What replicate does is it takes that weight tensor and it replicates it on all the machines and that keeps it synchronized, right?


This is a computation where we simply want to shard out the different to the different machines, but keep it synchronized.


And you can see if you do this, this is the operation.


Then the system knows, ah, this here is replicated across the machines.


So that means I'm going to distribute the data points according to this G dimension, according to the batch dimension and multiply it with this matrix.


According to this Einstein sum notation string on all of the machines.


And I'm going to keep this tensor in sync.


OK, so the system knows.


As opposed to that, you have you have the split tensor right here.


So the split, what it does is it splits a computation here, the dispatch X with inputs.


It splits it according to a axis index onto D different machines or into D different parts.


So you see here you calculate the how you should do the routing.


And the resulting tensors first dimension is this E dimension.


And then you say that should be split according to this first dimension onto D different places.


And these D different places are now separate.


They don't have the they don't have to be kept in sync.


Everyone has their own weights.


And now when you do this, you know, according to this E dimension, you can see because we know Einstein sum notation now.


You can see this E appears here, here and here.


So this operation is going to be applied element wise.


That means independent of each other in the direction of this dimension.


The system understands that since this tensor is sharded according to that dimension.


I have to execute this on each of these entries in separate with on each expert having their own weight matrix right here.


I hope this is a bit clear that their system makes it super easy.


You can basically do two things.


You can say this thing here is my classic parallelism where I want to keep it in sync.


And this thing here is where I want to split up and do different computation on the different parts.


And then they have also a general function that is more powerful.


Yeah, they and they you can auto partition and whatnot.


So they have a they have this.


We implemented the partitioner in the XLA compiler, which means that anything that can translate to XLA is a target for the system.


And that's, you know, TensorFlow and PyTorch can do this.


So technically, this can come to any of those systems.


But of course, who has their 2000 TPUs lying around to make use of this?


But no, I'm kidding. I mean, this I they here use it for transformers.


But I am very excited to see what people can come up with for the system.


I believe a system like this where it's super easy to to shard.


And they have some you know, they talk about, OK, we do the single machine compiler.


So the compiler is also fast and so on.


I don't even want to go into this, but this is very well engineered, it seems.


And they they they basically implement this for all of the operators.


So I'm very excited to see what people can come up with outside of the traditional applications.


I think there can be new types of models developed simply because we have a system like this that makes it easier.


So, yeah, I'm excited.


So here they show a bit how this works on the example of this Einstein sum notation.


So here we want to do this thing here, which if you remember, this is the operation where we want to route the input to these experts.


So we want to start with something that is sharded according to the batch dimension.


That means that we we have different different parts of the batch on different machines.


And we want to route this and finally end up with something that is sharded on the different experts.


So this is what the system does is first you have these here are the different shards, right?


You want to multiply this, as you can see, this and this right here means that this routing table is also sharded according to the same machines.


So you have the zero is all on the same machine, the one is all on the same machine and so on.


So what you want to do is you want to contract is there you want to contract according to this s dimension, right?


Which we have we have omitted right here.


And if you multiply that, sorry, OK, we omit the s.


So this is not much of a this is not much of a of a graphic right here.


But then they have this re-shard operation where they do and you don't have to worry about this.


So from here to here, there is this re-shard operation that just shards it according to the according to e.


Yep. I find this to be a bit more a bit more insightful.


So if you have something like this, this which is a regular matrix multiplication, right?


And you want to contract along b. This is exactly the example we had before.


So here is a situation where our tensor is sharded according to the b dimension.


And this tensor is also sharded according to the b dimension.


And you want to do a matrix multiplication of the whole tensor.


So what can you do? You're supposed to multiply these two matrices, but they are sharded on different machines.


Now, if you consider what you actually have to do is you have to multiply each row here with each column here and that in an element wise fashion.


So that distributes according to you have to multiply this by this plus this by this plus this by this plus the red by the red.


So you can simply multiply the zero tensors together, the one tensors together, the two tensors together and the three tensors together.


Each one will give you a full matrix and then you can simply add all of them in order to get your full result.


This is illustrated down here. So what machine one does, it simply multiplies its shard by its own shard of the second matrix.


Which will give it this thing here. And by the nature of how matrix multiplication is constructed, you can simply do an all reduce,


which means you reduce you sum across all of the machines, and that will give you the full result.


So this is a this is a an example of how this works. This is, you know, pretty simple.


And I believe you may have seen something like this already when you were looking at just parallelizing matrix multiplication and so on.


So this system handles this transparently, right? If you're sharded like this, this is what the system will do.


However, if you are sharded differently, the system will act differently.


So here is a system you want to do the same matrix multiplication, but the first tensor happens to be sharded according to the A dimension.


The second tensor happens to be sharded according to the C dimension.


And you want to end up with something that's sharded to the C dimension.


Now we have an additional constraint here that here you can see, we kind of assume that this full thing here fits into memory,


mainly because we want to obtain the full result you see here, A and C should not be sharded.


So we assume that we can keep that in memory. But here we want the final result to be sharded according to C.


Which opposes the additional constraint that it might be that the full matrix never fits into memory.


So how are we going to calculate all of that? We can't do the same trick anymore.


Now this G shard system apparently realizes itself when something is out of memory,


and it can do a smart move around being out of memory using a loop,