this is what's used typically for debugging purposes so as you're running


Introduction to Python 3 Programming Tutorial

your your program it's just going to output to the little console a bit of


information whatever you ask it to so for now we'll use a string so a string


is just encased by quotes like this so it can either be like double quotes or


you could encase it in single quotes you could even get pretty fancy and


encase it in triple quotes so anyway more on triple quotes later but for now


let's just print hello universe everybody likes to say hello world I'll


prefer the universe so now to run generally you'll do ctrl plus B but


it's gonna want to know what do you want to run it with initially and we're gonna


run it with Python here so I'm just gonna choose Python and then you can see


down here the output and so it just says hello universe so that is our very first


Python program super super basic but from here what we're gonna do is get


into some of the more you know basic principles and tools that you'll need to


actually write programs that are gonna actually do things for you back whenever


I first started to learn to program I thought all programs were these


graphical user interfaces and I thought I was always waiting for the moment for


the like the thing to pop up and it was gonna be a window and stuff I didn't


never really realized that actually most programs just kind of operate in the


background you don't really see them doing anything they're just kind of


running but you don't see them running so anyways that was always kind of weird


to me when I first started to learn so if you guys have any questions or


comments concerns whatever feel free to leave them below also I have a quick


shout out to my most recent channel members Alfonso, Papasani, C Hayes and


Scott Davis thank you guys very much for your support if you don't know what


channel members channel membership is you can click that like blue join button


there and it's just a way to support the creator you get access to videos early


among some other things like chat emotes and like a little badge next to your


name and stuff basically it's it's more so a way just to support the creator so


if you want to do that that is how anyway that's all for now in the next


tutorial we're gonna be getting a little deeper into some more cool kind of


logical things with programming so stay tuned for that and I will see you guys


Gradient Origin Networks (Paper Explained w/ Live Coding)

Hi there, today we'll look at gradient origin networks by Sam Bond Taylor and Chris G. Wilcox


of Durham University. So on a high level, this paper trains implicit representation networks,


but not on single data points, but on entire data set. It does so by using a latent encoding


of each data point. And it doesn't obtain that encoding through an explicit encoder.


But by simply looking at the gradient of the latent variable with when initialized at the


origin. So it's a bit of a weird formulation. And I've seen this paper uploaded on Reddit.


And the top comments would also would always say like, I don't really get it, I don't really get it.


And I thought, you know, maybe I'm completely wrong. But I can just give my opinion kind of


what's going on in this paper. Now also, most people on Reddit or a lot did say, I don't really


get it. But here is what I think is going on. And then listing something and that's there is where I


stopped reading. So as to not be kind of as to form my own opinion, I like to kind of understand


papers by myself. So again, maybe I'm completely wrong. But here is my opinion. If you like


opinions, hit the like button and subscribe if you aren't yet. And yeah, share this video out,


maybe that helps someone else understand. So this paper is a very short paper, it is four pages.


And it's a dense paper, it definitely can warrant, it definitely can warrant making a longer paper


out of it. Though that being said, it's an archive paper for now. So you know, there's nothing wrong


with archiving kind of unfinished work. But we're just gonna look at it and try to understand it.


Okay. So the abstract says this paper proposes a new type of implicit generative model that is


able to quickly learn a latent representation without an explicit encoder. So for that you


need to know what an implicit generative model is. And I've covered one type of implicit generative


model specifically the type that they're using here, what they're called a siren. So sirens are


implicit representation networks. And I've made a video about sirens. So if you don't know what


that is, go look it up. But very quickly, a siren will is a neural network to represent a single


data point. So each data point in a data set is represented by its own neural network, and the


neural network. So this might be a bit foreign to you. But usually you have some kind of image,


right? And it's simply represented as an as an array of RGB coordinates, right? It's, it's simply


an array of this is like one 0.5, and so on. So all the pixels are in this array, this is the


explicit representation of that data point. Now, this here is a long list, and it has some regularities


to it. So that's why you can also think of an implicit representation of the data point, the


implicit representation works as follows. You imagine again, your image, your image is made up


of pixels, and these pixels are on x and y coordinates. So this pixel right here would be


00. This pixel right here would be 01. And so on. Yes, a siren is or a generally an implicit


representation network is a network that takes in any x and y coordinate as the input. So the input


itself is the numerical x and y coordinate of that picture. And it passes it through a neural


network and outcomes the RGB value. Okay, and so an entire picture is represented by this neural


network, the neural network maps each coordinate to its RGB value. And here you can see that the


single a single picture can become an entire data set for this neural network. In fact, it has to,


because for a different picture, of course, there is a different mapping from x and y coordinates to


RGB coordinates. But this allows you to do multiple things. So first of all, this neural network can


be smaller than the explicit representation. Second of all, you can capture some regularity


in the data, specifically, sirens have sine waves as nonlinearities in the in the neural network


here, which is also a bit special, but lends itself very well to capture natural signals,


because natural signals are often, you know, repeated at different scales and derivatives


of themselves and so on. So I've covered this all in my in my video. And also, this allows you to


have a continuous representation rather than a discrete representation, like here, you just have


each pixel, now, you have a continuous representation. Alright, so these are implicit


representation models, or implicit generative models are these neural networks right here,


that map from coordinates to two colors. Now, what's the problem with this is, as we said,


you need one neural network per data point. Now, the idea that these people here go with is that


can't we do kind of the same thing, but except we have one neural network per data point,


we want to have the same neural network for the entire data set. So again, they want to have a


neural network that somehow outputs RGB coordinates. But now it's not for a single image. Now we have


a data set, okay, and the data set has many images like this is image i, this is image j, this is


image k. So what we could do is we could simply tell the neural network, the x and y coordinate


that we where we would like the RGB values to know. And we could also tell it which image it is


right, k, or i or j. And this will give us a neural network right here that can represent the entire


data set, because it always can see, ah, I want of image j, I want these these x y coordinate doesn't


help you very much, though, because it still has to learn for each image individually, how to encode


it, how to produce it. What's much more interesting is, if you kind of mix this with the kind of old


style, the kind of old style generative model. So in old style generative models, let's consider,


for example, an auto encoder. So in an auto encoder, what you would do is you would take


your image and you would put it through an encoder. And this encoder will give you a latent


variable z. And then you would put it through a decoder again. And that would give you an image.


So your generative model now is this part right here. And this z variable is your latent encoding


of this data point. Now, if you train these models correctly, be this a be this a, an auto encoder,


or a variational auto encoder, or the green part can actually just be a GAN, right? If you train


this correctly, then this z right here will be sort of a a latent encoding of the what the what


of the information in the image itself, okay, and that can generalize. So now I can input a picture


that the model has never seen during training. And the encoder will map it to a latent representation


that sort of makes sense that is able to reconstruct the image that I've put in. Okay, so


the your hope with these latent representation is, is that there is some kind of data manifold


somewhere in hidden in the in the entire space of parameters. And as long as you're on that data


manifold, you will produce a sensible data point. And this is kind of a continuous and so on. So


even though you've only seen a few during training, if you have a new one during testing, then you can


sort of, it will be mapped to a correct place on the data manifold, and it will produce a data


point again. And you've seen this, right, you've seen these interpolations in GANs, where you can


interpolate in latent space, and, and so on. The problem here is that, you know, in so in GANs,


we sample these things right here. So that's a different story. But in VAEs, we need this encoder


or in auto encoders, we need this encoder to obtain a latent representation for a given data point.


In GANs, there is no way if we have an image, there is no way to obtain the corresponding


z variable, if we don't have an encoder, right. And that's the problem we're tackling right here. So


here, what we want to do is we want to give the x and y we want to give the z, we we say we have


some way of obtaining a latent representation of one of the image right here. And from that,


we want to generate the RGB variables. Now the question is, think of again, the question is,


how do we obtain the z variable without having without having access to the encoder?


And that's, that's the problem of this paper. And this paper proposes a solution. So they say,


this is achieved with an implicit neural network that takes as inputs points in the coordinate


space alongside a latent vector initialized with zero. So that's the model that we saw. That's this,


this is sorry about that. This is this right here, it takes in the coordinates, this is the


coordinates, and it takes in the latent vector z. Now, this whole point with it being initialized


at zeros will get will get to that in one second, okay. For the fact right now is just that the


represent the implicit neural network also takes the identity of the image. So each image, the


image is always going to have the same z. And then we sort of say which x and y coordinate of


that image we want. So the z is per image. And then each image has all the x and y coordinates


of, you know, itself. So yeah. So if Yeah, you you, I think you can follow.


They go on, they say the gradients of the data fitting loss with respect to this zero vector


are jointly optimized to act as latent points that capture the data manifold. So this is where


this is where I already got lost reading the first time through. The results show similar


characteristics to auto encoders, but with fewer parameters and the advantages of implicit


representation networks. Okay, so we'll actually we'll, we'll jump to this right here. So this is


the, this is the comparison between a variational auto encoder and the gradient origin network. So


in a variational auto encoder, what you would do is you would have this explicit encoder right here,


as we said, and in the variational auto encoder, you don't obtain the latent representation


directly, you actually obtain the distribution in terms of the mean and standard deviation of the


latent representation, and then you sample from that distribution to obtain that latent


representation, I think the point here is simply to show that you first of all, you do need an


encoder, which you do need to train. And second of all, it's kind of a complicated process to get


that latent representation for the data point x. And then you need to decoder that generates an


image, and then you have the loss right here that compares the two that is used to train the


encoder and the decoder. Whereas in the gradient origin networks, what you do is you start,


you basically have a function f and the function f, it's a bit weird right here, the function f


uses two things. So this here is that z, which is termed zero here. But in fact, it's the latent


representation of the image, which is derived from the image itself. And I don't really know. So I


guess you can hear you can input this x is derived from the image itself by some way that doesn't


require parameters that is not learned. And it also takes in these coordinates, and it produces


that image. Now let's disentangle two things right here. What we're going to see is that


z is equally applicable to non implicit neural networks. So for the rest of this paper, now,


I'm not saying it's going to work as well, maybe it's going to work specifically well with implicit


neural networks. But we need to differentiate the these two things. So the first thing is explicit


versus implicit. Okay, we're simply going to view these as functions that take a z and give you an


x. Okay, if this is, this is most notably the explicit version, the implicit version is simply


that we're going to take a z along with all the x and y of the image. And we're going to obtain


the r, g and b values of all the images, right, which is equal to the x. So this this entire set


of RGB values is equal to the x, and we input the entire set right here. But essentially, it's simply


a function that takes in a latent representation of an image and gives you back a image. The second


thing, which is an entirely different thing, in my opinion, is how do we obtain a z from an x? So


how do we get to have an image? How do we obtain the corresponding latent representation?


And such that such that so this must be such that this function right here, the function that gives


you the x from the z will reproduce the x. Okay. So how do we obtain the correct latent


representation for any for any input data points? Two different things. Don't so I think they're not


dependent on each other, except, as I said, they might work especially well together or something


like this. Alright, so this becomes a lot easier right now in this formula. So this is the thing


ultimately that they optimize, they optimize the this thing, and it's introduced, like, I don't


know why they limited themselves to four pages here. And again, this is work in progress, as I


understand it, but it is it is not it's like cold water. It's like, you know, an expressive neural


network can be trained in this space to mimic this by minimizing the gradient origin network


loss function. That's that's it. That's what you that's what you get. And then you get the loss


thrown in your face. But let's deconstruct it. So this G thing right here. What's it? This is the


this is the loss that you minimize, okay, you can see that this is simply an integral of this loss


function over your entire coordinate space. So see here is the entire coordinate space. So this is


for a given for a given image, right for a given image f x, you would minimize this actually across


your across your entire data set. So you would minimize the parameters of f, f here is going to


be your generator neural network, your siren, whatever you minimize over the parameters of f


across your entire data set. Okay, so this is your standard loss function. That is a sum across your


entire data set. Cool. So what are you going to minimize, you're going to minimize each data point


consists of an integral over the coordinate space, which you can see of this loss function


right here. Now, this is simply due to the fact that this is an implicit representation. If this


were an explicit representation, it would simply be the loss function of that data point. Okay,


so don't don't be scared by the integral. I'm usually scared by integrals, I never get them.


And then I try to talk to them and be people be like, do you think you know, a Riemannian integral


or a Lebesgue integral? And I'm like, okay, but in in this case, this is this simply means that


you want the loss of each of the coordinates and you want to sum them up, right? Which is the same


as simply the the normal loss function with respect to a data point. This right here is the data point


itself. As you can see, this is the this is your natural signal. So this is the function that you


don't know. This is the true image function that maps the coordinate to the RGB space. In the case


of explicit representation, this here is simply x. Okay. And forget about this integral for now.


Cool. So we have a loss between x and whatever this is right here. This is a bit too long. And


whatever this is right here, you can see the loss function between two things. So what is this thing?


The loss function, I can tell you the one they use in this particular paper is the L two loss. So


this is simply the reconstruction loss between a data point and its its reconstruction. Okay,


so this part on the right is what's going to make the reconstruction you can see yes, our F here is


going to be our siren, our neural network that will take in a z. So f is one of these function


explicit or implicit that takes in a z and gives you x, the the reconstruction. Now the question is


what does f take in f takes in two things. First of all, the coordinates concatenated with the thing


on the right. And you remember, we said that instead of giving x y to the implicit representation,


we now give x y and z where z is the latent vector of the image we're trying to reconstruct. So if we


were to see this as a non implicit method, we can simply leave away this right. So we as we leave


away the x and y coordinates in a in a GAN or a VAE, we simply give it this thing right here.


Again, we're trying to disentangle the implicit network, the implicit generator from how we are


going to obtain the z. So this is not important. So what remains is this quantity right here. So


this must be our z for the image, okay, this thing. So what's this thing? I'm running slowly


out of colors. This thing is going to be somehow the negative gradient of something again, you have


the integral right here of the loss function. This again is x. This here again, we can leave this


away. We can leave away the integral and you'll start to see kind of a repetitive thing. So this


is going to be the gradient somehow of your loss function. With that, again, there is x and then


there is f of z zero. So this is somehow an x hat as well. But this is a special x hat, let's call


it x hat prime, or x hat zero, because the input is not z, but the input is now z zero. Okay,


this is kind of a complicated thing. So I'm going to explain what's going on right here.


Maybe in drawing. So what you want to do is you want to start out with z zero, which is an initial


guess of what your latent representation is, you do it without looking even at the image


at the data point, you simply start with one. And there are multiple ways to do this. And this


paper right here simply says we're going to see zero is just going to be a constant value,


zero, the constant value zero. That's why it's called gradient origin networks. Because


you always start with your z zero, your initial guess of your latent representation is the origin.


Okay, then you use f, your neural network to obtain a estimate, a first estimate of what your


image could look like, again, you have not looked at the image, you're simply taking the z zero and


you produce an image. Then you somehow somehow obtain a better representation z. And that you


use your f again to obtain x hat. And then from that x hat, you can now compare this to your x


and that will give you your loss that you back propagate. So two things here, you can see you use


f twice, which means that your loss if you back propagated, you must somehow back propagate to


both of these things. Okay, so this is the first, the first thing if you back propagate. The second


thing is what's this thing right here? How are we going to obtain somehow a better z. And the better


the better z is going to be obtained by basically looking at the gradient. So you've seen that we


have a gradient of z zero of the loss of x and f of z zero. That's that thing here is going to be


your z, z equals that. What does it mean? It basically means that so you've tried to produce


an image, but this is the real image that you want to get. And the loss measures how far apart you are


from that real image. How would you need to change your initial guess in order to make that loss go


down. So the negative here is to make the loss go down, because otherwise it will make the loss go


up. Okay, so it basically simply says, how do you need to change your z zero in order to decrease


the loss in order to get a better z for representing this particular image right here. And in the paper,


here is where I kind of disagree. Because in the paper, they say that they that this in a single


step, they give this gives you a this gives you the correct z or something like this. And I don't,


I don't agree. They say, with respect to the origin, we obtain a latent vector that minimizes


the reconstruction loss is obtained in a single step, thereby playing the similar role to an


explicit encoder. So this is true, this is kind of like an encoder, right? You simply ask what z


would I need to put in in order to make this representation be a better sorry, in order to


make the latent representation be a better latent representation for the particular image x. However,


if you compare, so what is this, this is essentially gradient descent in the latent space,


right? And the fact that we look at the explicit gradient is only because they started at the zero


point right here. The fact that they started at the zero point means that here they can just leave


away the following what if you were to do gradient descent, what you would do is you would say this


my z is going to be equal to z zero minus this thing, right now it looks much more like gradient


descent in the latent space, because you have some initial guests, and then you update it using the


gradient. Now there is no learning rate right here. So the learning rate is one in this case.


So this is and again, the z zero, because it's zero, you can just leave it away.


So this is simply one single step of gradient descent in the latent space.


In order to get a better z right here. However, this is not a this is doesn't it doesn't guarantee


you that in this single step, you're actually going to find the correct z or even an appropriate z


simply means that you're going to find a better z than z zero for that particular image. And


this can work, right. And again, because you back propagate to both of the F's, you say,


you basically say, I want my neural network, first of all, to reconstruct the data point better from


a given latent representation. And I also want my neural network to give me a latent representation,


basically to help my latent to help this procedure, you back propagate through the gradient descent


procedure. So you say I want my neural network to help me obtain a better latent representation if


I do one step of gradient descent. So therefore, it's not just pure gradient descent in that space,


it actually the back propagation makes it such that your neural network also supports that


supports obtaining a good representation in one step. Okay, now that we've disentangled this,


basically, you can see two things. First of all, you could probably get an even better


representation by doing multiple steps of gradient descent right here, maybe adjusting


the learning rate a bit, it depends, right, because you have to back propagate through all


the gradient descent steps. But pretty sure you could probably improve this by


doing multiple steps. Second of all, it doesn't really matter that this is a constant zero,


it gives you know, there's a cool name gradient origin networks, but you could probably start with


any constant or even here's the thing even non constant initial points, you could sample them


from a distribution and so on. And okay, so let's change like let's imagine changing z zero


to be sampled from some normal distribution. And then it looks much more like a GAN, right?


Alright, so here we go. I've cloned the repo and I've ran the code once just to make sure that the


data is downloaded and everything. And the code is, you know, pretty, pretty easy. So there is one


file. And I didn't do it in the colab because the colab was, I think a bit slow for me. But


I don't know if I caught a wrong runtime. But essentially, there is a bunch of setup code,


they know these siren layers and so on. And then you have the real deal thing right here. So you


have the step. So we do 500 steps. And in each step, we as you can see right here, we start with


zeros as z, then we put this into F concatenated with the coordinates. So the coordinates is like


a kind of a mesh grid type thing. We obtain the inner loss right here, we do a gradient with


respect so of the inner loss with respect to z, and then the negative gradient that's going to


become our outer z. So this z up here is z zero. And this z down here is going to be our true z


is going to be our true z from the paper, we are going to concatenate that again, with the coordinates


to obtain the G, which is the kind of reconstruction of x. And then our outer loss is going to be


simply this reconstruction loss right here. And then we're going to backward to all of the


parameters. Okay, so first hypothesis is that this here is simply kind of gradient descent. So what


we should be able to do is first let's run let's run this. So I've run this like that.


So this is shipping it to a GPU server. And as you will be able to see, the loss will be output,


and it's going to kind of decrease the loss over the course of 500 steps. And we can also look at


the samples. So while that's happening, what we can do is we can actually already prepare what we


want to do. So if this is really gradient descent, we should be basically just able to do this z


minus this gradient right here, because it's zeros, we would simply expect this to yield the


same loss. So we're going to do this, and then we're going to ship this off to the server again.


Sorry. So we were here. And okay, the logs failed. Alright, so this is called images. I have this


thing set up such that it's called logs. But you can basically see that the loss right here was


from 24 going to down to about 13 or so over the course of training. So by subtracting z minus the


gradient, we there really shouldn't be any change, right? Because z is zero at the beginning.


So again, we're going to run this. And while it's running, we're going to prepare


the different things. So my hypothesis is that we can maybe we could make this z here pretty much


anything. So let's do it. Let's put it into ones. Again, you see that the loss, I guess, you know,


we get an idea of kind of the noisiness of this thing. And 2119 and so on. We can in fact, over


here, we might be able to if we ship it to a different GPU might be able to run two things in


parallel. So this now is when we just start with ones instead of zeros. So let's see how that


happens. While that's the case. So you can see right here that we ended up at also about 1413.


This pretty much is the same if you you can we can look at the images that it's produced. So the


reconstructions look kind of like this a fashion amnesty, the samples kind of look like this.


And the interval interpolations, you can look at those as well. But we're mainly interested also in


the in the kind of loss right here, you can see that with the ones pretty much the same thing is


happening. So let's say we actually change this to a normal distribution. Okay. What does that do?


And while that's happening, we're going to revert this to the original zeros. And we're going to


investigate what happens if we just do more than one step of gradient descent. So in order to do


that, it's actually pretty easy. So this here is the gradient descent step, what we can do is we can


simply double that. Right? So now, if this is correct, I'm pretty sure this is correct. Okay,


the so the normal initialized isn't really the the hit right here, as you can see the lot Wow. Okay.


The normal isn't maybe it's because it's, you know, too large. I'm not sure. I mean, the other


thing is deterministic. So that's going to be like a lot easier. We can quickly go back and let's go


ones, let's go to normal. And let's like multiply it with like a tiny like point zero one or so.


I just want to see whether this works. I have no big hopes. Okay, so we are here again, and we're


going to make this into two different things. To do that, we're going to go back to the original


gradient descent. Two steps of gradient descent. All right, so now we have two steps of gradient


descent. And let's see whether that helps. Ah, okay, so the normal distribution already helps,


or is not worse. We simply initialize it with too big of a variance. The point zero one seems to be


some kind of magic number for normal distributions and neural networks. So on the right side over here,


and you can see we're a bit we're a bit, it's a bit off, but I guess with a bit of tuning, you


could do that. And it gets down to about the same loss as you saw. If we look at the images that this


produced, I'm going to guess it's, you know, they seem a bit worse, but it kind of works. On the


right side, however, if you do more than one step of gradient descent, wow, wow, we were you see,


we already started lower losses. And since this is gradient descent, we can also you know, there's no


need why the learning rate should be one. So let's try to are divided by a generous three, and then


by maybe six, like a decreasing learning rate seems like a rather good idea. And yeah, let's


just take the two steps with the decreasing learning rate. Oops. So you can see that the


loss now is way down, just because we did two steps of gradient descent. And the reconstructions,


I'm going to guess they're almost perfect. So we're now I guess we're overfitting a bit. So this is


now trading off kind of power of the encoder decoder and so on. But ultimately, yeah, so let's


just for the last part, just try to have this gradient descent with the decreasing step size


and see where that gets us if that gets us to even a lower reconstruction loss. And that will be our


investigation into the code right here. Okay, do do do do. Okay, we start with 19. Maybe we're


we're as good as before. That's fine, you know. But I hope I hope that kind of gives a bit of


evidence to my point that this is basically reversing a generator by using gradient descent,


which has been around for a while. And I happen to know someone who who once attempted to write


a paper about it. So yeah, but it's it's with implicit networks, which are pretty cool. So,


you know, maybe this might work especially well with them, given that the gradient of a siren is


a gradient and is a siren, and so on. Yep, as you can see, this works as well decreasing learning


rate. And now you can go nuts. Oh, nine. Wow. This is the lowest loss we've gotten so far. Right?


Yeah. So pretty cool. reconstructions look like things. Well, these are the best samples. I think


these are the best samples we've seen today. Maybe not. I'm not sure. Let's look at the


interpolations quickly. Yeah, this looks like interpolations. I mean, if you squint. Okay,


this was it for coding. See ya. Now, GANs have come with encoders before, or it much more looks


like a variational auto encoder as well. The difference here is we replace the encoder. So


this here is our encoder, right? This is our implicit encoder is simply gradient descent. This


has also been done before for GANs. So people train GANs, and then they try to find the latent


representation by back propagating. And some people even do this while some people do this


while training, they do gradient descent, and then either do or do not back prop through the GAN


the through the gradient descent procedure. So in a way or another, this is kind of sort of like


those ideas, not saying it is equal. And again, there could be like some special interaction


because you actually back prop through both these things. And there could be some special


interaction because these are implicit neural networks. However, I very much view these as two


different things. The cool, there is a rather cool derivation of that, where you can say, okay,


you can also use it as a classifier by basically doing this. And now hope you can understand this


much better. So what we'll have is we'll have the classification loss for sample x is going to be


your cross entropy loss between two things. Okay, well, can you please go down again? Thanks. So


your cross, your loss between two things is going to be the loss between your label y. So that's one


thing. And usually, you have the feature the logits on this side, right? Now you can see right


here, you have an F, that's probably that something that gives you the logits from your


features. And here, your features aren't going to be the data point itself, but your features are


going to be the Z variable that comes with the data points. So basically, you use this as a


feature producer. And the feature producer is made by again, minimizing this reconstruction loss. Now,


I'm not sure this is going to work really well for classifiers, because classifiers generally don't


require you to reconstruct things. And we know this, you know, people try to, this is like you


were to have a variational autoencoder, and then simply use that encoder as a feature producer for


a classifier, which generally doesn't work very well. But you know, you can you can do it right


here. And the the cool thing is that you can actually use the implicit representation network


F to give you features for the entire data sample Z. So you kind of freed from the coordinate


representation here, and you get kind of a latent, a latent vector back. So this is how you would use


an implicit neural network in order to do classification. That's, I think, you know,


pretty, pretty cool derivation of this. So here, they make some empirical claims, which I don't,


I don't want to go too much into. But there are certain advantages, certain practical advantages


of doing things like this, like you can have very, very few parameters to represent an entire


set of data, the interpolations here work nicely. As you can see, and I think generally, they make


the claim that this trains fast. And you can see after three seconds, it already has a lot of


information about the data set. And it does some sensible things. Okay, so the code is available.


And in fact, I'll probably enter into parse into this video. A let's actually test our hypotheses,


right? Let's test these hypotheses that I said. So first hypothesis is probably we can start with


something else than the constant zero. And second hypothesis is we can probably improve by doing


multiple steps of gradient descent in the inner loop. Yes, I this might be somewhere in this video.


Investigating Human Priors for Playing Video Games (Paper & Demo)

Hey there, what's going on? Today we're looking at investigating human priors for playing video games by Rache Duby, Pulkit Agrawal, Deepak Patak, Tom Griffiths and Alexei Efros.


So, there is a paper to go with this, but I actually don't want to get into the paper too much in order to not reveal too much of what's coming, but basically they're trying to investigate what makes video games work for humans.


So what do humans pay attention to? What priors bring humans into a video game? And the fun thing is they've created these games where they ablate these individual priors and we are going to play them.


So the original game right here, as you can see, is kind of this Montezuma's Revenge type of game, so you only need the arrow keys.


If you go to a bad blob like this, then you die and you can jump on them and you can use the ladders and the spikes, they'll hurt you if you jump on them and also if you fall down between the platforms.


So what you've got to do is basically get the key, then go to the door over here and badaboom. Cool.


So let's try out. So they basically ablate different things here. Masked semantics means that you don't know what the objects are anymore.


So you might go over here and you might be like, what's this green thing, can I jump on it? Oh.


So we're probably a bit biased because we've seen the game before, so we know that the pink ones are the bad ones and that's the key, so we should probably get it.


But you can imagine that it is a bit harder, but you could still solve it. Reverse semantics is very interesting if you play it for the first time, because all of a sudden now there's the coins. Oh, and the fire.


But I think humans could probably still figure it out with like some minimal trial and error. This ice cream cone, you realize, OK, now it gets interesting.


Because right now we've always had sort of we know that there's an object and there's no object on the platforms, but now these are masked.


So basically you don't know what's like a relevant object and what isn't. So I know that there's like a bad thing here and a bad thing down to the left.


So I'm going to guess these light, these light pink things are the bad things. Yeah. Yeah. These are the ladders. Cool. Bad thing right here.


We are rocking this. OK. Key. Where's the key? And the door. So it gets harder because you kind of have to remember the colors, right?


I know that the light pink ones are the bad squares. Still, still solvable. So let's jump over these on the left because these get really, actually it's going to here.


So masked affordances, what they're saying is that, OK, you can kind of from the way something looks, you can tell what you can do with it. For example, the platforms, you can jump on them and the background is sort of empty space.


So you know that there's nothing much happening there. So they trying to take that away by simply retexturing all the objects here such that you don't know how you can interact with them.


And it does get significantly harder because, OK, so these green ones are these green ones are the platforms right here. So but I can still see that. That must be the ladder. Right.


You can imagine if you were playing this again for the first time that this is significantly more difficult, but you still see the key and the green ones being the platforms.


We got this. Now it gets harder. Masked visual similarity. So this is where they say maybe as we did so far, maybe you as a human can kind of make out that things that are visually similar to each other can you can do the same things with.


Like we said, the green ones are probably the platforms. So they took it away.


Ah, gee. OK, so that must be OK. Can't go here. Fell down. Let's try again. This is a platform. Is this one here? Yes. These are platforms.


That was easy. Too easy. Too easy running into that blob there. OK. The ladders are still like this.


And OK, this gets harder as as you can. Gee. OK. I'm too dumb to remember from before.


I'm like the ideal subject because I don't remember. How did this work?


I'm going to solve this just so you know, even if this video gets to 50 minutes, I'm going to make it through this.


OK. Ah, here we can. OK. See, my short term memory is so bad. OK, we got the key.


Now just get over to the door. Door's over there. Yeah. OK. Now let's wipe the short term memory again.


Here, change ladder interaction. Where they basically say, OK, one of the things that you know from the real world is how these objects work in the real world.


So there's not really any pink blobs with evil faces. There might be spikes. Yes. But ladder is something you know that works.


So if you want to go up here, that doesn't work. So you kind of have to figure. OK.


So you have to go kind of left and right to go up the ladders. And so that one I actually tried before and I figured that out pretty quickly.


I think humans are able to figure that out fairly quickly because you kind of on the ladder. Right. You can actually go down easily.


You're on the ladder and then it kind of doesn't work. And then you kind of try to wiggle because there's two of them.


I don't think that's necessarily super hard.


And now it feels a bit like, you know, this Super Mario Maker thing where people just try to make levels as hard as possible and trick you with trick blocks and invisible stuff.


This is hard. So the direction of gravity. So now the left key jumps. Right. And here this key. So this is like this is extremely, extremely hard.


Because I have to like think about every move I make before I do it. And.


OK. No, no, no. This is so unintuitive for real. Yeah. Got it. Got it. OK. Really try this out. This is crazy. Yeah. Yeah.


OK. So the last thing is we combine all of it. I guess except the changed gravity and changed interaction.


So now all the priors, all the visual object priors removed. This is this is King's discipline right here.


OK. So we figured out where the blue. Cool. So where's the next? This is the next platform.


Where's the. OK. There must be like a. Yeah. Take that. OK.


But we know the next. So we can't really generalize from this because we know the next bad blob isn't going to be the same color.


Right. OK. The white. And this. I know there's a bad thing here, but we'd have to figure this out.


So basically kind of the point of the paper, I think, is to say that this is what you're doing to our. No spikes.


This is what you're doing to our algorithms if you're in the most case.


So they simply have to go and to basically try every single thing and remember what worked and what didn't work.


Now, of course, the algorithms can also exhibit like can also use the visual similarity.


That was the key. Yeah. Yeah. Let's go to the door door door. No, there is a there's like a bad thing here, right?


No spikes. OK.


So either we build these priors into the RL algorithms if we want to get them to human level or we have some sort of learning these priors before we let the people go onto a paper or I don't I don't know.


Or we just take it that RL algorithms have to figure all of this out by themselves.


So they ablate these things right here. You can see the masked object identity makes kind of the biggest difference in terms of time, number of deaths, the number of states explored.


Reverse semantics. I believe these are humans that are trying it for the first time and they're just like, oh, an ice cream.


So it can also hurt. Right. The algorithm wouldn't be super impressed by it looking like an ice cream.


But the human is very much and the crazy thing here, you can see exploration, the original game and then exploration in the no object prior game, especially if you play this for the first time.


This is just mad, like no friggin way. I would actually like love to see video games like this coming out.


This would be the worst selling video game of all times where dynamically it just removes these kind of priors.


But it's a I think it's a really fun way to investigate what humans learn and what they already bring into the game.


So here they have another game and they do this same thing on an RL agent.


And you see here the RL agent just don't care about any of these things except visual similarity.


So visual similarity helps the RL agent to generalize across the game.


So if you see a bad blob, the next bad blob will look similar and that's sort of kind of an invariance that we know they can exploit since they're using convolutional neural networks and so on.


But I think it is really drawing attention to the importance of priors, prior knowledge in reinforcement learning and human knowledge.


So in this game right here where you have these hidden rewards that the human doesn't see.


Right. But if they kind of touch it, they're kind of coins.


And the human performs way worse than the RL agent because the RL agent will actually try those things out.


And the human having the prior that the black thing, they don't see the yellow boxes, that the black thing is just empty space.


They won't even explore that.


So maybe, you know, that is something to think about with respect to building RL agents.


All right. I don't want to go into the paper too much.


It's a very cool paper, but we're here to play games.


And I invite you to read the paper, check out the website, try these games for yourself.


AlphaCode - with the authors!

Hey, this is an interview with the authors of the AlphaCode paper by DeepMind.


AlphaCode - with the authors!

This is a crazy system.


AlphaCode - with the authors!

It does automated competitive programming and is about as good as an average human in


AlphaCode - with the authors!

real competitions, which is crazy.


AlphaCode - with the authors!

In case you haven't seen it, I've made a comprehensive paper review of this paper in the last video.


AlphaCode - with the authors!

So be sure to check that out because the authors that I'm interviewing today have also seen


AlphaCode - with the authors!

that video and were able to dive right into the matter, answering any questions, any criticisms


AlphaCode - with the authors!

and so on.


AlphaCode - with the authors!

You're also able to get a behind the scenes look into what things went wrong during this


AlphaCode - with the authors!

research, things that didn't work out, things that were red herrings and much more.


AlphaCode - with the authors!

We also talk about how the project came to be and how the authors dealt with the immense


AlphaCode - with the authors!

media reaction that followed the release.


AlphaCode - with the authors!

Let me know how you like these types of videos.


AlphaCode - with the authors!

Having the authors on is a huge privilege and I'm absolutely sure you'll learn something


AlphaCode - with the authors!

useful from this conversation.


AlphaCode - with the authors!

If you like content like this, don't forget to leave a like, subscribe, tell me what you


AlphaCode - with the authors!

think in the comments and I'll see you around.


AlphaCode - with the authors!

Bye bye.


AlphaCode - with the authors!

Yeah, hi everyone.


AlphaCode - with the authors!

Welcome back.


AlphaCode - with the authors!

I'm here today with Rémy LeBlanc and Peter Choi, who are authors of the competition level


AlphaCode - with the authors!

code generation with AlphaCode paper.


AlphaCode - with the authors!

I'm just going to call it the AlphaCode paper.


AlphaCode - with the authors!

Everyone's excited about this paper.


AlphaCode - with the authors!

So much hype around it and it's very cool to have the authors with me.


AlphaCode - with the authors!

So Rémy and Peter, thank you very much for being here.


AlphaCode - with the authors!

Thanks for having us.


AlphaCode - with the authors!

Thanks a lot for having us.


AlphaCode - with the authors!

Yeah, we're quite happy to be doing this with you today.


AlphaCode - with the authors!

So the paper, obviously, you know, given that the machine learning community and the programmer


AlphaCode - with the authors!

community intersect in large parts, and then the competitive programming scene also is


AlphaCode - with the authors!

kind of known for not being the most humble.


AlphaCode - with the authors!

Obviously, let's say obviously, there was quite a bit of hype, quite a bit of media


AlphaCode - with the authors!

reception around the paper.


AlphaCode - with the authors!

Did you expect anything like this?


AlphaCode - with the authors!

And how did you experience sort of how the paper was received in public?


AlphaCode - with the authors!

I guess I can take that one for starters, Peter.


AlphaCode - with the authors!

So I think overall, we've been fairly happy with how the paper has been received.


AlphaCode - with the authors!

People have been talking a lot about the ideas that we put forward and the results that what


AlphaCode - with the authors!

we think is fairly impressive for what we're trying to do is nowhere near what might have


AlphaCode - with the authors!

been reported in some news outlets.


AlphaCode - with the authors!

So we did expect that there was going to be positive reactions, negative reactions, and


AlphaCode - with the authors!

a bit of misunderstandings, probably.


AlphaCode - with the authors!

But I think overall, we've been fairly happy.


AlphaCode - with the authors!

Yeah, I think we spent like a few hours, maybe even like a day or two after we released the


AlphaCode - with the authors!

paper just kind of watching with popcorn what was going on.


AlphaCode - with the authors!

And yeah, that was pretty enjoyable.


AlphaCode - with the authors!

But yeah, overall, I'd say I'm pretty pleased.


AlphaCode - with the authors!

Do you want to maybe just as an opportunity to do you hear like, cross over statements


AlphaCode - with the authors!

you said, you know, some people said a bit, you know, a bit more than what you actually


AlphaCode - with the authors!

did.


AlphaCode - with the authors!

So do you is there something that you saw that was like really where you say, no, this


AlphaCode - with the authors!

is actually this is wrong.


AlphaCode - with the authors!

This is too much, you know, rather than just selling it very prettily.


AlphaCode - with the authors!

Anything you sort of want to bring down to earth?


AlphaCode - with the authors!

I think the biggest thing that I noticed and like quite a common mistake was to like overstate


AlphaCode - with the authors!

our result as DeepMind, you know, has an algorithm which is as good as an average programmer,


AlphaCode - with the authors!

but like really the right answer is average competitive, you know, we get the same results


AlphaCode - with the authors!

as an average competitive programmer.


AlphaCode - with the authors!

And those are like huge, huge, there's a huge difference there.


AlphaCode - with the authors!

But you know, that distinction can be like a bit nebulous if you're not familiar with


AlphaCode - with the authors!

programming or competitive programming.


AlphaCode - with the authors!

And so that's the one the main thing I think would become the top of my list.


AlphaCode - with the authors!

Yeah, of course, like most of the most of your job as a as a software programmer isn't


AlphaCode - with the authors!

actually writing code, right?


AlphaCode - with the authors!

It's reading code, understanding code, thinking about how to achieve whatever it is you want


AlphaCode - with the authors!

to achieve, right?


AlphaCode - with the authors!

So we focus on a much, much narrower scope in this paper, where we have a very precise


AlphaCode - with the authors!

description of what we want to do.


AlphaCode - with the authors!

We have examples, we have constraints, etc.


AlphaCode - with the authors!

Which which to us is a very interesting proxy for problem solving.


AlphaCode - with the authors!

But it's very far from the full job of an actual developer.


AlphaCode - with the authors!

Yeah, I was I mean, I was.


AlphaCode - with the authors!

I think even with the with the correcting the record, it is still very impressive.


AlphaCode - with the authors!

And I think before we before the recording, we talked about that also, you seem to have


AlphaCode - with the authors!

been a bit surprised at how far you were able to get with this system.


AlphaCode - with the authors!

Could you tell us a little bit about the just the process of you know, how did you start


AlphaCode - with the authors!

out?


AlphaCode - with the authors!

What did you do?


AlphaCode - with the authors!

I mean, I've used for example, codecs, or copilot from GitHub.


AlphaCode - with the authors!

And I have to say it's like is really good.


AlphaCode - with the authors!

I think it's a game changer if the UI is cleaned up a little bit and models like this will


AlphaCode - with the authors!

be, you know, I think assisting programmers a lot.


AlphaCode - with the authors!

But how did you go from like that?


AlphaCode - with the authors!

Were you even aware of codecs copilot?


AlphaCode - with the authors!

And how did you get to to alpha code?


AlphaCode - with the authors!

And what did you expect?


AlphaCode - with the authors!

Right, so I think and I mean, I wasn't there from the very beginning of the problem.


AlphaCode - with the authors!

But I think we've always been focusing on a slightly different approach than what codecs


AlphaCode - with the authors!

and copilot are doing.


AlphaCode - with the authors!

I think we're really interested in this aspect of problem solving.


AlphaCode - with the authors!

And we're really interested in this aspect of generalization.


AlphaCode - with the authors!

Like we wanted to solve unseen problems and come up with novel solutions to things that


AlphaCode - with the authors!

the model hadn't seen during training.


AlphaCode - with the authors!

And so competitive programming was sort of a natural target for that.


AlphaCode - with the authors!

And then we started getting a bit of traction and we set ourselves what we thought to be


AlphaCode - with the authors!

almost an impossible goal.


AlphaCode - with the authors!

But we thought we needed to be ambitious to really push ourselves and push the methods.


AlphaCode - with the authors!

And so our level of confidence in whether or not we're going to achieve this fluctuated


AlphaCode - with the authors!

during the course of the project.


AlphaCode - with the authors!

Some points we had high points and we had low points.


AlphaCode - with the authors!

Some points were convinced we were going to succeed.


AlphaCode - with the authors!

Some points we had pretty severe doubts.


AlphaCode - with the authors!

But yeah, in the end, we managed to get all the way across the finish line.


AlphaCode - with the authors!

I think one thing I'd add to that is I think this is the first project where I, the first


AlphaCode - with the authors!

project I've worked on which had like quite a strict adherence to, you know, looking at


AlphaCode - with the authors!

a particular metric quite regularly.


AlphaCode - with the authors!

And I think that really helped us incorporate ideas that were happening, that were being


AlphaCode - with the authors!

researched within DeepMind and outside of DeepMind.


AlphaCode - with the authors!

So I think that was really worthwhile and something that we've learned to value quite


AlphaCode - with the authors!

a lot in working on these ambitious problems.


AlphaCode - with the authors!

It's cool if you have some sort of a North Star, right, that of where you want to get,


AlphaCode - with the authors!

at least you know where you want to get.


AlphaCode - with the authors!

I think with most projects, it's even ill-defined kind of where the end goal is.


AlphaCode - with the authors!

And I think that's probably half the game in academia and also projects as such.


AlphaCode - with the authors!

So I've made this little overview and intro to your paper.


AlphaCode - with the authors!

Did you feel that was accurate?


AlphaCode - with the authors!

Is there anything missing?


AlphaCode - with the authors!

Anything you want to amend on how the system works?


AlphaCode - with the authors!

Any wrong emphasis that I've set?


AlphaCode - with the authors!

I don't think there's anything wrong with what you described.


AlphaCode - with the authors!

And I was fairly impressed that you managed to sort of distill this massive paper down


AlphaCode - with the authors!

to a reasonable size in terms of the video.


AlphaCode - with the authors!

So yeah, no, I think I was quite happy with the way you described it.


AlphaCode - with the authors!

There's of course opportunities to get into more details by reading the paper itself,


AlphaCode - with the authors!

especially on the method section.


AlphaCode - with the authors!

But overall, it was really good.


AlphaCode - with the authors!

I was very impressed, as always.


AlphaCode - with the authors!

I generally love your videos, Yannick.


AlphaCode - with the authors!

So it's a really easy way to get an overview of the paper and decide if you want to read


AlphaCode - with the authors!

it yourself at all.


AlphaCode - with the authors!

And yeah, this was kind of not an exception.


AlphaCode - with the authors!

Thanks.


AlphaCode - with the authors!

I wasn't chasing for compliments.


AlphaCode - with the authors!

I was actually wondering if you had something to...


AlphaCode - with the authors!

Okay, so I think one point of the contention, I think we're all on board with, you know,


AlphaCode - with the authors!

we do some sort of a pre-training here on GitHub.


AlphaCode - with the authors!

We do some sort of a fine tuning on the problem we're interested in, right, which is these


AlphaCode - with the authors!

coding problems.


AlphaCode - with the authors!

But then I think the point of contention that a lot of people have is this sort of this


AlphaCode - with the authors!

approach of large scale sampling, followed by filtering, which is really different than


AlphaCode - with the authors!

how a human solves problem.


AlphaCode - with the authors!

This is a...


AlphaCode - with the authors!

I'm as a programmer, I don't blast out 100,000 different possible solutions and then, you


AlphaCode - with the authors!

know, run them all, not even in my mind, right?


AlphaCode - with the authors!

Not even... that's not even the way I think to sort of sample forward and then test all


AlphaCode - with the authors!

of these things.


AlphaCode - with the authors!

I'm actually impressed that this, you know, that the filtering step would give you the


AlphaCode - with the authors!

sort of the correct things right here.


AlphaCode - with the authors!

So my question would be, I'm willing, let's say, to disregard the fact that that's not


AlphaCode - with the authors!

mechanically how I do it.


AlphaCode - with the authors!

I'm willing to still consider the possibility that the model will actually, you know, given


AlphaCode - with the authors!

the attention maps and so on, actually does, you know, do something worthwhile more than


AlphaCode - with the authors!

just kind of random sampling, right?


AlphaCode - with the authors!

Because if it were just a random sample, I would never get a solution.


AlphaCode - with the authors!

So I'm willing to see that the model might be doing something.


AlphaCode - with the authors!

And then I thought, well, if that's the case, shouldn't I somehow find a representation


AlphaCode - with the authors!

of the abstract concepts inside of the latent spaces somehow?


AlphaCode - with the authors!

You know, whenever the algorithm is about sorting lists, shouldn't I find like list


AlphaCode - with the authors!

primitives and sorting algorithm comparison operators and something like the concepts


AlphaCode - with the authors!

that I would think of when implementing this algorithm or like a Dijkstra's nearest neighbor


AlphaCode - with the authors!

algorithm if I implement that, shouldn't I find these things?


AlphaCode - with the authors!

Have you thought of like investigating the model and see whether or not it kind of learns


AlphaCode - with the authors!

programming concepts by itself?


AlphaCode - with the authors!

Is that even, you know, possible?


AlphaCode - with the authors!

I mean, that's a very interesting question.


AlphaCode - with the authors!

We've done a lot of analysis on the model, but as we report in section six of the paper,


AlphaCode - with the authors!

it's either centered on the impacts of the end metric, like the solve rates, or we analyze


AlphaCode - with the authors!

the sample themselves.


AlphaCode - with the authors!

And Peter's done a great job, by the way, showing that our models don't really copy


AlphaCode - with the authors!

paste.


AlphaCode - with the authors!

But we haven't yet prodded the model enough internally to be able to answer that question


AlphaCode - with the authors!

definitively.


AlphaCode - with the authors!

If I had to venture a guess, though, I'd say it's very likely that these concepts are present


AlphaCode - with the authors!

at the latent space level.


AlphaCode - with the authors!

And as you just said, the best proof of that is that the model does actually come up with


AlphaCode - with the authors!

these relevant concepts and implements them to solve some of the problem.


AlphaCode - with the authors!

Right.


AlphaCode - with the authors!

So we have tree traversals, we have dynamic programs, we have sorting, all of these sort


AlphaCode - with the authors!

of things.


AlphaCode - with the authors!

So they're definitely there.


AlphaCode - with the authors!

It seems to me very likely that they're here and doing massive sampling alone cannot explain


AlphaCode - with the authors!

the solve rate that we have.


AlphaCode - with the authors!

I think another issue, though, is that probably the right concepts are there, but they're


AlphaCode - with the authors!

in there amidst many, many other concepts and picking exactly the right concept at the


AlphaCode - with the authors!

right time is actually really difficult.


AlphaCode - with the authors!

Yeah, I think I'd probably add something to that, which is, I guess, maybe the last point


AlphaCode - with the authors!

that Remy made is not even specific to the transform work that we have.


AlphaCode - with the authors!

When I read a competitive program problem, I've got five ideas in my head of what might


AlphaCode - with the authors!

work.


AlphaCode - with the authors!

So I think that wouldn't be that bad, even if there was a bunch of different things in


AlphaCode - with the authors!

there.


AlphaCode - with the authors!

One other thing I think I'd add is that, I guess, because we sample from the model autoregressively,


AlphaCode - with the authors!

the latents are actually changing as you do that.


AlphaCode - with the authors!

And so later on, the model may not have honed in on the concept of like, oh, I need to do


AlphaCode - with the authors!

a DFS here, or I need to do Dijkstra's algorithm until maybe like 50, 80% of the way through


AlphaCode - with the authors!

the problem.


AlphaCode - with the authors!

So I think if we were to do that investigation, we'd have to consider how that changes through


AlphaCode - with the authors!

the sampling procedure.


AlphaCode - with the authors!

It's not even clear where to look, basically.


AlphaCode - with the authors!

Is it at the end of the encoder?


AlphaCode - with the authors!

Is it during sampling?


AlphaCode - with the authors!

We don't know.


AlphaCode - with the authors!

Yeah, it is also, I mean, it connects to this larger problem of people arguing whether or


AlphaCode - with the authors!

not these models can quote unquote reason, right?


AlphaCode - with the authors!

And you explicitly in the paper also make an effort to connect this to abstract reasoning


AlphaCode - with the authors!

and so on.


AlphaCode - with the authors!

I think, you know, investigating things like this here could be sort of a proxy for really


AlphaCode - with the authors!

demonstrating, yes, there is actually something in these models that amounts to sort of symbolic


AlphaCode - with the authors!

abstract reasoning, even though we do sort of next token prediction.


AlphaCode - with the authors!

So I'm, yeah, I think it's fairly, fairly cool.


AlphaCode - with the authors!

I guess, can I jump in there?


AlphaCode - with the authors!

So I just say like one kind of more general point there, I think is that, you know, I


AlphaCode - with the authors!

definitely see this as it's like, clearly different from how I do, I solve a problem.


AlphaCode - with the authors!

But also, I think in machine learning, like, maybe, you know, the first step to doing something


AlphaCode - with the authors!

the right way is doing it at all.


AlphaCode - with the authors!

And I think that's kind of, you know, part of what we've achieved.


AlphaCode - with the authors!

Do you have plans to bring down this large scale sampling?


AlphaCode - with the authors!

Like is there any, are there any ideas floating around of, you know, maybe we don't have to


AlphaCode - with the authors!

sample a million things and then test them all?


AlphaCode - with the authors!

I mean, I think, of course, it would be somehow more satisfying if our model could just like


AlphaCode - with the authors!

one shot the problems.


AlphaCode - with the authors!

And I think getting higher quality average samples is a really interesting research direction,


AlphaCode - with the authors!

especially since yeah, every time you want to solve a problem, you probably don't want


AlphaCode - with the authors!

to have to try and begin different things, right?


AlphaCode - with the authors!

That's typically not how we work.


AlphaCode - with the authors!

But I think there's also something really interesting in this scaling that we observe,


AlphaCode - with the authors!

and the fact that we can actually get more and more good answers by simply by sampling


AlphaCode - with the authors!

more is something that's quite interesting to explore.


AlphaCode - with the authors!

And what's further interesting, I think, is that the larger model, like the model size


AlphaCode - with the authors!

seems to be also correlated with the quality of the samples in itself, which is also something


AlphaCode - with the authors!

I find cool.


AlphaCode - with the authors!

Yes, indeed.


AlphaCode - with the authors!

We see that the bigger the model, the higher we start and the steeper the slope basically


AlphaCode - with the authors!

in these sampling curves.


AlphaCode - with the authors!

So on average, the bigger the model, the better the sample quality.


AlphaCode - with the authors!

A lot of models have popularized or a lot of systems in recent times have popularized


AlphaCode - with the authors!

this idea of sort of having an additional model to do filtering output of generative


AlphaCode - with the authors!

models, right?


AlphaCode - with the authors!

Most famously, I guess, Dali, which uses the clip model to sort of rerank or filter the


AlphaCode - with the authors!

outputs.


AlphaCode - with the authors!

You here have a rather, let's say heuristic way of filtering the outputs.


AlphaCode - with the authors!

Is it even possible or considerable that you would sort of train another model?


AlphaCode - with the authors!

Or would that just shift the problem?


AlphaCode - with the authors!

I'm going to guess, you know, if training a model that can tell me whether a program


AlphaCode - with the authors!

is correct for a given solution, that's almost like solving the problem itself.


AlphaCode - with the authors!

But you know, we've seen that it generally helps to pair generative models with rankers.


AlphaCode - with the authors!

Is that something that is in scope here?


AlphaCode - with the authors!

Or is there a particular reason why that wouldn't work?


AlphaCode - with the authors!

I think that's a very reasonable suggestion.


AlphaCode - with the authors!

And over the course of the project, we've tried several ideas that are linked to this,


AlphaCode - with the authors!

notably training value functions, which could be used either as guides during the sampling


AlphaCode - with the authors!

process or as a re-ranking mechanism once the sampling is done.


AlphaCode - with the authors!

What we found though, is that learning a good enough value function remains extremely challenging.


AlphaCode - with the authors!

And so we're definitely interested in trying these ideas again.


AlphaCode - with the authors!

It's just that we haven't been able to make them work quite yet.


AlphaCode - with the authors!

And why that is, is still a bit up for debate.


AlphaCode - with the authors!

Of course, we have a rather small fine tuning data set, which might be part of the reason


AlphaCode - with the authors!

why, or maybe the action space is too big.


AlphaCode - with the authors!

We are still investigating that.


AlphaCode - with the authors!

Yeah, I wanted to add something to that as well, which is that I think, yeah, we definitely


AlphaCode - with the authors!

tried re-ranking a couple of times and it seems like, you know, a good thing to try.


AlphaCode - with the authors!

But the way that we eventually did a lot of that filtering was by executing the program.


AlphaCode - with the authors!

And that is an enormous boost.


AlphaCode - with the authors!

And I think whether we had a ranking model or not, we would definitely still do that.


AlphaCode - with the authors!

And there are ways of using the program execution that we haven't even considered.


AlphaCode - with the authors!

We just use the fact that the public test passes or doesn't pass.


AlphaCode - with the authors!

So I think, you know, potentially even continuing to use that or even expanding on how that


AlphaCode - with the authors!

happens, continuing to, how executing the program affects the filtering and ranking


AlphaCode - with the authors!

is also another kind of interesting, I guess, non-machine learning way to continue doing


AlphaCode - with the authors!

that.


AlphaCode - with the authors!

I'm all for non-machine learning.


AlphaCode - with the authors!

I'm all for not introducing more models.


AlphaCode - with the authors!

But you do point to a good question.


AlphaCode - with the authors!

There is this small set of candidates which comes from these large sets of potential solutions.


AlphaCode - with the authors!

And the filtering is a really important step there.


AlphaCode - with the authors!

As you say, you execute the programs against the small set of samples.


AlphaCode - with the authors!

Now this set is maybe four, maybe five test cases or something like this.


AlphaCode - with the authors!

And I haven't seen, maybe I've overlooked that, but I haven't seen anywhere in the paper


AlphaCode - with the authors!

where, did you investigate, you know, if we had 10 such public test cases, you know, how


AlphaCode - with the authors!

does that change?


AlphaCode - with the authors!

Or if we just had one, if we, how does the success of the model change with the amount


AlphaCode - with the authors!

of test cases you have at your disposal in the given problem?


AlphaCode - with the authors!

That's actually a really good suggestion.


AlphaCode - with the authors!

We haven't looked at that.


AlphaCode - with the authors!

I think in the end, the issue for us is we don't really have control over this quantity.


AlphaCode - with the authors!

And most problems have very, very few public test samples.


AlphaCode - with the authors!

Between one and three on average, I think.


AlphaCode - with the authors!

So we didn't really push this direction because we thought we can't move the needle on it


AlphaCode - with the authors!

at test time.


AlphaCode - with the authors!

But that doesn't mean that it wouldn't be informative to try to say.


AlphaCode - with the authors!

And if I had to take a guess, I would imagine that adding more public tests would be very


AlphaCode - with the authors!

helpful because it would make the filtering mechanism that much more powerful.


AlphaCode - with the authors!

So yeah, that's basically how I think about this.


AlphaCode - with the authors!

And of course we could try to generate more tests, but that's a very difficult problem


AlphaCode - with the authors!

in and of itself.


AlphaCode - with the authors!

Yeah, I think I had another thought on that, which is that I actually would kind of love


AlphaCode - with the authors!

to do that ablation, but actually not necessarily for the problem that we had, because as Remy


AlphaCode - with the authors!

said, we can't control the number of public tests we have.


AlphaCode - with the authors!

But there may be some applications of something like Africa code where you can control the


AlphaCode - with the authors!

number of public tests and knowing how that affects the ability of us to filter the samples


AlphaCode - with the authors!

would be super interesting.


AlphaCode - with the authors!

Maybe two samples is enough to get you exactly the right solution most of the time.


AlphaCode - with the authors!

I mean, unit tests come to mind, right?


AlphaCode - with the authors!

Just programming essentially by writing four or five unit tests for a function or a class


AlphaCode - with the authors!

that I want to write and then just let the model come up with a bunch of examples of


AlphaCode - with the authors!

for me to choose.


AlphaCode - with the authors!

Yeah, I think that would be, I don't know, like the future of programming looks more


AlphaCode - with the authors!

and more something I don't recognize from that, I think is very exciting.


AlphaCode - with the authors!

Is there some sort of, you know, between these two, is there some sort of adversarial setup


AlphaCode - with the authors!

that I could do?


AlphaCode - with the authors!

You have various models, like you have a model that generates new test cases, but at various


AlphaCode - with the authors!

stages, right?


AlphaCode - with the authors!

So for the clustering, you simply need to execute and observe the same outputs.


AlphaCode - with the authors!

Because I'm going to guess a model that makes new test cases doesn't necessarily make correct


AlphaCode - with the authors!

test cases.


AlphaCode - with the authors!

But is there also a model that makes test cases just sort of generates them, let's say,


AlphaCode - with the authors!

in a language model way, in a, you know, most likelihood way?


AlphaCode - with the authors!

Did you ever think of some kind of adversarial setup, given that DeepMind is a lot of in


AlphaCode - with the authors!

the space of like self play, and sort of this reinforcement learning setting?


AlphaCode - with the authors!

Is there opportunities here for sort of systems to challenge each other to get better?


AlphaCode - with the authors!

Yeah, that's, it's very funny that you mentioned that because the project started off right


AlphaCode - with the authors!

after the Alpha Star project, basically.


AlphaCode - with the authors!

And so we had our minds were full of these types of ideas, right?


AlphaCode - with the authors!

And so that's something I've actually been very keen on since the inception of the project


AlphaCode - with the authors!

more than two years ago, to bring some notions of self play, curriculum learning, etc.


AlphaCode - with the authors!

I think that that would be very exciting.


AlphaCode - with the authors!

Unfortunately, generating new problems is an extremely difficult task, because first


AlphaCode - with the authors!

of all, your problems need to make sense, they need to actually be solvable, right?


AlphaCode - with the authors!

So I can definitely see a world where we have many, many problems.


AlphaCode - with the authors!

And either they're way too difficult, or they're nonsensical.


AlphaCode - with the authors!

And the other thing is, we also have to come up with unit tests that work with the description


AlphaCode - with the authors!

of the problem, right?


AlphaCode - with the authors!

And we have a data set of 12 to 13,000 problems, if I remember correctly, which is probably


AlphaCode - with the authors!

not enough for us to train a really good generative model to ask problems.


AlphaCode - with the authors!

And so we haven't really tried up until now.


AlphaCode - with the authors!

This maybe, I think, one distinction I think is relevant there is that, you know, in Alpha


AlphaCode - with the authors!

Star, and in a couple of other self play setups, they are symmetric.


AlphaCode - with the authors!

So you kind of expect the both sides to be improving all the time.


AlphaCode - with the authors!

Whereas in our case, you know, it's less obvious how you might improve the problem maker over


AlphaCode - with the authors!

time.


AlphaCode - with the authors!

So, is there maybe there's a, I have no clue how these problems are actually made, because


AlphaCode - with the authors!

humans need to make these programs, right?


AlphaCode - with the authors!

If I look at a problem description like this, I'm like, this is insane.


AlphaCode - with the authors!

Not only is it very thorough, right?


AlphaCode - with the authors!

Also, I have to somehow make sure that I as a maker of the problem, don't make a mistake.


AlphaCode - with the authors!

And when I generate test cases, usually, you know, for the example inputs right here are


AlphaCode - with the authors!

kind of small, but then I need to test like all the edge cases, right to make sure that


AlphaCode - with the authors!

people have the correct algorithm, which means some are going to be very long and so on.


AlphaCode - with the authors!

So I almost have to write like a generator for, you know, these these long things, maybe


AlphaCode - with the authors!

there's an maybe there's a way to replicate that process of like how humans come up with


AlphaCode - with the authors!

these problems as because they're going to have like strategies and whatnot.


AlphaCode - with the authors!

They just they don't just sit there and go like, well, backspace, right?


AlphaCode - with the authors!

I don't know, have you looked into do you know how these problems are made, like on


AlphaCode - with the authors!

a mechanical level?


AlphaCode - with the authors!

So I think we've been focusing a lot on the solving aspect of things.


AlphaCode - with the authors!

And a lot less than the generating problems aspect of things.


AlphaCode - with the authors!

I have I have a healthy respect for the difficulty to generate problems that people can actually


AlphaCode - with the authors!

solve, right?


AlphaCode - with the authors!

I remember taking exams and thinking, oh, this is no fun.


AlphaCode - with the authors!

And then I know a lot of people who are teachers who have to actually devise exams.


AlphaCode - with the authors!

I think, wow, this is even less fun, actually.


AlphaCode - with the authors!

But yeah, I don't think we have a really good grasp on the human generative process


AlphaCode - with the authors!

for this thing.


AlphaCode - with the authors!

It would be really interesting to discuss with the problem makers to try to see what


AlphaCode - with the authors!

are the strategies and whether or not we can try to replicate that.


AlphaCode - with the authors!

And one possible direction would be to actually help them.


AlphaCode - with the authors!

But it would be quite cool.


AlphaCode - with the authors!

Yeah, I think that's sorry.


AlphaCode - with the authors!

I think that's a great idea, actually.


AlphaCode - with the authors!

Like I I'm really quite interested to go and ask them myself now, I think.


AlphaCode - with the authors!

Maybe like if I had to do I would look in a computer science textbook and like algorithms


AlphaCode - with the authors!

and then dress them up in some kind of story that seems to be like what a lot of problems


AlphaCode - with the authors!

are.


AlphaCode - with the authors!

But yeah, in terms of doing it mechanically, maybe that would be even harder than generating


AlphaCode - with the authors!

the solutions because like lots of people upload their solutions to GitHub.


AlphaCode - with the authors!

But I guess I expect there would be less data on how to create problems on GitHub.


AlphaCode - with the authors!

Yeah.


AlphaCode - with the authors!

Yeah, I was.


AlphaCode - with the authors!

I was exactly I was more thinking of there must be some process because also these people


AlphaCode - with the authors!

have to come up with new and new problems.


AlphaCode - with the authors!

Right.


AlphaCode - with the authors!

And there are so many algorithms and something like this backspace problem.


AlphaCode - with the authors!

It's very intricate, right?


AlphaCode - with the authors!

There's not really like an algorithm that I can just poof apply, like I really have


AlphaCode - with the authors!

to think through stuff.


AlphaCode - with the authors!

One of my questions is that you hear the test cases, the public test cases, they're kind


AlphaCode - with the authors!

of samples, right for you also to think through as a human.


AlphaCode - with the authors!

But very often, the testers, they also want to test not only whether you have the correct


AlphaCode - with the authors!

algorithm, but also whether you have the sort of correct runtime algorithm.


AlphaCode - with the authors!

Because you know, I can write an algorithm, you know, in I don't know, like, if I have


AlphaCode - with the authors!

an O of n squared, that might not be the algorithm the tester is looking for.


AlphaCode - with the authors!

So they want like the O n log n.


AlphaCode - with the authors!

I'm having trouble writing the O n log n algorithm, right?


AlphaCode - with the authors!

Because one is really easy to implement, and one is actually the challenging one.


AlphaCode - with the authors!

So they will make deliberately like very large hidden test cases, so that my naive algorithm


AlphaCode - with the authors!

would either go out of memory or out of time on the evaluation server.


AlphaCode - with the authors!

And this is something that you would not capture with just filtering on the public test cases


AlphaCode - with the authors!

as as your algorithm does, your algorithm would think, well, I've solved the problem,


AlphaCode - with the authors!

right?


AlphaCode - with the authors!

I've come up with a solution.


AlphaCode - with the authors!

The naive solution will probably even be the more likely one given the language model.


AlphaCode - with the authors!

And then right, and then it's filtering, it's clustering is like, well, all of this seems


AlphaCode - with the authors!

just fine, right?


AlphaCode - with the authors!

How do you have any grasp on how good you are on these types of problems?


AlphaCode - with the authors!

And is your model?


AlphaCode - with the authors!

Does it have some strategy to overcome that?


AlphaCode - with the authors!

Yeah, I think I can take that.


AlphaCode - with the authors!

The main answer here is that we just don't, we just don't do it.


AlphaCode - with the authors!

We when we actually like looking at what our real solve rate is, we had to do a lot of


AlphaCode - with the authors!

sort of manual checking of solutions to check that they were meeting the asymptotic complexity


AlphaCode - with the authors!

requirements of that we expected the problem to actually have.


AlphaCode - with the authors!

I think you you mentioned before the call or in your question about clustering to buckets


AlphaCode - with the authors!

by time or memory.


AlphaCode - with the authors!

I think you wrote that down.


AlphaCode - with the authors!

Did you have this in the paper?


AlphaCode - with the authors!

Or was this something I came up with?


AlphaCode - with the authors!

I don't.


AlphaCode - with the authors!

That's something that you came up with.


AlphaCode - with the authors!

Okay, yeah.


AlphaCode - with the authors!

Is this viable or is this like a bad idea?


AlphaCode - with the authors!

Yeah, I guess I just had a thought on that.


AlphaCode - with the authors!

I think it's quite a cool idea.


AlphaCode - with the authors!

Maybe that particular implementation of looking at time and memory usage of inputs like definitely


AlphaCode - with the authors!

is in the theme of executing the program and seeing what happens.


AlphaCode - with the authors!

So I think an idea along that lines is actually worth a go.


AlphaCode - with the authors!

One thing I would say is that a lot of these problems, I think, when you write the solution,


AlphaCode - with the authors!

which is asymptotically better, usually has like a big constant factor in front of it


AlphaCode - with the authors!

or a constant additive complexity.


AlphaCode - with the authors!

So you'd have to kind of consider that and whether that is going to adversely affect


AlphaCode - with the authors!

which solutions you're removing.


AlphaCode - with the authors!

The thing which actually is going to have eventually the asymptotic lower complexity.


AlphaCode - with the authors!

I think we could probably use it to cluster, right?


AlphaCode - with the authors!

Because if you had the same different asymptotic implementation, you would have different values.


AlphaCode - with the authors!

But choosing directly according to like trying to rank them, depending on their performance


AlphaCode - with the authors!

on very, very small unit tests, would probably...


AlphaCode - with the authors!

I mean, my intuition and our intuition, I guess, is that we'd have to be extremely careful


AlphaCode - with the authors!

how we do that and not to overfit too much to that particular metric.


AlphaCode - with the authors!

So something that I want to point out, though, is that, yes, sometimes we have what we call


AlphaCode - with the authors!

slow positives, which are correct, except that they're impractical.


AlphaCode - with the authors!

But still, I already find that to be quite impressive because some of these problems


AlphaCode - with the authors!

we go for the naive approach, but it's not completely evident that the naive approach


AlphaCode - with the authors!

would even work.


AlphaCode - with the authors!

So there's this thing like you want to...


AlphaCode - with the authors!

I remember a coding mentor told me about just make it run, make it right, make it fast.


AlphaCode - with the authors!

So we make it run and we make it right.


AlphaCode - with the authors!

Now all we'll have to do is to make it fast, which admittedly is a really difficult problem.


AlphaCode - with the authors!

Yeah, I think I wouldn't be too worried that the clustering might not work.


AlphaCode - with the authors!

I would be more worried that the language model itself might not even... might just


AlphaCode - with the authors!

jump on the sort of more likely naive implementation and never actually get to output the very


AlphaCode - with the authors!

different possibly more efficient implementation because these two things, they don't often


AlphaCode - with the authors!

look similar.


AlphaCode - with the authors!

They often look very, very different from each other.


AlphaCode - with the authors!

And yes.


AlphaCode - with the authors!

Yes.


AlphaCode - with the authors!

I think another issue is in our pre-training set on GitHub open source code, probably very,


AlphaCode - with the authors!

very fast efficient programming isn't the majority of what's on there.


AlphaCode - with the authors!

So it might be that there's a bias towards simpler, more naive solutions already when


AlphaCode - with the authors!

we start fine tuning.


AlphaCode - with the authors!

So of course we'd have to fight against that a bit.


AlphaCode - with the authors!

With respect to the sampling and whether or not you can output something, you have a lot


AlphaCode - with the authors!

of tricks to increase your sampling diversity.


AlphaCode - with the authors!

One of the most notable things is that you have this prefix right here, which I found


AlphaCode - with the authors!

quite genius.


AlphaCode - with the authors!

I think in general, the approach of including sort of unknown things like that you would


AlphaCode - with the authors!

only know at training time, like things about your labels into the prompts and then having


AlphaCode - with the authors!

that as sort of like a dial where you can control the model.


AlphaCode - with the authors!

I think that is a very cool, very cool idea.


AlphaCode - with the authors!

And I think you've shown quite impressively how that can help.


AlphaCode - with the authors!

You use it mostly to vary the outputs of your model.


AlphaCode - with the authors!

But that brings me like, given that we have to do all of these things to increase diversity,


AlphaCode - with the authors!

do you think maybe where our sampling procedure as such isn't a very good one because we have


AlphaCode - with the authors!

to do all these tricks?


AlphaCode - with the authors!

Like could we fundamentally remake our language models or our generative models to be more


AlphaCode - with the authors!

like diverse, let's say?


AlphaCode - with the authors!

Yeah, so I do think you're right and we're not equipped with the right tools just yet.


AlphaCode - with the authors!

Right now we have this very crude setting to tune, which is the sampling temperature.


AlphaCode - with the authors!

But this means that we have very little control over how qualitatively diverse our samples


AlphaCode - with the authors!

are going to be.


AlphaCode - with the authors!

So we're searching over the model distribution in an extremely crude way, which is basically


AlphaCode - with the authors!

pointing it into a general direction than say, OK, try to take as many sample courses


AlphaCode - with the authors!

as you can in that particular direction.


AlphaCode - with the authors!

But it seems important to me that we should be able to branch out in different directions


AlphaCode - with the authors!

only at fairly select decision points, not in every step.


AlphaCode - with the authors!

And we don't have a proper mechanism to do that.


AlphaCode - with the authors!

So we had high hopes for a top-K and nuclear sampling or for our sampling being guided


AlphaCode - with the authors!

by a value.


AlphaCode - with the authors!

But as we reported in the paper, this didn't really bring significant improvements.


AlphaCode - with the authors!

And I think another thing here is that we are sampling very independently.


AlphaCode - with the authors!

We're not taking past samples into account when sampling a bit more autoregressively


AlphaCode - with the authors!

at the level of samples could probably be an interesting thing to explore.


AlphaCode - with the authors!

I had one other point there, which since we sample from the model autoregressively, maybe


AlphaCode - with the authors!

this isn't really related to the diversity point, but to something in general.


AlphaCode - with the authors!

That's clearly not how I do things at all when I'm writing code.


AlphaCode - with the authors!

I usually write something, I write a sketch, and then I iterate over it in random bits


AlphaCode - with the authors!

of the code.


AlphaCode - with the authors!

So it's possible that that also is something that needs to sort of fundamentally change


AlphaCode - with the authors!

about the way that we sample from models.


AlphaCode - with the authors!

I haven't looked much at the outputs the model generates, which astounded me.


AlphaCode - with the authors!

Just seeing this and seeing it output from a language model is astounding by itself,


AlphaCode - with the authors!

but also it's very instructive.


AlphaCode - with the authors!

Even on the right, you even do a little bit of analysis and say, you know, these lines


AlphaCode - with the authors!

are this, these lines are this, these lines are this.


AlphaCode - with the authors!

Is this, did you generally find that throughout your solutions, I haven't looked at many more


AlphaCode - with the authors!

solutions, to be honest, did you generally find that code is interpretable?


AlphaCode - with the authors!

You know, very, very sort of instructive, or is this a particular problem that you've


AlphaCode - with the authors!

picked out and to show kind of like, oh, look, the model solves the problem in an understandable


AlphaCode - with the authors!

way?


AlphaCode - with the authors!

Or did you, was most of the output cryptic or understandable?


AlphaCode - with the authors!

Yes, I think I looked at a fair few, you know, individual solutions when I was doing the


AlphaCode - with the authors!

analysis for this paper.


AlphaCode - with the authors!

I think in general, so actually to be clear, like we did definitely pick this example as


AlphaCode - with the authors!

something that, you know, illustrates what's going on, but in general, you know, the model


AlphaCode - with the authors!

does produce things which you can read and understand what's going on.


AlphaCode - with the authors!

I think you have to, you know, and that's kind of expected in a way because we're training


AlphaCode - with the authors!

on human data, right?


AlphaCode - with the authors!

We're training to mimic the way that human programs look.


AlphaCode - with the authors!

So that's not crazy.


AlphaCode - with the authors!

But when we fine tune, competitive programmers write very unreadable code.


AlphaCode - with the authors!

So that's another thing to bear in mind.


AlphaCode - with the authors!

They will use a lot of type defs in C++, for example, a lot of crazy helper functions.


AlphaCode - with the authors!

And that's also something you see a lot in some of the solutions.


AlphaCode - with the authors!

You'll see these like huge copy pastes of code which like passes an input in an efficient


AlphaCode - with the authors!

way.


AlphaCode - with the authors!

A lot of that is dead code and doesn't actually get used.


AlphaCode - with the authors!

And that's consistent with some of the competitive programming, like real solutions.


AlphaCode - with the authors!

But yeah, I guess like in this, you know, maybe it's because we filter for public tests


AlphaCode - with the authors!

as well.


AlphaCode - with the authors!

Like in particular, the solutions which are correct seem to be fairly interpretable and


AlphaCode - with the authors!

make sense.


AlphaCode - with the authors!

But yeah, on rare occasions, like the implementation is quite difficult to understand.


AlphaCode - with the authors!

But yeah, I think if you want to look into that a bit more, we do have a tool, alphaco.dmind.com,


AlphaCode - with the authors!

which Remy and Julian worked on.


AlphaCode - with the authors!

And there's also some commentary on there, I think from Petter, who works at Google,


AlphaCode - with the authors!

about what the model is doing.


AlphaCode - with the authors!

And I think in the samples he looked at, generally, you know, he was quite happy that a lot of


AlphaCode - with the authors!

them seem to be doing something that you would expect in a reasonable way.


AlphaCode - with the authors!

I mean, it's distantly possible that you write something that just passes all the test cases,


AlphaCode - with the authors!

but isn't actually correct.


AlphaCode - with the authors!

Like with sampling so many things, like this, like, it might be not very likely.


AlphaCode - with the authors!

So it's definitely possible.


AlphaCode - with the authors!

And we did a fair amount of work actually generating new tests to try to make sure that


AlphaCode - with the authors!

that didn't happen.


AlphaCode - with the authors!

I remember somewhere, maybe a little bit under a year ago, we took a deep dive on our solve


AlphaCode - with the authors!

rate and we were trying to figure out whether it was the actual thing or whether actually


AlphaCode - with the authors!

we were gaming the problems.


AlphaCode - with the authors!

And we realized that there was a significant percentage of our solutions, quote unquote,


AlphaCode - with the authors!

which were gaming the system.


AlphaCode - with the authors!

And the possible reasons for that were that actually there was very little coverage because


AlphaCode - with the authors!

there were many tests, but the answer was always the same.


AlphaCode - with the authors!

Sometimes you have yes, no type of things.


AlphaCode - with the authors!

And you look at the private test and the answer is always yes on the 40 private tests.


AlphaCode - with the authors!

And so the model will try.


AlphaCode - with the authors!

If you sample from it a million times, it will try to just print yes.


AlphaCode - with the authors!

That's probably going to happen.


AlphaCode - with the authors!

And for other things, we just had very, very few tests.


AlphaCode - with the authors!

So we filter out the problems.


AlphaCode - with the authors!

We have too few tests, but we also mutated the tests to add new ones to make sure that


AlphaCode - with the authors!

this didn't happen.


AlphaCode - with the authors!

And I think we went down from, I don't remember if it was 40% or maybe even 60% false positive


AlphaCode - with the authors!

actual false positive rates to about 4% in our final data set, which is still significant,


AlphaCode - with the authors!

but we found that was a reasonable and acceptable amount of false positives.


AlphaCode - with the authors!

Yeah, I don't think I mentioned this in the video too much, but you have this kind of