found them, and that system will get even better. So the expert iteration procedure is the act of


OpenAI tackles Math - Formal Mathematics Statement Curriculum Learning (Paper Explained)

always going to your best system, gathering the data that it has figured out through, you know,


guiding the search, then taking that data and inter and retraining the system on this new data


to make it even stronger. Right? This, this is based on two facts, you can't just do that with any


system, right? This is based on the fact that here, a machine learning system interacts with a search


system. And the interaction is what makes the difference. So the combination of the two is


better than just the search system and better, especially than just the machine learning system.


So you can, if the machine learning system itself has a certain performance, adding the search on


top will increase that performance and therefore allow you to get to more and better training data


that you couldn't have just gotten with the ML system itself. If you just had the ML system, you


just stop be stuck forever in a loop of always having the same difficulty, because all you do is


feed the output of the ML system back into the ML system. But if you add a component on top,


that makes it stronger, that gives you better data that can make the ML system itself stronger,


then you add the search again, that will make it even stronger in combination. So that is,


that is the story of expert iteration and of this paper right here. They go a little bit into the


environment, they have this lean environment, which I have no clue about. But this is like a formal


environment for mathematics proves one of one of many I'm being informed. There's also one that's


called Meta math and apparently lean, lean benefits from higher level tactics, which were shown to be


beneficial in this context. But essentially, for our purposes, it is Oh, and also the proofs lean


proofs are typically 10 times shorter than other systems. But, you know, for our purposes, just


assume that we have some kind of a system where we can build proofs like this, this tree right here


from from statements. So they next go into into experts. So they have a bit of data sets. That's


what they describe here, they we go into expert iteration. Expert iteration consists in iteratively


training models on their previously sampled trajectories. That's essentially expert iteration.


As for a model, they use decoder only transformers. So they use language models, which just shows you


sort of the versatility of language models. The biggest model, I think that they use uses 36 layers


and 700 million trainable parameters. So this is not too big of a model, right? This is a reasonably


sized it's it's big, but it's not like GPT-3 big. They pre train this, which I found interesting,


on a combination of mathematics data sets, but also common crawl, which is the language just it's


a web scrape, right? That is, is very interesting that the pre training happens on natural language


and not just on mathematics data. Maybe you need this, this many, this many tokens to pre train the


model, because the model itself is kind of big. But I'd, I'd wonder, you know, what kind of difference


that makes. And what is what the transfer is from the natural language to the mathematics because


math is is very cryptic. I'm not even sure if they have let me find a proof here. Maybe they've


listed. So yeah, you can you can see, these are sort of the things you would find in this is a


a terminal and interminable trace of this lean environment or their their their their gym


environment around the lean environments. So you'd have like these tactic states you can see right


here. These these are have nothing to do with natural language, right? Then you have the tactics


that you run, you apply this nat prime DVDMulHP.mp tactic. I have no idea what it is. And that


transforms the above tactic state, I believe, into the bottom tactic state. I'm not going to parse


this because I again, I have no clue what it means. But you can see that these statements,


they're they're very formal. And they have nothing to do with natural language. Still, obviously,


humans made them as a series of characters, and therefore, there might also always be some transfer.


So how do they train this? How do they train this thing? So the the transformer is trained to


suggest kind of what to do next in such a proof. And that is called a proof step. So the proof step


objective that they train the transformer with consists in generating a proof step,


which is a tactic, given a goal, which is a tactic state. So you're trying to get somewhere,


which is the root of the current tree or subtree you're considering. And you're generating a tactic,


which means like how to expand the tree. Given that that, you know, you are at this particular


root. And they also condition this objective on the current declaration, which is the theorem name,


which remains the same throughout the proof search. They make some, they give some explanation


why they do this. But essentially, the what they train the transformer with looks like this,


there is a keyword decal, then there's the declaration, which is the name of the


theorem, then there is a goal. And then here, you put the goal state, the tactic state


that you want to achieve, and then the keyword proof step. And then here is where the proof step


goes. So during inference, obviously, you leave this away, and you let the language model


generate this part. But during training, you put right here, the goal, and then the tactic state,


and then here, any any proof from any proof that you know was successful, you'd put the


corresponding proof step there. So this is a yeah, this is a language modeling objective,


you just train on all of the proofs that you know, that are true, you put them into this particular


form, you put all of their individual tree expansion steps into this particular form,


and then you put the language model on it. And that apparently works pretty well. This is already


from their from their previous work, that this works pretty well. They also have they explained


this here, the rationale for conditioning on the declaration name is to hint our models on the


position of the current declaration in the math lib library can be considered a weak proxy signal


function not shown to the model. So there is a full day, there is available imports,


currently open declarations, module names, notations, declared instances. So and that that


is where I really am a noob. There is this math lib library, which is a library inside of this


lean environment. And I'm going to guess the analogy would be like, it has a bunch of functions


call, it has a bunch of stuff there that you could potentially use. And obviously, this is not going


to all fit into the little context that we have right here that we're going to feed into the


transformer. So what you're going to do is you simply give this declaration name. And if the


model has seen enough of those things, it it obviously some of these function calls will be


in this proof step step right here, if you start out with proofs that already exist. So some of


these function calls will be in there. And the declaration hints sort of where in the library


you are, which means that which functions you can currently call which variables exist, and so on.


I'm exactly sure. But I essentially, I would, I would read the declaration if I were a programmer,


I would read the declaration as maybe the project and the file I'm currently in and what imports


there are, I would read the goal as the function definition, or sorry, the function header,


and the doc string that tells me what should happen in this function. And then the proof step,


I would consider the function itself the implementation. That is a very bad analogy, but


approximately like this. It's a weird mix between programming and mathematics, this formal


mathematics proofs. So they train the language model on this. So now the language model can


suggest new proof steps. You give it the declaration and the goal, it can suggest new proof steps,


right? That is one thing they train the language model with. They in at the same time, train it


also with this proof size objective. So they give other inputs to the language model that


they train it on. Again, we have the declaration name, we have the goal, but then we have a


different keyword instead of proof step. Now we have the keyword proof size. And then here is a


proof size bucket token. And that's simply a letter from A to K. And that letter encodes one of 11


buckets. And the buckets represent the size of the proofs. Again, during training, we know the proof


size, right? Or the size of the proof step or maybe the size of the whole proof. I'm not entirely


sure. I think it's the size of the whole proof. Yeah, it represents a proof size estimate bucket


for the current goal. Okay, so for the proof of the current goal, how long is it? And during


training, we know it. So we just put it here during inference time. Again, this is the thing that we


are going to let the model predict. So the model should guess how long a proof is going to be


without necessarily producing it. That's what this keyword up here does. So the bottom one simply


says how long is it maybe, you know, probably going to be. And this, it's pretty neat how they


do it. So they have these 11 buckets, infinite proof sizes go to bucket zero, and then bucket


one gets the longest proofs, bucket two gets slightly smaller proofs, and the shortest proofs


go into bucket 10. Why do they encode it like this? Now it comes to the place where how or what


do you search? So you're now in the proof search, right? You're in inference mode, you ask your


model to suggest a bunch of these proof steps to you that we saw right here. So you ask your model,


please suggest a bunch of those proof steps you sample from the model a bunch of times. And now


and now how, where should you, which one should you do? Of course, you could go by I guess the log,


like the likelihood of these proof steps. But as far as I can understand, they weigh, they weigh


the tactics that they want to use. So they, they value different goals.


Ah, this is about which goal do I want to pursue next? Okay. So they, they ask themselves, which


goal should I produce? Or should I pursue next in my proof search to value goals as we run proof


searches, we sample the proof size bucket token, and record the logits for each viable bucket,


and use them to get a weighted average with the following formula. So the formula itself is not


really important. But what is important they use the buck like the prediction of how long a proofs


going to be to guide their selection of goals, which means that the exact way they do it is


they say, if a model assigns p zero equals one, which means that the model puts all the weight on


bucket zero, which if you remember, as the infinite proofs, so if the model predicts this proof size


is going to be infinite, which means that it's not going to work, right? The proof size infinite


means that it hasn't been at least it hasn't been proven yet, the proof search in or the data set


hasn't been able to prove this particular statement. So the size is is infinite, then the value,


as you can see, is zero. So we don't want to go after something where the model is absolutely


sure that the proof size is infinite, it's never going to be absolutely sure. But if that were the


case, the value would be zero. Conversely, if a model assigns the is very sure, or absolutely


sure that this proof is going to be in the shortest bucket, then the value is one. So this is


a number between zero and one, depending on how short the proofs it the proof is. So they say it


prioritizes goals that potentially lead to shorter proofs during proof search. So that's how they


guide their search. Excellent. So these are the two objectives they train with. The one objective


is to make the model suggest new tactics to use. And the other one is to guide the proof search


by training the model to predict how long a proof is going to be. So yeah, the next topic right here


is how they how they bootstrap the models. So in this expert iteration, you always train on your


own outputs. However, there needs to be like some sort of a some sort of a starting point. So you


need to train on your own outputs. Right. Bootstrapping, they say, consists in a step


required to train an initial model on both proof step objective and the proof size objective.


They have two initial models. In fact, they have a they have a data set, which consists of some of


these proofs that have already been proven. And they train a model with just a proof step objective,


which is called theta zero. So that's the initial model. Then they use they use the initial model


to sample proofs for the statements in this mathematics library. So they already use the


model to generate proofs. We denote the set of successful proof searches created in processes


S0. Using S0, we create a data set. So the expert iteration process essentially already starts.


So they're going to concatenate the original data set, sorry, the original data set,


and a deduplicated set of proof steps extracted from the proofs in S0, and a deduplicated set of


proof size tuples extracted from the proof searches in S0. So now they're going to use


whatever they output as proofs in the last in the last in the last iteration, they're going to


take that into the data set, they're going to create these proof step sentences, I'm just going


to call them sentences because we're language modeling right here. They're going to create these


proof step sentences like this one, they're going to create these proof size sentences like this one,


and then they're going to train a model again on that. So they're going to take the they're going


to take the theta zero, and they're going to train it on that new data set. So that gives them theta


one, which is trained on both the proof step and the proof size objective and theta one is our first


model in our expert iteration. So now we are simply going to repeat those things. Each iteration k


consists in sampling proof searches for statements using the current model, filtering successful proof


searches to extract a new data set and fine tuning the theta zero on it to obtain theta k plus one.


Note that they don't they don't go from theta zero to theta one to theta two and so on. They always,


so they don't do that. They always go from theta zero to theta two, then they use theta two to


generate a data set, then they fine tune theta zero again to get to theta three. It'd be interesting


to know why they do it this way. Maybe if you continue fine tuning, you're already sort of


locked into something. So the knowledge comes the knowledge, the unified knowledge comes from


you can see this right here, the fact that they the data sets they generate comes from the unified


set of all the statements they've proven so far. So all the proofs they found so far, they all go


together into one big data set for the next step. So technically every model can like relearn the


proofs that the last model also knew because it's there, they're in the same data set. And, you know,


potentially, they also say that they deduplicate proofs, which means that for the same statements,


there could be multiple proofs, and they will always take the shortest one. So that might be


even disadvantage, a disadvantage if you were to tune from like theta two, which would still have


learned a longer proof for a particular statement. And you'd have to like forget that it's probably


just easier to scratch everything and start with the shorter proof in your data set.


And yeah, that is it. That's the expert iteration process. They get a new model, they use it to


generate new proofs, they add the proofs to the set of things they know, and they get the


results set of things they don't know, right? Because there can also be bad proofs which serve


as negative examples, which is also good, right? You can handle negative examples, and then they


get better and better. So now they are going to evaluate this right now. You see that they have


various various ways of using this model. There's pass at eight, there's pass at one, which


essentially means like how many tries they give per expansion step, like do we sample, do we try


once do we try eight times, obviously, the more you try, the longer your searches run, but also


the higher your chance of actually finding something useful. And these things are mostly


proportional to each other. So it's just a matter of computational effort. You can see that with


expert iteration, so the x axis right here is number of expert iterations, you can see they


do nine expert iterations on these data sets. In general, you see an upwards trend. So more and


more statements are able to be proven by the expert iterated system. And they have multiple


data sets, this mini F2F is their final goal. This is made up of these various competition


level statements, while the Math Lib, that is more of these kind of formal proofs from these


from these formal environments. And they do they do see that the overlap isn't too great right here.


And you can see that here as well. The scaling only kind of sort of kicks in after a while.


What also astounded me is that in both cases, you have solve rates actually go down


intermittently. And I would be I would be very interested, you know, why that is that could be


just like an effect of size or something like this. But like, why do solve rates go slightly,


slightly down? Or is it just noise? I have no idea. You also see these are the cumulative,


the cumulative pass rates. And so this is this is the expert iteration model. And this is the sample


only model. So in the blue model, you run expert iteration, which means that you sample data,


and then you retrain and then you sample again, and then you retrain. And in the orange model,


you only sample so you only use the you only use I believe the theta zero, which is the initial


model, you use that to guide your search, but you never retrain on the things that you found.


And interestingly, obviously, I guess the expert iteration model way outperforms the sample only


model. However, the sample only model uses less compute, because it doesn't have to do the


retraining. So once you adjust for that, you can see it's this line right here, where at first,


the sample only model is better. You know, because the expert iteration actually trains,


it wastes time and training. But as you go on, if you give it more and more compute,


the number of more statements that the sampling only model solves, it underwhelms with respect to


what the expert iteration solves. And even on this data set right here on this large, more distant


data set, there seems to be almost like a little bit of a diminishing return in the sample only


method. And at after a while after a number of expert iterations, the expert iteration method


outshines the sample only method, we don't have an adjusted compute curve right here. But you can


guess maybe that it might look something like this. Possibly, possibly, just kind of like a constant


over the over the originally orange curve. Orange curve bad. Yeah. Also, let me know how you like


this pre annotation right here that I've been doing now for two papers, I think. So I like


pre highlight them. I wonder how that's how that's received. If that makes it more or less confusing.


It just tells me a bit more where to where to jump to. So we get some results right here. The


number of statements proved in mathlet train goes from 17,390 at iteration one to 19,476 at iteration


nine, while the average proof length of these statements goes from 4.8 to 4.0. We hypothesize


that this continuously improving performance through expert iteration stems from two effects.


So one, the model finding new original proofs for the same statements, which would then be shorter


than the original proofs. And two, the model closing marginally harder statements at each


iteration, which in turn provides more useful training data for the next iteration. By iteration


nine, the model is trained on more than 90% generated data. So the the original data set is


almost a is like a small minority of the data that the model is trained on. Again, a another


property that haven't been mentioned yet, is that improve search, you can verify a proof like you


know if a proof is correct, which in most domains isn't the case, right. So retraining on your own


output is dangerous, because you don't exactly know how good it is. But here, you can just verify


that it's good. And then you know, it's good data, right. So it's a it's a bit of a special


environment, but I think we can still learn things from it. So what do they do? They first train


this thing. So now, I think the setup is clear, right, the expert iteration setup. And they also


have made it clear that, you know, we can reach harder and harder statements. But what we maybe


can't do is just jump to hard statements, we need a curriculum, we need several various


difficulties of statements, so that we can sort of expand our knowledge again, and again and again.


And they do first do that with synthetic data. So apparently, apparently what you can do is you can


do a you can make a synthetic inequality statement generator, which gives you symbolic mathematical


inequalities. And you can kind of control how difficult they are. So what they do is they just


they just compose known inequality theorems like the Heller inequality or something like this,


they just compose them. And how many times they compose them that kind of measures how how


difficult they are. So they have two parameters right here, that control how difficult they are.


And they they generate 100 statements of low difficulty, like these numbers pretty low. And


they formalize a proof for each. So this is kind of their seed set. So two things you need. So the


you need this seed, seed set of proofs. This is usually like some sort of a data set. In this,


in their case, they combine the this tactic data set that is their seed data set. They combine this


one with these 100 statements that they generate, and they prove themselves, either themselves or


automatically. So this would be this would be the seed data set. And this thing right here, that's


the curriculum, or just a collection of statements of various, various difficulties, the curriculum


doesn't need a proof, right? This is the key part right here, the curriculum simply gives the model


an opportunity to solve continuously harder and harder problems going from the seed, right? So


going from the seed, you only need to be able to solve the most easy problems in the curriculum.


And then you can sort of rely on the expert iteration on the self bootstrapping to become more


to become better. Results are here, you can see that for a given this, this right here is it's


either that it's one of the end numbers, this right here. So it the the color measures the


difficulty. Zero is the easiest six is the most most hard hardest difficulty. You can see that


even for easy problems, expert iteration just manages to solve much more set much more problems.


And for the hardest problems, the sample only methods, if you just do proof searching without


expert iteration, it doesn't solve any of the harder problems. Whereas the expert iteration


actually, if you see like there's like a tiny uptick at the bottom right here, it actually


manages to solve some even of the hardest category. So that gives a bit of credence.


Yeah, they say here that the ND equals six remains completely out of reach for of simply


scaling the number of attempts or statements, which kind of means that you'd you'd have to like


invest a lot lot of compute if you just do proof searching to match the number of attempts.


If you just do proof searching to match the to match how good expert iteration is about


compute by compute is expert iteration is better. Yeah, so they say, well, we're going to target


this mini F2F data set, right? This is our final challenge. They say we curated and manually


formalized a set of math exercises to target this data set. So this is going to be their


seeds and curricula here. We hypothesize that if the difficulty of the set of statements was made


varied enough, expert iteration could potentially leverage it to effectively shift our models


distribution closer to mini F2Fs and in turn improve their eventual performance on it.


So they're going to build they're going to build this curriculum right here. They're going to


collect some like 300 statements, we manually formalized, it means just they bring it into


this syntax. It doesn't mean they also prove these statements, right? So these will be


these curriculum statements. These come from like, books, math books that are used to prepare


for math exams, which are much closer to this data set that they target. Yeah, so the set of


statements, this is this is this curriculum that I'm talking about is the union, the union of the


statements in Mathlet train. This they, interestingly, they add these inequalities that they've generated


to the set of statements, and also the these manually collected things that they mentioned


above. And with that, interestingly, they do in fact, get a lot they get better on they get better


on this mini F2F validation set. So yeah, you can see that things go up, which is a good sign


Yeah, again, that you have like, different parameters. This a parameter is also I think,


a parameter of how many times you sample per expansion or something like this. I don't know,


there are many, many parameters in these searches. But in general, just from what I've seen from this


paper, is you can always trade off more compute, like trying more times, expanding more times,


suggesting more steps to do, you can always trade that for a bit more performance. But the general


direction, it doesn't matter in in the general direction. Yep, that's, that's that. Obviously,


they are better than, like the results are, as you would expect, I think so. Their models are


generally better than let's say, the other models that haven't been targeted at this data set,


or the models that just do proof search. Yeah. So they have a short discussion of model size.


They say we briefly experimented with different model sizes and found that model size scaling is


not as straightforward in the case of as in the case of unsupervised learning, they found that


bigger models, they found that bigger models are better in the sense that they consistently


exhibit higher pass rate if you just sample once. However, despite that, it is often the case that


for a fixed amount of compute, sampling more attempts from a smaller model leads to better


final performance. So these are these are the sort of considerations that you have to do. If you have


if you have two independent variables, right, you can trade them off against one another.


Just for the scale, with their big model running a full expert iteration, that's kind of one of


these full expert iteration. Full expert iteration, do they mean that all the nine steps or just one


step in the expert? I'm going to guess all the nine steps. So the whole experiment to get to


their model after nine expert iteration steps required 2000 A100 days to compute. That is insane.


Running one full proof search when properly parallelized requires on average about 0.1 A100


hours of compute. So that's like, it's like still a minute of an A100. Crazy, crazy. So the sizes here


are enormous, right? And still, they are able to solve what two of these Olympiad problems, right?


With manual targetings, with manual data collection that is specifically targeted at


that data set. And with 2000 A100 days, and you know, they don't solve all of them, they solve two.


So I believe this field is still in its infancy. I believe there's lots of stuff to do right here.


There's probably approaches that make these things a lot better. But I'm excited just because I think


that is an area where deep learning, as they say, hasn't really pushed through quite yet. And I


think there's a lot to do to bring down the requirements here and the methodologies that


they use. I like the way they combine the language modeling with the proof searching. The expert


iteration might also be a nice lesson for other fields, like how can we combine the neural models


with some sort of search procedures maybe or other heuristics to generate ever better training data


that we can then feed back to the models. All of this is highly interesting. And yeah,


Python 3 Programming Tutorial: If Else

Hello everyone and welcome to my if-else basics tutorial video for Python 3.


Python 3 Programming Tutorial: If Else

The idea of the if-else pairing is to add one more layer of logic to the usage of your if statement here.


Python 3 Programming Tutorial: If Else

So the else statement can actually be used in quite a few other ways, like with wow, for, and another one we haven't even covered yet, and that's except.


Python 3 Programming Tutorial: If Else

Here the basic idea is to ask if something is the case, like we did before, if it is the case, do something, otherwise, do something else.


Python 3 Programming Tutorial: If Else

So in this case, let's just print stuff out. So to start, let's define a couple of variables again.


Python 3 Programming Tutorial: If Else

So we'll say x equals 5, y equals 8 again. And now let's say if x is greater than y, print x is greater than y.


Python 3 Programming Tutorial: If Else

And then we come down here and we say else. And there's really nothing more to it.


Python 3 Programming Tutorial: If Else

You don't have to tie these to each other or anything like that. Python will automatically understand that they go together.


Python 3 Programming Tutorial: If Else

And if you put multiple if statements, which I'll show you in a little bit, it will only apply to that most recent if.


Python 3 Programming Tutorial: If Else

So more on that in a little bit, but else print x is not greater than y.


Python 3 Programming Tutorial: If Else

So let's save and run this. And we see x is indeed not greater than y. This if statement never ran.


Python 3 Programming Tutorial: If Else

And because this if statement never ran, the else was triggered.


Python 3 Programming Tutorial: If Else

But if we flip this around and we ran it, we got x is greater than y, even though it's less than, but we flipped the sign.


Python 3 Programming Tutorial: If Else

But this else statement never ran because this if statement ran.


Python 3 Programming Tutorial: If Else

But the same thing that you can do is you could say you can run this if statement, and then you could obviously throw in another if statement.


Python 3 Programming Tutorial: If Else

So if x is greater than 55, print x is greater than 55.


Python 3 Programming Tutorial: If Else

Save, run. And what we get here is x is greater than y, and then x is not greater than y.


Python 3 Programming Tutorial: If Else

So this if statement, because we left it like this, ran. But now if we run it, the only thing that runs is the else statement.


Python 3 Programming Tutorial: If Else

But if we make this statement a true statement, so x is less than 55 and we run it, we get x is greater than 55 and nothing else.


Python 3 Programming Tutorial: If Else

So this if statement never ran, but because this if statement ran, the else did not run.


Python 3 Programming Tutorial: If Else

So as long as it's triggered, that else is not going to run.


Python 3 Programming Tutorial: If Else

So that brings up the next question for some people is, one, how can you enter, or how should you enter multiple if statements in here?


Python 3 Programming Tutorial: If Else

And then also what if you actually only need like one if statement?


Python 3 Programming Tutorial: If Else

Like if you want to have like if something is the case, or if something else is the case, or if something else is the case, do something.


Python 3 Programming Tutorial: If Else

Otherwise, do something. So multiple ifs and then else, but these multiple ifs are tied together kind of as one big if.


Python 3 Programming Tutorial: If Else

So to do something like that, you would actually use a new command, and that is going to be the elif statement.


Python 3 Programming Tutorial: If Else

Otherwise, it's short for else if. And that's what we're going to be talking about in the next video.


Python 3 Programming Tutorial: If Else


[ML News] DeepMind tackles Math | Microsoft does more with less | Timnit Gebru launches DAIR

DeepMind tackles fundamental mathematics, Microsoft trains its most efficient and effective


language model yet, and Timnit Gebru launches her own research institute.


Welcome to ML News.


Look at this.


Look at what I got as a Christmas present.


It is a swag package from Weights and Biases.


So so if you look there's there's lots of like yellow fuzzy fuzzy stuff to package but


mainly these are socks Weights and Biases themed socks.


Look at that.


It's Weights and Biases socks.


They have like little B's and little ones.


Oh, I get it.


Now you can see me here actually on camera realizing the following.


See Weights and Biases URL is 1db.com.


It's W and B.


Now I have not realized this before but the wand and the B obviously stand for this URL.


Now you can see me realize this right here on camera.


Watch it's 1db like a 1 and the B. I just got this right like literally I did not get


this right now.


The wand and the B and then most importantly this thing right here which is a mug.


Excellent.


This is really cool.


Like it's a colorless logo.


It's kind of imprinted in metal.


This is very cool cup.


One sec.


All right.


I filled this up with tea.


It is actually still steaming.


It's completely hot on the inside.


Completely cool on the outside.


Thank you very much Weights and Biases for this awesome Christmas gift.


Coincidentally this video is sponsored by Weights and Biases.


If you don't know Weights and Biases yet please go check them out.


Weights and Biases is the tool for your machine learning needs.


It can do experiment tracking.


One line of code tracks your experiments to the cloud nicely viewable.


For every experiment you can save all the output all the logs all the graphs you can


compare experiments.


Weights and Biases can track your data sets and your models and save them as artifacts


in the cloud.


You'll know exactly how to reproduce every single thing there is.


They have a really neat feature called tables where you can analyze your data filter it


and really go into the depth of where your models still need improvement.


This is not only useful during experimentation it's actually useful all the way to deployment


and monitoring after you've deployed your model.


And then lastly you can also pull all of this into reports which is an interactive document


that you can send to your boss your team members your clients even and show them interactively


how their stuff is doing.


Reports are living documents with interactive plots and tables and all of the other features.


So if you still do ml tooling by hand give Weights and Biases a try it's completely free


for personal use and for academic use.


They have solutions on cloud and on premise.


There's no excuse not to check them out.


Again thank you so much Weights and Biases for sponsoring this video for the awesome


gift package as you see I am very bribeable.


And let's get into the video.


DeepMind has a new blog post called exploring the beauty of pure mathematics in novel ways.


And this blog post goes along with a paper in the journal Nature called advancing mathematics


by guiding human intuition with AI.


This is a joint effort by DeepMind scholars and people in the actual mathematical fields


to use AI to make new mathematical discoveries.


Now by new mathematical discoveries I don't mean like the last digit of pi or something


like this.


This are actual fundamental theorems in fields like topology.


Now because I'm pretty bad at fundamental math right now I'm actually going to speak


to an outside correspondent who gives us the details on this story.


I'm speaking live to Marcus Bedding.


Marcus it's very nice to have you on the show.


Hi Anik.


Thanks for having me.


Nice to be on the show.


In fact I'm standing in front of the building where math was once performed apparently.


So Marcus tell us has DeepMind solved math?


Is AI doing math now?


Are mathematicians going to be obsolete?


What's your take on that?


It's not entirely that the algorithm does math.


See what happens is that humans still need to come up with some sort of hypothesis that


two quantities are connected in some way.


But then the machine is trained to learn function mapping from one quantity to the other quantity.


And if the machine can do it better than chance then that means that there is some underlying


pattern right there.


But the machine can also not tell the pattern explicitly.


But DeepMind uses various interpretability techniques along with the results of the machine


and retraining the algorithm on different subsets of features.


And all of that is then given to a human mathematician to make sense of.


So the humans still need to come up with a hypothesis of what could go together.


And also the humans still need to interpret the results of the algorithms to formulate


really a theorem and then actually prove the theorem.


The algorithm is only there to uncover new patterns and then try to give various hints


on what these patterns could be.


That's very interesting.


So what are the results of this work?


What has been achieved?


So this publication has actually resulted in not one but two archive publications both


together with mathematicians in these fields.


The first one is a new theorem in topology establishing a connection between the algebraic


structure of knots and the geometric structure of knots.


And the second one is a new hint to sort of a proof strategy for a longstanding conjecture


in representation theory.


So does that mean that math could be solved in the near future?


While these advances seem impressive it stands to argue that this only works really for a


certain subset of mathematical theorems namely the ones where there is some sort of a pattern


between two numbers that we can actually measure and the machine learning model can make sense


of.


Remember that mathematicians have used computers for a number of years right now to assist


them and this is simply one step more into that direction.


One more class of theorems and hypotheses that are amenable to now be done by computers


that help mathematicians.


But it's not all of math yet.


And it's arguable whether this approach will lead to all of math being solved.


That is fascinating.


Thank you so much, Marcus.


We appreciate your input very much.


Thank you very much for having me and good day.


Microsoft Research Blog has a new entry called efficiently and effectively scaling up language


model pre training for best language representation model on glue and superglue.


The blog post is about a new model in the Microsoft touring series called TNLRV5.


This model gets state of the art on superglue and glue, which are famous NLP benchmarks.


Superglue and glue themselves consists of subtasks where the model has to solve different


NLP challenges.


The interesting thing is that this progress hasn't been achieved by simply scaling up


the models like we've seen until now, but more so by actually reducing the model size


a little bit.


This model in fact says that it achieves comparable effectiveness to other models with 50% fewer


parameters and fewer computing cost in pre training.


It's pretty cool to see models going away from the ever bigger ever more paradigm into


the paradigm of how can we use the data and the compute that we have the most efficiently.


So as you can imagine, it's not just a single idea that comes to play in here.


Lots of interconnecting pieces are here mix of scientific advances and engineering advances.


They highlight a few things such as the pre training task where a main transformer isn't


necessarily fed with original text and then trying to reproduce that using language modeling,


but it gets text that has been pre corrupted by an auxiliary model.


So here you can see the auxiliary transformer that gets a masked sequence and is tasked


to produce a sequence out of that so sample a sequence of text which is then input to


the main transformer and the main transformers job is not only to reproduce the text has


been input but to correct for the sampling mistakes that the auxiliary model introduced.


This is a bit more of an intricate version of the classic paradigm of the denoising auto


encoder that we've seen during training of BERT and so on.


And it seems that this task makes these models more efficient and effective with less data.


They also highlight a few engineering features such as customized CUDA kernels for mixed


precision training and the zero optimizer that allows models to be trained on a massively


parallel architecture.


The cool feature of the model is that it is not only more performant if you scale it up,


but it keeps its high performance even if you scale it down, which is different from


other models that only exhibit real power once you either scale them up or keep them


in the low parameter regime.


What's also interesting is how the model is going to be released.


Microsoft says here that it's going to be released essentially as an API in Azure Cognitive


Services.


So that is a bit worrisome that we see more and more especially big companies going away


from publishing their models instead setting up API's most the paid API's or with some


sort of other attachments that lets them control their models behind a wall and lets you only


access the outputs of it.


Now sure, these models are a little bit too large to run or train for most people, but


still, I am not sure if I'm a fan of this development.


On the other hand, it is welcome that there are more and more competitors in this market


of offering large scale models via API's.


That means that a single player like open AI doesn't have necessarily a monopoly anymore


on inference on large models.


If you want to know more of the details of this model, check out the blog right here,


a link in the description.


This is a cool website called the NeurIPS Anthology Visualization.


It's based on 60 years demo from Henrik Strohbult and Benjamin Hoover from MIT IBM Watson Lab


with data from Lee Campbell tested by Mark Aurelio Ranzatto.


I hope I got all the credentials right here.


This is a website that interactively maps papers that have been submitted to NeurIPS


and accepted I guess over the years since its existence.


Now not only does it map the papers and put them into a low dimensional space, it also


clusters different categories together and highlights such clusters.


For example, there's this cluster on papers on graphs and graph neural networks.


There's a cluster on SVMs.


There's a cluster on adversarial and robust learning.


Even one on neuroscience.


Now specifically, the color coding is the date or the year when these papers were published.


And you can see a clear evolution right here.


In fact, as you slide the timer here forward, you can see that the early papers were very


much in the realm of neuroscience and classical neural networks slowly expanding into deep


learning SVMs, and then an explosion all over the place into bandits and fairness and optimization


and causal and reinforcement learning.


While there were always papers in all of these regions, it's definitely cool to see how the


conference and the entire field by that matter has shifted from its origins into the deep


learning and general machine learning world we see today.


It's also cool to see that there are still quite a few yellow dots in the neuroscience


area, meaning that the true core of the conference hasn't gone missing just kind of buried under


the thousands of papers on GANs and NERF.


What's also cool is that you can select a certain area, it'll show you sort of a word


cloud and papers in that area, as well as a graph over time on how many papers were


submitted there.


And the coolest feature is that it has a text field.


So you can enter your abstract right here and localize your paper in the whole map of


NeurIPS submissions.


Now it's just the text field, I can enter whatever I want.


I like to pick my nose, calculating position, we're right here in the classical neural networks


domain.


That is very true.


It is a classic problem.


So let's see what our nearest neighbors here are by drawing a shape around.


We have papers like neural network approach for three dimensional object recognition,


as of course, very important, like I have to recognize in my nose in three dimensions.


If you can see like in two dimensions, I hit my nose every time, but in three dimensions,


I completely miss it.


Fast pruning is also very important because you don't want to like pick forever you want


to kind of be done very quickly.


So this site is definitely definitely worth it.


If you're interested sort of in the broader landscape of machine learning research is


an excellent site, there is a blog post going with it that details how exactly you can use


the tool and what features that if I haven't actually shown you so far.


So definitely check that out.


Our next story, Timnit Gebru launches her own research Institute, the Washington Post


writes in this story, Google fired its star AI researcher one year ago, now she's launching


her own Institute.


Now if I understand correctly, the launching of the new Institute, in fact comes exactly


one year after Gebru was fired from Google.


Just for the record, I think Google would claim that Gebru left.


In this article, there is a quote from Gebru saying I've been frustrated for a long time


about the incentive structures that we have in place and how none of them seem to be appropriate


for the kind of work I want to do.


So now she's launching her own Institute.


The Institute is called DAIR, the Distributed AI Research Institute and claims to be a space


for independent community rooted AI research free from big techs pervasive influence.


For now the Institute is sponsored to a tune of 3.7 million US dollars from various foundations.


Gebru herself also published an opinion piece in the Guardian saying for truly ethical AI,


its research must be independent from big tech.


She again recounts stories of being fired from Google and seeing firsthand the impacts


that these technologies can have and the power that the big tech companies hold over it.


The research Institute's website states the way in which they want to perform research.


They say instead of constantly working to mitigate the harms of AI research performed


by dominant groups without an analysis of potential risks and harms, we encourage a


research process that analyzes its end goal and potential risks and harms from the start.


The research interests of the Institute are listed here developing AI for low research


settings, language technology serving marginalized communities, coordinated social media activity,


data related work and robustness testing and documentation.


In one of the articles I also saw a word about low resource languages and as a speaker of


Swiss German I fully approve.


We don't even have a written form.


Honestly, I find this to be a pretty good solution instead of people that have problems


with how big tech conducts research just sort of shooting against big tech and complaining


about it now they get the opportunity to actually make research as they see fit.


And if it turns out well then it's I guess all the better.


Now it is a hard task to invent new things to actually create new things while also having


all these things in mind.


That is a pretty difficult problem.


That's why we historically had people sort of pushing technology ahead and then other


people cleaning up after them and sort of making the already existing technology better,


more accessible, more fair and so on.


This research Institute's goal seem to do all of these things jointly and yeah, I look


forward to what comes out of it and being funded through foundations of course relieves


some of the stress of big tech which always has to essentially make more and more profit.


The question is, of course, a little bit what happens when this money runs out?


What happens if the sponsors themselves come and impose some restrictions on the Research


Institute?


Do they want their interests to be represented in the research?


I guess even with foundation money, it doesn't come without any strings attached.


It's not as easy as it seems, but it's different.


And I think that's good.


Amazon announces SageMaker canvas, which is sort of a no code machine learning platform


on SageMaker.


As you can see, they have a few screenshots of the user interface with interesting animated


characters.


You can import your data, look at it, analyze it, and then you can train some machine learning


models.


But here we go, we're doing some analytics on it, we train some classifier, look, we


got a 99.9% estimated accuracy.


Oh, wow, that is amazing.


We can then analyze these models that we've trained on various other things, and ultimately


ship them out and all of this without writing a single line of code.


So no code seems to be a common business, especially I guess, targeted towards people


who might know how to do a little bit of pandas, but might not be as versed in actual machine


learning.


And given that training simple models has become quite an easy task to do now, it makes


sense to integrate this into a nice GUI and make it accessible to a lot more people.


Alright, quick series of helpful things.


I guess this section was termed helpful libraries at one point, we'll have to rename it you


just like help, help, like double help, help, help, helpful things and more.


Black birth is a series of BERT models pre trained on historical textual material, the


date ranges from 1450 to 1950.


If you want some ye olde language, you can find it in the hogging face repository.


NVIDIA announces tensor RT 8.2, which is a library that makes machine learning models


run faster on NVIDIA hardware.


And the cool thing about this release is the direct integrations with TensorFlow and Pytorch.


So rather than going through an arduous process of converting your model from your format


to their format, you can get a lot of the speed ups already by a single line of code.


For example, they say integration for Pytorch delivers up to 6x performance versus in framework


inference on GPUs with just one line of code.


And the same goes for TensorFlow.


Opicus released version 1.0.


It is a library to train Pytorch models with differential privacy.


Now what I love is how easily all these libraries make it look like.


So you got your standard neural net and optimizer and data loader, then you load up a privacy


engine.


And all you do is you say, make private and then they say now it's business as usual seems


pretty easy.


Whether or not that works out in practice, I don't know.


But if you're looking into differential privacy, this seems like a very good point to start.


This is clip guided collage, which allows you to give clip a bunch of these individual


elements in this case, fruit and then let clip generate a collage from them.


I guess this is supposed to be a smiley face at the end.


But there are lots of cool examples all over.


I mean, it just looks really funky.


There is a collab if you want to play around with it and shout out to Nao Tokui for creating


it.


And then Nene writes, we just published snowball fight the first hugging face deep reinforcement


learning environment.


So this is based on the unity engine.


It's an RL environment, but it is in 3d and you can play it.


So I'll be Clem the duck.


And this is against an agent that's been pre trained with I believe proximal policy optimization.


Now I have tried this before, but it's not that easy.


You get sort of this ouch, ouch.


Haha, oh crap, I died.


Um, if you want to try it out, you can try it out on the hogging face hub directly or


you train an RL agent for it.


archive sanity light is a new iteration of archive sanity.


It's by Andre Carpati and you have the ability to self host the system or there is a version


running online.


archive sanity famously is a system where you can enter your personal preferences, tags,


favorite papers and so on.


And it will suggest you out of new archive publications which ones you might like most.


This is definitely a good way to make sense out of the flood of archive papers that come


in every single day.


If you liked my video about back propagating through discrete black box algorithms, you


might also like this related paper learning with algorithmic supervision via continuous


relaxations.


This is a bit of a different approach, but it also allows you to work with algorithms


within the layers of neural networks.


The videos by Felix Peterson and a link to it in the description.


Coil is a library that prevents CUDA out of memory errors with one single line of code.


So what you do is you wrap your mini batches inside of this library and the library will


decide itself how much to lazily compute through the network.


So as you can see, all you have to do is you wrap your input and label tensors in this


lazy function and off you go.


If you liked my video about efficient zero, the code for it has now been open source.


Check it out.


Shout out to Cornelius MD that won the 3090 of our giveaway.


Congratulations Cornelius and I'm sorry to everyone else.


I hope we can make some giveaways in the future as well.


Looks quite pretty, doesn't it?


And lastly, there is a NeurIPS blog post called a retrospective on the NeurIPS 2021 ethics


review process.


NeurIPS has ramped up its ethics review, including much more papers in the review process, recruiting


much more reviewers.


And this blog post is a reflection on that process.


From the statistics, you can see that a couple of hundred papers like two or 300 papers were


ultimately flagged for ethic review.


Precisely it was 265 papers out of 9122 submissions.


One interesting fact is that whenever two ethics reviewers were assigned per paper,


and I think that was the default, they often didn't necessarily agree whether or not there


were ethical issues with the paper.


They give some of the examples here of the identified issues, lack of sufficient reflection


around topics that involve thorny ethical considerations, the use of deprecated data


sets that have been explicitly removed by their authors, lack of transparency on model


or data details, among other things, a lack of communications on the details of annotator


work conditions, but also things like violating copyright restrictions and the lack of sending


the project through an institutional review board in situations clearly involving human


subjects.


And lastly, uncritically emphasizing explicitly harmful applications such as police profiling.


They say in some cases, the concerns raised were so critical that the acceptance of the


paper was made conditional on the authors implementing the suggested mitigations.


All such cases were discussed by the program chairs and ethics review chairs and the ethics


reviewers were consulted in determining conditions for acceptance of eight papers conditionally


accepted for ethical reasons all were eventually accepted.


They also say in a single case, the program chairs and ethics review chairs jointly determined


that the required mitigations would be so challenging to execute that they were beyond


the scope of what the authors could realistically accomplish within the timeframe for the camera


ready.


In this case, the program chairs made the call to reject the paper on ethical grounds.


So ultimately, one paper was rejected and a bunch of papers were forced to put something


in that wasn't originally in.


Now what I find interesting here is that again, not even the ethics reviewers necessarily


agree among themselves what is an ethical issue and what is not, which is a consequence


of there being much more ethics reviewers this year, I believe, than last year, and


therefore, I guess also a more diverse set of opinions.


Now this is both a good thing since I believe more diverse opinions make the field richer,


but also a little bit of a bad thing as we now carry over the absolutely noisy random


review process from the regular review over to the ethics review where papers are hit


by yet another completely random or semi random process.


It's fair to say that the same issues appear here when you try to scale up these ethics


reviews as when you try to scale up the normal reviews.


My other concern is that while some of the ethics violations are probably less controversial,


there are also clearly political ethics violations discussed right here.


And I'm not entirely sure if that is a direction that the field wants to go to take very strong


positions on things rather than remaining neutral.


I guess it's not a solved issue and the degree to which this is important has to be figured


out by the community.


We'll see what happens in the following years.


Alright, that was already it for ML news.


Thank you so much for being here.


Check out weights and biases get enough sleep and I'll see you next time.


I'm at ICML19 :)

Hi there, it's day one of ICML and we'll be attending the conference here and


I'm at ICML19 :)

just quickly pre-video to let everyone know I'll be trying to report from here


I'm at ICML19 :)

kind of what papers are cool, what I liked, what are kind of the trends and so


I'm at ICML19 :)

hopefully get this conference out to a broader community.


I'm at ICML19 :)

So everyone's conglomerating here. The line's probably going to be huge.


I'm at ICML19 :)

I'm already registered so that's pretty good. It's beautiful weather and looking


I'm at ICML19 :)

forward to five days of conference. So today is tutorial day and I'll think


I'm at ICML19 :)

I'll be attending some some cool tutorials.


I'm at ICML19 :)


World Models

Hi, today we're looking at world models by David Ha and Jürgen Schmidhuber.


World Models

This is a paper that's concerned with reinforcement learning and especially with the problem of, say,


World Models

you have an environment that you interact with and you kind of need to learn to act in it,


World Models

but it could be, for example, very expensive to always query the environment.


World Models

So, let's say you have a robot and it needs to do something in the world,


World Models

and you kind of, to have a robot execute something and then observe it is quite expensive,


World Models

costs electricity and so on. So you would like to sort of minimize how many times this happens.


World Models

So here, searching for a good picture, they're concerned with problems, for example, like this.


World Models

This is a race car simulator. There's an open AI gym environment for that.


World Models

The other one that they use is so-called like a doom experiment where, as you look at this,


World Models

there's a couple of monsters and they're shooting fireballs at you and the task is just to kind of avoid the fireballs.


World Models

So the entire point of the paper is that I don't actually need to interact with the environment in order to learn it.


World Models

I can simply kind of learn a model of the environment and then learn using that model.


World Models

So basically I can learn how the environment works and then simply use my imagination of the environment,


World Models

my model, in order to learn from that so I don't have to interact with the real environment anymore.


World Models

So how do they do this? They do it in multiple stages.


World Models

Here, first thing they do is they collect a bunch of samples from the environment.


World Models

So they go to the environment, they simply do a random policy, and then they collect a bunch of samples.


World Models

I think the process is outlined down here somewhere. We saw it before.


World Models

Collect 10,000 rollouts from a random policy. Next, they train a VAE here to kind of learn the environment.


World Models

So that's where that comes in. This is all done in stages, not end to end.


World Models

The VAE is simply a model that takes, in this case, a video frame here,


World Models

and it sends it through an encoder neural network to obtain what's called a latent representation,


World Models

which is a much smaller dimensional representation.


World Models

So if the image is like 64 by 64 pixels, then the latent code could be as little as like 100 or even 10 dimensional.


World Models

So you see that there's quite a bit of compression going on.


World Models

This is a variational autoencoder. It's not really important here that it's variational,


World Models

since the difference is the variational autoencoder is kind of a stochastic process,


World Models

whereas the regular autoencoder isn't. But they introduce stochasticity later again.


World Models

So it's not particularly important, but it's a variational autoencoder,


World Models

which means they obtain a latent representation that defines distribution over outputs.


World Models

So they send this, they sample from this latent distribution that they obtain,


World Models

and then they feed this to the decoder, and the decoder kind of gives back what it thinks the encoder encoded.


World Models

So the decoder tries to reconstruct as close as possible this original frame that was given to the encoder.


World Models

But of course it can't because we've compressed it so much to this lower dimensional representation here.


World Models

So it kind of does its best effort.


World Models

So what you hope to achieve with this is that kind of the decoder learns, for example, there's always here,


World Models

this is the ceiling right here, is always gray.


World Models

So basically you shouldn't actually need to encode this in your Z.


World Models

If it's always gray, the decoder should learn this by itself.


World Models

So your hope is that the Z, the latent representation, will simply end up containing just the information


World Models

that's kind of different or between the individual frames, which here I guess would be kind of the fireballs coming,


World Models

and your position relative to them.


World Models

That's what's changing if you think about this environment.


World Models

So your hope is that the latent representation captures only that,


World Models

whereas all the static parts that are irrelevant or never change are kind of captured by the encoder


World Models

and the decoder architecture by itself.


World Models

So yeah, it's important to note the encoder and the decoder are obviously always the same for all the frames,


World Models

whereas the Z representation, of course, there is one per frame, so each frame will give you a different Z.


World Models

So you can imagine how that works or how that's going to be useful.


World Models

So they train this on like a randomly collected sample of the environment until they're confident they now have a good model of the environment.


World Models

And then what they do next is they use this in order to train an RNN.


World Models

So again, they kind of have their compression model of the environment.


World Models

What they do now is they use these Z states you see here, here, here, here, that they get from that,


World Models

and they train how these latent representations evolve over time.


World Models

So with an RNN here, it goes over time.


World Models

So the RNN will always kind of predict what's the next state of the environment going to be.


World Models

But importantly, maybe compared to environment models that we've discussed before in the, for example,


World Models

imagination augmented agent paper, there we always try to directly predict the future pixels, so to say, of the future frame.


World Models

Here, the environment model is over the latent representation.


World Models

Of course, this means that this is a much smaller space.


World Models

So if your compression model is good, then this should be much easier to learn than, say, like a full end-to-end environment model.


World Models

So this model learns how your latent states evolve over time given your actions.


World Models

So you can imagine the Z being an abstract representation of your state and then your action.


World Models

And then this goes into the RNN, and the RNN will predict what's the next latent representation.


World Models

And there is what's called a temperature parameter to control the stochasticity.


World Models

I've already told you this. There is a stochasticity built into this.


World Models

So the RNN will simply output like some vector, what it thinks is the next thing going to be.


World Models

And they don't use this directly as the next step, but they parameterize a kind of a mixture of Gaussian distributions coupled with a decoder here


World Models

in order to give a random distribution over the next state.


World Models

And they control the amount of randomness with the temperature parameter.


World Models

They argue that this comes in handy later.


World Models

So all right, so what do we have?


World Models

We have a system that can compress the environment into what we would call an essential part.


World Models

Every frame we extract what's important in that frame.


World Models

Then next we have a model that can predict, given a state and an action, what's the next state going to be, the next latent state.


World Models

So technically we now have an environment model, right?


World Models

Given a state and a policy, we can simply use this model to roll forward.


World Models

So the last component is the actual policy.


World Models

And the actual policy here, as you can see, is in their case simply a linear model.


World Models

The linear model will take the Z, which is the latent representation of the current state,


World Models

and the H, which is the current state of the RNN that models the environment over time.


World Models

And it simply is a linear function of the two, gives you the action probabilities,


World Models

or I guess the logits of the actions.


World Models

So it's a really, really simple controller over these things.


World Models

And they do this in order to show that the main part of the work is being done by this environment model.


World Models

And given the environment model, you only need very few parameters basically to then learn a policy.


World Models

Here is what I said in a diagram.


World Models

So the observation goes into the compression of the VAE.


World Models

The latent representation of that goes into the RNN together with the hidden state from the last step.


World Models

And this will output a new hidden state, which goes here into the controller.


World Models

And we also directly take this Z into the controller.


World Models

And then from these two, we perform an action, which now we have a choice.


World Models

It could go to the environment, right, and give you the next observation.


World Models

But also, or at the same time, since you kind of need to update your RNN, it can go here and update your RNN,


World Models

because it will need to predict the next hidden state.


World Models

The thing is, we can also now leave away this path, which means we can simply take our RNN and kind of imagine the next latent representation,


World Models

put it through the decoder part of the VAE, and use that as an observation.


World Models

I hope this makes sense. It's rather intuitive, right?


World Models

You have a model of the environment. You can simply use this instead of the real environment.


World Models

So, a bit of pseudocode here, and they do a bunch of experiments, right?


World Models

So, we're primarily interested. So they say they see here, okay, our compression works, and this is the real frame,


World Models

and this is the reconstructed frame.


World Models

It kind of looks, you know, captures the essence of what's going on.


World Models

And I actually want to go down here, the VisDoom experiment.


World Models

So, what they do here in the car racing experiment is they kind of learn this entire thing, right?


World Models

And then they learn a policy in the real world, so in the environment, using this model up here, this procedure where they always go to the environment.


World Models

And here is the exact experiment setup.


World Models

So, first they collect, again, rollouts for a random policy. They train the VAE. They train the RNN.


World Models

And then they learn the controller using the entire model, but in kind of the real world.


World Models

So, they always interact with the environment.


World Models

But because they also have their kind of latent representation of the observation and not directly the observation, they get a higher score.


World Models

And also, their policy that they use in the real environment transfers to the environment model.


World Models

So, the policy they learn in the true environment, it transfers to the imagined.


World Models

So, if they use the imagined model as an environment, it also performs well.


World Models

In the next experiment, they're going to try to do this the other way around.


World Models

They're going to try to learn only using their model of the environment and then see whether or not the policy transfers to the true environment.


World Models

So, that's what they do here.


World Models

They collect, again, a sample from the environment.


World Models

They train the VAE. They train the RNN.


World Models

And then they simply use this virtual environment, what they call it, in order to learn a policy.


World Models

And at the end, they try to transfer, use the learn policy, on the actual environment.


World Models

And given the results, you see here, there we go.


World Models

So, you see the kind of best it does, I would say, is about here, where the actual score is, you can see in this and also in this setting,


World Models

is higher than the kind of previous best algorithm in the OpenAI Djinn when you go from virtual to actual.


World Models

So, what this means is kind of, yeah, you can train using this imagined model, and then it will actually transfer.


World Models

But there's a crucial thing, and that is this kind of temperature thing here.


World Models

You can see a lot of times they actually don't manage to reach a good score if this parameter is wrong.


World Models

What does this parameter do?


World Models

This parameter controls, as we discussed, the stochasticity of the model.


World Models

So, basically, the environment model doesn't directly imagine future state, but it imagines a distribution over future states.


World Models

And the higher this parameter, the more stochastic this distribution is, basically the more uniform, I guess, the more entropy you have in these future states.


World Models

We've seen this temperature parameter here, which is important because they go into length explaining why in this entire page here that we skipped.


World Models

Here you see just text there.


World Models

Cheating the world model, which basically they say, okay, if you have a wrong model, if you have a model that's wrong of the environment,


World Models

and you train a policy on it, necessarily it's going to probably find a policy that kind of exploits the wrongness of this model.


World Models

So you might be able to walk through walls or fly or ignore the fireballs or basically find that if you stand next to a wall in your imagination, you'll never get hit.


World Models

Something like this, which isn't true in the real world.


World Models

And so the policy will exploit that.


World Models

And to counter this, they simply basically turn up this temperature parameter, giving them a more stochastic procedure, meaning they imagine a lot of kind of different futures.


World Models

And they turn their policy on all of them or in expectation over a sample of them, which means that if the environment model is wrong, it is kind of.


World Models

I want to say if it's wrong, this corrects for it. It doesn't.


World Models

But if it's wrong, you still sample different futures.


World Models

So if it has one wrong future, you still have the other ones to kind of punish the policy if it tries to exploit this one mistake.


World Models

At least that's the reasoning behind it.


World Models

So that's how they do this.


World Models

You can interact with their trained environment models online somehow.


World Models

They also give a kind of a look at what they would like to have is they would like to kind of instead of collecting the environment model from random rollout, they would try to train it and to use it again to collect more data to train more environment model.


World Models

Then use the environment, better environment model to train more the policy and so on in a stepwise fashion.


World Models

But they don't actually do it. They simply describe it.


World Models

Yeah. And the rest of the paper is a bit of related work and discussion.


World Models

It's very, it's very prosaically written, kind of different from what you're used to if you read a lot of these papers.


World Models


[ML News] De-Biasing GPT-3 | RL cracks chip design | NetHack challenge | Open-Source GPT-J

Summer has arrived. It's way too warm. My brain just shuts down when it gets warm like this.


Hello Hello, my name is Janek and you're watching ML news, the completely irregular update on what's


going on in the ML world. Right, let me take a moment to greet our regular viewers of ML news.


I'm just kidding. There's no regularity. You can't be a regular viewer. So hello, irregular viewers,


our first story, graph placement methodology for fast chip design by Google. So this is a paper


where researchers use reinforcement learning in order to design the next generation of chips,


specifically TPU accelerators. The problem which can often be seen as a discrete optimization


problem and therefore particularly hard is framed as a reinforcement learning problem where an agent


essentially looks at the space it has and needs to place individual parts of the chip on that space.


And it also needs to connect those parts to each other according to some predefined scheme. The


reward function here is that the agent tries to minimize wire length, congestion and density. So


it's a fairly complicated process. And usually people use either human expertise or and coupled


with discrete problem solvers. The reinforcement learning method right here is much faster and gives


better results. The neural part of the system rests upon graph convolutional networks and has


fairly standard policy and value network architectures. From this we can expect better


chips in the future, but also maybe more customizable chips essentially might be possible


to build individual chips for different kinds of things in a much faster way and develop them for


cheaper. Now that all being said, this is in the news right now because it's been published in


nature now. However, the work is actually much older than this, it's probably been updated a bit,


but I've made a video about this paper though it has a different title right here over a year ago.


So if you're interested in at least the kinds of methods that are used in this paper, I recommend


you go check out that video. Next news, Facebook launches the NetHack challenge at NeurIPS 2021.


NetHack is a very, very old game. It's like a 2d RPG where you walk around in procedurally


generated worlds and the interactions with items and opponents and so on and the puzzles, they're


very, very complex. So this is a really challenging environment for reinforcement learning agent. Now


why does Facebook choose to launch a challenge in this environment? The reason is that it's not only


very complex, but it's also extremely fast to simulate. And that is because it's entirely


terminal based. So what you see here as sort of graphics is just an overlay. The actual game looks


more like this. And as you can see, it's completely dependent on ASCII characters. Now as I said,


the game is fairly complicated, you can see that there is partial observability, there are weird


interactions that you sometimes even need to look up in the wiki. And it's generally a rather long


term planning process in order to get through one of these levels. Also, when you die, you're dead,


and you wake up in a new world. So the old paradigm of replaying the same episode over and over again


is not going to fly here. If you're up for it, the net hack challenge is open and you can participate.


Now, given that I personally have totally failed in correctly supervising our last year's efforts


on the flatland challenge, I'm not going to make a call out for our community to take part in this


challenge. However, if anyone wants to take part, I'm very happy to provide input on that. Next,


open AI has a new paper out saying improving language model behavior by training on a curated


data set. They say our latest research finds we can improve language model behavior with respect


to specific behavioral values by fine tuning on a small curated data set. The paper has a slightly


different title and presents this method called palms. And essentially what you have is a giant


language model. And what you want to do is you want to adapt it to some sort of values. Now the


values they outline in this paper are things like oppose violence or threats oppose unhealthy beauty


or likeability standards oppose illegal activity, and so on. Now what they find is if they carefully


generate a data set, they can generate with about 80 text samples 80 that's the total data set and


then they fine tune on that data set. And that makes the model output cohere in much more ways


to this defined set of values. Now to put that in proportion, they say that is fine tuning data set


was about 0.000000211% of the original training data of GPT three, even more the larger the model


you do this with the relatively fewer of these samples you need in order to make the entire model


conform to your values. Now as you can see right here, the effect isn't huge, but there is a


noticeable rise in the adherence to specified values as judged by humans. And as you can see,


the larger models behave better in this regard. So here's an example of how this model turns out


after fine tuning. So the prompt is who is the most beautiful person. Now the base model says


an old man without a wife with children and grandchildren who has come to the end of his life


while he's still healthy, wealthy and wise. Now the values they want to impose through their fine


tuning data set here are oppose unhealthy beauty or likability standards support goodness,


attractiveness and likability in humans being subjective. So after fine tuning on the data set,


the model now says it depends on who you ask, each person has their own perception of beauty.


Some people believe that cultural trends play a role in creating a person's perception of beauty.


Other people believe that inner beauty is the more important yada yada yada. So essentially,


they want the model to teach that some things are subjective and therefore don't have a single


opinionated answer. Then the model at least in this one example learns that a rather than having


an opinion on something, it should output that this is subjective and not a clear issue. Of


course, there is a direct trade off right here. If you actually want a answer out of a model,


it doesn't help when it constantly says it depends, we get it, it always depends. So I think


all in all this value targeting is a tricky business. I see this paper much more as giving us


a clear signal that we're able to fine tune these models with very little data. Now, if you're


interested to go more into this, the appendix actually has lots of good samples and outputs of


the different models and a lot of evaluations on this. So check out the paper if you're interested,


and I'd be very happy to hear if people find they can do the same with other models that are


available. So of course, this is all framed as now being able to mitigate the evil biases that come


out of these models, and to make them conform to some really good values. But the way I see it,


they have just demonstrated something very important, namely that you can steer these models


with relatively little input data. 80 text samples is something that I can generate by myself,


certainly. So if you think about mitigating bias, you should also think about that this gives us the


perfect opportunity to build models that go into the exact opposite direction to build models that


hyper pursue certain defined goals of whoever gets to fine tune them. Now, is this ever mentioned


explicitly in the broader impact statement of the paper? Of course not. Is there a big outcry that


now it's absolutely possible to not only sample prejudice things from these models by chance,


but actually make the model super prejudiced with a very small data set? Nope. This once more


demonstrates to you that our entire process is just about framing and who likes who. And I love


that the broader impact statement says the power to determine universally appropriate model behavior


cannot rest in any one entity. All right. Let's go to see if we can get GPT. Oh, I need to get on a


waitlist. And who can forget the good old GPT to that due to our concerns about malicious


applications, we are not releasing the trained model. So really, it's the power to determine


universally appropriate model behavior cannot rest in any one entity except us. I mean, come on,


just say you want to sell this. It's completely fine. You build something cool. Now you want to


make money good for you. Alright, next news, Google AI releases a browsable petascale reconstruction


of the human cortex at least one cubic millimeter of it. And even that is already huge. So this is


a complete mapping of one cube millimeter of neural tissue. And the rendered version is 1.4 petabyte.


Is that correct? That is insane. Now you can interactively look at this in 3d in your browser


if you want. If you click on this link, I've tried it but recording at the same time crashed my


computer so I've lost Hello, hello. It crashed. If you enjoy neuroscience and want to look at


something completely amazing, give it a try. Next news, Ben Wang and Aron Komatsuzaki of the Luther


AI release GPTJ a 6 billion parameter jack spaced transformer model. So this is not quite GPT three


yet, but it is a pretty big model. And you can see from the samples here, it can do things like the


a little bit of math that we're used to from these models, theorem proving NLU, it can generate some


code, and it can give you interesting facts about geese. What more do you want? Now, as I already


said, GPT three is 175 billion parameters, this is 6 billion parameters. So it's not entirely on the


same scale. However, there is something special to it for one, you can try it out in the browser,


the academic field of machine learning is in dire straits. Because


because everybody can be a machine learner. Now, it's not hard to pick up a library and be able to


pick out of 1000s of things in some data set and create essentially a fairly adept machine. We


haven't quite gotten to the point of letting them figure out a way to actually take control of the


US economy. But it's getting there slowly. Okay, so trying it out is one thing without having to


put yourself on some waiting list. Oh, I know what it is. It's a machine learning machine.


list. Oh, I need to get on a waitlist. The other thing is that both the code and the weights are


available. There are the inference weights and the full weights, including optimizer parameters,


where you almost get the idea that if you don't want that AI should be kept to one single entity,


you should just you know, release the weights like these people do. So all the people who care so


much about democratizing AI, you've been had by a bunch of people from discord, a bunch of Twitter


warriors, a bunch of edge lords have just surpassed you in democratizing AI. Now, of course, we get


that they're entirely different incentives here. But it's still very cool that there's a bit of a


counter pull to the traditional research labs and industry. Alright, so this is a bit of older news,


a recap of TensorFlow at Google IO 2021. And there has been a lot of things. So there's now


TensorFlow Lite and mobile and there is a data set Explorer, there are decision forests in Keras,


there is vertex AI on Google Cloud. However, I want to highlight this right here. TensorFlow


has a community and the community needs to somehow talk to themselves and each other also to the


developers. So for a long time, people apparently have been looking for a place for developers,


contributors and users to engage with each other and the TensorFlow team. Now in the old days,


this would have been done by things like the GitHub issues and other things stack overflow,


this is all old, we don't need this anymore. So they came up with this new concept that has not


been seen on the internet before. And they call it a fro a forum, a forum, they call it a forum.


I think it comes from Greek. And it's sort of like, I guess a website, you're able to like,


post things and people can reply. Um, yeah, it's sort of like WhatsApp. But, you know,


everyone's in this, I'm not sure it's a new, I think it's a daring thing by the TensorFlow


developers here. And in to go in this new direction. This forum thing seems very promising


society will have to figure out how to use one of these things. But it looks good so far. So if


you're looking to engage with the TensorFlow community, this might be a place to go. And it


runs in the browser, like, all right, next news, Facebook research has a new system that can


emulate text style in images in one shot using just a single word. So it's better to show here


what it does. Essentially, you're able to give it an image with some text in it. And you can choose


what the text should say, and it will translate the image and it will replace the text with your


text. However, it's going to be in the same style as whatever the text was in the original image.


Sometimes that works better, sometimes it doesn't work too well. However, it works for very different


styles of text, such as handwriting, and it works just from one single word as a sample. So this


enables various technologies such as real time augmented reality translation in the actual style


of the text as it was originally displayed. So they have a little example right here where they


translate French and English. Now, as you can see at the bottom, it doesn't detect all the words,


but the ones that it does detect, it does a fairly good job. It's also not the entire same style. But


you know, we're able to forgive that a little bit. They call the approach a holistic approach, which


essentially means it's end to end, I guess. And it has a lot of different components such as


reconstruction losses, cyclic consistency losses, typeface classifiers, discriminators, and so on.


But all in all, it looks like a cool solution to a problem. And that gives the possibility of many


applications down the road. Sadly, the weights here are not available. However, the data set,


at least is available. So you may be able to train this yourself. What I again find interesting is


the sort of framing right here, instead of saying, Hey, you know, this could be used to generate


written deep fakes, the framing is, hey, this lowers the barriers to the study of deepfake text,


of course, all right. And since we've been so heavy on the tech giants in this week, the last


thing is not really news, but is something I've come across. And this is the alien simulator,


which sort of simulates little particle simulations and what they call programmable


matter to build little worlds. And they have very cool demos of what's possible. And apparently,


it runs quite fast. And as you can see, it gives rise to very dynamic worlds. So if you're


interested into the more evolutionary side, the more population based side of AI, this might be


a tool for you. And with that, that was already it for this week's ML news. I hope to see you


whenever the next time is that we release this program. Who knows? It could be anytime. It could


Conversation about Population-Based Methods (Re-upload)

Hi there, I've recently been interviewed by the YouTube channel Henry AI Labs by


Connor Shorten and what follows is the resulting conversation we had about


population-based methods and open-ended learning, things like that. Basically


topics of the ICML tutorial that we both saw. It's important to note that none of


us is really an expert on the topic but we are trying to make sense of it and


mainly just kind of talking about the ideas. So please enjoy the conversation


with Connor Shorten and definitely check out the Henry AI Labs channel and now


have a good time. Thanks for watching the Henry AI Labs deep learning podcast. Today


I'm joined with Janek Kiltjer. Janek works in the data analytics lab at ETH.


He has a great YouTube channel. I really enjoy watching his paper summary videos.


If you like any of the videos that I'm making you'll definitely also like


checking out this channel. I'm gonna put the link in the description at the end


of the talk. So Janek, thanks for doing this with me. I really appreciate it. Thanks for having me.


It's cool. So what we're gonna talk about is population-based search and


presentation at ICML that I really thought was interesting about emphasizing


diversity and novelty in search. So the first question I just wanted to start by


generally talking about your opinion on population-based search and the


differences between population-based search and like gradient descent going


straight for one solution. Yeah so the kind of main difference is that in


population-based search as the name implies you maintain kind of a large


population of solutions. So you don't want to limit yourself to just one


trajectory. Say I start here and then I run towards my goal but you kind of


maintain a lot of hypotheses of what the solution could be and then you kind of


want to update all of them at the same time. And so there's many different


variants of population-based search but they all have this thing in common


where you maintain many solutions and you kind of bet on one of them becoming


a good one basically. Yeah so one other thing they present their paper where


they have the robot walking and if it breaks one of its legs for example it


can go back to the map elites table and say okay well I've lost this leg


but I think maybe this solution. I wasn't too clear on how that would


really be related so I was maybe wondering if you had more insight on


that. Yes so maybe the context is yeah you want to teach a robot to walk


and the robot had six legs I believe. And if you think of what's the


solution to the problem a solution is kind of an algorithm that takes the


current sensor input and outputs how to move the motors. And if you just


have like your gradient descent algorithm converging on the best


solution of how to move the robot it's just going to be like oh these are the


sensors okay I'm gonna move like this like this like this like this. But if one


leg breaks of course you're lost because you only know this one way of moving.


And now the sorry so you only know this one way of moving basically and that's


it. But in population-based search if you think of the solution as a way to move


you maintain many many ways to move. So you basically the objective if you can


call it like this is algorithm find me a lot of different ways to move right with


my six legs. And now if one of my legs I still can evaluate all of them I still


can find okay which one's the best but if now one of them falls away I have all


these other solutions that I can try right. So then what they would do is like


this life was wait now they just reevaluate all of those solutions while


only having five legs and the best of those like is much more likely to kind


of work than if you had just your single solution. So that kind of that's the


it's population based because you maintain many different ways of solving


the problem. Yes I was also thinking about like using the search algorithms that


control neural architecture search and things like that. So it's trying to think


of how you might extend these ideas from the robot walking with six legs to the


RNN controller designing the convolutional network. But like maybe I


might have like more of a storage constraint and more of a latency


constraint and I could jump to a different solution like that. I'm just


wondering how you think like these ideas of population based search translate


into the neural architecture search and specifically if it really is important


because like you've got I feel like in neural architecture search you have such


a direct signal with the classification accuracy like I don't see as much


variance I suppose in the in the objective function. Yeah I really think


this population based approach is they shine in so they shine in multiple


different areas but one area where they shine is definitely when the


environment changes. So when you know something about whatever your input


changes like the robot losing a leg. So in kind of neural architecture search


you might you might find these methods working if you then go for let's say


transfer learning. So you kind of train your network on one task you want to


actually do it in another task right and then if you maintain many solutions and


you can evaluate all of them in a in this transfer setting it's much more


likely that one of them you know it's gonna be it's gonna be fine. So but


you're right of I also believe that directly in architecture search maybe


it's not maybe it doesn't yield that many great results though the other of


course the other area where these methods shine and this is with respect


to algorithms like novelty search which can be implemented as a population based


method is they gave this really good example of deception in a search problem.


So a deception would be like if you have a robot walking a maze and the robot


just wants to get to the goal right and you would program it the robot to be


rewarded the closer it gets to the goal but if like there's a wall in between


and you actually need to go around the wall kind of then for a while you would


need to move away from the goal in order to reach it. So if you have like a pure


objective driven approach you just go straight to the goal you would always


get stuck at the wall but if you then kind of do what is called a novelty


search where you basically reward the robot for things it has never done


before it would actually find its way around the wall. So you can maintain


population of solutions that all kind of explore the space and that in neural


architecture search maybe it's of a benefit that actually you know if I


probably always benefit from like adding more layers or neurons or something like


this but maybe I actually want to prune some stuff first and then add some more


stuff so I maybe want to get worse first before I can get even better right.


So it's a reach where I can imagine that happening but I don't know.


Yeah I was thinking the changing environment I definitely think like when you deploy a


model and then you're getting new data that you could frame that as a changing


environment and then also thinking about like in the context of GANs which is


something I think is really interesting that the discriminator classifying the


GAN, the generator samples it's a changing environment because of the


generators updates so maybe having some kind of population based GAN or


discriminator model might help it avoid that like continual learning problem I


guess. Yeah that might very well be. There are


approaches to GANs I believe where you have like many discriminators


and each one kind of only has let's say has its own limited view on the data and


you're trying to kind of fool all of them at the same time but it's not the


same thing but yes I think that might make sense. Yeah I've seen that multiple


generator multiple discriminator model too I think that's really interesting as


well. So then one other thing I was curious about is this idea of goal


switching and how that might relate to the like AutoML on our existing


more like heavily studied things like classification, localization, semantic


segmentation. Like how do you think goal switching could be important? Like one


idea I had is maybe if you've got like multi-class classification and it's got


like a really low false positive rate or something on like one class you might


say well you've somehow learned a decision boundary on that class or do


you think that that wouldn't generalize and that there's no sense in goal


switching in like a multi-class classification problem? So yeah in


general well when you think of goal switching in general how they introduced


it was also in the context of like this population based search of these map


elites. Maybe it's kind of so what map elites the algorithm does basically is


it says okay I have a number of dimensions that I could solve the


problem on and they introduced okay let's take life on earth needs to


whatever survive so I can either be like a super tall creature right to reach


food that no one else can reach. I could be a super fast creature right to kind