solutions are what I call payment based solutions, like give the reviewers money to review. I don't


[Rant] REVIEWER #2: How Peer Review is FAILING in Machine Learning

see how that fixes the incentive for you to reject anything, you just might write it in a bit more


eloquent style. Also, as soon as you bring money into the game that automatically excludes a lot


of people depending on how you do it, that aren't as affluent, which is certainly something we


don't want as a community. Other people are pointing to things like open review, which I agree


is a better system. However, it is still anonymous. So the same incentives exist. And it is still a


conference where you get a stamp of accept or reject. And once it's accepted, no one cares about


the reviews anymore. In fact, in open review, you can write as much text as you want. So the ACs are


even more overloaded with lots of texts to make their decisions. So something I want to highlight


is a thread by Thomas G. Dietrich on Twitter, where he basically suggests some sort of a wiki


and sort of a collaborative research wiki where you'd have a set of senior authors that basically


maintain that wiki that do a first check of papers and kind of match them against the wiki of what's


already known. I won't go through that here, I will link it. And I definitely advise you to read


it because it's very interesting proposal. It's a sort of utopian dream, I would actually welcome


if we all work together on increasing the knowledge of mankind in a wiki style way. However,


I think lots of people want their names attached to things. And even if you do what Thomas suggests,


and basically have people write papers, and then the editors integrate that into the wiki, it is


not clear how that system where the editors clearly need to be senior and experienced could


deal any better with the explosion of research that we're dealing right now, they would be as


overloaded as the current system plus who's going to be an editor, Thomas says becoming an editor


would be a very esteemed career path. And again, I completely welcome if that were the case in the


future. However, simply decreeing that something would be very esteemed, doesn't make it that way.


It's not fiat money. So as much as I would like that, I just don't believe it would work. And


especially I don't believe it would work right now. And I think it would be subject to the same


problems. So can we come up with a better solution? I think yes. But the way to go there is to align


what we want as a community with the incentives of people and not go against it. Because as soon as


you go against it too much, people will find a way around it. So the first thing I want to suggest


is we abolish conference publishing, this weird notion that you submit your paper to this conference,


and then all at the same time, a random process is happening and three random people give their


opinion while reading your paper for a couple of minutes. And then you get an accept after which


your paper is there never to be revised, or a reject, which simply means you try again seems to


be preposterous. I'm sorry. So people wonder, yep, how do we know when a paper is accepted? Who cares


about acceptance? Who cares? Why can't we just switch to citations? Citations is a pretty good


measure of how much people care about a paper. And yes, big names will get more citations, but


they do so now and they do so more effectively than ever. Why can't we just put our papers on


archive and then run some kind of page rank algorithm over the citations such that self


citations aren't worth as much. I mean, search engines figured out how to deliver you the most


relevant search result to a query 20 years ago, why can't we simply apply the same techniques to


research determining this work is quite relevant. This work is not that quite relevant. I get it


citations take time, and you won't immediately know after publishing. But I think that's a step


we can take, especially since conference publishing is also lagging like half a year behind publishing


on archive during which pretty much nothing happens. And then people say, Oh, but what about


peer review, peer review, peer review does not work. peer review is a joke in machine learning.


Okay, no one cares about the reviews, reviewers are nuisance, you have to get past them, all the


people still pretend to care that it means something that reviewers agree or disagree with


you. It doesn't. In fact, I want to get to a system where peer review starts at the moment,


where you publish a paper on something like archive, and then never finishes for the lifetime


of that paper. As new knowledge comes in from the field, the paper can be continuously re examined.


And if the paper turns out to be really important, more and more scrutiny can be applied to it seems


like a much better system than simply throwing the same amount of pretty random reviewers at every


paper and then giving it the stamp or not. So here's what I suggest we keep something like archive,


but amended with a commenting function. And the commenting can be pretty feature rich. So you


could incorporate plots and references to other things. This goes very much towards a kind of a


collaboratively edited wiki, but where people still put their names on things. So let's say I


publish a paper, someone else could publish a comment, which would be not less in quality than


a paper, it can be it can be a two line comment, it can be a full rewrite of the paper, it can be


an amendment. So I could have published a paper and someone else could say, Look, I've done your


code on a different data set. And here are the results, people could then cite my paper, or they


could cite comments and the citations will determine the relevance. The comments would also


be right there on archive. So every time someone goes to look at the paper, they'll see the comments


along with it. So if the paper has a big mistake, they'll basically see the comment that says, Hey,


this paper has a mistake, and I can prove it right here. And then they can maybe see a response to


that saying, No, you're wrong. And people can make up their own minds, we could build in some kind of


voting system like a stack overflow system for ranking comments. But instead of making this


stamp of approval thing a one time event by a random set of people, let everyone make up their


own mind and let people discuss and you can even have anonymous comments on these sites because


the comments will be evaluated on what they are writing and not who it is by now of course, if it


does turn out that commenting will become cool after a while, you can also comment non-anonymous


non-anonymously, and maybe get little medals that you get on Stack Overflow. I don't see that


happening. But if it does, the better now as a side suggestions, can we please stop publishing stuff


in PDFs? It's so like, why do we still do this, this many pages, this margin and so on? I get it,


some people still print out their papers. But websites are so much nicer to look at, and can be


made to print adequately. Let's start publishing research as HTML, not as PDFs. So remember when I


said the authors have a big incentive to not have comments on their paper, this pretty much goes


against that, right? So it is entirely conceivable that the authors will just start self hosting,


big company like Google could simply not publish to archive anymore, they could simply publish to


their own website, and remove themselves from the ability for other people to comment. Now this can


be solved technologically pretty easy by creating something like a browser plugin that if you find


a piece of research anywhere, it'll simply fuzzy match the title, find the appropriate comments to


that research as a unified set of comments across all of the internet. In contrast, conferences


should be conferences, it should be places where people come up, meet up and talk about relevant


issues that are happening right now. If I go to a conference now, most of the talks on the papers


is from research that is six months old or older, why don't we have conferences that are simply


consisting of invited keynotes, panel discussions, and things that are now called workshops where we


discuss current maybe unfinished research, have poster sessions for many more people, there's no


acceptance, there's no declining, if there's not enough room, do a lottery or something like this,


but make the conferences a place where science is happening, and not where we flash six months old


research. So why is this not happening? I already said that most of the incentives are actually


towards the current system as much as people complain about it. Now conferences are slowly


losing their reputations, as I said, because over time, people will catch on to the fact that the


signal being accepted at a particular conferences is more and more noisy. However, the system is


still upheld by most PhD students, for example, needing a certain amount of conference accepted


submissions in order to graduate. So what we really need is professors, and I'm calling on every


professor out there to start giving out PhDs, while absolutely not caring about the number of


conference accepted submissions that a student has. And that seems like something that's very


doable, because it requires individuals, professors to simply change their practices with which


they let people graduate. So that was it for my little rant on conferences and reviewer number two.


Please let me know what you think in the comments. I value your input very much. And I hope we can


get to a future where conferences are conferences and research is just done on the basis of its


What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study (Paper Explained)

Hello there. Today we're looking at what matters in on policy reinforcement learning, a large scale


empirical study by Google Brain. On a high level, this paper investigates a five different continuous


control tasks, and they train agents with all the different choices that you can make, basically,


on these continuous control tasks. So the different choices are like network width and height of the


value and policy network, learning rate, type of loss function, regularization constants, and they


train all of these agents and they try to parse out what works in general, and what doesn't. And


they have some surprising findings that number seven will surprise you. Yeah. Okay, so that's


the that's the study on a high level. As always, if you like content like this, consider subscribing


and sharing it out. That would be excellent. So they say the on policy reinforcement learning


has been successfully applied to many different continuous control tasks. While our algorithms are


often conceptually simple, their state of the art implementations take numerous low and high level


design decisions that strongly affect the performance of the resulting agents. Those


choices are usually not extensively discussed in the literature, leading to discrepancy between


published descriptions of algorithms and their implementations. So the sort of things they mean


here are the things that when you read the paper, the algorithm will be sort of described pretty


well on the main idea. But then if you look at the code, there's a whole bunch of hacks there. Like


on the Atari environment, you have to repeat certain actions, you have to introduce sticky


actions, then the question is, do you have like a random starts or the OS start at the exact same


time, therefore, the randomness of the level is not given, then you whether or not you normalize


certain observations. But we've had these things even in supervised learning or NLP, things like


this, we've had pre processing, I remember the resonant first resonant paper that beat ImageNet


to a significant degree over the last year's baseline. It was, oh, yeah, we have the simple


idea of the resonant. And they have an entire section where they go, oh, and we do this


normalization, we do this pre processing, we do this and this and this and this and this. And I


mean, there's an argument to be made for all of these, these choices. But often, it's hard to


disentangle, if the choice, choices of these pre processing things, or whatever the choices are


matters, or if the idea in the paper matters. And it's also very hard to compare different things.


So what they're doing here, so I would say this is not only a problem in RL, this is a problem


generally, they say, as a step towards filling the gap, we implement over 50 such choices in a


unified on policy RL framework, allowing us to investigate their impact in a large scale


empirical study. So large scale empirical study is basically means grid search over these choices,


kind of smart grid search, we train over 250,000 agents in five continuous control environments of


different complexity and provide insights and practical recommendations for on policy training


of RL agents. So as far as I could figure out the code and or and or the checkpoints of these


250,000 agents, or the code of this unified on policy RL framework is not available yet. And I


don't know if it's going to be but basically, what they're doing is they're building one agent. So


in usually you have this agent environment dichotomy, right here, you get observation and


reward. And here you get you give action, they build one single agent that has a lot of switches


that has a lot of like flags that you can say, okay, either Do you want this loss or this loss?


Cool. Do you want this regularization or this regularization? And if so, by how much right? And


so on. So I have this agent with lots and lots and lots of switches over 50 of these choices that


they implement right here. And they can basically turn each one on and off. And therefore they can


investigate these algorithms. So let's jump over the most surprising finding, which, okay, the most


I can tell you the most surprising finding is that the initialize the policy initialization scheme


matters significantly. Okay. That's what people maybe didn't know. What also matters a lot is the


learning rate, and things like the discount factor. But I think people in RL were already familiar


with that. I find it also interesting what doesn't matter, namely, most things seem to not really


matter too much. But there might be other explanations for this. Alright, so they say we


consider the setting of on policy reinforcement learning for continuous control. Now, this is


where I have a bit of a of a problem right here. Because the title is what matters in on policy


reinforcement learning, it's not what matters in on policy reinforcement learning for continuous


control. They do say in the abstract here, as you've already seen, in the last sentence that


they have continuous control environment, five continuous control environments. But yeah, I get


it, you need to make the title a bit click baity. But the title overstates a bit what this paper


says this paper basically says, what works in these particular five continuous control environments,


right? So they vary a lot of things with respect to the agent, but they keep the environments


relatively constant. And it's not five diverse environments, it's five, mu, joko, continuous


control environments that are very, very, very similar to each other in terms of their observation


in terms of how the world works, and so on. So consider this paper as an investigation in what


works and doesn't work for these five, and possibly for very relatively close environments. So that's,


that's, I think, my biggest trouble I have with this paper right here is sort of it overstates


what it what it says in the title. But, I mean, the investigation itself is done, I feel very,


very well. So they say they have a unified on policy learning algorithm, where they research


prior work took popular code bases made a list of common Lewis choices and then implemented


everything into starting from the seed RL code base, seed RL is kind of a framework for distributed


or for reinforcement learning in general. And they say, whenever we faced, we were faced with


implementation decisions that required us to take decisions that could not be clearly motivated, or


had alternative solutions, we further added such decisions as additional choices. So this I feel,


if I write research code, this is generally what I do, right, I write my research code. And whenever


I come to a place where I'm like, should I use this or this should use this optimizer or this


optimizer, I simply make a flag. And then even if it's just one choice for now, right, just make a


flag and parameterize everything. And that's, that's, that's the thing here, they parameterize


everything. But other than I would do now, then I would sort of sparsely explore the space of these


parameters. Now they do a more dense, dense observation or dense sampling of this space than


me myself would do with limited resources. Of course, being Google, it is possible to do these


kinds of things where you investigate all the choices. So they say here, difficulty of investigating


choices. The primary goal of this paper is to understand how the different choices affect the


final performance of an agent and derive recommendations for these choices. There are


two key reasons why this is challenging. First, we're mainly interested in insights on choices for


good hyper parameters. Yet, if all choices are sampled randomly, the performance is very bad,


and little training progress is made. So that means if you have if you have all of these hyper


parameters, then let's let's consider like a three dimensional hyper parameter space, then there are


combinations of hyper parameters that are very good, right here, maybe here. So there's this,


this cube in here, that's sort of very good, but the rest aren't really good. So if you just simply


sample from anywhere in the space, like here, or here, or here, or here, here, you will basically


never get anything that works, you sort of have to hit the combination correctly. And that's, that's


a problem in three dimensions, but it's way more a problem in 50 plus dimensions like they have here.


So they have to resort to a different strategy. They have to go basically start out from a good


configurations, where they say, they group these, we create groups of choices around thematic groups,


where we suspect interactions between different choices. For example, we group together all


choices related to neural network architecture. We also include the learning rate in all of the


groups, as we suspect it may interact with many other choices. Then in each experiment, we train


a large number of models where we randomly sample the choices within the corresponding group. All


other settings for choices not in the group are set to settings of a competitive base configuration


that is close to the default PPO versus V2 configuration. Okay, so what they're doing


basically is they're saying, now let's, let's consider these. So these groups, you can now think


of single dimensions in this space. So, or, yes, so let's consider the space of groups. Let's say


you have two different groups. One is the group of network architecture parameters, and the other


one is a group of learning behavior, like learning rate and training algorithm parameter. What they're


saying is they're saying, we know of a configuration right here, that is good. This is PPO versus two,


V, version two. And now what we're going to do is we're simply going to keep in each experiment,


we're going to, if we want to investigate the network architecture, let's say that's this axis,


we're going to keep all the other groups the same as this default configuration. And only investigate,


only basically move this point to the left and to the right. And we're not going to move it up and


down, we're going to keep the learning dynamic parameters of the other group or all of the other


groups, we're going to keep the same, and only move it in in the architecture parameter space. Now,


of course, this is not just one parameter, this, since they make these groups, this is a multi,


multi parameter. So at each point here, you can imagine like a little subspace of the inner group,


and they then sample from these. And that becomes much more feasible, right? So now maybe you have,


let's say you have 10 groups of five parameters each, you can densely sample five parameters,


like that's sort of possible, you cannot densely sample 50, but you can densely sample five. So


what you would do is you would keep the other 45 constant, that would correspond to this dimension


and all the other dimensions. And you would only vary within the group, which would correspond to


this dimension. But now you see that the problem again, of course, is that you're always starting


from this point. And you're basically only exploring along the axis of this of this group


space. Because you always keep one can't keep the others constant. And that basically, to me,


that means that these experiments are going to be heavily favored in, in terms of which of the


algorithms is closest to this to this baseline. Because if so, if I go with, with this particular


algorithm, I know that these parameters are the best for this particular algorithm, where if I now


use any other algorithm, these parameters might not be the best. And my only my only way of


adjusting to that other algorithm is by individually moving here while keeping others constant. So I can


basically only improve with it along one of the groups. I hope this makes sort of sense that it


feels like this experiment biases the results in favor of whatever is made, whatever choices are


made in this baseline. So keep that in mind. Now, that being said, PPO, of course, is very popular


baseline. So it makes total sense to use that as a as a base to explore from. But it's not like


they're doing an actual dense grid sampling of the space, they're doing a sparse sampling in the


group space, and then a dense sampling within each group. Alright, so they, let's go into the


experiments. The first thing they investigate are the policy losses. Now this is this is a rather


important topic. And that basically means how do you train the policy. And the choices here are,


of course, PPO, like we saw the proximal policy optimization, but there are also others, namely,


for example, policy gradient, you might know that if you learn about reinforcement learning, you will


inevitably learn about policy gradients, like the first thing you learn next to Q learning. And then


v trace is another sort of policy loss. v trace is optimized for distributed reinforcement learning.


And they have a bunch of others. And they hear they say the goal of this study is to better


understand the importance of the policy loss function in the on policy setting considered in


this paper was not to provide a general statement that one of the losses is better than the others,


as some of them were specifically designed for other settings. Now, I, of course, I agree with


this with this statement, it's nice that they repeated again, right here. So all the results


right here are just valid for these environments, or environments very similar to these. And you


have to keep in mind that the baseline parameters are PPO v two, and they only ever vary one group


from these baseline parameters. So that's why in this experiment, for example, it doesn't seem


too surprising that the PPO loss, as you can see, outperforms in every single experiment here.


Whereas the other losses underperform. So the recommendation is use the PPO policy loss,


start with the clipping threshold to point two five, but also try lower and higher values if


possible, because they have found and they have they have more experiments. And the appendix,


the appendix is full of these experiments, and you can go and look at them. So they,


but the general recommendation here for them is to use the PPO policy loss if you have these


continuous control tasks, and that there is a strong influence of this clipping threshold


that is in PPO. Second thing, network architecture. And that's basically you have you always have a


value network and a policy network. And the question is how many layers how deep and so


on? Should you make them? These things here are just MLPs. Since this is continuous control tasks,


you don't learn from pixels. As far as I understand it, you learn from the states or the sensors on


these robot simulated robots. Now, you got this here, they say, separate value and policy networks


appear to lead to better performance on four out of the five environments. And further regarding


network sizes, the optimal width of the policy m of the policy network depends on the complexity


of the environment and to lower to high values costs can cause significant drop in performance.


Well, for the value function, there seems to be no downside in using wider networks. Moreover,


on some environments, it is beneficial to make the value network wider than the policy one,


e.g. on half cheetah, the best results are achieved with 1632 units per layer,


da da da da da da da. The so some there, this is a thing that sort of crystallizes out of this paper,


because what you're doing is you have these one policy network and one value network,


like it's it's this dichotomy where the value network tries to estimate the reward, and the


policy network tries to maximize the value. So you have you have two learning things here,


you have this is learned. And this is learned. Now there is a certain degree of interaction as


the value network. Of course, the reward is dependent on your policy. So the value network


sort of has to take into account the policy when it estimates the reward. But it seems to be that


the policy network is the brittle or one, and therefore, more care has to be taken to optimize


it, whereas the value network seems to be a bit of more robust to changes. And we've seen this


already in that the the the loss choice for the policy seems to be quite important. And here also


the network parameters for the policy seem to be the things you have to actually tune per


environment. Whereas for the value, you can pretty much go you can pretty much get any wide network


will kind of do. Okay. So they say as for activation functions, we observe that tanh activations


perform best and relu perform worst, which is interesting, right? Because you would think that


in other deep learning tasks relu have become pretty popular and usually outperform these others,


other activation functions. But in this case, no, but this could also be due to other things,


because again, they go from these default parameters, which, for example, do not have


entropy regularization built in. And if you have a relu, where it's basically an unbounded function,


whereas the tanh is sort of a more or more bounded function. So that could be, you know,


there could be significant interactions here where they have split the groups, and then the choices


might be reversed if in the other groups, these two parameters were different. But for now,


apparently, that tanh activations perform best. The interesting thing here is they say,


interestingly, the initial policy appears to have a surprisingly high impact on the


training performance. So this is how you initialize the policy network. Again, policy network appears


to be the more brittle one and the one that you have to tune more. The key recipe appears to


initial the key recipe appears is to initialize the policy at the beginning of training so that


the action distribution is centered around zero regardless of the observation, and has a rather


small standard deviation. This can be achieved by initializing the policy MLP with smaller weights


in the last layer. So if you have this policy MLP as multiple layers, and then it needs to output


an action distribution, right. So in these in these continuous control tasks, you basically,


for each of the joints you have to affect, so you have like a little, you have like a little


walker here with four legs, and what's that? That's like eight joints or something. So you


have to tell this how much force it needs to apply to each of these joints. And as I understand it,


that's usually given by the network outputting a mean and a standard deviation, I might be wrong


here, but meaning the standard deviation for the distribution of action, that's going to be apply


here and then this is sampled from that distribution, the actual force is then sampled. Now they say you


should initialize the network such that the mean here is zero across or over your observations.


And the way to do that is to simply initialize this last layer here with very small weights. So


you and I think their recommendation is to divide to initialize this by 100 times smaller weights


than all the other layers. They say other choices appear to be less important. The scale of the last


layer initialization matters much less for the value MLP again, then for the policy MLP. Apart


from the last layer scaling network initialization does not matter too much, yada yada yada, there


appears to be no benefits if the standard deviation of the policy is learned for each state, or once


globally for all states for the transformation of policy output into standard deviation soft plus


and the expenditure performs similarly. So most of these choices in their case appear to be


relatively similar except the ones that they point out. The recommendation here is initialize the last


policy layer with 100 times smaller weights. Use soft plus to transform network output into action


standard deviation and add a negative offset to its input to decrease the initial standard deviation


of actions. To this offset is possible use tanh as both the activation function if the networks


are not too deep right here. This is probably where the relu would start to shine and transform


the samples from the normal distribution to the bounded action space, sorry, and to transform


using a tanh. Use a wide value MLP, no layers shared with the policy, but tune the policy width.


It might need to be narrower than the value MLP. Now this here, this no layers shared with the


policy, this might just this might be now a result that the policy is quite, quite brittle. So if you


can detach the value and the policy that might be of advantage. Which is also surprising, right? You


would think that these two networks if they are shared layers, they would learn more about the


environment, but apparently not. Then normalization and clipping. So you get a bunch of normalization


and clipping techniques, which is, for example, observation normalization, basically means that


whatever comes in, you normalize it to a given range. So that's usually you do that for supervised


learning, like if you have if you have MNIST digits, so this is a mostly black image with Okay,


can I draw on this with like a small portion of it is white. And what you want, this is usually in


the range of zero to 255. So you have zero to 255. What you want to do is you want to normalize that


search that it's in the range negative one to one, or alternatively, such that its mean is zero and


its standard deviation is about one. So people use both things and they tend this alone tends to


already boost the performance. So the fact that it's non that this is non negative, and the also


the fact that this number is somewhat higher than sort of in the zero one range, these are quite


important. And they're going to figure out that this is also important right here. So the


recommendation is always use observation normalization and check if value function


normalization improves performance. So for value function normalization, I believe you would you


would normalize the output of the value function. So instead of the value function telling you this


is how much worth something is, it simply can tell you sort of that it's more or less worth than


something else in a normalized range. gradient clipping might slightly help but is of secondary


importance. Okay, cool. Yeah, so all the other things also don't seem to matter too much, like


per mini batch advantage normalization, and gradient observation clipping. Yep. Then advantage


estimation. So advantage estimation in reinforcement learning is a basically the the value network


needs to be trained, right? So you take a step and a step and step and step and in each step,


you get a reward. And you get you perform many steps. Now the value network sitting right here,


needs to be trained to predict the total rewards that you can get from here on until the end of


the episode. Now, usually what you do is you can bootstrap this by sort of a temporal difference


thing in that you consider the you you consider a few steps into the future. And then you ask your


own value network what it thinks of the rest of the episode. So basically, you train you don't


train on the entire rest of the episode, you train on the difference between this and this. And then


you can get way more complicated where you actually ask your value network at each step what


it thinks and then you go to that value network while integrating this reward. But you also go to


this value network while integrating these two rewards and so on. And then your target becomes


sort of a mixture of all of these things. You can get super complex with these with these different


variants. And they say we compare the most commonly used advantage estimators and step GA and


vtrace and their hyper parameters. And their recommendation is use the GA with lambda equals


0.9. Okay. I feel this is not is not too, too, too surprising right here, because this this end step


is a very basic estimator. And the GA and the vtrace are better. And they say the GA and the


vtrace they appear to perform better. And they have not found a significant performance difference


between the two. So cool. Last thing though, this is second, second to last thing, almost last thing,


training setup. Now I believe this, this becomes more important. So they investigate choices related


to data collection and mini batch handling. So the number of parallel environments, the number of


transitions gathered in each iteration, the number of passes over the data, and so on. So this is


going to matter quite a bit. The recommendation is to go over experience multiple times. So what


you do in these environments is always you have a phase where you collect experience. And then you


have a phase where you learn from this experience. And so you collect experience, you start from here,


you collect a bunch of experience, you put all of that experience into a buffer, which is like a


database. And then you have these, what they're called traces, right. So all of these are now


episodes that your agent took. Now all of these episodes consists of many, many steps that the


agent took. So here is one step, here is one step, here is one step. And each of these steps are


going to be one training sample. So each of these steps and also here and here are going to be one


training sample. There are multiple problems here. The first and obvious one is if they if you just


leave them in order, then you will have very, very correlated mini batches. And that's not good. So


you want to kind of shuffle them around in here. Each time before you go to them, you can go through


them multiple times in different order. And that works really well. They say you should go over


your experience multiple times, since that doesn't hurt you. And it alleviates you from the necessity


to collect more data. The second thing they say is you should shuffle individual transitions before


assigning them to mini batches. Okay, we've, we've concluded that. And you should recompute


advantages once per data pass. Now what's the point? The point here, before we talked about, you


have to, you have these advantage estimators, which basically means you have to look for each step,


you have to look ahead a couple of steps, decide what the value of this state is or the advantage.


And in order to do that, as we have seen, you kind of look at your own estimation of that future


value. So you have this value is dependent on your own estimation of the future value. Now, of


course, if you just do, if you can only do this, if you have these episode traces, if you have these


blue episode traces still around, you know which step comes after which, you cannot do this anymore,


once this is all in mini batches and shuffled. So what some people do is they simply compute these


things once at the beginning, with the value network they have, and then they go multiple times


over this data. And just they shuffle, they might shuffle each time, but they keep these estimates.


And that's, of course, is more and more out of date, the more often you go over the data. So what


they recommend is, you should always go back to this set data set, recompute these estimates with


your current value network, then do the whole shuffling thing again, and then do another epoch,


and then basically come back to here again, and recompute the advantages. It makes a lot of sense,


right, but they also find that this actually makes a difference. For faster wall clock time, training


use many parallel environments and increase the batch size, both might hurt the sample complexity,


but they get you a faster wall clock time, which makes sense, right? If you have more environments


than you're going to collect more experience and more different experience, and that will speed up


your the time that you need for learning, you might collect more samples, though. So it will also


increase your flops. Tune the number of transitions in each iteration if possible. Okay. So next thing


is time step handling. What do they do? The choices related to the handling of time steps. So


this is the discount factor frame skip. So in these environments, you can choose to like ignore


intermediate frames, how episode termination due to time step limit are handled. And their main


thing here is that the discount factor is one of the most important hyper parameters and should be


tuned per environment and start with a point nine nine discount factor, try frame skip if possible,


there's no need to handle environments step limits in a special way for large step limits. Okay, so


the discount factor, which is also unsurprising, right, because the discount factor is basically


how how much you discount future reward, and that is inherently dependent on the reward structure of


the environment itself. So it's really unsurprising that this is a big, an important hyper parameter,


but it's good to note. And then last sec, okay, there's more second to last thing. optimizers,


they investigate different optimizers, we investigate two gradient based optimizers,


Adam and RMS prop, as well as their hyper parameters. And their result says, you should use


Adam with momentum, though I think they found that RMS prop isn't too much behind that. But they say,


you should tune the learning rate absolutely, which is also known in the community, right, you


can't you if you have a different problem, it might require a different learning rate, and they


find the learning rate to be a important parameter for an important parameter for these problems. So


you should tune it, but the other parameters of the of these algorithms aren't too much of an


influence, at least on these particular problems. And then the last thing is regularization. So in


regularization, they try different regularizing methods, such as entropy regularization, soft


constrained entropy should not be lower than some threshold, callback, libeler divergence between


reference distribution, and so on. And they say, we did not find evidence that any of the


investigated regularizers helped significantly on our environments, with the exception of half


cheat on which all constraints help. So they don't find a particular thing. But remember this, again,


this, for example, here, entropy regularization is used in the impala paper, which is which in


which proposes v trace. Now, they here only have an experiment where they change the loss to v trace


without entropy regularization. And in this case, they turn entropy regularization on with the PPO


loss, as far as I understand the paper. And there, you can already see that there is a space that is


not explored that is the setting of the original paper that introduced the thing. And I think this


this, if you can remember this study, this study, like are all GANs created equal, they concluded


that probably all GANs are created equal, especially like Wasserstein GAN isn't too much better than


anything else. And the author of the Wasserstein GAN paper was furious because they didn't, they


clearly said in the Wasserstein GAN paper that they add an optimizer doesn't work, and they had


to use RMS prop and then the RMS prop was not in that study included. So it seems that the


limitations of being able to really densely explore these choices is quite, it's quite hurtful in in


that you can only even though this is a super large scale study, and they trained so much, right?


You can only ever make very, very, very limited, very limited, sort of conclusions in these things.


And I would say, if you are in these types of problems, definitely consider their default


settings. Otherwise, what I'd much rather do is to just go to like a piece of code that implements


as close as an environment as possible to the one I want and take the hyper parameters from there.


In the appendix here, they describe all of the things that they've tried with the choices of


hyper parameters and all of the results. And you zoom in on like a random one, you already see


that the results oftentimes are very diverse, very wonky, very much like, maybe, you know,


this thing isn't so relevant, or there's large performance differences that are unclear between


the environments. So it remains to remains to be seen. But the main interpretation here is that


you're probably going to have to tune hyper parameters for a while on your own environments.


All right, yeah, the appendix is really long. And if you want details, I invite you to look at it.


Looping - Go Lang Practical Programming Tutorial p.12

What's going on everybody welcome to part 12 of the Go language programming tutorial series.


In this video what we're going to be talking about is looping in Go.


So it turns out Go, much like it doesn't have classes, it actually doesn't have a wow loop.


But if this isn't your first programming language you're probably very much aware that you could


write either a loop and the other, you know, you could write for loops and wow loops and


also you could have a wow loop that acts like a for loop and so on.


So anyways we only have a for loop but even like your most basic for loop example is really


a wow loop.


So for example let's say you've got your basic for loop which would be like for i colon equals


zero so we're initializing i and I don't really think I've stressed it even up to this point


but when you define a variable for the very first time you're initializing that variable


so you do colon equals.


But later if you wanted to reassign a value to i you would just need an equals you wouldn't


have to reassign i just for the record.


Anyway for i colon zero i less than ten so this is basically your wow this is like okay


i is currently equal to zero while i is less than ten what do we want to do let's just


i increment.


Okay so that could be our for loop and then you could say something like format dot print


line what's i not i o what is i so let's go ahead and run that real quick go run go tut


dot go and you get you know basically zero to nine so that just kind of prints out all


the values that's just your you know your hello world for loop example.


But there's quite a few things that you can do from here so for example not all these


things need to be included in your for loop so this would be a typical kind of like counter


loop but most of the time you're probably not writing for loops that look like this


and that's kind of my problem with especially with the language like go only talking about


the basics because you're probably not writing for loops like this so for example you'll


probably have a variable that's already defined so you would really probably have already


initialized i to be something or to be the result of some function so that would be gone


right and then you're going to probably rather than incrementing i you're iterating probably


through i most of the time you're going to be iterating like like what we have is a list


of and not a list a slice of urls right we want to iterate through that we don't really


necessarily want to do because again this is more like a while loop while i is less


than 10 but it can exist so for i less than 10 we could print i now if we don't do anything


i is not incrementing i would wind up you know this would be an infinite loop for example


so what we'll do is i plus plus so that's going to go ahead and increment i but one


by one basically for us so we can run that again this should be the same output so just


for the record you can do plus plus or you could say plus equals one that would do the


exact same thing the benefit of plus equals is that you could do something else like you


could do plus equals five for that matter whoops made it too big where did it go there


we go right and then it's just 0 5 done because it got to 10 before okay so then you have


like you could do i know i mentioned an infinite loop but if you wanted an infinite infinite


loop you could just do it this way do stuff okay so there's just no conditions right you


don't even need i equals zero so no conditions you could just start the for loop and it will


just go forever so this is the equivalent of a wild true so so there you have there


you have that so the other thing is like let's show some more examples of loops that are


more closely matched to reality so maybe you'll have x colon equals five and then maybe you


would have a for loop that's going to just like a while loop that's going to like so


for example like a lot of times you're going to do something while something is the case


but that something is outside of the loop sometimes so for example let's just do like


an infinite for loop which is just going to mimic our while loop for us let's go ahead


and format dot print line do stuff comma x whatever x is let's say x plus equals three


and then let's say if x is greater than twenty five trying to use python there break and


so a break is a way to get out of whatever loop you are in there so it would only break


you out of that loop so if it's if you have like nested loops it's only going to break


you out of that specific loop that you're in so anyways let's go ahead and we pull this


up let's run that do we not increment x oh we didn't save okay that's like what's going


on okay try again cool so do stuff five eight eleven and so on all the way to twenty five


and then it broke so at least in this case x is somewhat outside of this of this so another


way that we that you could have written this though obviously is let me to here so you


could have just said x colon equals five x less than twenty five x plus equals three


format dot print line do stuff x and that should give us the exact same result but even


though yeah that's going to give us the same result it's going to look a lot cleaner chances


are you're not going to write loops that look like that that's just my my estimation so


another example is like you know you wouldn't have like all these values and checks don't


have to really be for x like for example like what if we had a colon equals three four x


colon equals five you know you could have you could ask okay well while a is less than


twenty five x plus x plus equals three and then you know format dot print line do stuff


with x and then also a plus equals four something like that so you could have multiple variables


that are coming into play here where you're doing something to something else and then


obviously you know incrementing the thing that's basically that wow check that you're


that you're doing anyway we could run over a ton of for loop examples but I think the


best thing to do is to apply it to like a real a real problem that you're having so


what I'm gonna go ahead and do is just copy and paste the the xml code that we had up


to this point because again this is gonna be much more realistic especially when you're


trying to iterate over a data structure that you created which is going to be common so


in this case we're basically pulling that xml data and in fact let's just run it really


quickly just to see what we're returning it just visits the xml and pulls the url data


and populates a slice with that data and now we want to iterate over that data so immediately


you're not going to be able to just iterate over this data it's not really going to work


that way so instead what we're going to say is let's just comment out the print line we


could say instead for underscore because we're not going to use it and then capital location


in basically range of s dot locations format dot print line and then let's just print out


the results percent s and then a new line or maybe it should be new line percent s that's


probably a little cleaner let's do new line percent s and then let's just say location


okay now let's run that and I'll talk about it in just a second I just want to make sure


it runs okay so let me oh I see what we've done so we're just printing the line we need


to actually print f that'll format it for us now let's do a new one right okay so we


really haven't talked much about that but just for the record I don't know how I want


to do this we could say like for example format dot print f here sum r variable it's kind


of like what we did with the web development example except just with print line it wouldn't


work as well so let's just delete this real quick save come over here run right here are


some variables so that's how you can do string formatting basically anyway bring back our


for loop that was the mistake I was making not using print f anyways as you can see we're


actually now iterating over and outputting them these are string values and all that


so that's good so now let's talk about the range function basically what it's going to


do is it's just going to iterate over your structure and it returns basically two things


it's going to return the index value and then whatever that actual value is so again that


doesn't look anything like like your typical for loop obviously it is because basically


what's happened here is like generally you would have this is no different than for i


colon equals five what do we want to do right that's that's that's what we've done it's


just so happens to be that rather than five it's it's range on s locations and range returns


two values okay but it's still like if that's the first time you're running over this code


at least for me it just wasn't that obvious I don't know it just didn't look it still


looked pretty foreign to me so anyways that's what range is going to do it just it will


help you iterate over your own kind of data structures and also built-in data structures


as well and just know that just remember that it's returning both index and the value in


our case that we're just calling it location and then we can print out the location okay


so now that we've applied for loops to our code the next thing that we're going to be


doing is we would need to visit those site maps get information on the articles and we're


probably not visit the articles or probably just grab like the headlines or something


like that to keep it as simple as possible but we'll see when we get there anyways if


you have questions comments concerns whatever feel free to leave them below otherwise as


[ML News] Microsoft trains 530B model | ConvMixer model fits into single tweet | DeepMind profitable

Microsoft trains a model that's three times as large as GPT three, NVIDIA releases the third


iteration of their style gun model and DeepMind goes hard on ml for biology. Welcome to ML news.


You might have already heard this but weights and biases has just raised a series C round at


the valuation of 1 billion US dollars and is now officially a unicorn. Congratulations to weights


and bias is one of the absolute top products in the market. And I'm not just saying this out of


the goodness of my heart. They actually pay me to say this. So thank you so much to weights and


biases for sponsoring this video. Now how might this benefit you imagine weights and biases they


get all of this cash right now. They're just going to dump this on you in form of free product so you


can expect the weights and biases system to become more powerful, better looking faster, whatever you


want. And for the foreseeable future, it's probably going to be available to you for free as it is


right now. Hello. Yeah. Yes, yes, that's what I said. I mean, okay, I can say that. I mean,


are you sure forever is kind of a long like, I'm not sure I can make promises against the


nature of the universe. Like, all right. All right. Yes, I'll do it. Okay. All right. So


apparently, the products are going to be free forever for personal use and academia. Yes, forever.


That's the beauty of startup money. It's spend first and then earn back later. So if you don't


know what weights and biases is weights and biases is a general suite of tools for machine learning


engineers, machine learning researchers, and everyone in the lifecycle of ML products,


it can track your experiments, it can save your models and data sets, it can monitor


your runs, and it is with you from experiment all the way to deployment. It's usually in the cloud,


but it can be on premise. So if you want to take part in that sweet, sweet cash inflow,


go to weights and biases right now. And again, congratulations to them. They should absolutely


pay me more now that they have more. Hello, hello and welcome everyone to ML news. There's a lot to


go through. So let's get going. Microsoft trains Megatron touring NLG 530 B. How many words can


you accumulate to make a model sound really, really, really big? I guess we're gonna find out


with the next iteration. But for this iteration, this is a giant model. Now this is essentially a


decoder only language model much like GPT three, yet it is quite a bit bigger. So this model has


105 layers, its hidden dimension is over 20,000. And each layer has 128 attention heads. This new


model achieves various state of the art results in zero shot NLP tasks. And this blog post details


what it can do, and more importantly, how it was trained. So the training relies on this library


called Deep Speed by Microsoft, which is a library to train these large kinds of models split over


multiple computers. When I say multiple computers, I don't mean 12 Raspberry Pis. In fact,


this training is powered by 560 DGX a 100 servers, that's not 560 GPUs, that's 560 servers, each of


which has eight a 100 GPUs inside of them. And everything's connected by NV link and NV switch


and super duper Infini band. So this is an absolute beast. It trained with a batch size of 1920 and


achieves about 120 teraflops per second per GPU in throughput. Now the sheer scale of this is


absolutely crazy. And it's questionable whether or not humanity really wants to go this route of


scaling up in this manner. But I'm glad they did in this case, noteworthy is for example, the fact


that they didn't start out with a big batch size. In fact, they started with a batch size of 32.


And then gradually increased to the final batch size. Another noteworthy thing is that their


training data is based on the pile by Luther AI, which is an open source data set that came out of


the efforts of replicating GPT three, which noteworthy has not released their training data


yet. But like GPT three, the authors here pay close attention to the quality of their data.


So even inside the pile, they sample various proportions differently. And they also add some


things from common crawl and real news to arrive at their final data set. The article details


what kind of scores the model reaches on what kind of zero shot tasks if you're interested,


check it out. I don't know if the model will be accessible or whether this was just an academic


exercise or whether Microsoft wants to make money with it. I guess we'll see.


Nvidia releases StyleGAN three we've covered this paper previously it was called alias free


generative adversarial networks. So not much has changed since then. Notably, you can see the


comparison of StyleGAN two which had a very hard dependency on the position in the image. So you


see the hair texture sort of remains at the point where the image is yet StyleGAN three has solved


these issues largely, as you can see the entire objects move around independent of their absolute


position. So this gives rise to a lot more maybe controllable, maybe realistic pictures. So what's


new is that they have now released the code and the models to go along with this. And people have


already tried out a bunch of stuff, including putting these into notebooks together with clip.


So thanks to the people involved here and shepherd Eugenio Herrera and Catherine Krausen. So if you


want to try this out, remember StyleGAN two is trained on specific data sets. So for example,


here I have taken the faces data set, you're able to enter some sort of prompt here for clip. Now I


just entered the prompt eagle because I didn't know what was going to happen. So here's the start. And


let's see what happens. Okay. Yep. Yep. All right. I guess eagle means I'll just slowly disappear.


But people have come up with quite cool stuff here. Give it a try and see what happens.


Here's an interesting paper by Yuval Kirstein, Patrick Lewis, Sebastian Riedel and Omer Levy


called a few more examples may be worth billions of parameters, they analyze different NLP tasks,


and they discover that for some tasks, collecting a few labeled examples will in fact, increase the


performance of the model in a very drastic way compared to something like a zero shot performance.


Now, this is not the case for all models, though, which is the interesting part. So for example,


if you take something like open question answering, which is where the model has to recall


information or go look for information, then increasing the number of examples doesn't


necessarily mean that the model gets better. However, just scaling up the model pre training


it on more data that is worth a lot. But if you go to something like extractive question answering,


where you don't have to recall anything, in fact, you're given the Wikipedia article usually where


the answer is contained somewhere, and all you need to do is find the answer, then a few more


labeled examples are actually just as good as scaling the model up to drastic degrees. So the


authors hypothesize that in something like open question answering, it's really about how much of


pre training you have, which means how much stuff is stored in your weights. Whereas for extractive


question answering, it's much more how can you map the question that you've given to specific


words in the article. So the model can learn a lot even from very, very simple and few examples.


So this might be a thing to consider if you're in an area of NLP, and you may not have a lot of data


and you ask yourself, should I spend the money to get more training examples? Well, I guess it


depends on the task. Another interesting paper is something something strike through patches are all


you need. Emoji under review at iClear 2022. So the first question is have paper titles gone too


far. So this is an absolute meme paper, but the actual contents are really nice. Essentially,


the paper does a hybrid architectures between the vision transformers and the MLP mixers, they


hypothesize that at least in part, what makes vision transformers good are the fact that they


operate on patches and not necessarily the transformer architecture by themselves. So they


propose an architecture where you put the image into patches, but then it's just a mix between


depth wise convolution and point wise convolution, much like the idea of MLP mixer where you mix the


dimensions and then mix the locations repeatedly. With this, they're able to outperform the other


two models. And most importantly, this is to the best of their knowledge, the first model that


achieves the elusive goal of having 80% plus image net top one accuracy, while also fitting into a


tweet, our field is just memes now. And another paper that piqued my interest vector quantized


image modeling with improved VQGAN. This is an iteration on VQGAN involving vision transformers,


funnily enough after the last paper, so they go with a two stage approach where in the first


stage, they use a transformer encoder and decoder and in between a quantization layer. Now


quantization has been really successful in recent months. So it's not surprising that people make


strides when introducing quantizations into new places. This then is paired with an auto regressive


transformer that takes in the encoded codebook vectors or indices thereof and essentially learns


a language model over these. So you're taking a picture you encode it into latent space. And then


in the latent space, you describe it as a sequence of codebook vectors. And that sequence is


essentially a language by itself. And on this language, you can train an autoregressive


transformer. So now when you want to sample a new image, you can simply go to your transformer,


you can let it sample a sequence of these codebook vectors as they would appear in the data set,


you can use the transformer decoder to decode it and there you get a new image. Now the images of


this model look really nice. And that is actually my problem. The images almost look too perfect.


They look super smooth, they look absolutely crisp. And just these images right here, they seem so


clean that they're not even real anymore. Like I would expect these pictures on the front of like


a glossy magazine, a Time magazine cover, a National Geographic cover or something like this,


not just pictures taken by some person somewhere. Life Science writes William Shatner AI will chat


with you about the Star Trek actors life. Now this article is essentially about a product called


story file. The story file looks to be quite a cool product. What they do is they will sit you


down and film you and ask you various questions about your life that people may ask. Now you just


sit there and you just answer these questions. I guess this is going to take quite a long time. But


once you have this compiled, sort of like an FAQ about your life. And then what they do is they


provide you with this text interface or with a speech interface where you can now ask a question.


So what makes this different to a regular FAQ is simply that you ask a question and then it finds


the closest match in the FAQ list and gives you that answer as pre recorded. And then there's also


one time where Shatner says, why can't make any sense of that? And that's what happens when you


answer any other question that it can't map. So how much of this is really AI? Not sure, but it's


definitely good that they put AI in quotes when they titled the article. Google AI writes about


finding complex metal oxides for technology advancement. This blog post is a pretty cool


report about research that has been done in finding new materials. Material science is


notoriously difficult because essentially, we have no clue what happens if we mix two things


together that no one has mixed together before. And given the amount of things there are to mix,


most things haven't been mixed before. The authors here developed a new method of using an inkjet


printer to essentially print mixtures in various dosages into lines on a piece of I don't know,


cardboard paper, something like this, these are plates and you print out these metal oxide mixtures


in lines in various mixtures components or fractions, then you bake them and then you use


optical analysis to try to assess their properties. Now not all properties are accessible via optical


analysis, but you can use machine learning to try to suggest to you interesting compounds that you


might want to look further at. So out of the giant amount of possible combinatorial possibilities to


mix, they have come down to just very few that they needed to test further. So this is very much


like drug discovery, where also machine learning is now helping to suggest new compounds that might


be interesting to look at. So in the end, they found 51 oxide systems with interesting behavior,


only one of them had previously been experimentally validated. So all in all, pretty cool. If you're


into material science, give this article definitely a read. Next up TechCrunch writes,


Gretel AI raises 50 million US dollars for a platform that lets engineers build and use


synthetic data sets to ensure the privacy of their actual data. Gretel AI is a company that focuses


on data privacy on how can we make ML work in sensitive settings, how do we not leak private


data and so on. So one of their services is they let you abstract your data such that your ML


algorithms can still train but they will train on synthetic data that is guaranteed to be privacy


protected. Now just conceptually, this is a bit more challenging than it just might seem like any


information you pull out of data is potentially related to the privacy of the data where it comes


from even synthetic data even with various guarantees. As long as information is transmitted,


it seems like there might be a risk but these people are the experts so I'm not going to claim


anything here. And it looks like their tools are useful in a wide variety of applications. Now what


I love is their website where they have this demo called accelerate your tasks. And here's the


timeline that without Gretel you have to do Oh no, you have an idea you need to go ask your boss,


you need to copy sensitive data or you have to do all these things at once. And then with Gretel


wait, wait, watch that click here. Wow, idea integrate Gretel instantly synthesize or anonymize


data innovate. In any way, there's a blog post that goes along with the 50 million new funding


about why privacy by design matters more than ever. If you're interested, give it a read,


and I need to leave. Well, I got kicked out from my other studio. It's not technically my studio,


this is going to be resolved pretty soon. You'll see there's going to be a new studio,


it's going to be epic. Where were we? Oh yes, DeepMind has released two new works. One is here


on bio archive and one is a blog post by themselves. Though there's a paper to go along


with this as well. The first paper is called protein complex prediction with alpha fold multimer.


And this is a specifically crafted version of alpha fold to predict the folding of protein


complexes. So while the original alpha fold was made to predict how a protein folds from its


original chain of amino acids into its final 3d structure, the alpha fold multimer model handles


cases where there's not just one chain of amino acids involved, multiple chains will fold up to


create what's called a protein complex. And these are notoriously even harder to predict.


And these are notoriously even harder to predict than just single protein. So alpha fold multimer


contains various improvements that make predicting protein complexes a lot more accurate and improves


not only over baselines, but also over the original alpha fold. The second one is called


predicting gene expression with AI. And here we move from the land of proteins to the world of


genes. So in your cells, you have DNA and DNA is essentially a long strand of information. And from


this information, the amino acid chains that make up the proteins are read off and translated and


transcribed. Now it is really important to know which parts of the DNA are read and also how often


they are read and translated various things on the DNA can influence how different regions are read


off. For example, if one part of the DNA is coding for a protein, that region is generally called a


gene, then whether or not that gene is actually read off and how much it can be influenced by


factors such as how tightly the DNA is wound around proteins called histones. There are also various


methyl modifications of the DNA. And lastly, and this might be the most complex thing, there can be


what are called promoter and inhibitor sequences that are in front of the gene that influence that


gene. And these can be really far away. So imagine a really long text. And whatever is happening in


here in the text is influenced by like a single word or two words that come way, way, way before


it's like a uber German sentence. So how better to handle this than throw a giant transformer at the


problem. And this is what DeepMind did right here with the giant transformer trained on the DNA,


they can predict gene expression better than baselines. And this will improve our understanding


and prediction of what various modifications to the DNA will do. So if there is some sort of a


variant, then gene expressions can be predicted without having to necessarily test it beforehand.


Very cool. Give it a read. Kunihiko Fukushima has won the Bauer Award for Achievement in Science


for work on the neocognitron, possibly the earliest implementation of what would now be


called a convolutional neural network. So Fukushima's pioneering work is being prized with an award and


some prize money. And none other than Jürgen Schmidhuber has publicly released a YouTube video


to honor Kunihiko Fukushima for this work and for the reception of the award. Now Schmidhuber


actually has opened a YouTube channel as far as I can tell just for this video, or at least that


might be the first one. Now is Jürgen going to join the ranks of us ml YouTubers, it would be


amazing. I mean, this is de facto reaction content. So he's already halfway there. Now Schmidhuber


gives a glowing review of the work of Fukushima and what the influences of that work were. And


he generally seems to be pretty pleased with Kunihiko receiving this award, though about


halfway through the speech, he starts to switch from away from work of Fukushima to work of,


funnily enough, his own labs. Now I think the story arc he had in mind was to sort of give an


overview of what Fukushima had done, and then set this in relation to what is happening today. But


what is happening today is entirely framed in works of Schmidhuber's lab. Now, of course,


he's giving this speech. So fair enough, but with the exception of DanNet, which is a convolutional


neural network that is coming from his labs, and a year before AlexNet won several competitions in


computer vision, the rest of the talk is essentially disconnected from Fukushima's work


altogether, talking about LSTMs and how it's one of the most successful papers of all times talking


about how transformers were invented in the 90s by his labs, more LSTMs and a brief discussion on


DanNet, then going into how highway networks are essentially a precursor to ResNets. And at the end,


circling back to Fukushima's work. So it's essentially congratulations, his work was awesome.


Also, my work is awesome. Also, congratulations, his work is awesome. Now, if you're interested,


the entire speech is available on YouTube. And we of course, welcome Jurgen to the circle of ML


YouTubers. Okay, some helpful stuff for this week. Bayer is a benchmark for zero shot evaluation of


information retrieval models. This is available on GitHub. And it has various data sets and


benchmarks for information retrieval. The Bayesian Optimization book by Roland Garnet is out online,


it will remain free online. But this version is a sort of a pre print. And I think comments are


very welcome. So if you're into Bayesian optimization, or looking to get into it,


this is a nice resource. Imagineer by Nvidia is a Pytorch library for GANs that now also includes


the famous GAN craft. So if you've always wondered what your Minecraft worlds look like,


if they were real places, this might be the place to go. Mosaic is a new ML startup that came out of


stealth mode and presents itself as making ML training efficient. Notably, they came up with


two products. One is this experiment explorer, which pays special attention to not only your


accuracy and your loss curves, but also the cost and the efficiency at which your experiments run.


So for a given baseline, you can find out what is the cheapest way to reach the same accuracy,


what is the highest quality that you can achieve while keeping the same speed? What if I want the


same cost and so on. The other product is the composer, which is supposedly a library to make


training neural networks more reproducible. So you can drop in various extra algorithms,


such as learning rate schedules, or squeeze excite layers and so on. Now, do we really need another


neural network library? And how modular is all of this really, I guess we'll see how this develops


to me neural network training is seems to be still intricate enough that libraries are most useful


when they give you nice primitives that you can plug together, instead of ticking a couple of


checkboxes like here, I guess it's going to be pretty hard for them to make all of this work


together. On the other hand, it's going to be I guess, kind of easy for something like weights and


biases to also include a cost measure of training and be a real competitor to mosaic here. So I get


it these people make this their primary mission, but I think it's still going to be a hard fought


battle over the ML tooling space. I'm excited to see what happens. Tech Explore writes, Germany


unveils its first self driving train. Now self driving trains have been used in things like


airports and so on. But this is the first self driving train in Germany that runs alongside other


trains on the same tracks. So the report here is actually pretty funny in that it says these self


driving trains are more punctual and energy efficient than traditional trains. They offer


a more reliable service, they transport up to 30% more passengers and significantly improve


punctuality and save more than 30% of energy. Now what they're actually saying is that German


people suck at running trains, simply replacing human drivers, coordinators, schedulers and so on


with machines makes such a difference. That's on you Germans. That's not on the machines.


The New York Post writes Pentagon's first software chief quit because China has already won global


tech war. Pretty strong statement I have to say. So apparently he told the Financial Times there's


good reason to be angry at the US for falling behind. We have no competing fighting chance


against China in 15 to 20 years. Right now it's a done deal. It's already over in my opinion.


He claimed that the US like Beijing should have prioritized artificial intelligence,


machine learning and cyber capabilities over traditional military spending like building


new fighter jets. Now this is a stance one can take cyber security and cyber warfare are important


topics. But the article gets a bit weirder. He attacked Google for not working on AI with the


US Defense Department while Chinese companies are obliged to work with Beijing. The US also wasting


time debating the ethics of AI while China makes massive investments and issues such concerns he


said. Well, here's how it works. US companies and governments and military discuss AI ethics to


please one particular loud annoying part of the US public mirroring that Chinese companies,


government and military also discuss AI ethics to please a very loud part of the US public. I'm not


sure how serious we should take these warnings right here. It is of course an interesting


question on how much one should balance the very real concerns of AI ethics with the fact that


somewhere else in the world, someone might care just a little bit less about that and then overpower


you in 1020 years. And lastly, DeepMind becomes profitable. So apparently DeepMind is now


profitable for the first time was that has been hemorrhaging money in the past few years. Now the


article by tech talks here details how this is exactly happening. DeepMind doesn't have any


customers by itself. It's only customer essentially is alphabet. So the parent company is the only


customer, which means that DeepMind can essentially set any price they want and the customer is going


to pay it. So DeepMind going into the green might be more an accounting trick than anything else.


Probably the whole alphabet construct needed to save some taxes. And that was the most optimal


way to do it. The article goes into more detail on how hard and expensive it is to really do


reinforcement learning in the real world. And also the strategy DeepMind pursues where they pay a lot


of money to acquire the world's top talent. Now that being said, we have recently more and more


seen DeepMind venture into solving actual real world problems with things like alpha fold for


protein folding prediction and weather now casting, it seems like slowly it might make its way into


real markets. Alright, this was it for this week's ML news. Let me know what you think in


MuZero: Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model

Hi there, today we're looking at mastering Atari Go, Chess and Shogi by playing with


a learned model by Julian Schrittweiser and people generally from DeepMind.


So this paper is an extension to AlphaZero, the kind of famous algorithm that learned


to play Go and Chess simply by playing itself. And the kind of cool thing about this model


is is that it has a learned environment model. So what does this mean? Usually, if you have


a game such as chess, I believe there is a picture of chess down here. If you have a


game such as chess, and you want to learn to play it, you need to know the kind of the


rules of chess, right? So in chess, you have the rules like the pawn can move two or one,


right, the bishop can move diagonally, and so on. Similarly, in Shogi, or go here, you


know where you can place the stones and when you win, everything is clearly defined. So


what you can do is actually you can you can plan, right, you can now think of, okay, if


I do, if I do this opening, right, my opponent could do either this or this, or, you know,


this. And for each of the three moves, I'll have response. So if if they do, they move


this pawn, I'll go for like a gambit here. And if they move this pawn, then I can, you


know, move on something like this, right. So what in essence, what you have is a tree


search. So you start out with the state you're currently in, right? And then your opponent,


sorry, this should be your the state you're currently in your opponent has the option


of performing any one of these moves, let's say there are three moves. And then from each


of these three moves, you again have the option of performing any of these moves. And the


cool the good thing is in chess, you know, each exactly what they do, like if I move


my pawn, then the new board configuration will be the pawn will no longer be here, but


here, right. So you know exactly what's going to happen. You can calculate that you have


a perfect simulator, then other domains, you don't have that. For example, in Atari, all


you have in Atari is this, this screen, right? Maybe you have a little little submarine here,


right? You have some opponents, right? The opponent, I don't know what are the opponents


look like? The fish? I don't even know in this game, right? And you can, I think you


can shoot these coins to select? I don't know. Okay. In any case, and sometimes you need


to go up and there is like a health bar. So but but in essence, you only have this, this


this screen here, right? You don't have you don't have more. And if if you press a button,


you can you can you don't exactly know what's going to happen. You don't exactly know what


the pixel space will look like, as this shot moves forward. Right? I guess you could know,


but you can't, you can't use that to plan because the the kind of space is too big and


your actions may be not not clearly predictable. And when you win aren't clearly predictable,


there may be randomness. So all of this stuff, usually what people do is here, they do use


a model free reinforcement learning. We've had this discussion, you know, before. So this


would be model free. And while chess here, you'd go about model based. Now, mu zero does


is it uses a model based planning. But it learns the model. So it tries to construct


a model for this here, it tries to say, Okay, if I have this screen a here, right, my thing


is here, and I press the button, right, then probably my submarine is going to be a bit


more to the right. But it doesn't do this exactly. So this has been done before. And


this is what's what's kind of known as learning an environment model, where you map current


environment plus action to the next step in the environment, right. And this usually doesn't


work too well, because you're really trying to generate this entire pixel space here.


What the cool thing about mu zero is it doesn't do that. It doesn't predict the next state,


what it does predict is a hidden state. And let's draw the hidden state as a little cloud


here, it predicts a hidden state of the next step. And from the hidden state, it will predict


things like the reward, the policy, the value. And then it can use from that hidden state,


it'll predict the next hidden state. You and from that, it will again predict the reward.


So the base idea is you only predict, you only predict what you absolutely need to obtain


the values that are important for doing reinforcement learning. You're not trying to predict the


full environment, you're simply trying to predict whatever is necessary. And this here


is a learned quantity, whatever is necessary to predict what what your what your RL model


is going to need. So that's, that's the basic gist of it. And we'll look at how they do


it, or how they describe what they're doing. So basically, the picture a here is how mu


zero plans. So imagine you have a configuration, a current state, this is an observation, and


this could be a chessboard. This could also be a position in Shogi, but it could also


be a screen in an Atari game or a camera input of a self driving car, and so on. And the


first thing it does it encodes that observation using this h here, this, I believe they call


this a representation function, you encode that to this hidden state. Now the hidden


state, this is appropriately sized, the hidden state here is supposed to capture everything


you need about the state to predict the kind of RL quantities in the future, right. And


you learn this function h, which, in this case, of course, is going to be a neural network


in order to produce such a state. Now from this state, you do two things. First of all,


you have this function f here. And they call this I don't, I don't remember. But you have


a function to predict the following two quantities, you predict the value function at that state.


And the value function simply means, if you are in this state here, right, this is now


not a true state, but a hidden state. But still, if you're in this state, in this hidden


state that belongs to this observation, then in the future, you're going to make this much


reward on average, right with your current policy. That's the value function. So the


value function basically tells you how good it is to be in a given state, right. And then


the policy, this is a bit special, the policy is predicting how you would act in this state.


Now this is a bit a bit confusing, or it was to me when I first learned it, because we're


going to see over here, how mu zero decides on how to act, namely, it does this entire


tree search thing up to a certain depth, right. And then it creates this histogram. And from


that, it produces the action. But in order to produce to do this tree search, this is


exactly this picture a, this is the that tree search that is done. And in order to do that,


you need these p values, because we'll go there in a second, you need these p values.


And they cannot themselves again, do a tree search, right, that would be like infinite


recursion. So what you need is you need kind of an estimate, right? Like, if I were and


especially down, down, for it makes more sense, if I were in that state, how would I act,


right? If I were to do a tree search like this, so you simply build a neural network


that tells you with one evaluation without having to do the entire tree search down from


here, how you would act, this doesn't need to be a perfect approximation of how you would


actually act, but it needs to be good enough, right. So this simply tells you how you would


act in that state. And that's important, because what we do next is we use this policy to generate


this action. And this is a simulated action, this isn't a real action, because the real


action would go here to the next actual observation. This is a simulated action saying, if I'm


in this hidden state, right, my policy approximately would be this thing. And so I can sample from


that and say my action in that state would be this action. And so now I have a hidden


state and an action. And from that I can produce the next hidden state. Now, of course, if


I were to apply the action up here to the observation, right, action one, I would get


the next observation. And that is exactly how alpha zero works, right? You use your


simulator, your perfect simulator to take the current observation, the current state


with a given action that the order this policy gives you, and you produce the next state,


but we don't have a perfect simulator, right? And we don't want to learn a model that predicts


the entire state. But what we want to do is we want to predict the following. If we were


to take a one here, if right, we would get an observation, can we predict the result


when we would apply the function h to that, right, giving me s prime, right, this is observation


prime. So this function h here, which is the function that maps from observation space


to hidden space, if we were to apply this to the next hit to the next observation, we


would obtain some hidden state for that observation. Can we predict that thing, so we need a function


that maps from the hidden state, given an action, right, to the next hidden state. And


that's exactly what what happens down here, right? This function g here, maps exactly


this hidden state plus the action to the next hidden state. And also, also, at the same


time, it will predict a reward, right? Because in each step, you might get a reward. So each


transition here gives you a reward. And we're trying to predict that as well. Not that important,


especially for games like chess or Shogi, where there's only win or lose at the very


end. But they incorporate this here to also be able to play these Atari games and like


a broader range of reinforcement learning games. But in essence, that's what it is,


right? We're trying to predict the next hidden state. And now we can basically recursively


apply this. So from here, I have an idea of what my policy might be in that state, right,


my proximate policy, my kind of mini policy that only needs one evaluation, I can sample


an action from that policy. And if maybe it's action to here, and I can then predict the


next hidden state that I would be in. Also, the reward, right, and therefore, using this,


I can do like a tree search. So I can simulate future trajectories, right? First, like all


of these policies, I can sample from them. Right, I can sample from them gives giving


me different actions. So that that'll lead me down the tree different routes. So I can


simulate future trajectories in this tree. And at the end, I have a pretty good idea,


I can do this up to a certain depth, right? I don't have to do it until the very end,


I can. And then I'll have a pretty good idea of how my immediate the immediate future looks


like, right? Which actions lead me to approximately which states and for each state, of course,


especially for each bottom state here, I have an estimation of the value of that state.


So basically, I can, the easiest thing would simply be to whatever search, how many steps


is this one? This is zero, 123 steps into the future. And for each of these states,


I can devalue v here, v here, v, v, v, v, v, then I simply pick the action up the action


up here, I'm running out of colors, and simply pick the action up here, that will lead me


eventually to the highest value state. So that's, we of course, we've not incorporated


opponent plays here and so on. But that's the basic idea. You can do this more sophisticated


this tree search. And this is a topic that we might cover in a video about alpha go or


alpha zero. But in essence, you can do the same thing as alpha go or alpha zero, except


if you're not working with the simulator. But you're working with a learned model on


the hidden states of the true observations. So B is how you would actually act, right?


So for each observation here, we'd say you'd run such a tree search, and you kind of get


a histogram over visited actions. And again, we'll skip over that here. But this, this


is part of the alpha zero paper, and you decide on an action. And that will give you a reward


and a next observation. And that's how you act. And then you train these things, end


to end. So you, you train, you train the networks such that, of course, the reward, you know


what the rewards are, right, the reward prediction of G, you know what that should be right from


given a trajectory and action sequence, you know what the individual reward should be.


So that's, you can train G for that. First of all, you can also train to predict the


correct value functions like in classic reinforcement learning, you can do like an end step into


the future prediction, or you can play until the end sample trajectories and so on. And


the policy you predict you, you predict the policy, your approximate policy to to match


your true actions, right, because your true actions you've generated by doing this entire


tree search thing, which is, you know, that you're what you're actually going to do. So


you're training your approximate policy predictor that you use to run the tree search to match


as close as possible to your actual actions, right? This, in this fashion. So this policy


resulting from hidden state zero should be as close as possible to the action you actually


took in the observation that led to hidden state zero. Yeah, so this is how you search,


act and train using mu zero. And this is pretty, this is it, right? This is the rest is as


experiments. The rest is simply showing that they can handle these games, they can keep


the performance basically, of the simulator based alpha zero in, in games. Sorry, where


are the results here? Yeah, so in these games, in these left hand games, they can keep the


performance of alpha zero even exceeded here in go. And remember, they don't have a simulator


like alpha zero, they have to learn this model. And in Atari, they actually out compete the


current state of the art, which is I think, R2 D two, or impala. But it's some model,


I guess some model free RL baseline here on the on Atari. So that's pretty cool. And I


think that brings RL to kind of a new level with this hidden learning. And yeah, they


compare it against against multiple ones, although R2 D two is a different thing. All


right. Yeah, so that's that's that. For me, it's a cool paper. It's short. Read it if


you if you want. I invite you to also look at the additional experiments where they basically


ablate what they need is the learned model really as good or better as the real simulator?


Does it take as much time actually takes less time, which for for higher elo, which is pretty


cool. How many simulations are needed? Things like this. All right, that was it. I liked


WHO ARE YOU? 10k Subscribers Special (w/ Channel Analytics)

Hi there! We have just crossed 10,000 subscribers on this channel and that is


an absolutely mind-blowing number. To everyone who's subscribed, thank you! And


today for a bit of a special occasion I thought we would look at you. Yes, you


handsome! One of the 10,000 subscribers of this channel and we're going to dive


into the YouTube analytics and see who you are and what you like and how you


behave. So if you've never done YouTube as a creator this is what it looks like.


You can see right here there are 10,071 subscribers right now. And if you look at


the videos, Attention is All You Need is the most popular video on this channel.


It is also one of the oldest videos on this channel. Probably it's in many


university curricula now and there's like half a lecture allocated to it and


that is not enough to understand this paper. So people come to this channel. I


was still using kind of Adobe Reader at the time and etching in sort of... there's


only one color, it's super laggy but for some reason people like it and I'm not


gonna debate with people about what they like and what they don't. I might do


another one on Attention is All You Need just because I understand Transformers


much better nowadays and I think I could do a better job at explaining them more


clearly. So a lot of the NLP models tend to do very well as videos because I


think people are interested and practitioners are interested and they


just want to learn about these models and what they're doing. Also the Siraj


controversy, very popular. It was a fun event and a sad event. This video right


here, Deconstructing Lottery Tickets, it just outperforms all the other videos.


Absolutely mind-blowingly rockets over everything but if you look at the


retention, I only retained people for two minutes on average which is the


retention is the lowest of any of my videos and I could not understand this


for the longest time and then it occurred to me if you title a video


Deconstructing Lottery Tickets and you put a bunch of math on the thumbnail,


people are gonna click and then be very very very disappointed when you don't


tell them how to win the lottery and that takes them about one to two minutes


to notice like oh crap I can't make any money using this video. So if you go to


the general analytics tabs it's very interesting to see what people like. You


see the last 28 days I've uploaded every single day except here. This is a bit of


a gap. I have accidentally deleted the back prop in the brain video and had to


re-upload it the next day. So if you look at the last year the views have gone up


substantially. The subscribers have gone up but there are subscriber spikes right


here and these spikes are usually sometimes when some large personality


recommends this channel. The channel gains a bunch of subscribers that


doesn't necessarily translate into more views it's just people that click on


subscribe and then never care anymore. The metric that's most interesting to me


personally is watch time and as you can see your watch time has gone


substantially up in the last month which I find to be encouraging. One minute of


watch time means that I get to transmit one minute of information to the viewer


and that's what really matters to me. Of course if I'm doing a worse job at


explaining then the viewer has to watch for longer but that usually doesn't


really work out because they're just gonna click away. Alright to the fun part


the audience who are you? Now as you can see here 69% of you are not subscribed.


What are you doing not being subscribed? Though this has changed in last month


significantly. I believe now it's about half and half. What I find most


interesting though is that 10% of you have this bell notification on. So 10% of


you actively want to be disturbed whenever I upload a video. I am


incredibly flattered by this. This is the biggest compliment. Not even I do this


for the channels that I follow. Demographics also very fun. About 93% of


you tend to be male and about 6% of you tend to be female at least according to


YouTube statistics and that's a pretty good intersection of YouTube being


mostly male and machine learning field also being mostly male. If I'm doing


anything to attract any particular type of person please let me know so I can


diversify a bit. Everyone's welcome here. You tend to be above 18 which is good


because we have very very much adult content on this channel. I'm happy to see


that none of you is underage. I think that's just that's YouTube just don't


not reporting underage statistics. But most of you tend to be 18 to 45 years


old though to the older viewers you're of course also very welcome though. I'm


pretty sure that some of these is just because you were underage when you


created your account and you just told them you were born in 1923. So most of


you tend to come from the United States or India or Germany. This is very


incomplete list. I think the most people simply are in the other category of


countries which I think means YouTube just doesn't know. But it is cool to see


that India is so high up. One of the reasons I started this channel so the


main reason is because it forces myself to read papers more thoroughly if I have


to explain them. One other main reason I started this channel was because I


thought I thought there was a gap between what you could get from a


beginner's course any sort of Coursera course and where research is right now.


And to bridge that gap you basically have to go to a very good university. And


I know that most of the world doesn't have access to these very good


universities. So my idea was to kind of bridge that gap to make that person that


has a basic understanding of machine learning be able to be up to speed with