And it is this is called the value. So maybe it says, well, this position, I'm going to expect you to win roughly 75% of the time, right, which in expectation would be a value of positive 0.5 here, because 75% of the time you win and the rest you lose.


Assessing Game Balance with AlphaZero: Exploring Alternative Rule Sets in Chess (Paper Explained)

Let's say there is no draw and tic tac toe. So there's this value function. And the second thing is this p and the p is a policy function. So the p will and I've drawn this a little bit, maybe not super, super duper too large, but the p will tell you for every possible move you could make, which one should you consider even?


Okay, so it maybe it assigns this here, a point three, and this here a point four, but this here is like point 0001, and so on. So for every possible move that you could do, it will assign a number. And it's a distribution. So these numbers add up to one, but that's not important.


It tells you which moves you should even consider going forward, right? So p, in this case is a distribution over the next moves. And with those two things together, we can reduce our tree search quite a bit.


So now, instead of expanding all the tree, let's go back to the tree right here, you can ask your p, hey, p, which one of these three should I even consider? And maybe p says you should only consider those two. Okay, and then you go down.


And again, you ask your p, hey, p, which one should you consider? And p maybe says, well, here, you should consider those two here, you should only consider that this one. And this tree over here, we've already discarded this from the beginning. Okay, so this p right here, it guides your search, it tells you at each point, which moves should you consider?


And this, as you can see, reduces your tree dramatically. In fact, what alpha zero does is it simply says you have one second of time. Now expand as much as you can in this tree, given this one second of time budget.


And the second thing is the value. So what you would have to do expanding the tree is always to go to the end, right? So you always go to the end, where at the end, you have a fully filled board, I don't know here x, so you consider every possible situation, okay, here, maybe this, this player wins, as you can see,


you always have to go to the end. But in our case, we don't want to always go to the end, we'd rather explore more into like more branches than always go to the end. And this is where the value comes in. So at some point, you simply say, now I'm deep enough.


And now I'm going to ask my value v, there are slight differences with respect to alpha go and alpha zero and so on. But they all have in common that they estimate the value of the intermediate nodes.


Using this v model from over here, I have v as v was green. So they use this v model from over here to estimate at a certain depth. So v learns to look into the future. So everything that can happen from here, and it estimates and it says, well, from here, you maybe have a, you know, a point five value or maybe a negative point seven, and so on.


So v learns to assign these values to situations to states, which are these nodes right here, and P learns to suggest things to expand, right, that's alpha zero.


And then at the end, if you've expanded the tree enough and estimated well, then you have a pretty good idea what's going to happen in each of the branches that you considered right in each of these branches, you look into the future from here, you look into the future here, look into the future by doing this PV play.


And after one second after you've done, you know, a couple of hundred or 1000, or however many looks into the future, then you have a pretty good idea for each of the top level actions, what's going to happen in the future.


And you can simply pick the one that has the best future for you, according to your own model.


So that's what alpha zero does not so this is how you combine planning and neural networks, you want to do planning, but you can't because you can only go so deep.


So you use neural networks to first of all, reduce the number of branches you consider, because the neural network will tell you which ones are worthy to even look at. And second of all, you don't always have to plan to the end, because you can simply ask your neural network, how much an intermediate state is worth in expectation.


And this turns out to be pretty good.


Why don't we do this for every single problem? Well, we do for this, we do need a simulator. So you may recognize that right here, I said, we consider all the possible actions that we have.


And for each action, we know exactly what's going to happen. This is only possible, like in a board game.


It's not even possible in like a board game where you have a die to roll, or a card to draw anything that is random. There, there is a way to include this right here. But in this simple formulation, we need to know exactly with 100% certainty, what is going to happen if we take a particular action.


So this is only really applicable for the types of full information board games, where we can write simulators that are pretty fast, right. And even then, even though chess, you know, has lots of available actions and complications, it's nowhere near the complexity of like a, let's say, a modern video game or even or the real world is completely out of scope for now for these types of things.


Alright, so that was AlphaGo. Sorry, AlphaZero, which builds on AlphaGo, of course, and the rules of chess that we're going to consider using AlphaZero are the following. So there's no castling, no castling for 10 moves.


Pawns can only move by one square. Forcing a stalemate is a win rather than a draw. So you may know this in chess, if you do not checkmate the opponent's king, but only put them put the king in a situation where it cannot move.


That's called that's considered a draw. And I think even in the chess community, some people want to consider this a win.


There is torpedo, where pawns can move by one or two squares anywhere on the board. And semi torpedo, where it's the same but only from the second and the third rank. Pawn back, where pawns can move backwards and pawn sideways, where pawns can move laterally by one squares, but captures are unchanged diagonally upwards.


Then there is self capture, where it's possible to capture one's own pieces.


So there are, you know, slight, slight details here with respect to the 50 move rule and so on. But if you if you don't play chess, simply consider these are changes, minor in a lot of cases, minor changes to the chess rules that make the new rules either a superset or a subset of the original rules, but they are going to have quite some changes in for the play.


And we're going to look at what happens. So that's the entire research setup. As you've seen, it's alpha zero applied to these new rule sets. And under the assumption that alpha zero will solve these will become master at these games, which we can't verify, we can verify in chess because right alpha zero can beat people that have trained chess for all their life.


We can't verify it here. So again, this is an assumption. So the first thing I want to look at here, and this is going to play a little bit into my criticism of this paper.


It's a pretty cool paper, but I do have some concerns right here is the following the following charts. So they do, we don't consider how you train alpha zero, let's just say you can train it, you know, to whatever pretty good performance.


Here is how they evaluate. So they evaluate for each variant, they do 10,000 games played at one second per move for each different chess variant.


So if you remember, as we do our tree search, right, we expand the tree according to our P and we estimate the values according to our V. And we do this for one second in this first thing.


So in one second, maybe this here is the tree. So we have some sort of an understanding of what's going to happen in the future. You can imagine if we have more time, then we can expand this tree more and get a much more accurate picture of what happens in the future.


Okay, so they do 10,000 games at one second per move. But they also in addition to 1000 games played at one minute per move. So there's 60 times more time and you can imagine that will add quite a number of nodes here.


And, you know, if, if your P and V would be perfect, then it wouldn't matter as much how much time you have as long as you sort of have enough time.


But since they're not going to be perfect, since they're only neural networks, they're not God or Schmidhuber.


They cannot accurately, extremely accurately predict the future. So this planning, the more you plan, the more you actually look into the future, the bigger your tree becomes, the better moves you make.


So on the left, you see the distributions of wins, losses, and draws for one second per move, and on the right for one minute per move. So both white and black pieces here are played by alpha zero. So it's not alpha zero against something else.


This is playing against itself. And you can see in a, in classic chess, it's, it's quite, it's quite saddening actually, that this game which is so famous, you can see that in of 10,000 plays, 8,820 end in a draw.


Which means that if both players are super duper good, and, and play, you know, play against each other, it most likely is going to be a draw.


And this, I think, is the criticism even in human chess is that it's not really a decisive game in that it ends a lot of times in a draw.


So one of the motivations here would be, can we find a rule set that is maybe more decisive.


So that's one of the investigations they do in the paper. But you can see that there are actually so if you consider this torpedo chess right here, there it is more decisive, as you can see, in more times, either white or black wins right here.


And there are others which are even less decisive, like pawn back. So when pawns can move back, then players may just camp, they like move a pawn forward and move it back again.


And that will lead to a lot of closed plays and so on. Whereas torpedo makes you move much faster, you can advance your pawns much faster.


And that will probably lead to the end much faster.


So if you consider this on the right, so what changed the rules didn't change, alpha zero didn't change, it simply changed that we now let alpha zero think for longer.


And you can see that the decisiveness reduces dramatically. So whereas 88% resulted in a draw with one second per move, now 98% result in a draw with one minute per move.


And this is a trend throughout these games. And that's also what they say in the text, it is to assume that if you let alpha zero plan for even longer, that this trend will continue.


And ultimately, whatever rule set you make, the result is going to be a draw. If two, let's say perfect players play against each other, which is a bit saddening, right?


Because, yeah, that ultimately, ultimately means that all of these rules aren't decisive. It's only they're only decisive due to the fact that either one or the other players is way better or that in general that they are not they are not perfect.


So this is an appeal of a game. But there are certainly games that are decisive, even though both players are pretty high level. I mean, think of every, every competitive video game.


Yes, so that's a bit of my criticism, all of this, all of this needs to be analyzed in the background that what's actually happening here is that we're dealing with imperfect decision making, due to a limit in resources.


Okay, and this assumption now is already a little bit invalid, right? The assumption we made at the beginning, why I pointed this out is that alpha zero can solve these games, let's say to perfection.


And here, when we analyze the decisiveness and so on, it seems to be purely or largely a factor of how much time alpha zero has to think about the moves.


And these two things, to me, they don't really go, go together, because we don't know if for a different rule set, you know, the training is harder, or might take longer and so on, or that this exact one second makes a difference or not.


It's, it's just, there are so many variables here. And when you're dealing with, let's say, imperfect systems that are not trained to the end or evaluated in their full potential, you're always dealing with the fact that you stopped each thing at some intermediate point.


And that intermediate, where that intermediate point is, can influence the results drastically. Now here, it seems, at least the ordering isn't changed by much. But yeah, this is one, let's say one criticism.


The other criticism here, that that I would have, again, is the fact that if you consider something like torpedo, where you can move much, much faster, then yes, of course, let's say, I don't know, is it more interesting?


That's, that's the question right here. So they look at a lot of things like decisiveness, diversity, and so on. But the question is, is it more or less interesting to play? And I think that's what humans are really after.


And they're sort of trying to find proxies to this. I would argue if you play something like torpedo, the games may be much faster.


And so you get to the end faster, but also maybe might not be as interesting, even though it's faster, because the complexity is less.


And with respect to the decisiveness here, so if you have a game that's faster, you also need to take this into account, because here is another thing that is sort of an arbitrary choice.


As moves are determined in a deterministic fashion, given the same condition, diversity was enforced by sampling the first 20 plies in each game proportional to their MCTS visit count.


So what does that mean? That means that if you run alpha zero on the same situation, on the same tree, sorry, on the same board position, it will always come up with the same move, except for parallelism, inconsistencies, and so on.


But it will in a lot of times, it will come up with the same move. So how do you play 10,000 games? Because you can just play one game, because each game will be the same, because you simply tell alpha zero, give me your best move, right?


So it will just play its optimal strategy. And all the games will be exactly the same. So there's no reason why these should come out different. So they enforce diversity by saying, okay, okay, in the first 20 moves of a game, we don't actually take the best move, right?


Usually you have you have this distribution. At the end of the tree search, you have a distribution where you say, okay, this move right here is clearly the best move, I'm going to play this.


However, if this is one of the first 20 moves of the game, they say, no, we need a bit of diversity. So we're going to sample according to this distribution, rather than just play the best one.


Now this number 20.


It's just sort of decided arbitrary, right? And if you consider something like torpedo, it's a faster game.


So you're faster in opening faster, making your faster to the end game, maybe, even though they say, well, the game length isn't affected this much, it could just be that you're faster in a situation where you're kind of forced to do certain moves.


And maybe the difference in decisiveness here is simply a result of the combination of the faster moves in torpedo together with this, the fact that they just keep the 20 plies for each game.


Again, this is something that you need to consider when analyzing these results right here. And there are a number of these choices right here, like the one second or one minute per move, we sample for the first 20 plies before we play the max move that where I think the results of the study right here, they have rather limited interpretability if you if you ask me, because


because of these of these choices. Now, of course, they're still the results are quite plausible, believable. And the idea is really cool to explore these rule sets. But this was this is just my criticism right here.


So we'll go through the rest of the results pretty, pretty quickly, because a lot of people aren't chess enthusiasts. And we'll just pick out kind of the core messages that the paper is trying to get across.


So here the table, again, with respect to decisiveness, and you can see even for so for classic chess, it's a white has a 50. This is the empirical score for white under different game conditions. So 50.8% means most of the time, it's a draw.


So white wins with a probability of 50.8. Most of the time, it's a draw. And you see even like the most decisive variants torpedo right here is a 54% only.


So they they analyze different defenses and how the decisiveness is with respect to different defenses that are not really popular under classical chess. And the results are interesting if you play chess.


But I would say they're rather, they're kind of aha, okay, if you do not play chess, because they consider individual moves, and so on. What is an interesting part is this right here, where they look at, they look at one move that in classical chess, so e4 is a very, very popular opening, where you move your e pawn twice for white.


And NF3 is not a super popular opening. And here, they compare this in classic chess and in no castling chess.


This thing right here is a histogram. And the histogram shows you the log probability of opening sequences when you play the individual moves.


So what does this mean right here? If you play e4, then the distribution is something like this, which means that you have some sequences that have no entropy at all, which means that once you play e4, and maybe one move more, then it's almost, it's almost determined what you have to do.


According to alpha zero, you'll have like no choice except play these few next moves. However, if you play NF3, then alpha zero says, look, this distribution is much more to the right, which means that you have a lot more options here.


Now, again, this could be because the move is actually less decisive, because the move leads to more balanced, more interesting situations where you can continue.


However, you know, with many choices, it could also be because it's simply alpha zero simply doesn't know as well what to do, because it leads to more complicated games.


And you get to give each move one minute to evaluate alpha zero might just not be as good in those situations, because it leads to more complicated situations.


If it could search for longer, maybe this distribution would shift over here, just as well.


Again, we don't know because you only give this one second or one minute each time for both.


And again, this goes under the assumption of alpha zero is this perfect player.


However, back to what they want to say here, if you do this in no castling chess, you can see that this spike right here are all the these Berlin defense variants and castling this OO right here is a big part of that line.


If you do this in no castling chess, you can see that these two moves, now the histograms overlap much more, which means that and in fact, you can see in the in this number of possible moves right here that they come closer together.


So not only does the blue shift to the right, the orange actually shifts to the left.


And it basically means that whether you open with e4 or Nf3, you are going to have about the same complexity of game, the same number of moves available to you going from there.


As you can see right here, these lines are the moves available for white and black under the different rule sets. So in e4 here, especially as black, you do not have many moves available as white a little bit more, but also not more.


Whereas in no castling you do.


So again, small rule change, big effect on the possible moves that you can consider. And this is the type of information that you would want to have when you design a game.


And they allude to this also at the end here in their conclusions.


So the last thing is they also compare the material values of the pieces here in the different rule sets, as you might imagine.


So some pieces become much more or less valuable. I find it particularly interesting that if you do something like pawn sideways, or then where the pawns are much more powerful, of course, all the other pieces drop in value.


Again, these results are pretty plausible. So I don't want to trash the paper right here, because it seems like the results are, as I say, plausible and can give some cool insights.


So the chess master also gives his opinions on these different strategies that AlphaZero comes up with for the different rules.


And let's go through the conclusions real quickly.


So they say, assessing the consequences of rule changes in the game design process demonstrated on chess, where we've trained AlphaZero to evaluate nine different variants representing atomic changes to the rules of the game.


Training AlphaZero model on these rules changes helps us effectively simulate decades of human play in a matter of hours and answer the what if question, what the play would potentially look like under developed theory in each chess variant.


We believe that a similar approach could be used for auto balancing game mechanics in other types of games, including computer games, in cases when a sufficiently performant reinforcement learning system is available.


And yes, this is, I mean, this the application here would be for something like this. If you design a new game, then you want to know what you have some choice with how you can make the rules.


And you don't want to let humans become really good at each of the rules and then compare, you can simply give this to the algorithm, and the algorithm will tell you what kind of plays result from each rule set and then you can choose the one that you find most interesting or most maybe commercially viable and whatnot.


I actually see this much, I see this bigger than just games and this alludes a bit to the Salesforce paper on this AI economist, I think we can let AI, you know, tell us what happens if we change, for example, things like tax policy, or any any sort of policy.


I know, humanity is very complex to model and so on. And you're never going to have a perfect simulator, which probably makes alpha zero, not good, but in limited situations, like maybe also stock trading rules, and so on, you could definitely have situations where the rule set is too complicated to solve analytically, but you could give it to an RL algorithm and see what happens and whether or not you like the outcome and whether or not there are any, like, obvious results.


Are there any like obvious exploits that you did not see. So, this, I find, you know, pretty, it's a pretty cool approach and, and we should think of this in the future as we build systems that have rules in whatever capacity be this games or policy.


So, the, they say, okay, yada, yada, yada, we showed that there are several chess variants among those considering the study that are even more decisive than classical chess, meaning torpedo chess, semi torpedo chess, no castling chess and stalemate equals win chess.


So, we identified a rising diversity of opening play and the intersection of opening trees between chess variations, showing how different the opening theory is for each of the rule changes. Yeah, they, again, this, this diversity of opening play, it really rests on this assumption that alpha zero is a good player and sort of an equally good player in all of these variants, right?


Because if it's worse in a variant, it might not be as sure about the moves and that would just look like, oh, you have many possibilities but in fact alpha zero is just worse at it. And it doesn't know.


So, we also look at the intersection of opening trees, like if you change a rule, how does this change, change the, the kind of, how does this change the, the initial game? So, a lot of these grandmasters they learn by heart all of these opening trees, the initial moves of a game, how much would they have to relearn?


There is a negative correlation between the overall opening diversity and decisiveness, as decisive variants likely require more precise play with fewer plausible choices per move.


Again, this is one view, right? The other view is that there are rule sets that are just make it into a harder game, and then alpha zero given the same amount of compute is a worse player, and therefore it can't play as well.


Therefore, the games are less decisive.


And also, the opening diversity is higher because it doesn't know.


The game could be as decisive.


It might just be an effect of alpha zero.


For each of the chess variants we estimated, yada yada yada, okay.


No castling chess being the first variant that we analyzed has already been tried in experimental Blitz Grandmaster Tournament in Chennai, as well as a couple of longer Grandmaster games. Our assessment suggests that several of the assessed chess variants might be quite appealing to interested players and we hope that this study will prove to be a valuable resource for the wider chess community.


I don't know, is the chess community flourishing or going under recently? Because it seems to me like once a game is solved that hard by computers, I mean, it's still fun, but yeah, I just, I just, I guess Counter Strike is also solved by bots real hard.


It's still impressive when humans play or so.


Yeah, I don't know.


All of this is, again, if you're into chess, look into this paper, they have a lot of really interesting results that are not interesting to go into for the general community, but I believe this should give you a good impression of what you could do if you design a system that is built on rules.


[ML News GERMAN] NVIDIA GTC'21 | DeepMind kauft MuJoCo | Google Lernt Spreadsheet Formeln

NVIDIA erhaltet eine riesige Konferenz, DeepMind open-sourced den Mujoko Simulator und Google sagt voraus, was ihr in euren Spreadsheets zuschreibt.


Willkommen zu der deutschen Version von ML News.


Hallo und willkommen zu ML News, auch genannt Maschinelles Lernen Neuigkeiten.


Die erste Story ist folgende. NVIDIA hat eine Konferenz, die nennt sich GTC, findet am 8. bis 11. November statt und ist auch der Sponsor dieses Videos.


Ein Teil des Sponsorships war, dass ich das Video in Englisch und in Deutsch mache und deswegen hier sind wir.


Also, das ist das selbe Video in Deutsch, wie es in Englisch ist.


Wenn jemand jetzt verwirrt ist, if anyone is confused by this, what I'm speaking, switch over to the English version.


This is the German version. Without further ado, wechseln wir wieder auf Deutsch.


Die NVIDIA GTC Konferenz ist eine Riesenkonferenz. NVIDIA hat nicht nur die Keynote, wo sie über NVIDIA Sachen sprechen, was NVIDIA macht, was Neues ist von NVIDIA.


Das ist alles wichtig für Machine Learning, weil NVIDIA ist ziemlich, ziemlich große Firma und was NVIDIA macht ist relevant für alle Machine Learning Leute.


Aber die haben auch unglaublich viele Speakers eingeladen, die Talks geben, die nicht wirklich was mit NVIDIA zu tun haben.


NVIDIA will einfach, dass die Konferenz ein großes Event wird und das steht alles zur Verfügung gratis eigentlich.


Die Talks, die werden alle gehalten in dieser Konferenz online, man kann zuschauen, das ist gratis, aber man muss sich registrieren dafür.


Jetzt, wenn ihr euch registrieren wollt, dann könnt ihr entweder hier klicken, nicht so interessant, oder ihr könnt mein Link benutzen und ihr habt auch die Chance, eine NVIDIA 3090 zu gewinnen.


Wenn ihr meinen Link nutzt, die Karte ist nur für Leute, die meinen Link benutzen, also ihr compete nicht mit dem Rest von YouTube, sondern nur mit den Leuten, die den selben Link benutzen.


Und ja, es spricht eigentlich nichts dagegen. Ist gratis, ihr kriegt einen Haufen gratis Content und ihr könnt die Karte gewinnen.


Wie ihr sehen könnt, die Karte ist sehr gut, Frames per Second hoch, das heißt, man ist ein besserer Gamer, wenn die Frames hoch sind.


Ich glaube, so funktioniert Gaming, oder? Ich habe schon lange Gaming mehr gemacht.


Aber man kann die Karte natürlich auch für Deep Learning benutzen, wenn man gerade eine Pause zwischen zwei Fortnite-Battles macht.


Es ist wirklich erstaunlich, wie gut diese Karten geworden sind in den letzten Jahren.


Und deswegen, ironischerweise ist es von Cyberpunk, ich glaube, die Grafik ist auch wirklich das einzige, was an dem Game gut war, schlussendlich.


Okay, NVIDIA sagt, ich soll diese Keynote mehr hervorheben.


Also was wird in dieser Keynote passieren? Das wird die beste Keynote, die alle je gesehen haben.


Vielleicht erinnert ihr euch an die letzte Keynote, wo Jensen Huang gerendert wurde.


Das war in ML News, ich habe das reported.


Und NVIDIA hat einen großen Deal daraus gemacht, wie viel Arbeit sie da reingesteckt haben.


Ja, sieht wirklich gut aus. Aber dann mussten sie das irgendwie zurück schrauben, weil es stellte sich raus, dass nur 14 Sekunden der anderthalbstündigen Keynote wirklich gerendert waren.


Der Rest war der richtige Huang und es war ein bisschen confusion da.


Ich weiß nicht, was passieren wird in der Keynote, aber es wird epic.


Habt ihr gesehen, dass wenn ihr auf Twitter GTC21 als Hashtag benutzt, wird so eine kleine Lederjacke daneben gerendert.


Ich bin mir ziemlich sicher, dass NVIDIA für das Geld bezahlt hat, geniale Marketing, geniale Business entscheidet, von Twitter Hashtags zu verkaufen.


Genius.


Und hier ist was, was ich noch gesehen habe und was spekuliert wird, dass vielleicht in der Keynote drüber gesprochen wird.


Das ist diese Omniverse-Plattform von NVIDIA.


Ich habe die nicht gekannt, aber die ist ziemlich cool. Das ist so ein Real-Time-Rendering-Framework und damit kann man in real time Sachen machen, die bis vor ein paar Jahren wirklich große, große Mengen an Rendering gebraucht hätten über Tage hinweg.


Das sieht alles ziemlich, ziemlich cool aus.


Ja, vielleicht kommt das in der Keynote, vielleicht nicht.


Das da... who knows.


Die Konferenz selber, ich habe schon gesagt, 8. bis 11. November. Die Keynote ist das Highlight von Jensen Huang am 9.


Aber die Speakers sind wirklich auch gut und die meisten Talks sind natürlich in Englisch.


Ihr könnt hier die Konferenzschedule anschauen. Es gibt über 500 Sessions hier, also es ist wirklich ein riesen Ding.


High Torch selbst hat 15 Sessions, TensorFlow hat 12 und die Industrien und Topics über die geredet werden.


Die Liste ist länger als ich ein Screen habe, also genau.


Es gibt dann eben auch Conference-Workshops und Trainings, die sind die meisten in Englisch, aber das sind Hands-on-Training, praktische Trainings.


Das kostet ein bisschen Geld, aber da lernt man wirklich von Instruktoren, Instruiert, wie man zum Beispiel ein Transformer Natural Language Processing Applications baut.


Oder die Fundamentals, die Fundamente des beschleunigten Datenwissenschaften.


Mein Echtzeitübersetzer funktioniert nicht so ganz.


Genau, das ist eigentlich schon alles für dieses Segment.


Kein Grund, die Konferenz nicht abzuchecken. Viele coole Speakers, die Keynote ist sicher interessant.


Letztes Jahr war ein virtueller Jensen Huang dabei.


Wir sehen, was dieses Jahr wird.


Also registrieren und toi toi toi, ich hoffe ihr gewinnt.


Und wir sehen uns in den nächsten News.


Die Mind hat den Simulator Mujoko gekauft und Open Source den.


Mujoko ist ein Robotics Simulator. Ich kann keine deutschen Wörter sagen.


So, hier seht ihr einen zum Beispiel so einen Real Life Flip von so einem Spinny-Ding. Keine Ahnung, ein Zirkel oder sowas.


Und Mujoko ist bekannt dafür, dass es akkurat und schnell ist.


Also es ist eine gute Piste, die man sich nicht verletzen kann.


Es ist eine gute Balance zwischen genug von diesen richtigen physikbasierten Sachen zu Capturen.


Zum Beispiel diese Gyroskop Effekte hier. Man sieht die hier in Zero Gravity.


Die Drehachse wendet sich immer wieder um 180 Grad.


Viele von diesen Simulatoren, die zum Beispiel für Gaming gemacht sind, die beinhalten diese Effekte nicht.


Mujoko beinhaltet diese Effekte und ist trotzdem schnell genug, dass man es für Sachen wie Reinforcer,


Control und so weiter benutzen kann.


Und das ist natürlich, was DeepMind in den letzten Jahren gemacht hat.


Mujoko ist wirklich benutzt in der Robotics und Control Community.


Aber das Problem war nicht nur, dass es nicht Open Source war, sondern auch, dass die Lizenz dafür einiges an Geld gekostet hat.


Und das hieß, dass unabhängige Researcher oder Researcher in gewissen Labs, an Unis nicht unbedingt den Zugang hatten, um damit zu arbeiten.


DeepMind hat jetzt den Simulator gekauft und hat den Open Sourced.


Man kann sehen, DeepMind hat schon einiges gemacht an Arbeit.


Und hier seht ihr einfach, wie flexibel dieser Simulator ist.


Genau, es waren einige Werke im Gange, diesen Simulator zu ersetzen mit Open Source Software.


Aber wie gesagt, die Balance, die Mujoko hat und die Arbeit dahinter, ist beträchtlich und scheint ein einfacher Weg, den einfach Open Source zur Verfügung zu stellen.


Wieso genau DeepMind das macht, weiß ich nicht.


Vielleicht gute PR ist auch mal was Gutes oder vielleicht haben die einen anderen Grund.


Wie auch immer, wir freuen uns, dass der Simulator zur Verfügung steht und viele mehr Leute wirklich ins Continuous Reinforcement Learning reisen.


Das fand ich noch cool.


Genau, der Rest des Körpers steht einfach komplett still.


Und das Bein.


Der Punkt ist nicht, dass es super duper realistisch ist, sondern einfach, dass die Interaktionen zwischen den Objekten realistisch sind.


Und das macht Mujoko ziemlich gut.


Also, wenn ihr solche Simulatoren mögt, wenn ihr Research damit machen wollt, checkt ihn out.


Er ist in C geschrieben, das heißt, man kann das überall kompilieren, läuft überall und es gibt Integration dafür, für ziemlich jede Sprache insbesondere Python.


Okay, PyTorch hat ein 1.10 Release, das ist ein neues Release der PyTorch Library und zum Beispiel gibt es jetzt CUDA Graphs in PyTorch.


CUDA Graphs, das sind nicht Graph-neuronal-Netze, sondern das sind Graphen von CUDA Kernel.


Also, ein CUDA Kernel ist zum Beispiel eine Matrix-Multiplikation oder eine Addition von zwei Sachen und die CUDA Graphs API, die macht jetzt, dass man nicht nur diese einzelnen individuellen Kernel starten kann von der CPU, sondern man kann einen ganzen Graph davon starten.


Also, vorher musste man wirklich für jede Operation auf der GPU, musste die CPU sagen, jetzt mach bitte eine Matrix-Multiplikation, jetzt mach ich bitte eine Addition und so weiter und das hat ganz schöne Latenz gekostet.


Und jetzt mit dieser API kann die CPU eigentlich einen Befehl schicken und sagen, bitte mach eine Matrix-Multiplikation, gefolgt von einer Addition und so weiter und so fort und solche ganzen Computation Graphs eigentlich an die GPU schicken.


Das macht die GPU schneller, muss nicht warten, keine Kommunikation, alles cool und das ist alles jetzt zur Verfügung in der neuen PyTorch Release.


Darunter gibt es auch andere Sachen, zum Beispiel dieses torch.special-Modul, das kopiert das SciPy Special Modul, also wenn ihr das SciPy Special Modul benutzt habt, bis jetzt den NumPy und SciPy, dann gibt es es jetzt auch in PyTorch.


Diese Projekte, das Ziel ist wirklich, dass die NumPy und SciPy APIs replizieren und zugänglich machen zu Deep Learning.


Das letzte, was ich highlighten will, ist die Parameterization von dem NN-Module. Das macht das, zum Beispiel wenn ich in der Vergangenheit die Normalization eines Modules austauschen wollte,


dann musste ich immer das Modul subclassen, eigentlich reimplementieren und die Normalization ersetzen mit einer anderen Normalization, wenn ich das machen wollte.


Und mit dieser neuen Parameterization API habe ich die Möglichkeit, ohne das Modul komplett neu zu schreiben, verschiedene Sachen darin zu ändern.


Und das heißt, es ist einfach mehr zugänglich für Wissenschaftler, um damit rumzuspielen und ziemlich einfach neue Ideen auszutesten.


Ziemlich, ziemlich cool. PyTorch 1.10 ausprobieren. Yes, please!


Google hat eine neue SEPA-Release. Das Paper ist schon ein bisschen älter, es war in ICML und es heißt Spreadsheet Coder.


Formular Prediction from Semi-Structured Context. Und was es macht ist, es funktioniert in Google Spreadsheets und es sagt voraus, was ein Benutzer gerne hätte als Formel.


Also sobald man das gleich tippt in einer Zelle, sagt das jetzt voraus, was für eine Formel ich gerne darin hätte.


Und es ist ziemlich, ziemlich gut, wie das Paper zeigt. Beispiel hier, zweimal weiß es, dass ich gerne die Summe möchte und der Benutzer hier in der dritten Zeile schlägt es nicht mehr die Summe vor,


sondern eine Formel, die wirklich den Prozent von Veränderungen hervorgibt.


Das Prinzip funktioniert etwa gleich wie wenn ihr in Gmail oder in Google Docs die Tab-Completion gewöhnt sind. Auch hier wird etwas vorgeschlagen und mit Tab kann man das kompleten.


Das funktioniert eigentlich so, dass das System ganz viele Sachen in Betracht zieht, zum Beispiel die Values in den Zellen, die um die Zelle herumliegen, die man möchte, die Row-Headers, die Column-Headers und all das.


Und daraus wird dann abgeleitet, was man gerne für eine Formel hätte. Zum Beispiel, weil die Row-Header hier total heißt und die Werte hier oben dran eigentlich in der Column stehen, versteht das System, dass ich gerne die Summe hätte.


Aber hier für die D-Spalte sieht es, dass hier keine Werte oben dran sind, das heisst wahrscheinlich will ich nicht die Summe und hier habe ich einen Header, der heisst %Change und das gibt dem System eine Indikation, was man gerne hätte.


Das ist jetzt zur Verfügung in wirklich, das ist General Availability für Google Spreadsheets, das heisst jeder und jede der Google Spreadsheets verwendet, kann dieses Feature jetzt benutzen.


Das ist ziemlich cool, dass Research relativ schnell von Paper wirklich zu einer Produktimplementierung geht, das passiert nicht so oft und es ist ziemlich cool und geben das Google Spreadsheets ist ein gratis Produkt, da steht es jetzt eigentlich allen zur Verfügung.


Dieses System selbst ist einigermaßen komplex, also man sieht, da gibt es ein Row-Based und ein Column-Based BERT-Encoders, dann Convolutions, Skip-Connections, bis man wirklich ein Aggregate Embedding hat von dem umliegenden Kontext um eine Zelle und daraus wird dann mit einem LSTM decoded, was man gerne für eine Formel hätte.


Ich glaube das braucht schon, hat schon einen gewissen Engineering Effort gebraucht, bis die wirklich an dem Punkt waren, wo das so funktioniert hat, aber ziemlich cool, dass es jetzt auch wirklich funktioniert.


Wir haben gewisse Ablation Studies gemacht und wenn ihr interessiert seid, am besten checkt das Paper out, Spreadsheet Coder, Formula Prediction from Semi-Structured Context.


Das nächste ist HandTracking.io, das war ein Projekt, das auf Reddit ziemlich viel aufmerksam gebraucht oder gezogen hat und das ist ein cooles Projekt, das HandTracking macht im Browser eigentlich und die fokussieren spezifisch auf gewisse Gesten, die man mit der Hand macht, zum Beispiel das Finger-Pinching oder die Faust machen und die mappen diese Gesten dann zu Aktionen, die man machen kann.


Und hier, die Aktionen sind Zeichnen und den Bildschirm löschen, das kann man auch ausprobieren hier im Browser, hier wenn ich die Faust mache, cleart das den Screen, wenn ich die Finger pinche, es funktioniert nicht super duper, vor allem wenn man ein bisschen schnell ist, aber wer sehen kann, ziemlich cool.


Also wenn ihr Applikationen habt für HandTracking, das scheint auch ziemlich lightweight zu sein, das funktioniert im Browser hier und funktioniert mit irgendwie 40 frames per second oder so, obwohl ich noch ein OBS Recording und zwei Screens habe, also ziemlich cool.


Plus es ist MIT License, es ist auf GitHub, das heißt ihr könnt es anpassen wie immer ihr wollt. Excellent.


Sartorius ist eine Zellinstanzsegmentierungschallenge auf Kaggle. 75.000 US Dollar Prize Money, der Task ist, ihr kriegt so ein Mikroskopiebild und ihr müsst dann die Zellen segmentieren, das heißt die einzelnen Zellen sagen, wo die sind und wo die nicht sind und anscheinend ist das für gewisse Arten von Zellen wirklich ein ungelöstes oder nicht gut gelöstes Problem im Moment.


Also wenn ihr gerne was mit Computer Vision machen wollt und das auch wirklich real life applications hat, das dann eingesetzt werden kann, vielleicht Sartorius ist für euch.


Kaggle ist verfügbar für alle, scheint eine coole Challenge zu sein, gibt ein bisschen Geld zu gewinnen, ja.


So dieser Teil ist über Libraries, über Bibliotheken, die ich diese Woche gefunden habe und die irgendwie hilfreich sein könnten. Das erste ist Control Flag, ein self supervised idiosyncratic pattern detection system for software control structures.


Das ist eine selbst supervised idiosyncratic Muster Erkennungssystem für Software, wie sagt man das? Software, das ist Software auf Deutsch. Kontrollstrukturen.


Also das ist eine Software, ein System, das in einem self supervised Art gelernt hat, Source Code zu lesen. Das heißt, da war keine Supervision, da waren keine Labels, wo jemand gesagt hat, hier ist ein Bug, hier ist kein Bug, hier ist ein Bug, hier ist kein Bug, sondern das hat einfach GitHub angeschaut und gelernt, was so gängige Muster sind, wenn Code geschrieben wird.


Das ist nicht genau das Gleiche wie OpenAI Codecs oder sowas, weil OpenAI Codecs scheint einfach ein Language Model zu sein. Das System hier ist wirklich mehr auf Source Code ausgerichtet, auch sprachspezifisch.


Wie ich das sehe, im Moment gibt es das für die Sprachen C und Verilog, aber man kann das einfach auf anderen Sprachen trainieren.


Das passt wirklich den Source Code selbst und repräsentiert dann dieses geparste, diesen Syntax Tree in einer Deep Learning zugänglichen System oder Form.


Und dann kann es entscheiden, ob eine bestimmte Sequenz von Source Code üblich oder unüblich ist.


Und wenn es unüblich ist, dann kann man den Benutzer notifizieren und sagen, hier ist wahrscheinlich ein Fehler, hier ist wahrscheinlich ein Bug, schau das nochmal an.


Das ist ziemlich unüblich, was du geschrieben hast. Also das kann man wirklich benutzen, nicht nur um Bugs im Sinne von wirklichen Fehlern, wirklichen Syntaxfehlern oder so zu finden,


sondern auch für vielleicht komisch implementierte Algorithmen oder Sachen, die einen Memory Leak geben können und so weiter.


Also ziemlich cool, wenn ihr C schreibt und nicht sicher seid, ob ihr das gut macht, dann Control Flag ist vielleicht ein gutes Projekt.


Code für den Code Checker ist auch available, das heißt, wenn ihr das auf eine andere Sprache erweitern wollt, gerne das trainieren wollt, dann ist das auch verfügbar.


Also das einzige, was nicht geht, es wird wahrscheinlich nicht Bugs detekten, die andere Leute auch oft schreiben, weil dann ist es ja wieder ein übliches Muster.


Aber man kann nicht alles haben.


Salina von Facebook ist eine Lightweight Library für sequenzielle Agenten. Das ist eine Bibliothek, die sehr einfach macht, komplexe, sequenzielle Entscheidungsprobleme zu modellieren.


Zum Beispiel Reinforcement Learning Agenten, aber nicht nur. Die haben verschiedene Beispiele, zum Beispiel A2C ist ein einfacher Reinforcement Learning Agent


und obwohl es ein einfacher Agent ist, diesen zu implementieren, ist trotzdem immer ein bisschen kritisch, ist immer ein bisschen schwierig und Salina macht dies recht einfach, wie man es hier sehen kann.


Es scheint eine gute Balance zu sein zwischen Einfachheit von Implementierung, aber nicht sehr superstrick auf eine gewisse Domain.


Also es geht weit über Reinforcement Learning hinaus.


Also wenn ihr Probleme habt oder Code habt, der sequenzielle Entscheidungsprobleme macht, aber die bisherigen Reinforcement Learning Libraries waren ein bisschen zu restriktiv, zum Beispiel,


dann könnte Salina irgendwas für euch sein.


Auch mit sequenziellen Sachen ist Y-Data Synthetic, das sind Datengeneratoren für synthetische Daten.


Synthetische Daten braucht man oft, wenn man zum Beispiel auf Testdaten trainieren möchte und nicht auf echten Daten oder wenn man nicht sehr viele echte Daten hat und mehr davon machen möchte,


wenn man zum Beispiel unbalancierte Klassen hat und von der einen Klasse einfach mehr Daten haben möchte, dann greift man oft zu Generatoren, die von den richtigen Daten lernen, aber dann synthetische Daten generieren können.


Das kann man auch verwenden, zum Beispiel, wenn die echten Daten Privatsphären geschützt sein müssen und so weiter.


Da gibt es sehr viele Möglichkeiten.


Diese Library hier, die ist insbesondere ausgerichtet auf Tabellendaten und auf Zeitserien Daten.


Und die sind oft schwieriger mit sowas wie ein GAN zu modellieren.


Wir wissen bis jetzt, wie man Bilder macht.


Wir wissen auch, wie man Textgeneratoren macht.


Aber synthetische Daten für Tabellen und für Zeitserien sind oft ein bisschen noch unzugänglich.


Und diese Library hier macht es relativ einfach.


Zum Beispiel hier trainiert diese Library für ein GAN, für dieses Credit Card Fraud Dataset.


Und man kann sehen, wenn die Training Steps mehr und mehr werden, wie dieser GAN diese hellblaue Klasse besser und besser abbilden kann.


Und dadurch kann man auf diesen neuen synthetischen Daten trainieren anstelle von den richtigen Daten.


AIM ist eine Open Source Experiment Tracking Library.


Es gibt viele davon, ich weiß.


Aber dieses hier ist wirklich ein aktives Projekt, The System.


Also wenn ihr gerne sowas wie Arch Linux habt, wenn ihr eure eigenen Bootloaders schreibt und gern Machine Learning macht,


dann könnte dies hier ein Projekt sein, wo ihr vielleicht auch was nicht nur benutzen, sondern vielleicht auch was kontributen wollt.


Dieses neue Release handelt spezifisch den Fall, wenn man wirklich viele Experimente hat.


Das war anscheinend ein Problem für die in der Vergangenheit, ist jetzt nicht mehr.


Man kann hier sehen, Average Run Query Execution Time über 2000 Runs ist unter einer Sekunde.


Also da gibt es viele, viele Sachen, die hier neu sind.


Ich weiß, es gibt schon ein paar von diesen Trackern, aber das scheint wirklich ein Projekt, wo man, wie ich gesagt habe, auch was zu beitragen kann.


Die Roadmap hier hat noch einige In Progress Items, einige Checkboxes zum Füllen und es integriert auch mit den meisten großen Frameworks, wie ihr sehen könnt.


Also wenn ihr gerne rumhackt, wenn ihr gerne neue Sachen habt, vielleicht auch beitragt, AIM könnte was für euch sein.


Das letzte ist Robust Bench.


Das ist ein standardisiertes Benchmark für Adversarial Robustness.


Also wenn ihr im Feld von Adversarial Examples arbeitet, robuste Modelle trainiert oder robuste Modelle versucht anzugreifen mit neuen Attacks, dann ist dieses Benchmark für euch.


Ihr könnt hier einfach eure Defense oder eure Attacks reinpluggen.


Die evaluieren das gegen andere über 80 state of the art robuste Modelle in ihrem Model Zoo.


Es gibt ein Leaderboard und das Ganze scheint recht einfach zu sein und vor allem standardisiert.


Das heißt, man kann wirklich vergleichen von Paper zu Paper, was oft in der Adversarial Examples Welt sehr, sehr, sehr schwierig ist.


Also Robust Bench, check it out.


Ein bisschen lustiger, CBS Local San Francisco reported, dass es anscheinend eine Straße in San Francisco gibt, wo die Waymo Cars reingehen, dann gibt es ein Dead End, also dann gibt es eine Sackgasse und dann drehen die Wagen um und gehen wieder raus.


Und das passiert etwa einmal alle fünf Minuten.


Die Autos hier auf dem Video, die haben alle Leute hinter dem Steuer, das heißt manchmal haben die Leute sogar eine Hand am Steuerrad, aber ich denke mal, die machen hier Testfahrten für selbstfahrende Autos und niemand hat wirklich eine Ahnung, wieso so viele von den Autos ständig in diese eine Sackgasse fahren und dann wieder umdrehen.


Die Fahrer wissen nicht wirklich, was abgeht, die sagen einfach, das Auto ist so programmiert, dass es da reingeht oder sowas.


Ich kann mir vorstellen, dass das Routing, die Karte, die die Wagen intern haben, die da durchführen möchte und dieses Dead End, diese Sackgasse nicht wirklich eingezeichnet ist und aus irgendeinem Grund das Update, dass dort eine Sackgasse ist, fehlschlägt.


Und deswegen gehen einfach die ganze Zeit Autos da durch, weiß ich nicht.


Aber die Zukunft mit selbstfahrenden Autos wird wahrscheinlich relativ cool werden.


Ich nehme an, es gibt dann Wettbewerber, die die meisten Autos in einer Sackgasse stecken lassen können und so weiter.


Wird wahrscheinlich witzig.


Okay, das letzte, was ich gesehen habe, ist dieses kleine Video hier, das ist von der Firma, die heißt Blue River Technology und ist super super cool.


Die nennen sich selbst ein bisschen die Boston Dynamics von der Agrikultur, von der Landwirtschaft und es sind die selben Kontrollalgorithmen, die wir in Drohnen und in kleinen Autos usw. sehen oder in anderen Roboter.


Aber es sieht einfach 10 mal so cool aus, wenn das auf irgendeiner Landwirtschaftsmaschine oder auf einem Traktor läuft.


Einfach 5 oder 10 Tonnen, die da auf diesen zwei kleinen Rädern perfekt balancieren, ist wirklich wirklich impressive.


Das Businessmodell der Firma selbst ist nicht wirklich das Balancieren auf zwei Rädern, sondern generell die Artificial Intelligence, die künstliche Intelligenz in die Landwirtschaft zu bringen.


Die Website hat viele Fotos, wo alle Leute lächeln und alles ist gut und die Sonne scheint und die Natur ist toll.


Ich glaube wirklich nochmal Leute, die lachen, ja alle sind froh, da zu arbeiten ist das Beste.


Ich glaube wirklich, dass das AI in der Landwirtschaft eine gute Chance hat, viel Positives zu bringen.


Wir können zum Beispiel umweltfreundlicher arbeiten, effizienter arbeiten, wir können mehr aus dem Boden herausholen und den Boden gleichzeitig besser erhalten usw.


Ich glaube da gibt es schon viel zu tun, ich weiß nicht, keine Ahnung, ob Blue River wirklich eine gute Sache ist oder nicht.


Ich fand einfach das Video relativ cool.


ALiBi - Train Short, Test Long: Attention with linear biases enables input length extrapolation

Hello there. Today we'll look at train short, test long,


attention with linear biases enables input length extrapolation,


also called Alibi by Ophir Press,


Noah A. Smith, and Mike Lewis.


So on a high level, this paper replaces the position encodings or position embeddings


of transformers by a new very simple system that enables


these transformers to extrapolate to much longer sequences at inference time


than they have been trained on.


So you can train on quite short sequences and then inference will not suffer,


will not degrade even if the inference sequence length is much longer than the training sequence length.


This goes from two times longer to 10 times longer to more.


So this builds on what people have learned on position encodings in the last few years,


what works and what doesn't, and it sort of advances this one more step.


There's still room for improvement after this, but it's quite a simple thing to do.


The code is available. I'll link to it in the description.


And it seems like it might be worth a try if you implement transformer based language models


and you want to infer on longer sequences than you've trained on, give this a try.


As always, if you enjoy paper reviews, don't hesitate to subscribe


and tell me in the comments what you think.


All right, let's get into it. So what's the problem?


The problem is position encodings.


As we've said, transformers were released in 2017 by the original Attention is All You Need paper


and they already dealt with the question of position encodings.


Now, why is that?


That's because a transformer fundamentally isn't a sequence model per se, it's actually a set model, right?


So let's say you have a sequence of tokens.


And in this paper, we exclusively deal with sort of autoregressive text generation,


but there's no actual reason why this is the only case where this should be useful.


But that's what we're dealing with.


So you want to predict the next token from a series of tokens.


So here you have five tokens and you want to predict the next one that comes after that,


and then the one after that, and then the one after that, and so on.


So since a transformer essentially transforms a sequence of inputs into an equally sized sequence of outputs in every layer,


the transformer other than a fully connected network, the transformer itself doesn't really know per se where a particular item is.


So for example, for this node right here, the transformer would generate the query,


and then match that up to keys that are emitted here, and then it would route information via the inner product.


However, it doesn't matter if this node here, for example, is here or over here,


if it has the same key, the information routing happens the same way.


Ergo, to the transformer, it doesn't matter where the inputs are.


So essentially, it's dealing with the input sequence as a set and not a sequence.


Now recognizing that the original transformer already had to deal with position embeddings,


meaning, you know, if, let's say every sequence element comes in and initially, like the initial sequence, you give every token an embedding.


So these are your standard token embeddings that you know from Word2Vec or GloVe or something like this.


So initially, you give every token a similar embedding.


Now let's say these two tokens here are actually the same token.


So the cat and the ant, okay, maybe not.


But so two words can be the same, right, in the same sentence,


even though they might mean a bit different things because they're at different places.


So what you want to do is you want to augment these embeddings right here by position embeddings.


And the position embeddings can be as simple as simply appending, let's say, okay, to any of these vectors, I append one dimension, I simply write the position in it.


So this is value zero, this is value one, this is value two, I simply append the dimension and I put the number there.


This won't work too well, because we're sort of in linear space and numbers between zero and one and so on.


So there are various schemes how to do this.


The first scheme that the original paper came up with is this scheme of the sinusoidal encodings,


which means that if we, let's go down here, this is our sequence, how do we make the position encodings?


And they said, why don't we, or let's make six, why don't we have multiple dimensions of position encodings?


So our position encoding is a vector.


Now, let's say that the one dimension, we simply index a really long sine wave.


So the sine wave would continue back here, a really long sine wave by the position.


So this token would get, so here is the zero, right, this is a sine wave.


So the first one would be assigned a zero, then this one would be assigned like a 0.5, this one like a 0.7, 0.5, and so on.


Right, you see like, so, but then these aren't unique, right?


For example, this and this, they have the same one on the first dimension, let's say.


Well, in the second dimension, we'll do a sine wave, but we'll make it double, double as fast, like this.


Okay, and now again, we index all the tokens by where they are.


So this again would be zero, this maybe 0.7 here.


Now this would be also 0.7 maybe, and now this would be, this is almost, this is like 0.1.


So now you can see this vector here is already different from this vector here.


So as you build up your sine waves, you can make them even faster, right?


And even faster, as you build that up, you eventually get unique representations for each position.


But also, the advantages, and that's what the original paper hypothesized, is that now the transformer can reason sort of about distances between tokens.


So it can say, well, if two things are relatively close, you know, in this top most dimension right here,


I can be reasonably sure they're kind of close together.


But how close together? Well, if they're also pretty close in the lower dimensions, then they're probably right next to each other, right?


Or it can say, well, I want something that's like, you know, medium size apart from this word that I'm on.


Not right next to it, but, you know, kind of away.


So it would look for something that's kind of different in one of these dimensions.


So the hypothesis was that, you know, with these things, it could reason about absolute and relative positions from the tokens to each other, right?


It doesn't have to learn that relationship between word one and word three, and word two and word four separately.


It could actually just learn at one point the relationship between any two words that are a bump apart in this dimension, and then that would replicate across.


And it could potentially also extrapolate.


However, this didn't turn out to work really well.


And that is for two reasons.


At least this paper makes it seem like that's for two reasons.


The first reason is that it doesn't like the embeddings themselves don't really seem to extrapolate that well.


So the functions that are learned from these embeddings, it's not like they transfer to longer sequences as much.


That's the first point.


The second point is these vectors that we build up here, the position encodings.


What they were doing is they were simply adding them to the vectors that are the word embeddings.


And you know, that works fine, I guess, especially if you also train the word embeddings at the same time, the model can sort of circumvent that.


But as you go up the layers, you have to carry through this information.


So now all your computations within a layer have to first of all deal with what are the meaning of the tokens and how they relate to each other.


But second, it would also have to carry through this positional information to the upper layers.


And that's where more follow up positional encodings made a sort of a difference in that, for example, they said something like,


well, we don't want to just add them to the bottom.


We also kind of want to inject them into every layer separately, right?


We inject them here, we inject them up here and so on.


So the model always has access to the position encodings firsthand and doesn't need to carry through this information.


So this is one of the improvements that has happened.


The second improvement is to simply switch up the sinusoidal encodings by themselves.


And that's a thing that we're going to see today.


And the third is actually related to the first one a little bit, is that if you say I'm going to inject the position information everywhere,


it also matters where and how you inject the position information.


So as you might know, if there is an incoming embedding here, for every token, we're actually going to create a query, a key and a value.


And the trick seems to be that if I only inject the position information into the query and the key and not the value, right?


If I inject it into the query and the key, I influence how information is routed here that influences that.


But then the actual information that's transmitted to the next layer, those are the values.


And I do not inject the position information into the values at all.


Therefore, the information that flows from layer to layer to layer has no positional information in it at all, at least not directly.


Because the values remain position information free.


We inject the position information at every layer into the queries and the keys or the computation that we do with them.


So these are the sort of improvements that came together in the last few papers.


They compare different embeddings right here.


So this sinusoidal is the original one.


Rotary embeddings as they're used in GPT-J.


T5 bias as it's used in T5.


And then their new one, Alibi.


And here you can see this model, for example, is trained on 1024 tokens in its training distribution.


However, when they inference, when they make new inference on longer tokens, you can see right here, everything performs quite well.


This is perplexity. Lower is better.


If you go longer, the sinusoidal embeddings shoot up immediately.


So they fail immediately.


Also, the rotary embeddings, they don't seem to cope super well.


A bit more, but not super well.


So even if you go double the sequence length, they sort of fail.


The T5 bias is better.


But the T5 bias is a learned embedding, takes more memory and needs longer to compute and to train, which is a disadvantage there.


Also, it degrades relatively quickly.


And then the Alibi embeddings that they suggest, they are not learned.


They are fixed embeddings like the sinusoidal and the rotary embeddings.


But they can deal with way longer sequences right here.


So they keep up the speed of not having to learn embeddings.


They keep up the not wasting memory on things because they're not learned.


They don't increase the computation time.


And they manage still to bias the model in a way that it can extrapolate to much longer sequences.


So how does it do this?


Yeah, so here you can see memory stays relatively low, doesn't increase.


Inference speed stays relatively high, training speed stays relatively high.


How does it do this? Here is the main model, the main way that we do this.


So, as I said, we're dealing with auto-regressive language modeling, which means that we're dealing with causal attention.


That's why only a triangular matrix appears right here.


There is in my mind not really a reason why this can't be extended to full self-attention.


In this case, you just fill in sort of the rest of the triangular matrix right here.


But consider again our model of transforming a sequence to another sequence.


And just view one single token like this token right here.


This token produces Q2, query 2.


And it pays attention to all of the keys in the input sequence, right?


This is the attention mechanism.


The query is multiplied with all of the keys to decide where it should get its information from.


Now, if we simply do it like this, and this is with the causal attention,


it can only actually pay attention to all the keys that come before it.


So, query 2 would be multiplied only by key 1 and key 2, and not key 3, because it can't look into the future.


So, if it were just that, then as you can see from this calculation, there's no notable difference between these and these, right?


It depends only on what the key is to decide on the information, not the position at all.


Now, what we do is pretty, pretty simple.


We simply add the distance between the two positions.


So, for query 2 and key 2, this here, the distance is 0 because they are the same position in the sequence.


So, this is token number 2 in layer L, and this up here is token also number 2 in layer, I'm terrible at doing L, L plus 1.


Okay, if it's the same token, we don't do anything.


Other than that, we add the distance or we subtract the distance right here, multiplied by a number m.


This is really a number, so I was also surprised, m is a number, just a number, like 0.7 or something like this.


So, you can see, the further into the past a given key is, so the further into the past, the more is subtracted from the attention value.


Remember, these things here are attention values.


These things decide if this is high, that means that, oh, key 3 is really relevant for query 3, right?


If this is high, it means key 2 is really relevant for query number 5.


And what this here does is it simply says, well, however, the further in the past it is, the more we are simply going to subtract from that value.


So, whatever value you compute, however important it is, the further in the past, the more we're simply going to subtract from it.


And we'll do that in a linear fashion.


So, if your token is here and you look back, then it sort of degrades linearly.


You just subtract more and more and more and more from that value.


You can go negative as much as you want.


Why? Why does this make sense?


I was first a bit confused. I'm like, wait, you just subtract?


Like, it seems like you might want to multiply or something like this.


But remember, once, for example, for query 2 here, we built the multiplication.


Sorry, this is a bit heavy.


We built the multiplication of query 2 and key 2, right?


This is an inner product.


And we also built the multiplication of query 2 and key 1.


Now, what do we do with the two things?


We do a softmax, which means that these are numbers.


And they go into a softmax, which is going to give us a distribution.


The softmax is something like e to the query 2 key i divided by sum over j e query 2 key j.


So, they go into an exponential function.


And now you can see why subtracting something makes sense.


Because essentially here, we're working, this is log space.


And therefore, subtracting something in log space essentially means that you multiply it or you divide it by a constant.


And you divide it multiple times or by a higher constant, the more in the past it is.


There we go.


If this will be the histogram without the biases, you know, with the biases, you simply say, well, whatever is more recent,


so the more of the right ones is going to be even more important after the softmax.


Of course, it's normalized.


So, this gains in importance and this would drop in importance, whatever it is.


Right. Even if it were, even if it were, this is higher initially than this, it would just decrease whatever is in the past and sort of remain whatever is close by.


Actually, it decreases everything, but it decreases whatever is in the past more.


So, it's just a bias that says whatever is in the past is less important.


Now, I told you this m is a number.


So, how do they pick the number?


And they simply come up with a scheme.


They were just like, okay, so first of all, here's the formula.


So, for routing to token i, you take the query, multiply it by all the keys and simply add m times this vector right here.


Now, I'm not sure if, you know, the order needs to be, the order needs to be correct.


So, I guess if this is the vector right here, the keys have to be sort of reverse order or something like this, because this is the most, this adds to the most recent token, this to the second most recent token and so on.


So, here is how they choose m.


m is different for each layer.


Right. No, m is different for each head.


Sorry.


m is different for each head.


So, they say, okay, if we have eight heads, the slopes that we use are the geometric sequence, the geometric sequence that starts at half and multiplies each element by a half to compute the next element.


For models that require 16 slope heads, it's a bit different.


So, as you know, transformers, they have multiple heads.


So, if the, if this attention computation is essentially split, so you have incoming signal and the attention computation is essentially split over multiple heads.


The attention computation is done somehow here and then it's averaged or added together at the end.


And they're simply saying, well, this m number in these different heads should be different.


Because it might be more useful to have a harder slope, it might be more useful to have a flatter slope.


So, they come up with this scheme where they say the slope is one half and the slope here is one quarter.


The slope here, like it's slightly less slopey, here it's slightly less slopey and so on.


So, they have these almost like different options and I quite like that.


Because I think whenever you have sort of parallel things in your architecture, like multiple heads for attention,


and it's my personal opinion that you should do something to make them different from each other.


Otherwise, you just sort of rely on noise and you build an ensemble, which is cool, right?


Ensembles are cool.


I think you can make them more effective if you say all of these different options, they're slightly different in how they work.


And the model can therefore choose a bit which one to utilize most.


Now, you can you could still replicate those if you want more capacity or anything like this.


But I'm generally a fan of doing something like that.


So, all the heads have slightly different slopes, as you can see, in how important or how unimportant they make the past.


And these slopes are predefined by them and that's it.


So, yeah, that's that.


The M is one number per head in the fashion that we've shown.


And it's really simple.


The drop off is completely linear, right?


And the simplicity might be the key right here, because now we test whether this extrapolates in the experimental results.


And you can see that this extrapolates quite well.


So, I already shown you before, of course, the perplexity in what they've shown.


But here is another test on the Wikitext dataset.


So, again, we have perplexity on the Y axis and the square dots you see, they're always the classic sinusoidal embeddings.


And they are always trained on as long a sequence as you test, because we've already seen if you make the sequence longer, they just fail.


So, here the comparison is really you train on a sequence and that is exactly the length of the testing sequence.


So, they should be perfectly adapted to that length.


Now, the top line is the new embeddings trained on 512.


So, the top line is trained on this size, yet if you test it, it already performs better.


Now, what do you make of what?


I don't know what you make of this.


Like the claim is somehow, well, it's just a better position embedding by itself, because you can see here it's already better.


I don't know.


Maybe this is also just experimental, like machine learning experiments in papers always making the baseline worse than themselves.


But what we can say is that you can see it generally the perplexity decreases or remains constant as you up the scale, even if you've trained it on a small length.


And when you actually train it on larger lengths, so this line starts here, the one they trained here, obviously, I guess they could test it on shorter sequences, but what's the point?


You become even better because you've trained on longer sequences, right?


And again, you see the same pattern also with the one that you trained on very long input.


So, in general, you see on long texts, the perplexity decreases as you train for longer, obviously, right?


So, it still has an effect, you still want to train on as long sequences as you can, because that will gain you in performance.


However, it's not too bad if you train on short sequences and then extrapolate to longer ones with this embedding.


In contrast to the sinusoidal embeddings that just completely fail when you give them anything longer than like 1.1 times the training length.


And they have various comparisons about perplexity and how many words per second.


Here is a cool plot that shows, if you train on the same length as the sinusoidal embeddings, you get much lower perplexity and only a tiny bit of a slowdown, it seems.


Probably because you inject the position encodings into every layer.


By the way, have you seen here, the position encodings, they only go to the query and key computation.


They don't go into the values at all. We don't add them to the embeddings at the beginning.


So, this is exactly one of the things we've talked about at the beginning.


So, this is how they sort of incorporate one of the learnings of the last years.


So, because you have to do this every layer, it's a tiny bit slower, but you gain a lot in perplexity.


And if you go to train with smaller sequences, obviously you're going to be faster.


And as you can see, your perplexity, it doesn't suffer too much.


In fact, in their experiments, again, take it with a grain of salt, but in their experiments, it is even lower than the full length training with the sinusoidal embeddings.


So, they go into, as I said, into various experiments right here.


In general, their message is always the same.


There is a weird phenomenon where the perplexity actually gets better as you go beyond your training length.


And they attribute this in part to the so-called early token curse phenomenon, where it depends sort of on how you split your evaluation data.


And if they modify that, they see that, at least as I understand it, they can say that,


okay, if for some evaluation protocols, we actually don't get better, so it's probably due to this early token curse.


But nevertheless, the perplexity stays flat, or you don't suffer that much if you train on short sequences.


Hey, this is Janek from the future.


Just a short addendum here to make it clear.


And they also describe this in the paper.


What is probably happening isn't that the transformer is all of a sudden able to reason about much longer contexts.


But what is probably happening is that it still only looks at the most recent context because the more distant past has been down weighted so much by these biases that it becomes irrelevant.


But nevertheless, it still enables the transformer to handle these long sequences.


And potentially, if something's really important in the past, it can pick up on that.


All right, back to the video.


So all in all, I think this is a very, very simple, cool paper.


I want to see in practice really if this works out, if this does something.


Again, they've only tested on language modeling, autoregressive language modeling, where I'm not exactly sure why they haven't tested it on other things.


Maybe they have and I've just not noticed it, though.


It should work in other things.


But only time will tell if this is really worth something, if this is really useful in practice.


If there are so many cases where you can only train on shorter things, yet evaluate on longer things.


That's why I would be also interested in non-autoregressive language modeling tasks.


Because if you have to, say, answer a question about a document, right, it's much more about integrating whole information about the document or finding relevant things in the document.


And there I'd be interested in the discrepancy between training and inference.


All right, this was it.


I hope you sort of understood what it is.


Check out the code.


Apparently, it's really pretty simple to include this in any sort of existing transformer.


And yeah, tell me what you think.


That was it.


Home Lab Build - P.2 - Rack has evolved!

So, I've done some stuff since we last spoke.


Where we last left off, I had just finished building the NAS and had added a 10 gigabit


switch.


The next addition to the rack is a 48 port patch panel.


The patch panel is used to cleanly redirect the ethernet cabling back to the machines


in the actual rack.


Essentially it's just there to keep things clean in the front and just for like organization.


There's lots of ways that you can manage ethernet cables, I just happen to like the way this


looks best.


Next, I added my 16 port gigabit switch which I'll be using for anything that doesn't need


to be super speedy.


Also this is where the internet comes in since my internet isn't faster than 1 gigabit anyways.


This also allows me to save a port on the 10 gigabit switch so I just need to have internet


in on that switch and the 1 gigabit switch has internet in and internet out essentially


just to the 10 gigabit switch but could also go out to other switches as well.


Finally, I just slapped my older 100 megabit per second switch in the rear.


I don't actually think I'm going to be using or needing this so I'm really just storing


it back there to be honest but this is also power over ethernet or PoE so I might end


up using it later for cameras or raspberry pies.


It's just there if I need it but also I'm just storing it back there.


So much of the stuff on the server rack doesn't even cover like half of the depth so there's


actually a lot of storage area back there.


There's a lot of things I might wind up putting in the back of the rack.


One annoying aspect to having multiple machines is multiple inputs and outputs.


You can certainly remote shell or remote desktop to control and operate them but there are


always downsides and nothing beats a direct connection with a desktop UI in my opinion


especially when I'm working with multiple operating systems and things that have some


sort of UI requirement.


And while I can work completely within the shell I would just usually rather work with


a UI anyways.


So along with consolidating power and networking I'm also doing this for all the other machine


I owe.


To do this I'm going to be using a KVM switch.


KVM stands for keyboard video and mouse and the switch part lets me switch between multiple


machines using that same keyboard and mouse for all of them.


The way this works is I plug my main mouse and keyboard into the KVM switch.


The KVM switch that I have has 16 VGA out.


Then I use these VGA cables that split out to VGA and USB and connect those to my machines.


Most of my machines don't have VGA out so I'm using DisplayPort and HDMI converters.


The interesting thing here to me is the KVM switch is actually reporting the mouse and


keyboard information out over VGA before it splits out to USB for the machine.


I didn't actually know that was possible.


Before mounting and wiring everything I just wanted to make sure it would work as I intended


and hoped for so here I've just connected a Windows and an Ubuntu machine and can switch


between the two of them with the push of a button.


Some KVM switches also come with remotes but I plan to use my KVM switch close enough to


just press the buttons.


So since everything's working in the KVM switch goes.


I didn't have a monitor mount quite yet so I just sat the monitor on top of the NAS unit.


It doesn't look great but it works pretty well actually and it fits.


Next in goes the universal rails to support the box machine.


This is my 2x RTX 8000 machine though I've put one of the RTX 8000s in the Puget machine


temporarily.


The box workstations can come with mounting hardware but at the time that I got the box


machine I didn't need it so I didn't get it.


One nice thing about actual sliding rails which you can get for the box machines is


they don't take up any space beyond the size of the casing that you're mounting but in


my case the box machine is going to have to sit on top of universal rails.


This means that there's the added thickness of the rail itself and then the rails that


I'm using are also variable in size so in the middle they actually double up in thickness.


While small this ends up adding up to being something like two-thirds of a centimeter


or so in added thickness which can honestly make or break a mount fitment especially if


you're using this mounting method in succession.


So in this case this added thickness is indeed enough to block me from actually getting the


box in initially.


So first it's time to make some adjustments to the casing.


Immediately I removed the top cover this was always going to have to come off.


I think it's really only here if the machine is being used in workstation form but also


because of this waviness it does add a little bit to the width of the machine when it's


in like rack mount orientation and then I also just slid off the side panel to the actual


machine.


With the official mounting hardware I am pretty confident that the side panel could actually


stay but like I said we've got a few extra millimeters with the rails that I'm using


so we need to free up any bits of space that we can.


Also I already know there's some wiggle room when it comes to the monitor mounting so I


loosened the NAS screws a bit and lifted that whole machine up just slightly but that ever


so slight bits that we can move those things that accounts for millimeters so I moved that


whole machine up slightly retightened everything and then retried the box again and in this


setup the NAS will wind up kind of serving as the boxes side panel and the only time


it will come off is if I pull out the box to do work on it or something like that so


this should still keep it closed and dust away and all that.


And with that the box machine finally fits.


So from here I wire up the box to the KVM switch and I boot that.


At the moment the box machine is actually on GTA data collection duty for the self-driving


car project so it's currently running Windows.


The next order of business is to install a 1U drawer with cable holes in the back.


I am at least for now going to be using this to house the keyboard and the mouse that controls


the KVM switch and all of the machines.


And I can confirm it works for my very official business.


While the monitor can just sit on top of the NAS and be fully functional it would look


much cooler actually mounted on here so here I have an adjustable depth monitor mount.


So I disconnected the monitor getting ready to mount it and there's trouble in paradise.


The flat part of the monitor stand comes off really easily but the neck part seems stuck


and I'm pretty confident that I'm going to have to like take the monitor apart just to


get this neck part off so I start unscrewing things and prying away and sure enough once


I get inside I can see that there's definitely no way this neck would have come off without


opening up the actual casing of the monitor and unscrewing it which is kind of weird for


a monitor with VESA mounts but so be it because this monitor is special.


I bought this monitor four years ago for one simple reason.


It was the cheapest 1080p monitor available on Amazon.


Little did I know that four years later it would actually be just about the most perfect


fit possible for my server rack.


Especially nowadays with monitors and their casing being rounded or curved very often


it's actually really challenging to find one that fits between the 19 inch posts and is


just a nice simple flat square monitor.


You can also look for actual rack mount monitors and such but what you find is extremely painful


and extremely expensive.


Also just noting the photos that are used on these product pages is kind of weird like


who's watching American football or nature documentaries on their server rack it's just


kind of weird.


So besides monitors there are also entire KVM consoles that act as the KVM switch, monitor,


keyboard, and mouse and they're all contained within one unit.


The problem is these are all proprietary in like way too many ways so if like the screen


fails what are you gonna do?


What if the keyboard fails?


Etcetera.


They're all made by companies that I have no trust or faith in, they're extremely expensive


and always have really shaky reviews and feedback.


The one benefit that they have is they can take up as little as one U of rack space and


oftentimes that is of the utmost importance.


My KVM switch alone takes up one whole rack unit and then the monitor just in size is


a bit over six units.


You could stuff in things in two of those units and they would just be like blocked


by the monitor but no matter what this whole setup that I'm doing is going to take up a


good chunk of unit space and that's why there are solutions that only take up one rack unit.


But this is my rack and 42U is way more than I need right now anyways.


I got 42U just because somehow it was the cheapest variant of the rack at the time and


I really enjoy having direct IO access and I do quite literally want to be able to monitor


my machines in the rack so for me this is just the perfect setup.


Next while I do plan to convert a few more machines to be mounted in the rack I don't


quite yet have the cases for them but I'd still like to utilize the rack networking


power and the convenience of the KVM switch.


Plus I'm also just really enjoying how easy it is to move everything around as one big


unit so I grabbed some scrap wood to cut and make a floor for the rack just temporarily.


So after cutting those pieces I've got a nice simple floor to rest machines on.


Again I promise this is temporary.


So now things are definitely starting to come together but there is one glaring issue here.


I don't currently have anything I want to put here at the moment.


I could put in that other switch that I put in the back but that switch is blue and it


just would look weird so for now in goes a blanking panel.


I can figure out what to do with that later.


Finally I'm adding a 2U drawer.


I'm going to be using this to store rack specific things like cage nuts and some spare parts


for the cases and the builds that are actually in the rack.


The rack itself also came with quite a few cage nuts and bolts but I'm starting to run


low so I wanted to get some more and this is what I got.


I actually ordered the same ones.


They're the StarTech cage nuts and bolts but they really look like they were salvaged from


like a sunken ship or something and the packaging and everything was fine.


Like the actual shipping packaging but the packaging here like I don't know where these


came from but pretty crazy.


So I ended up buying these AC Infinity cage nuts and screws from the recommendation of


someone instead.


I just had zero faith in like sending these ones back and then reordering the same ones.


Like I'm pretty sure they're all going to come from like the same warehouse and I just


felt like there's too much risk that the ones that I would get would just be super rusted


again and at least these AC Infinity ones have no rust.


Anyways, the rack is getting pretty far along now.


Moving forward I would like to add some UPS slash battery backup for sure for the NAS


maybe for some of the other small things along with converting a few more machines that I


have to rack mount and then it's really just a function of kind of cleaning up cabling


maybe adding some sort of global rack monitoring with a Raspberry Pi or something like that


but otherwise I'm quite happy with how things have turned out so far.


So that's all for now.


Hopefully you guys have enjoyed questions, comments, concerns, suggestions, whatever.


Feel free to leave them below.


Otherwise I will see you all in another video.


Addendum for Supermasks in Superposition: A Closer Look (Paper Explained)

Hi there, today we'll look at super masks in superposition again. So this is part two of this


paper by Mitchell Wurzman and Vivek Ramanujan. And here's the reason why there's a part two. So after


yesterday's video on this paper, I like I couldn't sleep because I really felt that I had left out


some important aspects that I wanted to touch on during the video. Now sometimes during videos,


I look at the clock and I realize like, oh, crap, the video is already like an hour long. And I know


people are watching on 2x speed anyway. But still, it's like too long. And I need to wrap it up really


soon. And what I felt were pretty important messages about this paper got lost. So specifically,


I want to address three different things. First of all, they have like a formal analysis, not a


formal, but a kind of more rigorous analysis of what their modified G objective does. And I also


want to give some intuition in that because I felt I felt I really hadn't done a good job at that.


The second part is that the the two different ideas right here being the super masks and the


superposition. And I think my opinion is sort of that these are two separate things. And they really


have nothing to do with each other. And I think that didn't really come through last video. And


the third one being the broader impact statement of this paper, which I you know, I usually kind


of gloss over it and go like, haha, but I hear there is an important point to it. So yeah,


we'll get to that. Alright, so again, not a new paper today, I realized this, but I think it's


worth kind of diving deeper into this paper, which is a very cool paper. You know, so so don't don't


get me wrong right here. And I feel mostly I haven't done a good part at explaining it like,


literally lying awake. Okay, so let's go to the first point. We had this. So if you hadn't seen


the first video, super masks and superposition basically says that we want to do lifelong


learning. And we want to do lifelong learning by lifelong learning is the task where you have a


bunch of tasks in sequence and you learn them in sequence. So one after the other. And basically,


the goal is to not forget tasks. Once new, you learn new tasks. And this model does it by always


building one of these super masks for each task that is applied to the same randomly initialized


base neural network each time. And you know, by by keeping the super mask around, you won't forget


the task. And then at inference time, if you're given the task and just retrieve the mask, if


you're not given the task, you can do this superposition trick where you apply all the


masks in a superposition. And then you look at sort of the gradient of an entropy function in


order to decide which task reduces the entropy the most. So which task is the most certain about a


particular data point, and that you you kind of infer that that's the task you're going to go with.


So instead of the entropy, which is, you know, well reasoned, they had this other objective they


call a G, and G basically looks at the it's really strange, it looks at the superfluous neurons. So


they also add the superfluous neurons, these s neurons right here. And they, they did g objective


will only look at the s neurons in order to decide whether or not that's the correct task. And it's


basically just the log sum x of the s neurons. And we had some intuition about them being, you know,


all small and so on them being like outlier detectors. But there is an entire chapter in


the appendix where the authors do a sort of more in depth theoretical analysis of that, which,


you know, I, it's not not necessary to do this for them. So I really enjoy, I enjoyed reading that.


And that gave me sort of the better intuition of what this g objective does. So here they say,


the aim is not to formally prove properties of the algorithm. Rather, we hope that a more


mathematical language may prove useful in extending intuition. Okay. So again, that's,


that's pretty cool. So they start off by saying, you have your neural network is basically W, and


the, the sorry, the it's, it's this phi right here. And the W are the last layers weights, which


compute your logits. So y is going not to be your class, but y is going to be your logits. And p is


going to be the probability vector over your class, which if the you calculate this via a softmax is


going to be the following expression right here. If you have a mask, right, then at least in the


last layer, you can in you can infer it as this right here. So you multiply the mask by the last


these weights, and then that gives you your logits. So they say here with they initialize


the weights right here, actually, they initialize the they have no bias term and initialize the


weights by this constant. So plus minus this constant, it's not really necessary to do that,


but they do it right here makes the analysis also a bit easier. I guess it just works more well.


If you have these masks in superposition, of course, you want to add all of these masks with


the respective alpha weighting factor, then multiply by the weights, and that gives you


your logits. So note that this this doesn't necessarily only only have to be the last


layer's weights right here. You can view that as any sort of weights of the neural network if you


formulate this phi correctly. So you don't think that they only apply the mask to the last layer,


they do apply the mask to the entire thing. Alright, now the the important part here is


what happens if we look at the derivative of G with respect to one of the alphas and take the


maximum negative derivative of that G, which is that mysterious function that only looks at the


at the at the superfluous neurons. So what they want, they kind of construct this G by principle.


What they say is, we want a function g that mimics the supervised loss, right? We want a function g


that is kind of equal like the supervised loss if we had the task ID, right? And that's, that's


pretty cool. Because, you know, the, the supervised loss, you sort of need all the information you


need the label, you need, you need all the all the you need the task ID. So the supervised loss


is unavailable. But we want a function g that in its gradient mimics the supervised loss. So they


go about constructing this right here. They say, lemma first lemma, it's possible to construct a


function g such that the gradient matches the gradient from the supervised loss for all the


supervised loss for all s neurons. So for all the superfluous neurons, specifically, we want that


the gradient with respect to the logits, if the gradient to the logits is equal, that means the


gradient to all the rest of the network is equal because the rest of the network goes through the


logits, right? The gradient through the logits is equal to the gradient of the supervised loss


to the logits for all the superfluous neurons and zero otherwise. So they say the zero otherwise is


pretty easily done in math, you know, simply set it to zero, and in the actual code, which you can


achieve like this, where m indicates the superfluous neurons. So this is just, this is just multiplied


here, and the other ones are detached. So there is no gradient flowing. This is the property that


we only look at the superfluous neurons. And now we are going to show that the gradient is going to


be equal. So they say if you had the supervised loss, which means if you have the label, then this


would be your cross entropy loss, okay, so you cross it divides into this part where you need


the label, and then this part here, you don't need the label. Now you can pretty much say, look,


the label is certainly going to be one of not the superfluous neurons, because the superfluous


neurons are superfluous, they are never the correct neuron. So this is always going to be,


you know, not the neurons we look at. So the gradient, certainly, this is always going to be


zero, because we never we, wherever the gradient is flowing, that's not where the where this is one.


So the gradient of any superfluous neuron is just this thing right here. And that's exactly why they


build the function g. So the function g has this exact gradient, the function g, if you derive it,


has that same gradient as the supervised loss for the superfluous neurons. Okay, so it's sort of


magic. But it's not, you know, it's not magic. So they need two more assumptions here to have to


get the following properties. So the for the first property, now because now we want to have g be


identifying the correct task. So we've already constructed g now we want to show that if we


really do this, the gradient with respect to the alphas, then if we do it for a wrong task for the


tasks that it's not the task of that particular data point that goes into computing g, then we'll


get a value that's probably lower than zero. However, if we plug in if we derive, but with


respect to the alpha of the correct task, then we get a gradient a negative gradient that's higher


than zero. Okay, so we're now going to prove that this with high probability really allows us to


distinguish the correct task from the wrong task. We need two assumptions right here. Assumption one


is we assume that the mask learned on task i will be independent from the data from task j. If the


task data is from task j, then this are independent random variables. Okay, so it sort of means that


the the tasks themselves are kind of independent. But it's not it's it's not the same requirement.


But you can think of in the case of permuted MNIST or so this is some it's given except if


you consider this kind of frequency of brightness and so on. But if you have independent task,


I think that this is given. That means that the features right here and the masks are independent


variable. If if the data is from task j, then the features and the mask from task i are independent


variables, sorry. The second assumption you need is that we assume that a negative weight and a


positive weight are equally likely to be masked out. Okay, so this again you can think of with


some regularity. This is certainly going to be to be given in a randomly initialized neural network.


Note that when the features are zero, which will be the case for zero mean random features.


Yeah, so yeah, before I said this was your neural network, this is your random neural network,


right? And then you mask that and so on. If this is a randomly initialized neural network,


then you can make a case that the expected features of those will be zero.


It doesn't need to be the case, but you can you can construct it such that it is. So if you have


the two things, right, if you have those two things, then you can prove the following.


If the data x comes from task j, then when you derive by an alpha that's not of task j,


you get a number that's smaller than zero in expectation. And here, the crucial part is you


reframe this gradient, you reframe, reframe, reframe. And what you'll see is that this here


comes out. So this is a sum and each element of the sum is going to be greater or equal to zero,


which means that this thing is greater or equal to zero, which means the negative thing is smaller


than zero in lemma h1. Now we're going to look at lemma h1 to get an intuition of what's going on


right here. So lemma h1 says if j is the true task, and i is not equal to j, then this quantity


here is greater than zero. Alright, I restarted my tablet and we are back. So what's kind of the


the intuition behind why this quantity here would be greater or equal to zero? And honestly, in order


to make it a bit easier, I first want to look at whenever i equals j. So whenever j is the true


task, and then i equals j, then we can sort of think of the opposite, like why, why this should


be smaller or equal to zero. So consider this, this is the run the feature of the network of you,


right? And then the EUV connects that to the to the mask at point v and the mask at point at that


point, UV is either zero or one depending on the training. So this, this xi right here,


that's going to be the from the initialization, but the mask is going to be zero or one depending


on whether that feature contributes, sorry, whether this entire thing here contributes


positively to the task or not. So the secret right here, why we can make claim that this is greater


or lower than zero, is going to be that the mask can only be zero or one, it cannot be negative one,


one, it cannot be negative one, right? So if the mask is zero, then obviously this thing is going