A notable ingredient right here is that here is where we actually bring in the human, the


Symbolic Knowledge Distillation: from General Language Models to Commonsense Models (Explained)

human annotated data into this process of automated knowledge graph generation, because


we need to train that critic.


Critical teachers or not, a student can outperform the knowledge source.


So this is about that the student model, the exceed the quality of GPT-3, which so if you


simply prompt GPT-3, you get some of these triples, right?


Yet the student models that are trained on these triples that come from GPT-3 outperform


GPT-3, which can make sense since GPT-3 is a general purpose language model.


And these student models are specifically trained on that particular kind of data.


And also, I have to say the student models, they are their GPT-2.


So in the student model, what you would do is you have your corpus, you have event relation


inference, event relation inference, right, these are your samples.


This is this is all text, essentially, right?


So the relation, you can abstract that in a either a single token, or you can make it


into a text as they did.


So they feed that into a GPT-2, which is something that you can train.


And that GPT-2 is trained to take in an event and a relation into the context, and then


generate the inference, much like GPT-3.


But now you actually train it specifically on this particular data structure and data


set.


And the GPT-2, you pre-train it, of course, on language modeling.


And it could be that some of the effect that the students model exceed the quality of GPT-3


might be due to the fact that it starts out already from a GPT-2 checkpoint.


It's a possibility, like there's a possibility that that also plays into the game right here.


Machines can now win over humans for automatic knowledge graph construction.


So that is a little bit, it's a little bit shady, since the critics you train are still


using humans.


But I would agree that at least the paper shows that there are better places to use


human knowledge than letting humans come up with a text corpus.


Because these text corpora can be generated pretty easily using large language models


and proper prompting.


And if you do that, then you can use the human knowledge to filter whatever the language


models output, and that might be much more effective.


So this was it for this paper.


I hope to not only show this paper, but show, give you a little bit of an idea of what all


is possible with these language models and proper prompt engineering.


And I think this serves as a little bit of a recipe for many, or a lot of things to come,


a lot of NLP tasks to be done could be tackled in this particular way.


Alright, so yeah, let me know what you think in the comments.


Fast reinforcement learning with generalized policy updates (Paper Explained)

Hi there, today we're looking at fast reinforcement learning with generalized policy updates by


Andre Barretto, Shavo Ho, Diana Borsa, David Silver and Doina Precu. So on high level,


this paper proposes a framework for reinforcement learning, where you have many tasks at the same


time. And they propose a framework where they learn many policies at the same time that can


or cannot correspond to these tasks. And then their argument is that if you now have a new task


that you haven't seen before, you can easily construct a solution to that task from your old


policies, basically mixing what you learned about your old tasks. And it's a pretty general framework,


and we're going to look at it. In my opinion, it's it's pretty cool for certain settings. However,


I think it kind of breaks down the more general you go, which I guess is expected of such a


framework. But it's, as you can see, it's kind of math heavy, but we'll get into the examples and


what it's potentially useful for. Alright, so that was it on a high level. If you like content like


this, don't hesitate to subscribe to the channel and share it out, leave a like and tell me in the


comments what you think. I'm still reading all of them. So I will see it. Cool, let's dive in.


So they say the combination of reinforcement learning with deep learning is promising


approach to tackle important sequential decision making problems that are currently intractable.


Well, they're taking they're talking about, you know, things like mostly these game playing


AI is like go and things like this. So where this combination of deep learning with reinforcement


learning has really shined, or shown, whatever, one obstacle to overcome is the amount of data


needed by learning systems of this type. So again, if you look at these systems like AlphaGo,


they need a simulator, and they need to collect enormous amounts of data, even more so with


systems like the Dota AI, the OpenAI5, Dota or Starcraft playing Alpha Star, I think it's Alpha


Star. They need so many simulations in order to learn about the tasks because they always start


from scratch. In this article, they say, we propose to address this issue through a divide and conquer


approach. We argue that complex decision problems can be naturally decomposed into multiple tasks


that unfold in sequence or in parallel. By associating each task with a reward function,


this problem decomposition can be seamlessly accommodated within the standard reinforcement


learning formalism. Okay, so what are they saying right here, they are basically saying that if you


have a task, let's say you want to get what's the from here to here. And that's very complicated.


Let's make it complicated, super duper complicated. You can basically subdivide that task into


multiple sub tasks, right? So here is like, left turn, right turn, go straight, left turn,


go straight, right turn, and so on. And each of these sub tasks, you can see the two right turns


here might share a lot of common information, there could also be tasks that are at the same


time, like you need to go forward and jump can be decomposed into going forward and to jump.


Now, they're saying is if each of these tasks now has its separate reward function in the


environment, like for some reason, the environment tells you this, by the way, is task, task one,


and you're going to get a positive reward if you do a right turn. And this down here is task two,


the left turn task, and you're going to get a positive reward if for that task. So the entire


task state can be decomposed into a vector. So in our case, here, we have maybe a vector with


three elements, okay, the three elements correspond to turn right, go straight, and turn left. And now


you're this, this right here is your reward vector. So we're no longer talking this framework,


we're no longer talking about just a reward, we're talking about a reward vector. Now, each of these


tasks is going to give you its own individual reward. So let's say you're here, and you're


actually turning right, this is going to give you a reward of one for this task, but rewards of zero


for the other task. Okay. So the environment will somehow tell you which tasks you you get reward


for. Now, there is a notion where you can map this back to a single number. And that is the second


thing they introduce here. So the second thing they introduce here is this thing they call w.


So w is going to be a mixing vector, w is going to be a vector, I will call w right here, this is


the reward vector, w is going to be the vector that tells you your final reward. So here, we're


going to do an inner product. So we're going to transpose this and multiply by w. And w mixes


these rewards and comes up with your final reward right here. So this, this is maybe the reward


vector, this is the reward number, how are we going to call this reward number. So in this case, w


would have to look something like this, let's say, this is an example. So the task right here would


be to only do right turns. Now, this is not a really nice example, we're going to see some


nicer examples later on. But you can see that now the environment is specified as a vector of rewards,


and you can create a specific tasks like turning right, simply by adjusting how you mix these


different things by this vector w. And this is going to be the key ingredient here. So they


discuss your general reinforcement learning, the reinforcement learning lingo. And I think we've


gone through this a number of times just very, very quickly. In reinforcement learning, you're


given these transitions, you are in a state, you take an action. And that leads you to get a reward


r prime, and you get into a state s prime in the next state, they say the reward is given by the


reward function. So the reward is purely a function of where you are and what you do and


where you get to. Now, most reinforcement learning problems, you can actually kind of forget about


this part right here. Because, well, it is, it is kind of important, but you could


most reinforcement learning problems, the reward is simply a matter of where you are and what you


do. And this can be a random variable, there can be randomness. But maybe it's easier if you for


now think about the reward simply as a function of these two things. So what you want to discover


is a policy pi, where you input, you input where you are, and the output is going to be what you


should you do in that situation. Okay, that is a policy. And associated with each policy is this


thing called a Q function. So you can see right here, the Q function of a policy


is going to be a function of where you are and what you do. And this is a bit confusing. But


it basically means that you are in state s. So you are here. And you have, let's say three options,


action one, action two, action three to do. Now the Q function tells you the Q function, this is s,


and the A's are the numbers, okay. So let's say we plug in the state s. And for a we plug in number


two, what it will tell you is, if I am in state s, and I perform action number two, then how valuable


is that for me, and value is defined by all the reward that I'm going to pick up from now until


the end of time, or the end of the episode, it depends. But let's say until the end of time,


well, how much how much reward am I going to pick up from now until the end of time?


It's a bit of a vague, not a vague question, but a difficult question. I can tell you how much I


could estimate how much reward I'm going to pick up in the next step, because I know what action


I'm doing, I'm performing action number two. But what happens after that? Who knows? So that's where


this policy right here comes in. This policy right here says, so the full definition of the Q


function is if I'm in state s, and I perform action a right now, and after that, I follow policy pi,


what is my reward going to be right now? It's well defined. So right now you do action


a. And after that, you do whatever action the policy tells you in that specific situation.


Okay, so that's the Q function. And you can pretty easily see that if you have a Q function, right,


if you have an accurate Q function, you can get a good policy by simply always going with the action


that gives you the highest Q value. Because it's because of a recurrence relationship called the


Bellman equation. This thing right here. So your Q function basically decomposes into the reward in


the next step, as we said, plus whatever happens after that, and whatever happens after that is


just by the nature of how the things are defined is going to be the Q function of whatever the


policy is telling you. So you can get a pretty good policy by always doing whatever action


your Q function tells you is best. This step of calculating the Q function is called a policy


evaluation. And this paper here is going to generalize these notions. Sorry, so this is


a policy evaluation. And then the act of selecting an action is going to be a policy improvement.


These are just names, okay. But we need to know them because the paper introduces two new things.


I'm going to where do I highlight policy evaluation? I don't know. But here they say


this is the policy improvement. Okay, here, policy evaluation, policy improvement, these are the two


steps. So the first step is calculate the Q function. The second step is to select an action.


And you can see how these things interlock, namely, we can calculate the Q function of a given policy,


and we can improve that policy by selecting whatever action is best for the Q function.


This paper generalizes this, and you can see that there is a little a little R right here. So the R


is just a specific way to reference the reward function used right here. Okay, and you can see it


here as well. Now, usually, we have one policy and one reward, right. And so what we do is we improve


the policy. And that leads us to better evaluate the Q function for a given reward function. And


that leads us to improve the policy. Now, this paper is going to transform this into the following.


We have many policies. So we have policy one, policy two, and so on, until policy,


P. And we also have many reward functions, reward one, reward two, reward three, and so on until


reward, let's call that R. So we have many different tasks, right here. And we have many


policies. Now, in essence, they don't need to have some anything to do with each other for the theory


of this paper. But I can simplify this a bit of how they see the world. So let's say you have an


agent, and the agent has been trained on simply that first task right here, and has been trained


using classic Q learning, reinforcement learning whatnot, and that results in this particular


policy. And then the agent just from scratch, you restart it again, you run reinforcement learning


just on reward number two, and obtained policy number two, and so on. So you do this for all


these rewards individually, okay, so you give the agent a new task, and you ask it to learn a policy


for that task. Now, you're in a situation where if you are in have a new task, so our new, the


question is, do you again, need to train a new policy? And the answer for this paper is no,


because we have all these policies, we don't need to train a new we can simply mid mix and match


these policies that we already know to obtain a good solution for the new task. So how does the


paper do it? It does it? Yeah, it does it in the following. It defines the successor features,


okay? Maybe it's in maybe it's better if we first go to an example. So the example they give here


is the following. Otherwise, this I guess this might sound just a bit too abstract. Okay, so you


have this world here, the agent is the thing here in yellow, and it can just move. So its actions


are move left, up, right, down, this, this is one step, okay, in the environment, there are two


different objects, one object is a triangle, and one object is a square, okay. So there are a number


of tasks we can define right now in this thing. So we define tasks according to a reward function.


So the reward, let's say the reward one is going to be one, if, if it picks up a square, sorry,


the square and zero else, just if it picks up a square on any given step, we give it a reward of


one, we don't care about the blue triangles, okay. And then reward two is going to be the opposite,


it's going to be one, not the opposite, but one if it picks up a triangle, and zero else. So you can


see the good policies right here. So pi one is a good policy for reward one, because it just goes


and collects these red things doesn't care about the blue things just goes and collects them by two,


it goes and collects the blue things doesn't care about the red things, okay. So let's imagine that


you have run reinforcement learning twice, once for reward one, and once for reward two. And now


you have two policies, okay, so you have two policies, this will lead to pi one, this will lead


to pi two. And now I give you the third task. Now the third task is a bit special. It's one, if you


pick up a square, and it's, it's zero else, except it's negative one, if you pick up a blue thing,


or the order of these is kind of wrong, but it just for visual representation. Okay, so now you're


asked to pick up the red things, but avoid the blue things. Pick up as many red things as you


can avoid the blue things. And again, as we said, the question is, do you now have to run reinforcement


learning again in this agent with your simulator using like Q learning or something like this,


from the start? Or can you come up with a solution, just given these two policies that will


perform well on the on this new task? Okay. And we're going to see how they do it. So what they do


is they use successor features. So these successor features. I've done a video about successor


features, and I'll link to that, you can look at that. But essentially, essentially, the successor


features are defined like this. And for that, we need to know what this thing is, right here, they


simply call this a feature function. Okay, it's very, it's very ambiguous term. A feature function


is a function that takes in a transition, so state action, next state, and maps it to a high dimensional


vector. Note this is almost the same as a reward function, except the reward function simply maps


it to a number. Now, this is mapped to a higher dimensional thing. Again, I want to, I kind of want


to leave out the next state right here, just to make things easier on you. So a feature here


can be many, many things. But the structure of the features is going to be such that the reward


function is going to be this feature times this w vector. So it was a bit, a bit not correct before,


when I said the reward is now a vector, the reward of a particular task w can be seen as


the inner product between the features and the task vector. So w specifies the task and the


features, well, they specify the features. In our case, it can be it can be fairly simple, namely,


yes, I was I was definitely wrong at the beginning. So the feature functions right here


is which object do you pick up, okay. So we define the feature function as one, zero, if you pick up


a square, and we define the feature function as zero, one, if you pick up a triangle. And now you


can, and we define it as we define it as zero, zero, if you pick up nothing, okay. So we define


zero, if you pick up nothing. And now you can fairly easily see that the reward of each task can


be simply calculated by mixing the features accordingly. Okay, so reward one is going to be


simply the feature times a one zero, which is the w vector. So I can specify a task by giving the


appropriate w vector. And now you can see that if this is my reward function, my agent can go out


into the world, if it collects a square, it is going to be rewarded right here, if it collects


a triangle, even though the features indicate that it collected a triangle, it doesn't care about it


because the W is zero right here. If I now want to give it the new tab, the same is true for our two,


if I now want to give it a new task, our three, right. And you remember the reward function


right there, I can achieve that reward function by simply multiplying the same features, the exact


same feature functions by this vector right here. Okay. Remember, there is a slight difference


between the reward function and the feature function. In this particular example, the idea


of the paper is that the feature function can be rich in, in expressivity and, you know, tell you


all sorts of things about your current state and the reward function is just a number, right. And


then the reward is specified by simply linearly mixing these features. So the structure imposed


by the paper here is that there are such a thing as a feature. And any task can be described by


mixing these same features. Okay. That's, that's the issue right here. So the features


are going to be constant across tasks. Whereas the W defines the task.


All right. So the, the goal here is that if you have learned many, many things during your tasks,


what you want to do is you want to learn this feature representation that is the same across


all tasks. And then you want to simply have the W specify how to mix these features to get the


reward. Now, of course, this is a very strict, very, very definition. Not, not a lot of things


will fall into this unless you make the features like exponentially big, of course. However,


they do discuss whenever a task doesn't fall into that. So I hope you're with me so far. This is the


first kind of restriction we impose on our worlds that we can tackle with this framework. Namely,


that all of our worlds have all of our tasks in this world have to be a linear mix of the same


features. If that's given, then our then we can derive policies for tasks that we have never seen,


we can derive good policies by doing zero learning simply by specifying the task, we can have a good


policy for that task from the policies we've already learned for the other tasks. Okay, so


the reward three is now simply this. And yeah, notice it's not the same as the reward function,


because the reward function had one if you pick up the square negative one if you pick up the


triangle and zero else. So the zero, we don't have to specify here because it's not part of our


features. Right, so you can see that the reward function is given simply by that. And we can now,


as I said, derive a good policy for this reward by looking at the other policies, even though


none of these policies has ever learned to avoid anything. So it makes it defines these successor


features right here. So the successor features is much like the Q function, you can see the signature


is almost the same. So as a Q function tells you how much reward you're going to get if you do the


action a and then follow policy pi, the successor features, almost the same thing. However, it


doesn't tell you what rewards you're going to get, it tells you which features you're going to get.


And which features by that we mean the sum of future features. Now you can see this sum this


a little bit this it of course it comes from the fact of the linearity up here. So it's not really


an additional restriction, but simply to clarify what this means for your environment, your


environment has to be able to be looked at in terms of these features and these features that


need to be cumulative. Again, that comes from the fact that it's linear, but to see so a feature


like I want an even number of steps or something like this would be terrible. Because and they're


going into things like this later, but it would be terrible because here we have the sum. And


as soon as you if you have a feature that is very high, if you have an even number of steps, then


or if you have a feature that counts the steps, you will never be able to do well. Because if


you have a feature that counts the steps, it simply counts up and up and up and up,


depending on how many steps you do. And your reward can never be specified in terms of a mix


of these features. And therefore, your successor features are going to be useless. But in our case,


where it's where feature one is pick up is how many of the sorry, after rephrase,


our feature one is whether or not you pick up a square. Therefore, if we sum it up,


our successor feature one is going to be the number of this is this is a pound sign, the number


of squares that you pick up. Okay. Similarly, our feature two is whether or not you pick up a


triangle in a particular step. So our successor feature number two is going to be the number of


triangles that you pick up over time. I can see that the successor features is kind of the analogous


of your Q function. But it is not in terms of a single number, the reward, it is going to be in


terms of these features, which is an entire vector. Okay. And because we've constructed this in a


linear way, you can also pretty clearly see that the Q function is inherently related to the to


the successor features, you can obtain the Q function by simply multiplying the successor


features by your task vector w. Now, a lot of you might be wondering, where does this w come from?


And in our initial case, we're just going to frame everything as being given, right? So we're given


this this w, we're, we're defining everything from our godlike perspective for now. So don't think


all of this is learned by now. Yeah. Alright, so how can you now derive this, this magical new


policy? Okay, so we, let's say we have this policy one, and we have the policy two, and they, and you


have the this features that you've learned constantly over both tasks. In fact, you're it's


given right? It is pi function, we give it we impose it that the feature one is whether you pick


up a red square feature two is whether you pick up a blue square, then we know that the reward


functions can be achieved by doing the w. So this here, your w is going to be one zero, and your w


here is going to be 01. And we now we want a good policy for task three. And we know we can achieve


this by the one negative one, w, how can we derive a good policy? And this is this algorithm, this


general policy evaluation, a general policy improvement. So it assumes that you, as we said,


you have many, many different, many different policy. So here you can see policy one,


where's policy two, here's policy two, and so on. It assumes that you have many different


features, and therefore many different successor features. In fact, you have a vector of them,


right. So here you can see feature one, feature two, and so on. And it also assumes that you're


in a current state, and you have many actions at your disposal right now, action one, action two,


and so on. Okay, so this is all the past, you've already defined your features, you have learned


these policies. And now you're given a new w w new, in our case, it's this one negative one. And


we want the best action. So we are in state s, we are given this w, we want the best action. Now,


here's a method where we can simply calculate the best action in terms by not reinforcement


learning at all in this new task. So by structuring things like this here, so what does it really


say here? It this thing says, we are going to evaluate all of these different cells of this


tensor right here. So we're going to determine what is the successor feature number two for policy


pi one in state s, if I right now do a two, this is very abstract. So let's say you're here,


and action, action two is actually going to the right, okay, so you're here, oh, this was yellow,


it doesn't matter. So this is so this is action one, this is action two. So action two is you go


to the right, okay, you can you can see that this will let you pick up will let you pick up a


triangle now here, that's action three, and so on. Okay, so what's this number going to be? So we are


in state s, as we said, we do action two. So action two is going to pick up a triangle, the triangle,


the picking up of a triangle means that our pi for this step, or sorry, our five for the step is


going to be zero one, okay. So our successor features, this is not the features itself,


this is the successor features, the successor features decompose into the next step, plus all


the next steps that we can follow. Okay, so all the steps that will come. So what are these


features going to be is it's going to be the sum over that plus everything that follows. And I can


take a little bit of a guess here, which means that this number, so we only care about feature


two right here, this feature, feature two, this number is going to be one for the next step,


because we are going to pick up a triangle if we do action two. But then after that, we're going to


follow policy one. And policy one has been trained to pick up the red squares and not care about


triangles. So I'm going to guess that every now and then, it will kind of step over a triangle,


but it won't fall, we won't, you know, explicitly go look for them. So let's say the episode was 10


more steps, but the board has like 100 squares. So and it has like three triangles on it. So let's


say that's like three 10s in expectation. Okay, so this is going to be this is going to be the


number that we're looking for. We're doing this for every single one of these cells. Okay, this


this thing is going to do for every single one of these cells. And this is very similar to


evaluating q functions, except we're evaluating an entire vector right here. That's the difference


to simply learning many q functions. So if you were to evaluate only a q function, then you would


only have this first matrix this first block right here. Okay, but you have feature one,


feature two, and so on. So you calculate everything in terms of these features. And then


by linearity, you can mix it with that vector. So in our case, this is going to be the one negative


one, which will give you the q functions, right from what we've seen before you obtain a q function


by simply mixing your successor features with your with this task vector. And if you have a


q function, you can pretty easily determine which action you should take. Now you have here a q


function with respect to every single policy, but you can simply take the max right. So the max


across all of this will determine will determine so you take the max across all the policies,


which will give you the q function for a particular action over all policies that you consider.


And then you can simply take the arg max of that and determine the action you should take.


Okay, so it's a pretty big evaluation. But if you do this, that means you don't have to do


reinforcement learning on this task, it simply determines which action right now is the best


given everything that I know from these old policies about the task. And that's not going


to be like the optimal policy, per se, but it's going to be one policy that's pretty, pretty good.


And you can actually prove some things across that. So they do this right here. And you can see


that here is what Q learning does on this new task of picking up the squares and avoiding the


triangles q learning takes a while to get there. However, if you do what they are suggesting,


and you know, you give the W, you can supply the W, almost from the beginning, you see right here,


almost from the beginning, it is at a high reward. Now q learning surpasses it eventually. But


it's pretty impressive that without doing any learning, you are immediately good. Right? Now,


the caveat here, of course, is that they already need these policy pi one and pi two given to the


algorithm. And that comes from previous reinforcement learning trials. And they say


that they give these trials as many steps as q learning uses. So they give them this these


amounts of steps on these other tasks. So the comparison here is a bit shaky, if you ask me.


But the point made is that if you haven't a new tasks right now, you can obtain very good solutions.


And you don't have to do anything. Okay, and these solutions can be the basis for new reinforcement


learning, right, you could start q learning off right here, and then get here much faster,


potentially, and so on. So the next objective right here is that now we have defined the tasks


and we had we know what these features are. And we know how to mix these features as imposers of


the task. So what happens if we only have the reward function, we specify the task only in terms


of their word functions, but we're kind of looking at the features and we're like, agent, please


figure out yourself how to apply these features in order to make the reward high. And that's what


this thing is right here, this GP and GPI with regress w so you don't no longer tell it what the


w is. It needs to infer it through reinforcement learning, right? And it's not really reinforcement


learning, but what it does, where is it? Yeah, it's simply it because all of this is linear,


and this thing here is given. So always remember, this thing here is given. And these are the


rewards that you obtain, you can simply do a regression to figure out the w of the task. Now


that's going to take some time. But as you can see right here, it is going to take a lot less time


than simple than doing Q learning from scratch, notably because you have good features. So this


is some this is this gets closer and closer to transfer learning, right? If you imagine that this


right here is your pre trained neural network, and you simply learn the last layer of it,


you freeze this, you do transfer learning, fine tune the last layer, here we are. So


it gets closer and closer. And you'll see this trend right here. So it's pretty cool what you can


do. But basically, I think it's a lot of math around a framework. And the more and more you


relax the kind of impositions that they need for their framework, the more it gets back to simply,


well, we do reinforcement learning, at least in my estimation. So before we look at that, this here


is a pretty, pretty cool experiment, where they look at how the different tasks can be achieved


if you give different policies. So you'll have noticed that we have always given these two


two tasks one zero, and zero one, these were our tasks that we trained on. And then one negative


one is the task we evaluated on. Okay, and you might object and say, wait a minute, these these


two tasks, you know, they're pretty good, as let's say, pre training tasks, because, and it's


basically the standard basis, right? And any other tasks can be mixed from those. So these are


orthogonal vectors in this vector space. So you're being pretty generous to this system. What


happens if we're not as generous? So that's what they do here. So they have different policies,


and they evaluate how much you can learn with these different policies. So the way you have to


read this diagram is right here, it's going to be the one zero axis as they will, they label it


right here. And this is going to be the zero one axis, and this is evaluation. So every direction


on this circle defines a task. For example, this task right here, as you can see, is going to


define the task of picking up both the squares and the triangles, right? Whatever you pick up,


you get a reward. However, the task down here is going to be please pick up the squares, but avoid


the triangles at all costs. Okay, and now they're going to look what happens if we supply two


different policies to choose from. Remember, we're in this situation, we're again in this situation


where we give everything and we give initial policies, we give the task vector. And now it's


about deriving a good policy just from looking at the old policy. So no learning. As a baseline,


as a baseline, you have Q learning, which into a given direction, tells you basically how long Q


learning takes or how far Q learning gets with a given amount of steps indicated by this one, two,


three, four, and so on. Yeah, you see, I think this is this in how far Q learning gets with these


dotted lines right here. So Q learning gets this far with 10 to the I don't know, four, and then


this far, 10 to the five, and so on. So these are comparisons. You can see that on the outside Q


learning is going to beat this these methods. But our hope is going to be that of course, if we have


this zero shot generalization, it's much better than running Q learning for really long if we get


close to it. So the green thing is what we've already seen policies one and two will give you


a fairly, you know, good, fairly good extent right here. So what does it mean? It means it can solve,


it can solve pretty much everything from here, here, this task, this, this task, this task,


it kind of falls off once we go down here. So once we go to the avoid section, it sort of falls off


because it has never learned to avoid. Now, still, we can of course, do the avoidance by simply


imposing a negative collection. But negative collecting and avoiding aren't exactly the same


thing in these in these environments, right? Because avoiding can also be going really close


to something but not hitting it while collecting. It's not the inverse of collecting, the inverse


of collecting would be like run away as far as as far as possible. So we can expect that we've


only ever learned to collect, we're not going to be super good at avoiding. Then the other extreme


is when we give policies three and four, I haven't told you but you can see it right here. Policy


three is explicitly to collect one and avoid the other. While policy four is the opposite right


here, avoid the squares, collect the triangles. And now this policy, this policy is should be


pretty good on all of the tasks in between. As you can see, it has the biggest extent right here.


And that also makes sense. By the way, there's nothing down here because the task of avoiding


both things doesn't really make sense because you can just stay where you are. Because there are


also these these squares where there's nothing. But you can see that the mixture of those is


quite potent. So already we can see even though these span a basis, in fact, an orthogonal basis


as much as these, because of the nature of the features that we define for the task,


they are not equivalent in mixing after so we can be more generous, we can also be less generous.


If we only provide policy five and policy five is simply to pick up to pick up both objects,


then we're going to have a pretty hard time when it comes to avoiding things. So you can see it


can do fairly well picking up the various things in a positive manner. But as soon as we cross this


line into the like this horizontal line into where it's about avoiding a particular object,


it's not it's not the choices of actions we have from policy five aren't going to be super good


at that. And they do another they do another thing right here. So that the left thing is where they


say, it's important which policies we provide. And the right thing, they want to say something like,


it's important. So they want to say, if we provide more policies, that can be advantageous,


because we basically have more options to choose from. Okay, so now they start off with policy


four. And policy four is simply avoid the squares, collect the triangle, you can see it performs


fairly well over here, where it's all about avoiding the squares and collecting the triangles,


as soon as you get into, you know, collecting, or even here, the opposite directions, it's pretty


bad, right? That's the red thing. And now they add policy two to policy four. So policy two


is going to be also to collect the the triangles, but to just neglect the squares. And that will


also do a bit better. Why does it do better? Because it's better at collecting, because this


policy here also needs to avoid. And this policy here doesn't care. So in the regimes where it's


better to not care than to avoid adding this policy, adding these options is going to be good.


And you can see that there's a general expansion here, as we add more policies. However, I want to


point out that, for example, here, this black thing, which should be technically superior to


the blue thing, because it contains, as you can see here, all the policies that the blue thing


contains plus another policy. I don't I don't know if my vision, but I'm pretty sure here,


the black thing is inside the blue thing. So that means there can also be a disadvantage


to adding more policies right here, because maybe you got you have too much to choose from.


And so, right here, what we say is we add a policy that is all about collecting the squares,


and it is performing, it is actually decreasing the perform the addition of this is decreasing


the performance on tasks where you have to avoid the squares, which I'm not sure if, if that makes


sense. Again, the opposite of collecting isn't avoiding, but I'm just pointing this out. And


this isn't really mentioned in the paper, the paper simply says, see, we add policies, and


therefore we are getting better. I'm not, I don't agree with this, given these results, or maybe


it the plotting, the plotting is bad. Alright, so they say, okay, more policies better, which


I disagree with. They also say, oh, we can, as as much as we can regress the W, right, we regress


W, we figure out the task, we can even learn the successor features. Okay, we can not the successor


features, the pi functions that lead to the succession successor features. And you can see,


if you do it with the true W, you're really good at the beginning. If you do it with a regress W,


we can see that before. You can, you, so this is the small version of this plot right here. This is


like this section, I think, yeah. You know, you improve. However, we can also learn this pi


function, we can also learn the features, where if we're not given the features, maybe we can


learn the features. And they say, well, we can do this with but also by regression. So here,


what we can do is we can find the function that minimizes the function and the W along with it


that minimizes this error right here. Okay, so you're finding the function and the W that that


matches this error. And this now really is like learning a neural network. I mean, you know, so


I get I get it, you have the I here and the W doesn't depend on the I and so on. But you're


getting more and more back to actually simply learning nonlinear functions, mixing them linearly


right here. And I think that's going to be kind of the crux of this method. The fact that the more


complicated your problems are, the less you're going to be able to do this kind of stuff. And


they even go as far as to say, well, what if like before, we the reward is actually something like


whether or not you have collected an even number of triangles or squares, then they say, well,


you can simply not have a single W, but you can find a function W. And now the policy is a function


of the function of W. And you can do potentially the same regression problem. But as you can see,


it gets so now you this right here is going to be a function of state. And so you can see that


more and more, it simply goes back to basically Q learning. Again, the only difference here is that


you have this intermediate features. But I think you can simply view this, let's say, as a hidden


layer in a neural network. I get it somewhat held constant across sums and so on. But you know,


I I like the method in terms of, you know, in terms of the analysis. So if you are given all


this stuff, it seems pretty cool that you can derive new policies, its implication for lifelong


learning, they say, look, here, you have a bunch of tasks in your database that you've already


learned on your agent is going out into the world, it faces a new task, it can use this thing,


it can use this thing to obtain a new good policy for that task, it can then use reinforcement


learning, or L to refine that policy. And then it can simply save that policy into the database.


So it keeps expanding and expanding this thing. So it keeps adding rows and rows and rows right here


of new policies that it's learned over the course of its life. So once it's facing a new task,


it can just kind of draw from its experience and derive a good initial solution. However,


the actual analysis only works, I feel, in quite limited circumstances. And if you want to relax


these limited circumstances, then you need to basically regress and regress and regress away


from away from their setup. And I'm not sure I'm not sure where this is going to go. If this is


going to be a general framework for people, it seems like it because it's pretty easy. But


then also, it seems like most of the world doesn't really fall into this category. In fact, this


divide and conquer approach. I'm not sure but from divide and conquer, I always imagine something


like you subdivide and subdivide and subdivide until you know you are at some kind of basic task,


they still only go for no single task like this here, the tasks are somehow in sequence. And I'm


not I think we should really think about hierarchical RL. Now this can be a good first


step right here. But most hierarchical RL, even the ones that specify themselves as fully


hierarchical, like we can do many layers, they rarely go above two layers or three, like like


one, one meta layer and one actual layer like this one right here. They rarely go further,


maybe they go two layers, but that's about it. I've seen very little in actual hierarchical or


divide and conquer reinforcement learning just because it's so hard to train. Yeah, all in all


cool paper. And if you want to get into the math a little bit, I think it's pretty easy math. Once


you kind of set your goals on what it's actually meant to achieve. If you just read from the


beginning, all these reinforcement learning papers, it seems a bit like, why? Why are we


doing this? Right? You just oh, could we define this? We define that we define this? And you're


a bit like, yeah, but why? So often it pays in these papers to go at the end to the examples,


and then come back to the theory, knowing what they want to achieve. Alright, that was it for


Defer - Go Lang Practical  Programming Tutorial p.20

What's going on everybody welcome to part 20 of the Go language programming tutorial series.


In this tutorial what we're going to be talking about is the defer statement in Go.


So what we were talking about in the last tutorial was, hey we're using this kind of weight group and all that and it's really cool,


but what if we run into something that causes this function to error and we don't actually get to tell the weight group, hey we're done.


Well then we're going to have a problem, we're going to be waiting, right, and we don't want that.


So this isn't a problem that you're going to come across with just doing Go routines in weight groups, for example,


when you want to close connections when you're done or close files or whatever,


if you hit an error before you get to the close, you're never going to actually run that operation.


So in Go they have the defer statement, so the idea of defer is it will be evaluated whenever it gets hit,


but actually the defer statement won't run whatever is being deferred until the surrounding function either completes or panics out.


So once it's been evaluated, if that function does happen to panic out, all of the deferred functions will run.


So if it hits an error or if it doesn't hit an error, basically whenever that function is all set and done,


it's going to go ahead and run whatever you decided to defer.


So to exemplify this, I'm going to go ahead and clear, clean up here, we'll come back to this,


I would recommend you save this because I'm just going to copy and paste it over when we return back to this,


but I just want to use a more simple example for now.


So let's say we've got a func, we'll call it foo, and foo basically just does a format.println,


and this is a function that does stuff, who knows what, and then func main can run foo.


Okay, so if we wanted to, we could say, okay, while this function is doing some stuff,


we want to make sure when it's done doing some stuff, we want to make sure it runs format.println done.


So this is going to be evaluated immediately, but it's only going to run when the rest of the foo function is done.


So let me pull this up, and let's run that real quick, and we can see, yeah, it says doing some stuff, who knows what,


and then it's done running.


Now what if we added another, like what if we threw in another defer statement, so let's do, are we done?


So the way defer statements work is in a first in, last out order,


I'm not really sure why it's that way, but basically that also means last in will be first out.


So what we're going to see when we run this is doing some stuff, are we done, and then done.


So just to show that, sure enough, that's how it ran.


So we could also exemplify that with like a for loop or something like that.


So for example, we could say for i colon equals zero, while i is less than five, i plus plus,


we could just say defer format println i.


So save that, run it, and sure enough, it basically counts down.


So getting back to our main example, I'm just going to kind of copy and paste it over, copy, paste.


So this is our original example where we kind of came into this.


So this one, it works, and it waits.


So just to show, right, just runs those two go routines, waits for them to finish,


but if wait group dot done never runs, we're in trouble.


So what we could say is rather than running wg.done down here, we'll just cut that,


and then we defer wait group dot done to the top.


So then we can save that, and now we can rerun the function or the program,


and now we've got hey there, hey there, hey there.


Okay, so that is how we can defer things and why we might want to defer things.


And if we're going to be talking about the defer statement,


we might as well also talk about panic and recover in the Go programming language.


So while we can use defer to make sure something happens,


we can also use defer to recover after a panic has occurred.


So you could defer some sort of function that's going to handle for whatever the error was


that we might have actually hit.


That way your program doesn't stop running, or at least it doesn't have to.


You could let it stop running, but you don't have to do that.


Anyway, in the next tutorial, that's what we're going to be talking about.


We should have no problem covering both panic and recover in the next tutorial.


So panic is a way that we can say, hey, we've hit an error based on some sort of condition,


so we can panic out, but also, like I was saying, defer doesn't necessarily only run if the function is successful.


Defer will also, all the deferred statements are going to run if the function also panics out.


So if the surrounding function panics out, all the deferred functions will also run.


So then within those deferred functions, we can recover from panics.


So anyways, that's what we're going to talk about in the next tutorial.


If you have questions, comments, concerns, whatever, feel free to leave them below.


Listening to You! - Channel Update (Author Interviews)

Hi all, this is just a short channel update. Recently, I've been conducting a lot of surveys


of people and asked a lot of people to comment on things about the channel. And I want to give you


an update on how that's going. So as you might have realized, I've had the great opportunity


to bring on a lot of authors firsthand on the channel to explain their papers, explain what


they think and sort of the behind the scenes stuff of the research. And this is this is amazing. I


would have never thought that so many people would want to, you know, come on and share things with


the audience. But you know, here we are. It was really cool for the people, I guess, to come on,


because they get to share their work. It was really cool for me, because I got to interview


the people. And then after that, I would make the paper review, which would be shorter, more


condensed, because we'd already covered so much in the interview. And I thought that would sort of be,


you know, a good piece of content. However, it was not so good for you. A lot of you and I've read a


lot of comments, I've conducted surveys, you might have come across them on YouTube, on Twitter, and


so on. A lot of you miss the old style paper reviews, the longer paper reviews, and you pointed


out some crucial things. First of all, it is really difficult to be critical of a paper when you make


the paper review after interviewing the authors, because that's what I would do. I would let the


authors explain the paper to me, essentially, so I know even more when doing the review. And then


after that, I'd record the review. However, it'd be a real dick move if I were to bring up some sort


of criticism in the paper review that I didn't bring up in the interview, right? Because, you


know, what am I going to do interview the authors and then be like, well, but this part here, this


is really crap. And then the authors have no chance of responding. I mean, it's not a good way


of doing things. So I was not able to be as critical as I would be when I would just approach


the paper for myself, not that I want to be critical, but it was just a different atmosphere.


So I've decided going forward that I would do the paper review first in its full length in its sort


of classical way, and then show that to the authors and then interview the authors. This


allows us to get into the criticism and into the meat of the paper much more quickly. And also a


little bit more of that behind the scenes stuff, it will make the interviews a bit shorter as well.


And I think that will just be an improvement for everyone. It does represent a bit more work for


myself. But you know, that's life. Yeah, it's essentially whatever I did before, plus the


interviews plus the most dreaded part, which is like scheduling and organizing all the people,


which is really not something I'm good at. But I'm trying. So if you are something that's


kind of like expecting an email from me for like four weeks, I'm sorry, I'm really sorry. Yeah,


what's still not clear to me is whether or not to release the videos in one part, or to release the


review and the interview separately, maybe back to back on two different days, or whether to release


them apart from each other, like the review as soon as I have it, and then the interview later,


people are kind of split between the two methods. And we'll just have to experiment a bit. So going


forward, there will be classic paper reviews. And if there is an author coming on, the author will


be able to react to the paper review. Not always, it's not always going to be possible. It does


require more work on deadlines for me. And I don't always have time to prepare the review before I


interview, but I'm trying as best as I can. So there are about two or three videos in the backlog


that still have the old format. And then after that, we're going to switch to the new format,


and it will be glorious. I really want to thank everyone who's contributed to finding this to


tell me what they think to you know, all the commenters, all the people on discord,


all the people who took part in surveys, thank you very much. I want to do as best as I can want


to make the best use of your time want to make the best use of the author's time. And I hope this is


just going to lead to greater content. Please, as we continue to experiment with stuff, let me know


what you think continue to tell me what is best for you continue to tell me what you didn't like.


GPT-2: Language Models are Unsupervised Multitask Learners

Hi, today we're looking at Language Models, Our Unsupervised Multitask Learners by Alec


Radford, Jeffrey Wu, Revan Child, David Luan, Dario Amadai, and Ilya Sotsgiver from OpenAI.


This paper has generated a bit of hype in the last few days, so I wanted to go over


it, basically take a look and take a look at the surrounding, let's say, controversy.


So let's actually have a look at the blog post that OpenAI released along with this


paper.


They say, we've trained a large-scale unsupervised language model which generates coherent paragraphs


of text, achieves state-of-the-art performance on many language modeling benchmarks, and


performs rudimentary reading comprehension, machine translation, question answering, and


summarization, all without task-specific training.


So this sounds quite suspicious at the beginning, but we're actually going to have to look at


how they do this.


It sounds really good being able to do a rudimentary translation without any training on translation


itself, just learning a language model.


But this has been continuing of a trend in recent kind of time where we see that the


better your language model gets, the better basically your model gets on all these kind


of language tasks.


All right.


They go into this and we'll see how they do it.


So basically what they've done is they've taken kind of a bigger data set of language


model data set, which is about 40 gigabytes of internet text.


They say this is here on the top.


So it's one of the largest kind of text data sets there is unsupervised.


And they also taken one of the largest language models.


So their largest transformer-based model has 1.5 billion parameters.


So they take huge amount of data, huge model.


They train this on...


They train the model on the data and what comes out is like a giant super language model


that is able to perform all these cool tasks.


So here they have like a bit of a sample.


So what they can do is they can basically...


So the way a language model works is you always try to predict the next word based on what


you've already seen.


So you kind of query it by giving it some starting words and it needs to continue the


text from there.


So here system prompt on top you see, in a shocking finding scientists discovered a herd


of unicorns living in a remote previously unexplored valley in the Andes Mountains.


Even more surprising to the researcher, the fact that the unicorns spoke perfect English.


And then the model continues.


The scientists named their population, the population after their distinctive horn, Ovid's


unicorn.


Four horns, silver white unicorns were previously unknown to science.


Now after almost two centuries, the mystery of what sparked this odd phenomenon is finally


solved.


I mean, you can read this.


It's really, really coherent text and it's quite surprising.


I think it's like slightly cherry picked, but still the fact that a model is able to


do this is unprecedented, especially since it's like a general language model, not specific


to the task of continuing news articles about unicorns or anything.


So yeah, they go into these findings, we'll see them in the paper and they also say that


yeah, they can now do all these kinds of tasks like question answering, reading comprehension


in a zero shot fashion.


So at the end here, they say what it's capable of.


So let's say AI writing assistance, more capable dialogue agents, unsupervised translation,


blah, blah, blah.


They also say a few kind of, let's say bad implications.


Create misleading news articles, impersonate others online, automate the production of


abusive or fake content to post on social media, automate the production of spam or


phishing content.


They liken it to a system called deep fakes, which generates really well crafted, let's


say videos of people.


So they frame it in a way as this could be used for dangerous things.


And they say they aren't releasing, they're only releasing the small version of GPT-2


along with the code.


They're not releasing the data set training code or the GPT-2.


This is the big model, the model of weights, right?


And they do this, they cite safety concerns.


So I mean, the community basically is going nuts over this statement, this decision not


to release basically the code or the model or the data set to the world.


And so if you search on Twitter for hashtag GPT-2, then everyone basically has an opinion


of whether or not this is a good thing or not, apart from people testing it out.


So they've given access to like a selected set of journalists to an API where they can


query the model.


So there are some samples flying around.


And basically people are just debating whether or not this is a good thing or not.


And I mean, it's just hilarious to go along with this and to just read all the people


having opinions.


I mean, I've given my opinion as well.


Just chime in, it's a fun ride.


Basically like this post here on Reddit, machine learning says, should I release my MNIST model


or keep it closed source fearing malicious use?


Today I trained a 23,000 layer ResNet, got a 99% 6% accuracy on MNIST.


I'd love to share the model, but I fear it being used maliciously.


What if it is used to read documents by the Russians?


What are your thoughts?


I mean, this is in essence, I mean, in essence, it's that, right?


It's like, yeah, come on.


So I can just give my opinion upfront.


I think a lot of things came together here.


And I think that this being OpenAI, being kind of this initiative, I mean, it's never


been there before, they're not an academic institution.


They're not a company, but still, they're researchers, they wanna have a career.


So there's lots of pressures on them.


There's pressure to further publish.


So I think that that's kind of an underlying motivation to not release your model and your


code and your data set is you actually, there's a lot of work in this and you actually might


wanna get more than one paper out of it.


So keeping these things to yourself is basically a surefire guarantee you're gonna get another


two, three, four, five papers out of this data or model or whatever.


It's also a good way to kind of generate press if you basically say, oh, we're not releasing,


but we have this really good model.


There's one thing on Twitter, I mean, you probably can't find it, but it says like,


step one, my model is state of the art.


Step two, my model is state of the art, but generalizes better to other tasks.


Step three, my model does the same thing, but with fewer parameters.


And step four, my model is so good, I can't even talk about it.


Yeah.


So basically, I think a lot of things came together.


This press generating the pressure to create more kind of papers out of it and genuinely


security concerns.


I think being open AI and the open AI kind of was established as a way to, let's say,


their statutes pretty clearly say we want to open AI and research it in ethical use.


And you have backers like Elon Musk that talk all the time about kind of safety related


issues in AI.


I think there's a lot of pressure on these people to, with everything they do, basically


have an ethical component.


All right.


So everything they do is kind of scrutiny to this, does this have ethical implications?


And where can we basically stand out from the rest of the community by doing something?


It doesn't need to be more ethical, it just needs to be different with an ethical reasoning


behind it.


And I think this is it.


I think there's a lot of things coming together.


I don't think anyone like maliciously thought, oh, this, you know, we're going to do this,


it's going to generate so much press.


Or I don't think anyone actively thought, ah, we'll just keep it secret.


We're going to get so much more papers out of it.


I honestly think that the reasoning was there's a lot of pressure to do this ethical things


when there's not...


If you think about it, it's a good language model and it can generate text, but you can


also hire people to generate text, to generate fake news, to do phishing and spam.


It's just a bit more efficient now.


And yeah, so it's unprecedented that you don't release this research, kind of Cold War style.


So it's not really dangerous to release this and it's just delaying the inevitable basically.


But I think that the pressure was on and they made a decision that they thought was in line


with their values.


And I think this just neatly aligns with the underlying, the other benefits to them.


All right, so let's dive into the paper.


The paper is actually not too much content there.


What they basically say so far is that a lot of these papers on these tasks, they say the


dominant approach to creating ML systems is to collect a data set of training examples,


demonstrate correct behavior, train a system to imitate, test its performance on IID samples.


So they basically say there's kind of the single task training on single domain data


sets is a major contributor to the lack of generalization observed in current systems.


So they basically say these language systems, they don't generalize.


Like a QA system might be trained on a QA task, but it has nothing to do with the task


is basically a little bit different.


And even in multitask learning, they say multitask learning is a promising framework, but also


it's kind of, say, it's still nascent and there's only very few different tasks to do.


So they basically say you need basically a big, big unsupervised model and it will implicitly


learn all of the kind of special tasks.


So they say there are approaches that basically learn these language models, but then still


require supervised training.


So basically fine tuning.


This has been the, this is for example, the BERT paper we discussed in the last video


or two videos ago that learns a giant language model, but then does fine tuning for each


of these tasks and gets really well.


What they want to do here is basically simply learn a language model and then investigate


whether or not the language model can perform downstream tasks in a zero shot setting.


That means without any parameter or architecture modifications or no fine tuning.


So what they do, so basically what a language model is, if for those who don't know, it's


if you have a sequence of text, let's say A, B, C, D, E, these are words, let's not


use actual words, the cat sat on the mat and so on.


And you, and you, a language model is you kind of remove the end of the sentence at


some point and ask the model what comes next, right?


That's a language model.


I mean, there's different kinds of language models, specific language models, but that's


the basic, the basic thing.


So the, the, you just ask the model what's next.


So you can, you can do a lot of unsupervised training because you don't need a labeled


data set for this.


You simply need a text corpus.


And that's basically all they do.


They use transformers, which we've also discussed in the Attention is All You Need paper.


So if you, if you don't know what transformers are, go back and look at that.


Yeah.


All right.


So basically they say a lot of these special tasks like translation and question answering


can be framed in language model way.


For example, if you simply input, if this is your text, translate to French and then


you the English text and then the French text, right?


And then at, at test time, basically you leave away the French text.


You simply ask the language model what comes next, right?


If, and its input is translate to French and then the English text.


This is the translation framed as a language model task because you can specify the task


that the language model has to do also as language.


So this is quite, this is quite an interesting approach.


And one they exploit here and they basically say, well, since in a large and diverse corpus


of web pages that they collect here, some, there's going to be some websites that basically


all do translation from English to French and the model can learn from that.


So here in this paragraph, they basically list examples of naturally occurring demonstrations


of English to French and French to English translation found throughout the training


data set.


So basically this is, this is how the model could learn.


Let's just look at one.


I hate the word perfume versus it's somewhat better in French, parfum, right?


So there's a way in just an unsupervised setting where the language model could learn, right?


If you just cross out this word parfum at the end and you just ask the model what comes


next, right?


The model sees, I hate the word perfume versus it's somewhat better in French, period, colon.


Then the model has to put something there and the most logical continuation is to put


the French word for perfume.


All right.


So that's kind of how they frame translation and these other tasks in language model way.


So they talk about the training data set, which is a major component here.


They say they make a new training data set because all of the current ones aren't sufficient.


They say most problem source of diverse and nearly unlimited text is web scripts such


as Common Crawl.


While these archives are many orders of magnitude larger than current language modeling data


sets, they have significant data quality issues.


So to say content are mostly unintelligible and so on.


So they basically describe here how they scrape a new web scrape, which emphasizes document


quality.


They go on Reddit basically and scrape all outbound links from Reddit that have received


at least three karma, which means that, you know, three upvotes for a post of a link,


which basically means that three humans agreed that this was a good link.


So that's how they collect the data set.


The resulting data set web text contains the text subset of the 45 million links.


They then kind of clean this and scrape it down and remove some stuff and they end up


with a large corpus.


And then they talk about how they represent the input, which is byte pair encoding style.


It's not exactly byte pair encoding.


It's a byte pair encoding inspired encoding.


We might make a video about this by itself because it's really interesting.


But basically you can think of it as tokenization and pre-processing.


Then they say they show their models, so architecture hyperparameters.


Basically these are their models.


This is the smallest one.


This second smallest one, they say it's the same size as BERT, so the language model by


Google that we've looked at.


And then the largest one, 1.5 billion parameters.


That's huge.


And yeah, they say it's 10 times larger than the previous.


So the first one is their previous model.


And this now is the GPT-2 model that gets all these nice results.


So they do experiments.


They do experiments on language modeling itself.


So they train on their corpus and then they evaluate on a bunch of other language modeling


corpus.


So these up here are language modeling corpuses.


And the state of the art is in this row.


And then you just look at basically the bottom row compared to their largest model.


This is perplexity where it says PPL.


And I think this here is accuracy.


So perplexity, lower is better, which you can see here.


The previous state of the art was 39 on Wikitex 2.


They get to 18.


With accuracy obviously higher is better.


So the kind of previous accuracy in Lombarder was 59.


They get to 63.


Basically they improve everything except for this one billion word corpus.


And they also explain why.


They say this is the most heavily preprocessed text and so on.


So they basically are really good at language modeling even though they train on a different


data set.


That's the point.


The point is they train on their own corpus and then they go and just evaluate on the


test set of these new tasks.


And they become better basically than the models that trained on the training data set


of that particular task.


So they do a number of further experiments where they basically show that the model has


now learned, kind of implicitly learned a number of different tasks.


So let's look at, for example, summarization.


Just want to show an example of how you can do this.


So summarization.


Summarization task is you get a long text.


You need to produce a short text.


And that short text is then compared to short texts that humans wrote when the task was


to summarize the long text.


And you get points on how much your text overlaps with these human texts.


So they say we test GPT-2's ability to perform summarization on the CNN and Daily Mail data


set.


To induce summarization, here's what I find interesting.


We add the text TLDR after the article and generate 100 tokens.


And they say they need to reduce repetition and so on.


But basically this is the way you can frame summarization by text input.


So I find this just kind of a really nice way to think about these problems.


The fact that instructions of the task can be given as text.


This is a very nice example here.


So basically you put the entire article.


And so here is the CNN article, blah, blah, blah.


It's super long.


And then here you put TLDR, which is for those who don't know, it's too long, didn't read.


So people use this phrase to indicate that then they will write a short summary of whatever


was before here.


They will either put this at the beginning or at the end of a long text to say to people,


okay, if you don't want to read all this, just read this down here.


It gives you the gist of it, which is exactly summarization.


So if you then take this away and ask the language model, what's here, right?


Basically throughout the training corpus, the language model will have encountered such


pieces of text with a TLDR in it.


And the language model might have learned that whatever is down here is a short version


of whatever is up here.


And thereby, if you then ask the language model, what comes next?


Here, right?


The language model might learn, aha, I need to summarize whatever is above.


That's my best shot at completing, at answering the question, what comes next?


And yeah, so they get surprisingly good results from this.


So they say, on the commonly reported Rouge 1.12 metrics, the generated summaries only


begin to approach the performance of classic neural baselines, just barely outperform selecting


three random sentences from the article.


But still, while qualitatively the generations resemble summaries, they often focus on recent


content from the article or confused specific details.


So this is kind of a task where it kind of worked, but not really.


But I just find it's really interesting that it kind of, how they framed the task, and


how this can still, so it still kind of works.


And that's the gist here in all of these tasks is also with translation.


They obviously don't get near the performance of a system specifically trained to do this


task.


But they also always say, it kind of works, right?


It sort of learns something.


And their entire point of this paper is to say, well, look, the diversity of tasks the


model is able to perform, and I would say kind of perform in a zero shot setting, suggests


that high capacity models trained to maximize the likelihood of a sufficiently varied text


corpus begin to learn how to perform a surprising amount of tasks without the need for explicit


supervision.


So, yeah, their entire point is if we train on such varied data that spans the entire


range of human language expression, the kind of tasks we want these systems to do will


be learned implicitly.


So basically, it points to let's get an even bigger corpus, let's get even bigger models,


and we might get even better unsupervised zero shot way in these kind of special tasks


and general language understanding.


All right.


So that was basically it.


I've jumped over a lot of points, but I encourage you to look into this, to look into the specific


experiments.


They're really interesting, the way how they framed things.


And just shout your opinion around about whether or not the publishing is a good thing or not.


It's really funny.


I love it.


How Apple scans your phone (and how to evade it) - NeuralHash CSAM Detection Algorithm Explained

Hello there, today we're going to look at CSAM detection, the technical summary of Apple system


in order to detect child abuse material of users before they upload it to iCloud.


So I recently reported on this in ML News. And this story, of course, not my story, but the


general story has sparked a lot of controversy around the world with respect to privacy of users


and Apple essentially coming to users phones to scan the phones for illegal content and so on.


So now we have the technical summary, where Apple details exactly what's happening and how they're


trying to both preserve user privacy, but at the same time, essentially catch people who create


and share these types of materials. Now, needless to say, I think everyone's on board with reducing


the spread of these materials. The question is what kind of trade offs we're willing to accept


in order to make that happen. And the trade off here is mainly privacy of people, even though the


system is designed to mitigate it, there are still weak points that where the system can be attacked,


the system can be used for purposes that it was not intended. There are other problems. On top of


that, at least in my estimation, the system can be evaded fairly easily. So, you know, you combine


the system can be evaded fairly easily with, we're going to implement the system that potentially has


pretty, you know, really nefarious consequences, if someone gets control of it, that is not a good


actor. I don't think, you know, we'll have to think about the trade offs of doing these types


of things. And yeah, that's just that. So we'll go through the report, we'll go through how the


system works, how Apple describes it, and we'll go through the strengths and weak points. And you can


make up your own minds about that, even though I'm going to, of course, try to bias you in a certain


way. So keep that in mind. Alright, so we get here a, essentially, it's a sort of a white,


technical white paper, giving us a description, first an overview, and then a description of


these various techniques. So there's going to be like a neural part with it, which is sort of the


machine learning interface to this whole system. Since we're dealing with images,


that's, you know, that the front end, essentially, then we're going to deal with a whole bunch of


cryptography slash security stuff, which tries to preserve user privacy as much as possible,


while still allowing Apple to detect who shares this material. Okay. So here are the requirements


of the system as far as Apple sees it. So first of all, the the detection, so this is CSAM, it


stands for child sexual abuse material. And the system specifically is designed to catch,


identify and report iCloud users who store known material in their iCloud photos accounts. So it's


very limited in scope. In fact, Apple does not scan your entire phone all the time for anything


that you might have. It scans the things that you're about to upload to iCloud. And as we're


going to, in fact, see it, it just computes as you upload to iCloud, it computes the security


voucher and uploads that along with the material. And it only is supposed to detect known material.


So there is a database, the database is provided by the National Center for Missing and Exploited


Children. And that database, as far as I can tell, Apple doesn't even have necessarily access to that


database itself. But for sure, they only so they they're not going to train a detector to, you


know, classify abusive material per se, like, so they're not going to catch new material until


that new material is entered into this database. So this is essentially saying we have a list,


we have a big list, the database of things that we collected from, you know, confiscated phones or


whatnot, collected from these websites. And we are simply going to check if in your iCloud account,


there is any of those things, right? Any of any of those matches, then you have one of these known


things, then we're going to report you. Now, the the challenge is, of course, to preserve user


privacy. So here are the requirements that they set themselves to is they set upon themselves.


Apple does not learn anything about images that do not match the known CSM database. Now, this is


hard, right? Apple can't just go to your iCloud account and and scan all the images. Otherwise,


Apple would know what the other images are. So as I understand it, things in your iCloud account


are encrypted anyway, so Apple can't do that, right? So it can't just, you know, compare images,


because otherwise, either you'd have to send the abusive images to the user's phone, which kind of


defeats the purpose and then compare on the phone, or you have to send all the user's photos in clear


text to the server. And then Apple would essentially see all the user's photos, which is


also not okay. So we're going to have to get a bit creative here. Second, Apple cannot access


metadata or visual derivatives for matched images until a threshold of matches is exceeded for an


iCloud photos account. So it gets even more complicated, right? If you have apparently, like,


if you have one image, they're not going to they don't want to, they don't want to report you yet,


they are going to set a threshold, let's say five images, like if you have five matches in the


database, then you know, it's very probable that you're engaged in actively sharing or consuming


this material. And therefore, we're going to report you, you know, like if it's below that,


probably their lawyers, their lawyers can't make a good enough case. And so they're going to say,


if it's below a threshold, we don't want to be able to decrypt this, right? We only want to be


able to decrypt all of the things once the threshold is exceeded. So this is yet an additional


constraint that we have to somehow work with, we have to design an algorithm that is going to


we have to design an algorithm that allows us, we cannot decrypt anything until we have enough


threshold exceedances. You know, excess excesses. Well, what's the word? I don't know. Okay,


let's go through the other requirements more quickly a bit. The risk of the system incorrectly


flagging an account is extremely low. In addition, Apple manually reviews all reports made to these


to the to the Institute to the government to ensure ensure reporting accuracy. Now,


this is a good goal, right? However, I think we've all encountered websites that told us that some


decision was manually reviewed. But it's pretty, it was pretty clear that it wasn't right. So


this is this is a goal, we know that as soon as there's like pressure, as soon as there is, you


know, something more important going on, as soon as the system is overwhelmed, they are just going


to swap out humans for for robots. I don't know how much pressure there needs to be for these


humans to be swapped out. But still, at least initially, they're going to review all of the


reports they make. Then users cannot access or view the database like this, yeah, this should be


fairly obvious. And users can't identify which images were flagged as being in the database by


the system. So you can't design an algorithm that only, you know, transmits data to Apple once a


match is found, because then the user would could inspect the network on their device, and they


could figure out which of the which of the images is problematic, and apparently notify their


whatever their friends or something. So you don't want that you want the users essentially to


upload all their stuff, they never there's always a bit of data that goes with it. If there's a


match, they don't initially know about it, I guess until the police knocks at their door. So these


are the requirements. Okay. So this is a is an overview. What we have is we have this database


of the database of this material, what we're going to do with this database is we're going to


compute some hashes from it. So these are hash now a hash essentially is simply a representation


of a piece of data that is shorter, but still uniquely identifies the data. So if I have a hash


function h, and I input image a, I get out hash a, if I input image b, I should get out a different


hash b. And if I input image a again, I should again get back back a. Okay, this is a classic


hash, their hash functions are designed to if you input the same thing, you want to get the same


thing out. If you input a different thing, you want to get a different thing out. And ideally,


the thing on the right side, the hashes, they're much, much, much shorter, so much less data than


the original data. This works because I mean, theoretically, it shouldn't work, right. But it


works because most, most images that are possible in the data space aren't actually images. So the


the amount of images that can exist as natural images is way lower than you know, that the pixel


grid would allow. So there's a lot of compression potential. So the hash function is supposed to


output the same thing. If you input the same thing, output the different thing, if you input a


different thing, that's a classic hash function, we use hash functions when we want to check like


the integrity of files. So in a classic hash function, if you change even one bit, the hash is


going to change as well. That's how you see someone tempered with some some file or something like


this. Here, we're going to use a little bit of a different kind of hashing, we also use these


functions, but we also use this neural hash, which is going to be more fuzzy and geared towards the


fact that we deal with natural data with natural images. In any case, what we're going to do is


we're going to hash these, these images. And we're going to do a step that's called blinding, we'll


look at that. And we put them on the client device. So the client device has the database, but in a


hashed format. So looking at the hash will actually not tell you anything about the original image. So


this is the requirement, the user does not see the images that are in the database. Okay, that'd be


terrible. In fact, okay, like the regular user doesn't see anything. But even if you inspect your


device, you couldn't find that data because it's hashed. Now, on the client device, we take the


image of the user, we, we compare it to the database. Now we can do that since the hash function


output the same thing, if you input the same thing, right, if we run the image through the same hash


function, if we run the image through the same hash function, we can simply compare with the


database and see if there is something in the database that matches this images hash, and then


we know a hot that images in the database, it's a match. And then we can upload that to the cloud.


However, that would violate another one of our requirements, namely, the user could learn which


of the of their images match the database. So we'll have to, as I said, we'll have to get a bit


creative. So what we do is we don't check for a match on the device. What we do is we produce this


call so called safety voucher. The safety voucher is essentially comparing the image to the database,


but it leaves out like one step in the process. And that step can only be done by the server. So


so it's like a comparison, but you leave out the last step, it's actually not possible for the


client device to do the last step of the comparison that would actually evaluate if something fits.


And that's going to be done on the server. This technique is called private set intersection


matching. And on the server, you do the matching, if there is a match, you you know, you flash a red


light, except there's the additional constraint that you need to have this threshold requirement.


So you want that you can only decrypt the things of the user if a threshold is exceeded. And that


is yet another technique called I think threshold secret sharing or something like this. So we're


going to look at these components one by one. First, the neural hash. Now I told you about hash


functions. And I'm going to repeat that the issue about a hash function is if you input the same


thing, it should output the same hash, it should output the same number. So here you can see an


image on the top and the neural hash at the bottom. So this is the hash. So when we input


the same image, we want the system to output exactly this number, not a similar number exactly


this number. Now look at the image in the middle, would you say this is the same image or a different


image now in the context of detecting abuse material, this is the same image, like it displays


the same thing. We want our system to be robust to these transformations, because otherwise, these


people, they could just change the image a little bit. And then the hash changes, right, they could


make it a little bit brighter or darker, they could just re encode it, they could resize it a little


bit, and they would evade the detection. And that's what makes it difficult. What we can do is


we can train neural networks to handle these kinds of things, we already have the techniques. So the


two images you see here on the left, they should output the same neural hash. And the image here


on the right, which is a different image, it should output a different neural hash.


So what we're going to do is we're going to design a neural network in their case, it's a convolutional


neural network, says it right here, a convnet, you input the image into a bunch of layers. And then


at the end, you get out a vector. Okay, so you train this neural network, and you can do this


via contrastive learning, this is essentially self supervised contrastive learning, such that if you


input this image, and this image, their vectors are going to be fairly close together. And then


if you input this image right here, its vector is going to be, you know, a lot different. So the


vectors of images which are close in up to some transformations should be very, very close. This


is standard self supervised learning, you teach the network to be robust to these kinds of


transformations, you enforce that the vectors that the neural network outputs are close by each other


when you input these distorted images. And the network should also learn that images that are not


distortions of each other, it should go far away. So we can do this. But you'll notice here, the


requirement is not fulfilled. Namely, they don't, the neural network doesn't output the exact same


vector, it outputs only, we can only train it to output vectors that are really close by each other


if it's a similar image, and really far apart if it's a different one. So how do we get this


discrete ness in here, and that comes through locality sensitive hashing. So locality sensitive


hashing is essentially a method in from from kind of the big data world to do approximate nearest


neighbor search. And there is various techniques for doing this, I'm going to present you one of


them, which I, from what I read, this is what they do, it might do something slightly different. But


essentially, what you do is you define random hyperplanes. So one hyperplane might be this,


and you know, in our case, it's just going to be a a line, a 2d hyperplane. Sorry, a 1d hyperplane


in a 2d space. One might be this, and one might be this. Okay, so those are your your three lines.


Let's number them. This is number one, this is number two, this is number three. And let's also


label the sides of each. So this is the positive and the negative, positive and the negative,


the positive and the negative side of that. So now what what can you do is you can check for each


vector on which side of each of the three hyperplanes they are. So this vector right here,


it would be on the positive side of plane one, it would be on the positive side of plane two


and on the positive side of plane three. So what this vector would actually be, you can


even visually see they're in the same corner in the same slice of the space Whereas this vector


right here, it would actually be on the positive side of plane one, and on the negative side of


plane two, on the negative side of plane three. So here you can see it doesn't work for all vectors,


you can see it doesn't work for all vectors, right? Two vectors could be really close together,


yet a plane could just cut through them. In that case, you would not find those two. But if you


know, if you choose the number of planes correctly, their distribution correctly, then with very high


likelihood, if you have two images that are very similar, and the neural network, in fact, outputs


vectors that are close together for them, they will end up in the same bucket. So this here is


going to be the discrete neural hash of that image. Now, they then stick that since this might still


be a fairly high dimensional representation, depending on the hyperplanes, they stick that


into a classic hash function. So in order to reduce the number of bytes, and also in order


to make it less possible to in fact, reconstruct an image from the hash, because from these hashes,


it's still actually possible to reconstruct the image depending on the dimensionality, right?


They feed that through more hash functions in order to derive the neural hash. And there you see it.


The neural hash for these two images, if we have trained the neural network correctly,


should be the same in really like the same the same discrete bytes, whereas the neural hash for


this image will be different. So that's how you detect and depending on how you train the network,


you can catch most of these distortions, the network will also generalize. So even if some


person comes up with like some transformation that you haven't specifically thought of,


if you've done a good job at training, there's a good chance that you'll catch that transformation


as well. So this is how we derive the neural hashes. Now, from the neural hash. So our first


approach could be, you know, we take our big database of illegal material, right? So this is


an here's an image, here's an image, there's images, we run all of them through this exact


same neural hash procedure, and we get a neural hash out of it. And then for a user, we take their


image, we also run it through neural hash, right, that gives us some vector, and then we simply


compare to the neural hashes of the database, which we have with us, this would work, okay. But


as we said, this violates some of our requirements. Therefore, what do we do? So it's a bit more


complicated. The server, the Apple has this database, or presumably, they at least have these


hashes, these ones of the database, right? What they're going to do is they hash them, they hash


each of them one more time, with let's call that H prime. So they hash each of them one more time,


one more time, with a hashing function that only they know, right? So they have the hashing function,


it can also take like a private key. So there is a private key. And they call this the blinding


step. Okay, so there's a hashing function that only Apple knows. Now, if your image if the user


image goes here, they it gets like some sort of by the way, these lines, they are short for like,


they're short for a vector of zeros and ones, right? So if I draw a line, it's like that's a


it's a hash of an image. Now, if I have a hash of a user image, what I have to do is I have to send


it to the server, because only the server has H prime, right? As this hashing function, and then


the server can compare the two things. Alright, so now this. So now this is, this is this is better,


this fulfills our requirements better, in order to also have the other requirements included,


here is what we actually do. So what the server does is it derives the neural hash for each image


in the database. And then it does this blinding step, okay, so you receive a blinded hash from


each image that the server knows that and then you order the things you order the hashes


according to the neural hash. So how you how can you do that? You simply look at the neural hashes


of each images and you put them in order, right? So yeah, you just sort them. So the order of the


images is going to be according to the neural hash. So if I know the neural hash of an image,


I can determine what row in the database it is stored at. However, the row is of course,


a much shorter number than the neural hash itself. So I can't reconstruct the neural hash,


if I just from the row number. But I can, if I have a neural hash, I can know what row in the


database, the blinded hash for that image is stored. Okay, so for the server, this essentially


is double information, like this information comes from the image and this information also comes


from the image. However, for the client, what the client now does is you get the client the device,


you get the image, you compute the neural hash of the image. Now with the neural hash, you, you do


multiple things. So what you want to do is essentially you want to send the neural hash


to the server, along with a payload. And the payload, just imagine it contains the real image,


you put the real image into the payload, you upload that to the server, right, so the server


can actually compare. But this would violate a bunch of our things. So what do you do, you take


the neural hash, you look up the row, remember from the neural hash, you can look up which row


it the blinded hash is stored at. Now, we have two cases, if the user image is an actual illegal


image, right, then this blinded hash will be the actual blinded hash of this neural hash. So if I


were to run this through h prime on the server, I would actually get the blinded hash. However,


is the if the user image is not illegal material, you know, it will still have a neural hash, like


you can compute that for any image. And it will still determine a row to look up because you know,


you'll get a row, you'll just probably get some random row. It's a it's a function that's only


designed for the hashes that are in the database. So if you go to it with a hash that's not in the


database, it'll just give you some row. Specifically, if you apply h prime to the neural hash,


it will not output the same blinded hash. How can you now abuse this fact, such that the server


cannot learn anything about your image if your image is in fact not illegal? Well, what you do is


you look up, you look up the row using the neural hash. And you use whatever is here in that row as


a private key as an encryption key to encrypt the payload. And so you send you send the neural hash


to the server, and you send the encrypted payload to the server. Remember the payload, let's say


the payload contains the actual clear text image. So we only want the server to be able to look at


the image. If in fact, it's an illegal image. Again, let's play our two. Is there a diagram what


happens on the server? No, let's play our two scenarios here. So the server gets this cryptographic


header derived from the neural hash. The first thing it will do is it will run the neural hash


through h prime, the server can do that, right? It will obtain it will obtain the blinded hash for


that for that particular neural hash. Now, again, if in fact, this is an illegal image that should