of sequence to sequence tasks namely it deals with sequence to sequence tasks


Imputer: Sequence Modelling via Imputation and Dynamic Programming

where there is a monotonic alignment. So you see that this is given here the


first word is corresponding to the first word here the second to the second and


the third to the third. This is not always the case in machine translation


you know different languages have different sentence structures so for


example in French this would be je t'aime. And you can see that the first word is


still the first word however the third word has become the second when the


verb goes to the end. So the imputer would not be able to deal with this task


very well. A different task where the imputer would be useful for would be


something like speech recognition. So if someone were to speak the words I like


you and you would measure the waveform of that it would look something like I


like you. Right so if you have this waveform let's actually make some chunk


samples out of this. Let's say this is a sample right here and here is a break


here and here. Right so we have five samples on the bottom right and you can


see pretty easily that this sample here this is the I and then this is silence


this is the like this is silence and this is the you. So the imputer deals


with these kind of sequence to sequence tasks where first of all there is a


monotonic alignment sorry monotonic alignment and second of all this is an


engineering constraint where the length of the input sequence X is larger or


equal to the length of the input sequence Y and you'll you'll see why


mainly because we rely on being able to compute this alignment here. The


alignment of input samples to output samples right. You can see that the


monotonic alignment is actually given fairly well in speech recognition


because if something is later down here it is also later in the sequence up here


that is a monotonic alignment and also usually we have more wave samples it


then we have words in the output sequence so that would be a task for the


imputer. Now let's think about how we would do something like this. So let's


put X at the top here and we said X has five tokens in it and let's put Y at the


at the bottom right. Y actually has three tokens right so this here is I like you


this is the waveform right and we want the I like you at the bottom so what


could we do right first of all what the imputer does is it represents I like you


not as this form right here but as a form where you have the same length as X


divided into the same amount of things and then it does the following so for


this this is an example


right this is how it would represent Y right it would say I have as many chunks


on the top as on the bottom I know this chunk here corresponds to this token


then this here to this and this here to this and then these are these


intermediate ones right so you can see these correspond to those these these


are silence right here now it doesn't always need to be that there is always


one token and a silence then the token and a silence the task of the imputer is


actually to see whether this is more likely than for example I like and then


silence silence and then you right so the task of the imputer is to distinguish


these two from each other and then of course also produce the actual tokens


now if you think about how would you go about taking X and producing something


like Y so this is why let's call it till this is the actual Y right but you can


see that this here is a deterministic function in one way it's actually not a


deterministic function in the other way and that becomes interesting when you


have to compute a loss for this but how would we go about doing this right what


we could do is we could just take a big transformer BERT right that's actually


draw an arrow we could just take BERT and we could simply so in BERT you have


in if you if you construct it correctly you have as many input tokens as output


tokens so what we could simply say is for each of the outputs that we get we


simply make this as a softmax classifier over our vocabulary with the silence


being one special token and we simply classify each of the outputs into this


vocabulary this would be one step right so we could do one step BERT bang bang


input to output and there is more they're more sophisticated approaches to


doing this in one step like a CTC but ultimately we could just do one step but


then you'd have the same problem like for example XL net if you haven't seen


my XL net video I recommend it that they exactly take the problem if you do this


right then at the moment where you decode the word like you have no idea


that there is an I over here all you know is the the vector you have here


that you sample the I from right but this could be a distribution where I is


pretty high but some other word is also pretty high so this process over here


that samples the word like has no idea which of the two here you actually would


sample so it cannot condition on it so it is the the assumption here is that


the sampling of the word like is independent of the sampling of the word


I and of course that's not the case the you need to know what word is there if


you want to sample the word like otherwise you can end up with some very


confusing sentences so this one step process is pretty quick but it has the


drawback that there are these conditional independence assumptions and again I


invite you to watch the XL net video if you want to dive more into this problem


the second thing you we could do is we could just decode one after another


right so we could say all right I'll make sorry I'll make my five slots here


and I just leave them empty for now and I'm just going to decode the one that I


am most sure about and let's say the the speech at the back here is is very clear


and you say other I'm I know this is a you right so I'm going to fill in you


right here right and make this alignment that this goes here this is the you I


still don't know what the others are but now what I did they do a second step and


in the second step I get as an input not only the original input like this this


thing here but I also get the fact that I already decoded the word you to here


right in this step so now I say given that I already decoded the word you


which one am I now most sure about and I'm might be most sure about to say I'm


most sure about this now being an eye because there's a you at the end and


this kind of sounds like an eye so an eye here right it goes to the next step


and then the next step it already has the information that it decoded I and


you and now it's a might say okay given these that's so probably this thing so I


here probably the thing here the thing right here is silence right makes the


most sense I can I hear some noise but there's already a word after so now I'm


pretty sure that this here is a silent token and you go this until the end


until you're actually at this so this here would be n step decoding this here


would be n steps of decoding which now no longer has the problem of these


conditional independence assumptions but of course now you have the problem that


you need n steps right the imputer does something in the middle of this the


imputer will as you can see here it will form this into blocks right blocks of


size B and this is the empty symbol here right and what it will do is it will


make a step where in each block for each block it will conditioned on the


previous alignment and conditioned on the input it will decode whatever it


feels it is most certain about in each block and then it does this for as long


as there are still empty tokens right you can see here the first block and


then in the second step it will decode this this this and this so the imputer


can trade off between the conditional independence assumption of the one step


BERT and the full conditional independence assumption of the n step


decoding right so it will compute this alignment and the actual tokens at the


same time in this process so how many steps does this take this takes now B


steps and this is pretty cool because B is the block size so this is independent


of the sequence length so it is able to compute this alignment and output in a


constant number of steps right so you're by modulating this B you're now able to


trade off speed versus let's say performance in the imputer and this is


pretty cool so I think actually I think the the bigger point to understand here


is how to actually use the assumption that there is a monotonic alignment


right because if there is a monotonic alignment and if this thing is given


here then you can do this you can do this representation right here with the


silence tokens and that allows you to basically represent the output in a form


that is of the same length as the input and do this kind of token by token


decoding while still allowing you to have variable lengths output as long as


they're smaller in length than the input so that's pretty cool and then the the


next pretty cool thing is the fact that they do this in blocks now of course my


issue with this so this is how the system works my issue with this is how


the system is trained so if you think about how you train this you must train


this first of all the loss function right has to revert this and how they do


it as they marginalize you see this down here you want to marginalize over all


the possible alignments right here and so this is how you train you sample an


alignment from the alignment policy and this alignment policy is I think they


have some heuristics of how they construct the alignments during during


training or you have experts actually giving you this alignment I think they


use in the speech recognition they use something like CTC to give you the


alignments from the alignment policy and then you have a masking policy and I


think they also did just do random masking and then they use that for


training and then they marginalize over the alignments this I'm pretty sure is


not the same distribution as the decoding procedure I just described


right so the decoding procedure if you do this in B steps right that means each


of the step is dependent on the step before so that means the distribution of


whatever you whatever the imputer sees is actually dependent on itself while


these people are proposing a training framework where you have here you have a


heuristic in order to come up with the training sample alignments and here you


have a random I think a random masking policy that comes up with the with where


the empty tokens are so this is not the same distribution and then also it


marginalizes over all compatible alignments which I'm I'm pretty sure


this is not the same distribution this is not the correct loss distribution


they have some math to show that in expectation it's the same but yeah this


is this is over there over there roll-in policy and roll and X sir and and


marginalization this I don't want to go too deep into this I've given it some


thought but it will make this video too long and boring if I actually go into


the details here as suffice to say I invite you to look at the loss


computation and ask yourself if you think that is the correct way to produce


the data set for training given how you do the inference later the architecture


of the imputer is actually pretty similar to BERT in that first of all


well okay you're dealing with audio in the input so you're going to have some


convolutional network here and you also need to take as an input the prior


alignment that you've already produced right so this you embed and but then you


simply do an attention network a transformer which will which is pretty


close to to the BERT example we've made and so I mean they stress that that


their that their loss is actually a lower bound on the loss so I shouldn't


be I shouldn't be too hard when I say it's not the correct distribution they


do minimize something some loss that actually makes sense but yeah I mainly


wanted to go over the over the how the imputer works and how the it is


structured and I think it's pretty cool and it lends itself very well to these


tasks and most of all I like the fact that it exploits the these assumptions


here so not all tasks fit these assumptions but if a task does fit the


assumption then I think it should be you know it it should be fairly obvious that


one should exploit that in order to perform better all right that was it for


PCGRL: Procedural Content Generation via Reinforcement Learning (Paper Explained)

Hi there, have you ever wondered how video game levels are made? Yeah, me neither. But


this paper has. And in this paper, you can see a reinforcement learning agent that has


learned how to make video game levels in various ways. So this is implemented for this game


here where the goal is simply to make the longest maze. This game here is an adaptation


to the legend of Zelda, where you have to get a key to the door. And this game here


is called Sobocon, where you have to put all of the crates onto the green squares in order


to solve it. So it's a puzzle game. Alright, and this is done via reinforcement learning.


So the paper we're going to look at is called PCGRL procedural content generation via reinforcement


learning by Ahmed Khalifa, Philip Bontrager, Sam Early and Julian Togelius. Now this paper


is basically just a fun paper, and I feel and it shows how to frame a problem in terms


of reinforcement learning, and then how to solve it. It's pretty straightforward, it's


fairly short. And I you know, the codes available and all so you can go check it out yourself.


They say we investigate how reinforcement learning can be used to train level designing


agents. Okay, so usually, usually, you we do reinforcement learning for playing games


in themselves. And now we use reinforcement learning to train an agent that can design


a level. So we don't design the level itself. straightforward, we design the agent that


designs the level. And what's the advantage here? The advantages, of course, the agent


could then potentially generate multiple different levels once we have trained it. I say this


represents a new approach to procedural content generation in games, where level design is


framed as a game itself. So the design of the level is now the game. And the content


generator itself is learned. By seeing the design problem as a sequential task, we can


use reinforcement learning to learn how to take the next action, so that the expected


final level quality is maximized. This approach can be used when few or no examples exist


to train from, and the train generator is very, very fast. Okay, so this is the outset


of the of the problem formulation. Now, we're going to go through the steps you have to


do in order to make this work. There are a few things that I think this paper does quite


well. And so the first thing is you actually have to frame the problem in terms of reinforcement


learning. So what is reinforcement learning is pretty simple. In reinforcement learning,


you have this agent environment split, okay, so at each step, the environment is going


to send the agent an observation. So the environment is going to send an observation to the agent.


And the agent needs to take an action in response to that. Now, something happens in here, we


don't we don't worry about that. And the environment is going to send the next observation, okay,


that is a result from taking this action. And it is also going to send the reward for


this action. So at each step, the agent gets an observation, and the reward for the last


action took and it has the output the next action. Now, the environment, of course, has


to somehow decide how do I represent the observation, this is the representation, how do I transform


one observation to the next observation given an action, okay, the action comes in and transforms


the last state to the next state. And then how do I give the reward? How do I calculate


the reward? So these things are the things you have to decide on the observation space,


the how the reward is calculated, the action space, and how an action transforms one representation


into the next representation. So this is what we're going to look at the different variants,


we're not going to look at specifically how reinforcement learning is done. Because once


you have an environment like this, you can just plug it into a standard reinforcement


learning algorithm, and it will solve it for you. Okay, so that's the power of basically


having standardized or representations. So the observation space of this problem is going


to be pretty simple. All the games we're dealing with here are in this, oh, I already did some


drawings are in this, in this framework of this grid world game, okay, so you have this,


you have this grid, this level is subdivided into this grid. And that naturally corresponds,


of course, to a 2d matrix. Now, each point in this matrix has a number, and the number


describes what type of tile this tile is. So as you can see, right here, the one is


going to be a wall, while the zero is going to be empty space. Three here is one of these


crates, and two is the player. So you get the point, right, each number corresponds


to a type of tile. So far, so good. That's the observation space. Now, what is the action


space? What is the what the agent can do? At each step, they say the agent can change


one of these tiles, okay, so you can change one of these tiles, let's say this one right


here, it can change it to a different one, or you can just leave it. So this is a wall


right now. And it makes the problem fairly interesting to have a wall right here in the


middle. So we're since we're looking at this tile, we might just want to leave this right


there. We could also change it, we could actually change it such that to a to a two such that


there is another player right here. Right, let me Can I even draw this? This yellow isn't


such that there is now two player tiles in the game. So would of course be an invalid


level. And the reinforcement learning agent ultimately should learn to produce valid and


good levels. Okay, so at each step, you can change one tile. And of course, the goal is


to make a better and better and better level over time. Now how how do you choose which


tile to to change? That's a thing you have to define. And they define three different


ways where in which the agent can choose which tile to change. In the narrow formulation,


they themselves so the environment chooses the frame the the tile to change. So the environment


will say now you can change this tile if you want. And then the next step it will say now


you can change this one if you want. Now you can change this one if you want. Now this


is completely random, okay? How the environment actually doesn't have to be but the environment


chooses and that is problematic for the agent because the agent cannot kind of predict which


tile it can change next. And therefore, it cannot really plan ahead how it wants to change


the level it can only make very, very local, very greedy choices. It can be like, Oh, I'm,


I'm right here. I might I'm actually build a wall right here. Yes, that sounds good.


But it can't. So an example is maybe you want to make the level more interesting. Maybe


you think that the crate up here is a bit close to this field here, you have to push


it onto this field. And that's fairly easy, right? So you just push it like up and then


to the left. Actually, it's not that easy, because there's the wall right here. And you


have to go around. Actually, you probably have to push this down. But let's say the


level is too easy. And you want to move the move the crate to a bit, like let's say here,


in this, in this framework where the environment tells you which which, which tiles to change,


you can basically, once you come across this tile, you can delete it, okay, but then you


kind of have to wait and wait and wait and wait until at random, this tile where you


want to put it is selected. And this might actually never happen because the episode


might be over. And if it never happens, you are in an invalid level, right? So the the


the agent here is basically forced to greedily make the level valid before it can make it


interesting and then it can only make it interesting in sort of local ways. So the second way that


the second formulation here is the turtle formulation. Now, you might know this from


the turtle graphics, where basically, you have this, you have this little turtle thing,


and you can always move it either like, you know, down, up, left or right. And then you


can always put a dot or not put a dot and thereby you can like trace out things. This


is like intro to programming. Same here. So now the agent is given a starting square,


and it can choose to change it or not, but it can also choose how to move to the next


square. So to the right, up, left or down, it can so it can choose so you can go along


and say, Okay, now I'm here, I want to change it to a two. Now I'm here, I want to change


it to a two. Now I'm here, and so on. So it can basically do things like build long walls


and things like this. So you can plan ahead more considerably. But still, if you regard


the problem from before, if it wants to place the misplaced place to create to a different


location, it can like if maybe it's here, okay, the agent is here. And then it can say,


Okay, I want to not change but move not change and move not change, move and then it can


delete and then it has to move over here step by step until it can place it again. It's


so it can plan ahead considerably longer, actually, it could just move straight over


because the agent itself is not constrained by walls. So it can move ahead quite a bit.


But it's still kind of localized changes because it can move one tile at a time, right? And


if in between the episode ends, it's again, an invalid level. So the third formulation


is the most powerful formulation. It's called the wide formulation. And this is where the


agent at each time step cannot only choose how to change the tile but can freely choose


the next tile to change. So it can it could say, in one step, it could say, I want to


delete this tile. And then in the next step, it could say, I want to place it right here.


Okay, so this is so it can plan ahead considerably. So how you design the action space is very


important for how your agent or for what your agent can possibly learn, and how easy it


is for the agent to learn because it's going to be pretty easy for this agent to learn


to move crates like this, where even though the other agent that moves one tile at a time


can also do it, it has to plan ahead for longer. So it has to sort of invest more of the reinforcement


learning power into doing these sorts of things. But of course, it's being more constrained


also means you have less actions at your disposal. Like this last agent, it has a lot of actions


it can do, it can choose any tile at once, right. So that can also introduce considerable


exploration dilemma. And you have to trade these things off when you design things like


that. Alright, so this is the action space. Now, how the observation involves evolves


into the next observation should be fairly clear. I mean, that's already given by the


action space. If you ask yourself, if you're in this in this situation right here, and


the agent deletes the crate, then the crate is no longer there. Then, so if it changes


this to a zero, then it's just empty space now. Okay, so that's, that's fairly obvious


here. Now the last thing we need to do is the reward calculation. What reward do you


give the agent? And you can here you can give the agent the reward either, let's say at


the very end, you cannot give it a reward for the entire episode and give it a reward


at the very end. reinforcement learning algorithms are able to deal with this to a certain degree.


You can also decide to give it at each step. Now the way they do it here, I believe is


they give it at the end, and they have multiple components to the reward. So the reward in


this case is how well the level fulfills certain goals that the programmer sets. So the goals


in SoboCon are basically the rules of the game. And that means there is only one player,


if there are two or none, then the reward is less. There are at least one crate, and


there are as many crates as green fields. Okay, so here you can see there are only two


crates, but three green fields. So the agent will get a penalty for producing a level like


this. And then the last thing is the level has to be solvable. And for checking solvability,


the the authors of this paper simply employ a SoboCon solver, they have a SoboCon solver,


that is like a tree search algorithm that tries to solve the level if it can't solve


the level, then the level is invalid and the agent gets a worse reward than whenever the


level is solvable. Okay, so how you design the reward is also very important if you only


give give like a one reward when all the goals are fulfilled and give a zero reward as soon


as one of the goals is not fulfilled, a reinforcement learning agent is going to have a very, very


much difficult time to learn that. So you have to kind of design the reward such that


you help the agent realize what's important. So maybe if there's only one crate missing,


but you know, in fact, the level is solvable. Except for that, you know that maybe one green


field is going to be empty, then you could still give a fairly high reward, but you could


just give a higher reward when the level is actually solvable, like or all the rules are


fulfilled, fulfilled, and there is a crate here. Okay. The other thing to notice here


is that in this case, you actually do need a solver for the level since it's a puzzle


game. And that means your level your agent is only going to produce levels that are as


difficult as your solver can solve. So that's going to be a considerable problem. But that's,


you know, a limitation here. But all of their rewards are hard coded, so to say. So there's


the reward is given by the environment. So now that we have observations, which are these


matrices right here, we have actions, which and we actually have three different ways


of formulating actions, and we have reward, they can simply plug this into a standard


reinforcement learning algorithm. Now they have one last thing that they have, which


is this change percentage parameter. So what they say is they give the agent an initial


state, and then the agent is allowed to change it around like here. So on the left, you have


this initial state, okay, this is sort of a random initial state and you allow the agent


now to change it in this stepwise fashion, and you always update the agent. By the way,


the agent, as you might imagine, the agent takes this matrix right here and puts it shoves


it through like a few convolutional layers and then decides on an action. I almost forgetting


that this this is so obvious by now that yeah, the agent is like a standard deep learning,


taking in a 2d doing some convolutions and then having like a policy output. Yeah. So


you shove this into a proximal policy optimization algorithm, which is a standard reinforcement


learning algorithm and you allow it to change these things. Now what they do is they only


allow the agent to change the levels by so much because what they say is, if we start


out from these different states, we would, you can decide on two things, either the you


can train the agent to find you the best possible level ever, right, but then it would sort


of ignore the starting state, it would just learn which level gives me the highest reward.


And it would just change all the tiles always to that, right, it would just try to change


the to always reach that best possible state and forget the start state. So they say, okay,


the the last constraint is the agent can only change like 20% of the tiles at most. And


after after that, we end the episode or we just don't allow the agent to change anything


anymore. It needs to first. So if it changed this here to empty space and wants to change


something else first needs to change this back and then it can change something else.


So you can do that. So this constrains the agent and kind of teaches it that in order


to get a high reward, it must sort of adjust the starting state to something that gets


higher reward. And that's one way of making the levels that you generate more diverse.


It's sort of a unique problem to this particular kind of reinforcement learning problem. Because


all sometimes, like most of the time, you just want to find the highest reward, whatever.


But here, you also want to, you know, maximize diversity of the levels you generate. And


therefore, you could say that's, that's a pretty good, you know, that's a pretty good


constraint to put into that. So that's a thing I like here about this paper. This, this change


percentage constraint. Now at inference time, you can change that. So at training time,


you only change whatever 20%. But at inference time, you can technically let the agent run


for longer, as you can see, I think here, they just let it run until it, you know, find


something good, like this one right here, fairly good from the starting state. And you


can see it sort of still adjusts to the starting state right here. So you can see that these,


it connects the two dots on the top. So the goal is to make the longest possible maze


or a long maze. So it connects these two, you can see here, also, this one connects


them. So, and then it goes out here and connects to this one. So it's fairly good at relying


on this starting state. Now, you can see that these turtle and wide representations that


can actually choose where to go and where to change something are considerably, or,


you know, more more powerful than this narrow thing, especially if you look at this level


right here, which again, is the importance of designing the action space well, is going


to directly affect the outcome that you're going to have. All right, and you see the


same thing here for this Zelda game. Now, here you can see the starting state often


involves, let's say, here you have two players, and you have three keys, and that's an invalid


starting state. And sometimes the the door cannot be reached, sometimes the door is actually


not even there, like here. And you can see that the agent, all of the agents sort of


learn to make at least valid levels, where you have the player and the door and the key


right here, being able to reach everything. So that's, you know, fairly, fairly cool,


because counting is one of these things that the neural networks aren't necessarily super


good at. So it's nice to see that, you know, they can, they can, here they have two players,


and they they're deleting one of them. Here, they have three crates, and they actually


make it such that the number of crates and the number of green tiles agree. So, you know,


that's that's fairly cool that this comes out. And here, you can see the different power


of the algorithms. So in this binary problem, and this is the Zelda problem, this is Sobocon


problem, you can see that as you allow the agent at inference time to change more and


more of the level, the percentage of levels where the agent gets a good gets a good level,


like succeeds in building a valid level, goes up and up. And now this, as I already said,


this narrow representation here appears to be a bit less powerful than the others. Interestingly,


in Sobocon, the best one is this turtle representation where you can only change one tile at a time,


and not the more powerful wide representation. That's probably because I'm going to guess


that's because the either the reinforcement learning algorithm was isn't, you know, powerful


enough or their representation like the CNN is maybe miss architecture a bit. You know,


technically, this representation should be able to achieve higher scores, but not as


easily because as I said, the action space is so much higher. So it's more difficult


to learn, but ultimately, it should learn it better. Alright, so this is this was this


paper. It's, I think it's fairly cool and fairly fun to view it from this particular


perspective. And they discussed that the future could be that humans solve this together.


Because usually, when you have assisted level design, you would have something some sort


of like an optimizer running to optimize the level you're working on directly, like it


say, okay, make something here and it would sort of run for a while. And that takes, you


know, takes time. Now this here, this agent at inference time is very, very fast. So it


can, you know, work together with humans. So the human would say, for example, Oh, here,


please make a wall right here, because that's going to make the level more interesting,


but make it such that the level is still, you know, interesting and solvable. And then


the agent can, you know, go across do some things that's going to be super fast. And


agents and humans could work together at this. Now one drawback, of course, is that in a


puzzle game like Sobocon, you know, you have to make sure the level is solvable. And here,


luckily, you can employ a solver, but as the puzzles get more difficult, that's not, not


super like, that's not going to be the case that much. And also, they remark that most


of the levels generated are fairly easy, because their reward only depends on whether or not


the level is solvable by an easy solver, right. So you could give some reward for how difficult


the level is. But then again, that depends on your solver. So an interesting next step


would be to evolve these, or to train these. As you train reinforcement learning agents


to solve these kinds of games. So kind of do a curriculum learning sort of a GAN setting


between level generator and reinforcement learning algorithm, like reinforcement learning


game player to to sort of evolve levels and agents at the same time. I think it's sort


of like this poet approaches, except you would directly learn. I think that would be next


nice direction for this work. In any case, the code is available, you can even plug in


your own games and make your own levels. So check this out. And with that, I'll see you


On the Measure of Intelligence by François Chollet - Part 3: The Math (Paper Explained)

Hello and welcome to the third part on On the Measure of Intelligence by Francois Chollet.


Now, this is a multi-part series. If you haven't seen the first two parts, I recommend to watch at least one of them.


They're somewhat overlapping. But we've basically gone over the history of intelligence measurement


and the foundations of what a measurement for intelligence for an AI system should look like.


And today we're going to get into the formal definition of the intelligence that Chollet proposes right here.


So this sentence here pretty much sums up what kind of what we're interested in.


The intelligence of a system is a measure of its skill acquisition efficiency over a scope of tasks with respect to priors, experience and generalization difficulty.


So these are the things that we've established so far, basically.


The intelligence of a system, that's the thing we want to measure, right, is a measure of its skill acquisition efficiency.


So how fast does it acquire new skills? Important here is that we are measuring it over a scope of tasks.


So it's not arbitrary skills. It is a scope that we define.


And this is going to be mostly the the human scope, the tasks, the scope of tasks that humans can solve and are sort of different at.


And what we need to factor in our priors, which is what is already built in to a system, because that doesn't count as intelligence that's already built in.


If your ability to solve a problem is already built into you, you don't have to use intelligence to solve the problem.


Second, experience. If you have had lots and lots and lots of experience at the particular task you're asked to solve, you don't have to use intelligence.


You can simply rely on your experience. And the third is generalization difficulty.


And that's a property of the task. So if the task is very difficult to generalize.


So if it's very difficult, if the task itself is very difficult, then achieving a good score at it should count as having higher intelligence if all other things are equal.


So this is going to be the basis. And today we're going to watch Shirley define these things into a basically into a number that can give us the intelligence of any system with respect to these things.


So that's the program for today. If you like content like this, share it out and tell all your friends and leave a like so that YouTube knows that you do like it.


OK, so the conceptualization of the entire system is like this.


There is a task and we're going to consider a series of tasks, of course.


But if we just look at one task in our scope, there is the task and the task outputs these situations.


OK, which in a machine learning term, these are like your training examples.


And on the other side, there is the intelligent system.


OK, now the intelligent system in a pure machine learning side, you would factor this as the task gives the intelligent system a something like a training sample or in reinforcement learning, it would be something like an observation.


And the intelligent system, you know, gives something back like a response.


Here we have a kind of a in between step.


The intelligent system doesn't actually give back the response to the situation. The intelligent system generates a skill program.


So the intelligent system will generate a program that can map the situation to a response.


And that skill program should be able to run on its own.


OK, so in the classic machine learning sense, if we look at supervised learning, for example, the intelligent system would be the would be like a ResNet, ResNet plus SGD.


OK, that is an intelligent system.


And if it is output to is it is able to generate a skill program.


So during training, what happens during training?


During training, the intelligent system is able to intervene in the skill program at each step.


So the situation comes in and then the skill program does something.


But the intelligent system can at any point, it can kind of intervene and update the skill program and generate a new skill program for the next step.


Right. So there's a situation the skill program gives a response and the task gives feedback in form of a score.


In machine learning terms, this would be your training sample.


Your training sample comes in, your neural network gives a response, which are the logits of the classes.


Right. Then the task gives a score to that, which in the supervised learning case is the label or the loss function as a feedback to the intelligent system.


And the intelligent system using SGD would update the skill program for the next step.


OK, so at each step, the intelligent system can update the skill program.


That's why the intelligent system in this case is the architecture of the neural network and the procedure to update the weights, not the weights themselves, but the procedure to update the weights.


And the skill program here, those would be the actual weights of the neural network or like the instantiation of the ResNet with these particular weights.


OK, now at test time, we sever this connection right here.


So this this is now severed at test time. At some point, the training is done.


The task says, OK, now training is done. And then the intelligent system will produce one last skill program.


And then this connection is cut and the skill program must by itself answer to these situations.


It cannot like the intelligent system cannot intervene anymore.


And in this loop here, it situation response, situation response.


This goes over for a number of steps and all the scores during that time are counted and tallied up.


And at the end, you know, the higher the score, the better.


So the intelligent system must at this end step produce a skill program that by itself can achieve a high score.


OK, so there's always this training phase first and then there is the test phase.


Now the training phase, the situations that we get in a training phase, they are called a curriculum in this in this world.


In our world, this would be something like a training data set.


But this this is curriculum. It's slightly more intricate.


But just the notion here makes sense, right? The intelligent system produces the skill program.


OK, so there's a lot of formalisms right here, like, OK, the task has like a situation generator and that maps the task state to a situation.


So the task can have a state and the skill program can have a state and the intelligent system can have a state.


And I don't like this is all a bit too formal. You don't really need to understand it.


Except if Francois Chollet is watching this, I think I have found I'm not sure if it's a mistake, but you say the intelligent system here consists of three objects.


So it generates the skill program according to its internal state.


OK, and it generates the skill program. And when it learns, when it learns, it updates the internal scale internal state according to.


Let me if I can find it. Right here. A self update function.


So this is how the intelligent system can update itself, so its own state.


So it takes the internal state of the intelligent system and outputs another internal state.


And this is the you know where I said the internal the intelligent system at each training step, it can observe what happens and basically react accordingly.


So it takes the situation, the response and the feedback and its own internal state as an input.


Now, what do we have here? It takes the situation, which would be in our case, the training sample, the response or the logits that the neural network has produced,


the feedback, which is the loss and its internal state.


OK, now what I argue is basically that it should also get the internal state of the skill program as an input right here, because the skill program can have an internal state.


All of this, like the response can be a stochastic procedure of the skill program.


And I guess it's not necessary because you can sort of infer it, but I think the framework would be more complete if the internal state of the skill program at that time were part of the intelligent system update procedure.


Just you don't. OK, this is not relevant. This is just me bickering.


Cool. Let's actually jump all of this. This is boring. This is very boring. OK, blah, blah, blah, blah, blah. Lots of definitions.


All right. Quantifying, generalization, difficulty, experience and priors using algorithmic information theory.


So these things that at the beginning we said that we want to define intelligence with respect to, we are now going to quantify using algorithmic information theory.


Algorithmic information theory in this case right here that we're using, it's not very complicated.


The main quantity is this H, the algorithmic complexity. The H of S is the length of the shortest description of the string in a fixed universal language.


So it's the length of the shortest program that outputs the string when running on a fixed universal Turing machine.


So basically, if you have this string S right here, S is a bit string, the shortest program that can compute S or, you know.


So in the worst case, that's the string itself. But if the string is like 01010101 all the way, you can just say 01 times 50.


And that would be like the shortest program to produce that. It is an information theoretic concept here.


But in essence, you can just think of it as a measure of how long is the program that I would need to write to output a given, to produce a given output.


So that's the algorithmic complexity. And then the second quantity right here is the relative algorithmic complexity, which is almost the same thing.


It's how long is the program that I have to write, so the shortest, we're always talking about how long is the shortest program that I have to write that produces S1, but is allowed to take S2 as an input.


So it can always ignore S2. That's always a possibility. So if S1 is like a super easy string, you can just output that. But if S1, let's say S2, here is 01001.


And S1 is 01001, 01001. So it's just twice that. So you could sort of output that string here, you could write a program that just outputs this, or you could write a program that just says 2 times S2.


Okay, so the length of this is not part of the program, the program is just 2 times S2, because it's allowed to take S2 as an input.


Okay, so this is the algorithmic, the relative algorithmic complexity is how complex is the program to get from S2 to S1. So you can almost already see how that will relate now to generalization.


Okay, so a few quantities that we need to consider are a task called T here, then Sol T theta is the shortest of all possible solutions of T.


So task T has a solution of threshold theta, which the threshold theta is just, this is like the minimum score we need to achieve in a task.


We consider tasks according to thresholds, like, you know, you need to get a score of 9000 in Pong or so.


So the shortest of all programs that will optimize, that will solve the task to a threshold theta, which is the shortest scale program that achieves at least theta during evaluation.


And the other quantity is this train Sol opt TC. This has a lot of quantity right here, you can see the task, we want to be optimal, but with respect to a curriculum.


Okay, the curriculum is the training data. So this quantity is the shortest optimal training time solution given a curriculum.


So it's the shortest scale program that achieves optimal training time performance over the situation in the curriculum.


So this right here is if we could, if we had an oracle that told us here is how to solve the task in general, like the task of determining cats from dogs in images.


This would be the program that does it. Okay. You know, overall, over the entirety of the task, all cats and dog images there are. That's the solution.


Now this quantity right here means sort of the one neural network that is best at disturb determining cats from dogs in this particular training data set, this curriculum C.


Okay, so this is the the one neural network that is hyper optimized in this particular training data set.


And now we assess the generalization difficulty. So the generalization difficulty is going to be a measure of how hard is it to in a particular task to generalize to the whole task from the curriculum C.


And that's going to be the relative algorithmic complexity to go from this quantity to this quantity, both quantities we've just explained.


So it basically means if, if I had the perfect solution on the training data set, how much how much more complex is it to get from that to the perfect solution on the entire T of data, or you can also guess on the test data set.


So if, if this is really easy, so if the training data set already perfectly captures all of the data there is that this quantity is zero, like the out the program.


I don't need to write a program, I already have the solution. Right. And you can see here we divide by the age of Sol T.


But however, if the training data has no information whatsoever about about the about the solution to the general task, or if I just so horribly overfit on the training data such that it doesn't help me at all for the general task, then this quantity is zero.


So this quantity is in zero one with with, sorry, the quantity is one, of course, yes, because the shortest this thing up here will be equal to just H of Sol T, because this doesn't help me in that case.


And then this ratio will be one. So generalization difficulty of one basically means that the training data solution doesn't help me at all.


This this particular training curriculum is useless because I'll just overfit so horribly that I will not learn anything about the task or I can't learn anything at all.


And generalization difficulty of zero. Oh, that's yeah. No. Yes. Generalization difficulty of zero basically means that all of the solution is already contained in the training solution.


And I require no work to get to the to the test set solution. Okay.


This is I mean, I do this train test set. This is all a bit more general as it is written here, but I think it's a good a good way to think about it.


Okay, so the point here he makes is that is that so.


Yeah, he makes this example right here where he has these two data points where X minus point seven five has a label false and X point one five has a label true and the shortest possible solution will not help you to generalize to the to the other things.


So the nearest neighbor program would be better prepared for future uncertainty, but would take significantly more space to write down.


So there's there's a trade off there is direct trade off to how much you optimize on the training data and how much generalization capability you have.


Okay, so the next quantity we want to assess is developer aware generalization difficulty because so far we've only considered generalization difficulty with respect to the task itself and to the to the curriculum.


But what you could do is you could simply, you know, you producing this intelligent system, you could simply build in the solution to the entire task into your intelligent system.


That means it could completely ignore the training data and still perform pretty well on this thing, even though, even though the training data itself the algorithmic complexity.


It tells you nothing about about this so the generalization difficulty would be very high in the measure up here.


But so you would think wow, this intelligent system solves this task really well, but it's because you've baked the solution to the task into the system, and it just ignores the training data.


So the developer aware generalization difficulty is going to capture that and basically punish you for building the solution, the final solution directly into the task so this is the intelligent system right here at time zero.


This is basically whatever you pre build into the intelligent system. This is it hasn't interacted with the training data yet.


This is simply the state at the very beginning. So this is all the priors you build in.


If you build a ResNet, you know it has certain you know it has convolutional filters and so on.


That's a certain prior on the translational invariance if you build a an alpha go system that certainly has the rules of go built in to the system and it has this Monte Carlo tree search which biases it towards a certain kind of learning and so on.


So all of this is captured in this quantity right here.


This basically means that how if I am given the optimal training solution as before, and also the initial state of the learning system.


How much more work is it to get to the solution of the task. And here you can clearly see if I have already built the solution into the system.


So if I'm building a tic tac toe learning system, I call it the learning system, but I like build in the optimal strategy from the beginning into my system and it just ignores the training data.


Then this thing here would be low because it takes me, it takes me a lot of work to own we only have the training data to get to the solution.


But it takes me very little work if I also have the initial state of the system because the solution would be encoded into the initial state already, right.


So any prior you put in there will be captured by this. Okay. So, otherwise it's the same, the same metric.


Zero means it's very easy to generalize to the entire solution. One means it's even like it's given the training data solution and the system you give me that it is very hard for that system.


Now consider here this quantity actually depends on the system you put in. All right.


Then we need two more things, which are priors and experience. So this was the difficulty. This was how difficult is the task as such if for a given system and a curriculum.


Now, what we want, we want to characterize priors and experience now priors are pretty easy.


What are, what is a prior, a prior we can capture by simply looking at the difference between how complex is the solution minus how complex is the solution if I'm given the initial state.


This is almost the same as before, but it now only considers what you built into the system, right. There is no training data anymore.


It simply says, if I have, you know, if you give me your, the source code of your learning system, can I, if I can already read out the solution, then this quantity right here will be zero.


There is zero complexity to get from your initial state of the learning system to the solution of the task and therefore this entire quantity would be one. That means the prior, all the information is in the prior.


However, if your learning system is a very general learning system, like it's a, it's like a standard reinforcement learning algorithm with almost no assumption about the data, then this quantity right here would be very low.


Sorry, it would be very high, of course, because the initial system doesn't tell you too much. So it's still a lot of work. If I gave you the source code, it's like, well, this is very general.


This doesn't tell me anything about the task and it would require a lot of work to get the solution of the task and therefore the quantity up here would be very low and therefore this would be close to zero.


So that means there are no priors in this intelligent system for that given task.


OK, and the quantity is always how to reach the threshold. The solution is always with respect to a threshold in scale. So you must reach like this many points.


OK, the important thing that Chollet notes here is that the priors capture not the amount of information in the program, in the intelligent system, but the amount of relevant information for that task.


You can make a super duper complex intelligent system. The only thing that matters for this quantity is the amount of information that's relevant to get to the solution of the task T.


The last thing we need to capture is experience. Now, experience basically means how much during this learning phase. Now we just talked about at the outset, like the state at time zero for the priors.


Now you remember we interact with the task for a number of time in this training phase, right?


And the question is, of course, if we are given a longer training phase, it is easier to generalize generally.


Right? More training data makes it makes our life easier, makes it easier to generalize. And the intelligence is inverse proportional to that.


So a system that had all else being equal, that has less training data, but is performing as well on a task as a system that had more training data, that system that had less training data, we consider to be a more intelligent system because it can generalize more efficiently.


So we need to quantify experience and experience now in the same kind of in the same train of thought is going to be the difference between two quantities.


So the first quantity is this. So here we consider at each time step T. OK, so at each time step T, we have the intelligent system.


And we have this thing here called data. Now, data is everything that the intelligent system gets at that point in time.


So the intelligent system is here at time step T, and then it outputs the skill program and that skill program gets a situation and gives a response.


And this gives a feedback and all of this data that's called data.


OK, it's basically you can think of it as one additional training example.


You're at time step T and you're given one additional training example. The experience is going to quantify how much information is in that one additional example.


And then we're going to sum this up over time down here, which basically means over the entire course of training, which is this curriculum C,


how much information did you get out of the training data at each step?


That's going to be your experience over the course of training, which again, this is the sum over the experience that you got at each step.


And the experience at each step is simply the following two things.


So we're going to assess is how difficult is it at time T?


So you've learned for T steps, how difficult is it to go from that to the solution?


So if you might have had some training data, right, and you score a certain you score a certain score, like 80 percent on the on the test set.


So that's basically how difficult it is.


It's like you still make 20 percent of error.


That's your difficulty.


And then you get one more training sample, this data here.


Now, you can ask again, if I know everything I'm knowing my intelligence system, but I also get one more training data point.


Can I? How easy is it now to arrive at the solution of the task?


And now you can say, oh, with this training data point, I now can correct some of my mistakes and I only make like 18 percent of error.


OK, so the difference here would be like two percent.


So that's going to be your experience is going to be worth of two percent of errors.


OK, now the important thing here is that it is it is different if we we could have just written here minus H of, you know, Sol theta T, given the intelligent system at time step T plus one.


Right, because the intelligent system at times that people as one has had that data point at time step T and incorporated it.


But that's not that's not the same thing here.


We in in this step right here, when we say how difficult is it?


We assume that, you know, God or Vapnik himself tells us how like the optimal way to use that information.


OK, whereas this thing here, it's not a given that the intelligent system will use that information in the most optimal way.


So this is basically the difference between how difficult is it to get from the intelligent system to the solution and how difficult is it to get from the intelligent system and the data point at time T if you could make optimal use of that data point to the solution.


All right. So this this is going to be an assessment of how much experience you've had in the in the sense of had you been able to incorporate the experience properly at each time step.


Because yeah, because otherwise, you know, you you couldn't compare the experience if two systems had had the same experience in the same task, it should mean they had had the same, you know, data points in the same order or in in a simplistic sense.


All right. So this is all we need. Intelligence.


Boom. This is it. So there's a lot of stuff here.


Okay, intelligence of an intelligent system with respect to a scope. And there are two definitions right here.


One is for optimal skill at each task, and one is for threshold of skill.


Now we're going to focus on the threshold as we said, we at each task we require something like you must achieve 8000 points, and we're going to consider the shortest programs that will get to at least 8000 points.


Now, there's a bit of confusion in the notation here as this, I'm pretty sure this quantity right here, you know, should be called something different because it's you know, it's the T is here. And then there's this here, this refers to this, and this shouldn't be out of here, this should be meaning something like thresh.


I'm pretty sure this is just a name, like here, the name opt. So, um, yeah.


So in any case, the intelligence is of an intelligent system with respect to a scope of tasks. Okay. And the first thing we do is we're going to average over the tasks in the scope.


So we consider all the different tasks, and each task has a weight associated with it. This, this is the threshold and skill that we want. And this is sort of a mapping, this is a conversion rate, because this might be, you know, 9000 points at Pong.


And another task might be, you need to achieve point two, and that's really good at point two is really good. So this W for each task is simply going to map it to a like a uniform coordinate space of, of, of points of skill level of that particular task.


Okay, so we're going to average over tasks. Now you can, I guess, disregard this, this is not super, this is just scaling. We're going to average over tasks. Now in each task, we're going to consider all curriculums that get you to this threshold.


So all curriculums that get you to the threshold T for theta T for task T, which means sort of means all the possible permutations of training data sets for that task, right?


It's more general than this, but we yeah, we want to assess all the, all the different ones.


And as you can see here, there's P of C. So this is an expectation, this is the probability of that particular curriculum. This is, this is the expectation over data right here.


This is the expectation over the training data distribution, okay, in the classical machine learning sense.


So we're going to take the average across all tasks over the expectation under the training data distribution. So we're good so far. And usually right here, we would put something like the empirical risk, right?


The minimum, minimum loss, min loss function, min theta loss function over my, over my C, over my training data set, okay, but not in this case.


Because we now want to consider the priors and the experience and discount that from the difficulty. And that's what's written here.


So this is the developer aware generalization difficulty. This here is the amount of information that's already contained in the priors.


And this here is the amount of information that's contained in the experience in that curriculum.


As you can see here, the experience is in that curriculum. So basically, a system is more intelligent if the task is harder for that given system and that given curriculum, okay?


So that makes intelligence up. A system is more intelligent if it gets to a certain threshold with lower priors, okay?


If the priors are low, this whole quantity is high. And the system is also more intelligent if it gets to the threshold with less experience, okay?


So if the experience here is lower, it is counts as more intelligent. All right.


In this in this quantity, and this is written all in the text here, it has some properties in that it for example, it down values actors that in the same curriculum, like in the same training data,


they if if an actor learns faster, like it learns earlier to reach the threshold, it would assign more intelligence to that actor and so on.


It's kind of sometimes it's hidden over the it's hidden in the definitions. For example, these curricula are not all the same as the curricular specifically curricula that you need to reach this certain threshold.


So it's not always doesn't always sum up to one with this probability here. That's why it's not exactly an expectation. Let's call it an expectation in quotation marks.


But in the general sense, that's it. So insist the intelligence of a system is over a scope of tasks.


The the expectation in quotation marks under the training distribution of the generalization difficulty accounted but accounted for discount, we discount the prior knowledge of the system and the experience that the system has had. Okay.


And that's it.


It says, P plus E prior experience represents the total exposure of the system to information about the problem, including the information it starts with at the beginning of training.


Okay, so if this is high, then the system is not very intelligent or is not if a system that has more of this but generalizes to the same level as another system is considered less intelligent than the other system because it has had more exposure to information about the problem.


It makes a lot of sense, right?


So schematically, the contribution of each task is the expectation over skill times generalization divided by priors plus experience. That's kind of in words, what we looked at.


So it gives over a number of key observations. And at last, he goes over consequences, or basically a recommendation for what a benchmark should look like if we regard it in this light.


Now, of course, these complexities and so on, they're not exactly computable. Right. So it's like how much exactly the shortest the length of the shortest program is is not exactly computable, but it can inform our notion of how we should test intelligence.


Okay, so what to expect of an ideal intelligence benchmark.


First of all, it should describe its scope of application its own predictiveness with regard to the scope. So that means the validity, it should be replicable, it should be reproducible.


It should measure broad abilities and developer aware generalization. Sorry, it should.


It should set out to measure broad abilities and developer aware generalization.


Okay, so that means it should not be solely measuring skill or potential.


It should not feature in its evaluation set any tasks that are known in advance, either to the test taking system itself, or to the developers of the system.


And that, of course, refers directly to the developer aware generalization.


And it should seek to quantify the generalization difficulty it measures, or at least provide qualitative guidelines with regards to its generalization difficulty.


It should at least be made clear whether the benchmark seeks to measure local generalization, broad generalization, or extreme generalization.


So we've seen this in part one.


Taking into account generalization difficulty minimizes the possibility that a given benchmark could be hacked by solvers that take undesired shortcuts that bypass broad abilities.


It says it should control for the amount of experience leveraged by test taking systems during training.


It should not be possible to buy performance on the benchmark by sampling unlimited training data.


So this this already rules out sort of any, let's say image recognition or NLP benchmarks because there we can always just feed in more data, the more unlabeled data from the internet or even labeled data.


Like if there's a benchmark that, you know, is on computer vision, I can just pay more humans to label more data and then I will be better at that benchmark.


The benchmark should avoid tasks for which new data can be generated at will.


It should be in effect a game for which it is not possible to practice in advance of the evaluation session.


That's going to be hard, right?


It should be it should explicitly and exhaustively describe the set of priors it assumes.


Any task is going to involve priors.


But in many tasks used for evaluation today, priors stay implicit and the existence of implicit hidden priors may often give an unfair advantage to either humans or machines.


So this is, for example, if the test is like a speed test.


A lot of times machines are going to be way faster than humans because the hidden assumption in a speed test is that kind of your nerve conductivity is the same across all test takers.


And the last one, it should work for both humans and machines fairly by only assessing the same priors as possessed by humans.


And he refers to core knowledge, which we saw in the last part, and only requiring a human sized amount of practice time or training data.


So this means if we want to compare humans and machines, machines can often incorporate way more data than humans.


So the tasks in the benchmark should only like the amount of data should be such that a human could process that data.


Now, of course, that that sort of also means that any task where basically you collect data during your life is all also sort of ruled out a bit.


So that means the AI benchmark task can't be like cook a pan of spaghetti or something like this.


Yeah. And in the end, he says these recommendations for general AI evaluation wouldn't be complete without a concrete effort to implement them.


In part three, we present our initial attempt, which is going to be the ARK dataset and the ARK Kaggle challenge.


But that's a story for next time. I hope you enjoy this and at least got some bits of it.


It's very abstract. This measure of intelligence, of course, it can never be computed exactly.


But the fact that someone is trying to formalize and it's not the first time this has been trying to formalize.


But this I feel it's quite understandable and makes sort of sense. And I'm interested to see if people come up with actual approximations to this quantity that you could actually compute sort of.


Analyzing Models with TensorBoard - Deep Learning with Python, TensorFlow and Keras p.4

What's going on everybody and welcome to part 4 of the deep learning with Python,


TensorFlow and Keras tutorial series in this video and the next video what we're


doing is talking about how we can analyze and optimize our models using


TensorBoard. Now TensorBoard is a way for us to visualize basically the


training of our model over time and basically you're mostly going to be


using this to watch things like accuracy versus validation accuracy and loss


versus validation loss but there's also some more tricky tricky things that we


can do that maybe we'll cover more in time but the idea of TensorBoard is to


just kind of help you shed some light on what's going on in your model. So with


that let's go ahead and get started the first thing you might notice is I'm just


adding these two lines in here it's really not important for this tutorial


but if you're curious you can do this and this is basically so like any model


even small models are gonna try to hog a hundred percent of your VRAM so but if


you wanted to run like multiple models at the exact same time you could


specify a fraction of your GPU that you want that model to take up in this case


a third mostly I'm just doing this so when I rerun a model sometimes it takes


a minute for whatever reason especially if I'm using like sublime or something


like that it just takes a minute to clear out the VRAM so anyways and then


when I'm recording I've just got no more space left and things crash but but also


if you're trying to run multiple models like with the Python plays GTA series


and stuff like that this was super useful so we could run many models on


the same GPU at the same time like an object detection and the self driving


so on so just a little tip if you find yourself needing that anyway let's get


started so the first thing I want to go ahead and do is also a little bit of


housekeeping we need an activation function after this dense layer it was


kind of silly of me to forget that someone pointed that out and yeah that's


definitely gonna be very problematic basically without the activation


function it becomes like a linear activation function which is pretty much


useless in this case if anything is probably causing trouble like we're


definitely not trying to do any sort of regression or anything like that so


doing that right before the dense layer ouch so we don't really want to do that


so let's go ahead and save this and even with just that change we should find


that accuracy is quite a bit better now I'm gonna let that run and then I'm gonna


bop on over to the tensor flow documentation for the various Keras


callbacks so the way that we can interface with tensor board is with a


Keras callback in specifically we're going for the tensor board one but take


note of the other ones that exist here there's quite a few you can early stop


based on parameters you can change you can do like a learning rate scheduler


you can also reduce a learning rate when things plateau trying to think if


there's anything else the the model checkpoint I forget if I pointed that


one out already it's early in the morning everybody anyway model checkpoint


that's super useful so you can actually checkpoint a model like as it's training


so this is super useful like if you got a model and you don't know how many


epochs to do and you don't want to like over train the model right so you might


say okay a hundred epochs or 50 epochs and you know it's probably gonna max out


around somewhere between 10 and 20 but what you can do with the model


checkpoint is you can save based on a very like various parameters so you can


save on the best loss it's ever seen or the best validation accuracy and so on


so super useful but anyways right now what we're looking at is tensor board so


with this tensor board callback we can save based on the various parameters for


now we're just gonna focus on log dur that's it you're not gonna touch any of


these other parameters but they're pretty cool and maybe at the end of the


next video I'll briefly touch on some of these but anyways let's move this


over here and let's get started so the first thing that we want to go ahead and


do is bring in tensor board and so we're gonna go from tensor flow dot Kairos dot


callbacks we want to import import tensor board okay so we've brought that


in and I just want to point out validation accuracy 72 and then in


sample accuracy basically 79 so much better than before and again that was


only three epochs right yeah anyways okay so once we've done that the next


thing that we want to do is we're gonna be trying to train at least we're gonna


train a new model and then also anytime you're using tensor board it's just a


good idea to give your model a useful name so like in this case we might call


our model you know 2 times 64 conf netter you know something like that that


lets us know what model was that again because you're gonna train a bunch of


different models so I'm just gonna come up to the very tippity-top here and I'm


gonna say name equals and then we're just gonna call this cats verse dog CNN


64 by 2 good enough and then I'm gonna go ahead and add one more little


parameter here that format we don't have time yet but time dot time and in fact


let's go ahead and convert that to an int time dot time now the reason why I


want to do this is sometimes you'll either forget to change the name or


you'll just train that model again for whatever reason and it's useful to have


the time stamp in there because that just makes it totally unique so you


don't end up overriding a model and worse if they have the exact same name


you're not really overriding you're actually appending so it'll do this like


funky zigzag in the graph and you just you don't want that so we've got a name


I'm gonna zoom in a little bit just make things easier also let's make some more


space okay once you've got the name now we're ready to specify the actual


callback object itself so tensor bull tensor board is going to equal and it's


a tensor board objects and then we're gonna say these wherever we want that


log dirt at B I'm gonna say logs select slash and then dot format name one of


these days I really need to get into was it f strings is the latest way to format


strings I think anyway and this is a little outdated but forgive me so once


we've got that callback defined and also keep do take in mind you could customize


these callbacks or create your own callbacks and maybe one of these is the


parent class and then you're gonna modify it somewhat so you can do all


kinds of funky stuff with this so just keep that in mind once you've got it all


you need to do is pass it into your fitment and the way that we do that is


by passing in callbacks equals and this is a list so it's a list of all your


callbacks in our case we just have the one tensor board so I'm gonna paste that


in fix my styling there save and let's go ahead and run that and in fact I


think what we'll do is we'll run for ten epochs I'll go ahead and save it


hopefully we don't crash the video as this goes type error in it is it log


underscore dir probably yeah log underscore dir is the parameter there


let's try one more time must jump oh all right looks good we are training okay so


as we train you should now have a new directory right here called logs and if


we click on that we can see oh there's our our model right now so to view it as


it's going you want to open up your command window either a terminal or


command prompt rather and at least on Windows the way you do this and you can


just type CMD from there just make sure you're in that main directory right so


you're in the directory that houses logs now to get tensor board running just


type tensor board dash dash log dir law log dir equals and then logs what if


that has an underscore too yeah okay that must be why I didn't do the


underscore okay so then it says okay it's running at this address so you can


just like copy and paste that address if you want into a browser or you could


just manually type it if it's as simple as mine was it looks like mine for


whatever reason let's see logs log dir logs for some reason mine is not showing


up deep learning basics logs I'm looking at the log there wonder if I'm running


another tensor board no hmm let me break this tensor board log dir equals let's


try without the quotes I don't think that'll change it sometimes it can be


picky but I don't think that's gonna be it oh maybe that was it anyway on


Windows it can get kind of finicky on you it's weird anyway so no quotes on


Windows sorry about that for a while you had to specify like a little variable


too it's very annoying okay so there's our model all nice and done and you know


so basically what you can do first of all is let's see I think this is just


simple reg ex yeah so so what we can do is we can just do this and we can


visualize all the graphs together so let's do this so here you've got your in


sample accuracy and you've got your in sample loss and then over here you've


got your out of sample accuracy and you're out of sample loss and what


you're hoping for is that in both cases accuracy goes up loss goes down now when


we see something happening like this we can clearly see after the fifth epoch


validation loss starts creeping back up interestingly enough the validation


accuracy still improved about two points but it's questionable whether or not


this is actually better it probably is because that's out of sample it'd be


nice if we had a much much maybe a larger out of sample but eventually as


this creeps up it's this will eventually fall like if we did maybe 20 epochs now


the question you might ask is well why why after why is this happening well the


model is only getting the you know in sample training data so at some point it


goes from generalizing and really seeing patterns to memorizing all of the input


samples or many of the input samples so so what you're gonna want to do is watch


always validation loss at least for me when I'm like evaluating a model I'm


looking almost purely at validation loss and really nothing else so now what you


could do is you could begin to kind of change things up so for example we could


do one we could do a model without this dense layer at all so let's go ahead and


just remove the dense layer and this time let's do let's do 20 epochs


I don't know if I'm gonna let it finish this might take a while I don't want to


just run up a bunch oh we ran out of memory fabulous no okay so it crashed


okay let's see here hopefully that didn't kill my recording I will find out


I'm gonna be very angry if that killed me so what were we on tensor let's see I


gotta find this stupid file that we were working on so I think it didn't even


close after all that too


all right 20 epochs okay we'll try it let me try to run it one more time it's


a stupid recording at the same time also maybe in the future I'll run via the


console for some reason sublime doesn't want to like clear away even though


you've cancelled it and you're done it just doesn't want to clear it away so


that's kind of annoying okay so as that's running tensor board is still up


I have no idea what I do with it maybe it crashed I don't know where I put it


I'll just open another one I guess HPC okay so here is the next model in


training I wonder do we save that we didn't want that's looking really


similar yeah okay no no dense layer


interesting well we're actually doing worse you probably also would have


wanted to update the name to be like no dense or something like that just keep


that in mind you always want to be thinking about that also why is our


regular expression suddenly not working how about any word there or any letter


okay so here we have at 10 at the 10th epoch see we're still in samples worse


out of sample is actually slightly better in most cases which is curious


and the loss seems to be doing much much better we had we made it you know


basically the 10 epochs without without too much issue let's refresh again this


normally will do it every like 60 seconds or 90 or something like that


okay we're done okay so we can see here the validation accuracy kind of starts


to plateau which makes sense because after the eighth epoch in this case it


starts creeping up we can also see jet in general loss was kind of worse than


in sample but out of sample loss was better so again personally I would say


without the dense layer is clearly victorious because I'm more interested


in what's going on with the out of sample than I am in the in sample because


it's just too easy to cheat on the in sample like you can always like it's


almost like the better the in sample looks to the out of sample like that's


clearly the worst the model is like that that means over fitment is occurring so


this is what I tend to look for just just for some other quick notes like if


you just wanted to look at one you could check or uncheck or you can also like


just click this like if you had like 50 you could click just one quickly you can


toggle all runs down here this is the smoothing so as you can see there's like


these like subtle lines out here no clue what what just happened there I think


are something's wrong actually oh yeah I don't the graph definitely broke for a


minute anyway you can change the smoothing so this would be like no


smoothing so it's very raw and you can see how jaggedy everything is and then


or you can add just like a ton of smoothing to really smooth things out


I'm trying to think if there's anything else then if you had like a bunch like


like for example if you only wanted to find your 64 by 2 models you could type


that in but in this case we only have that many but let's say you want the run


at 344 it should just find you that that one specific for example so you can do


stuff like that to you know filter through a bunch of a bunch of models


okay I think that's enough for now in the next tutorial we're going to talk


about optimizing a model because we could go back and forth and manually do


things then rerun things and do things again but it's gonna be much more simple


to to just kind of automate that entire process as well so that's what we're


gonna be talking about in the next tutorial if you got questions comments


concerns whatever feel free to believe them below also we have a discord it's


just discord.gg slash centex come in there hang out with us chat with us ask


[ML News] EU regulates AI, China trains 1.75T model, Google's oopsie, Everybody cheers for fraud.

The European Union seeks to regulate AI. Chinese researchers train a model 10 times as large as GPT3,


Google makes an oopsie and Jacob Buckman appeals to the community to please commit


more academic fraud. This and much more in today's ML news. Have fun.


So lawfare rights, the European Union unveils its proposals for the artificial intelligence


act seeking to regulate AI and harmful uses thereof. So what does this actually mean? First


of all, how do they even define AI? They say artificial intelligence systems means software


that is developed with one or more of the techniques and approaches listed in Annex


one and can for a given set of human defined objectives generate outputs such as content,


predictions, recommendations or decisions influencing the environments they interact with.


In Annex one, these things are described as either machine learning approaches,


logic and knowledge based approaches, or statistical approaches. So in essence,


I think there is an easier name for all of this under one hat. It's called software.


If you think that's a bit far reaching, don't be worried. European Union divides different AI


applications into different categories of risk, ranging from minimal risk to unacceptable risk,


and prescribes different things you'll have to do if your application falls into any of those


sections. For example, if you're in the high risk category, you have to do a conformity


assessment, which either you can do yourself, or you'll have to submit to some sort of regulatory


body. Now rest assured that these regulatory bodies are of course not going to be staffed by


lobbyists from the exact corporations that are going to apply for exceptions to the rules right


here. If you're in the unacceptable risk category, which includes things like facial recognition and


social scoring, you are prohibited from performing these things. Of course, there are going to be


exceptions as well for things like law enforcement and so on. Safe to say in its quest to regulate


everything under the sun, and if they could the sun itself, the European Union's regulations have


always only brought benefits to humanity. I mean, aren't we all just so much better informed about


how our data is used now that every single website has a yes, I accept the cookies banner


that certainly helps your helping European Union. Thank you very much. So for now,


this is a proposal but safe to say the European Union will probably go forward with regulating AI


in some capacity. In an article in ScienceMag, Derek Lowy writes machine learning deserves


better than this common pitfalls and recommendations for using machine learning to detect and


prognosticate for COVID-19 using chest radiographs and CT scans in which the authors identify over 2000


studies of which they finally select 62. And say a review finds that none of the models identified


are of potential clinical use due to methodological flaws and or underlying biases. Derek Lowy


elaborates on this and goes on a very good rant against how machine learning practice is not


living up to the scientific standards of the fields where it is applied to and very often


it's just used to get some papers published without actually bringing benefit to the field.


In one example, he says one commonly used pneumonia data set turns out to be a pediatric


collection of patients between one and five. So comparing that to adults with Coronavirus infections


is problematic to say the least, you're far more likely to train the model to recognize children


versus adults. In general, the studies fail in doing things like revealing key details about


the training and experimental sets, not performing robustness or sensitivity analyses, not performing


external validation work, not showing any confidence intervals, and many more. And being


in the machine learning field, obviously, this is the case. So if you are looking to apply machine


learning to any fields that's not core machine learning, please get familiar with the common


practices in that field to generate valid scientific contribution. Though we all know that


valid scientific contributions probably isn't the main motivation of most people doing these kinds of


things. I love this comment by Derek Jones who says you have completely misunderstood the purpose


of machine learning in academia. Machine learning provides a means for people who don't know anything


about a subject to publish papers in the field. All that's needed is some data, some button pressing,


and the ability to convincingly sprout technobabble and getting lucky with reviewers couldn't agree


more. Next news, Ping West writes that a Chinese AI lab challenges Google and OpenAI with a model


of 1.75 trillion parameters, which is 10 times the size of OpenAI GPT-3 model. Now we don't know too


much about this model, it is apparently trained with Pytorch and uses a fast mixture of experts


architecture which allowed Wudao to be trained on both supercomputers and regular GPUs with


significantly more parameters. The mixture of experts architecture generally is more of a


sparse architecture akin to Google's switch transformers. So directly comparing the model


size to GPT-3 is not exactly valid. But this model called Wudao is a multi modal model and


its individual parts can do things like caption generation, generating poetry and even generating


images from a description. And in all of these things, they appear to outperform the current


models that Google and OpenAI have right now. All this comes out of the Beijing Academy of


Artificial Intelligence. And the researchers not only seek to build models for language and images,


they say we are also building Tian Dao as a model for physics and Tian Yan as the model for life


sciences, adding that the end game plan is to fuse all of them together, making AI not only work


inside computers, but also across the universe. Not sure what that means, but sounds exciting.


Of course, we were already impressed when a team earlier this year out of Huawei released Pungu


Alpha, which was slightly bigger than GPT-3. But this here is of course another level and


we're excited to see what comes out of scaling models larger and larger. All right, next, the


BBC writes, Google apologizes for ugliest Indian language search results. So there's this image


going around tweet by PC Moan googling ugliest language in India, the Google question answering


system triggers and replies with apparently a language that exists there. Now, not so long ago,


all of us understood that Google is a search engine and gives you things that it finds on the web and


that this here might just be a slight but humorous failure of technology, we would all sort of have


a laugh about that, whether you spoke this language or not. But apparently in today's time, it is very


fashionable to absolutely freak out when something like this happens and point out how valuable this


language is that it has a long tradition and that is so harmful to the people who speak this language.


And you just kind of have to ask yourself, what's up? Are people actually upset about this? Or are


people just pretending to be upset about this and working themselves up because they can get some


internet power from this. So I happen to have right here. Now actually, I happen to have here a


bucket. And this pocket actually contains all the damage that was done by this search result. So if


Oh, it's empty. Oh, so I mean, come on, what is this upset culture? I mean, even if this has upset


someone, the ability of Google to quickly serve you this kind of information is, you know, pretty


good. We recognize that, you know, sometimes it picks up something from the internet. And we all


understand that this is not an authoritative answer. Don't pretend that this is somehow a


source of truth. Alright, let's try this out. Best machine learning framework. Apache Spark. Oh, wow,


I didn't know. Well, my mind just changed. craziest machine learning researcher.


Jeff Hinton. Oh, who knew most hand some deep learning. Learning researcher.


Carpotty. Now, of course, I'm not saying we should not criticize Google for doing things like this,


Google has apologized and fixed it. But I do think there is a giant overreaction to these things and


blowing out of proportion about actually how important this is. And also a real overstatement


of how many people are actually affected by this except for getting outraged on the internet.


Next news, ZDNet writes McDonald's wants to democratize machine learning for all users across


its operations. By users, they mean internal teams, so don't get confused. And by democratize,


they apparently mean just apply. So in the quotes from the McDonald's execs, you'll find things like


we want to enable more end to end automation and machine learning operations in general. And we


want to continue to implement governance, and also cost control measures in order to make sure that


we're doing from the business perspective continues to make sense. And also the way we do is, is we


bring all the data into an S3 bucket, where data lake is enabled, which helps us to do data versioning


and also build scalable and performance feature engineering pipelines in the platform. And further,


we've not only identified the tools, the technology, we've done the legal paperwork,


which can always be a hassle, but also identified use cases, built the models and deployed them.


What are you doing? This is zero information. How can people say so much without saying anything at


all in terms of content? In the last paragraph, you'll actually find McDonald's will include


carrying out very fine grained sq level forecasting for its restaurants, automated marketing and


personalization related activities beyond what he refers to as good machine learning for marketing.


So they want to predict your behavior, I want to sell you more stuff, I want to use machine


learning to give you diabetes faster. Why can't you just say this at the beginning? In any case,


I wasn't aware that McDonald's was deep into machine learning. But obviously,


it makes sense, you know, good for them. Next up analytics insight, right AI is helping you


make profits by predicting cryptocurrency prices, all the buzzwords in one thing artificial


intelligence cryptocurrency latest news. Now the article is pretty short. But if I may brag for


just a bit on our discord, you'll find a link in the description. We have had forever a community


project channel called stock market prediction. I highly recommend you check that out because we've


been doing that stuff for ages. Alright, if you've seen my AI generated music video or are in the


space of generating images using the clip model, you love this trick. Aron Komatsuzaki writes that


there is a simple hack. If you just add unreal engine to your text prompt, these systems tend


to generate much higher quality images. For example, here it looks really cool. So try it out


or look at this thread. There are many more examples right here. In general, I love how


prompt engineering is really becoming something that people pay attention to. I think there's a


lot of potential that is as of yet on top. And in our last news, people are paying a lot of


attention to Jacob Buckman's article, please commit more blatant academic fraud. Now of course,


this is a bit of a sarcastic take on the recent news about collusion rings in ML,


which we've covered in last week's ML news. Now I have to say since last week, I've had my ears a


bit more open to these kinds of things. And I can promise you this happens much more often than you


think. Now the point of this article claiming please commit more blatant academic fraud is to


contrast it with the low level not so blatant academic fraud that the community is already doing


day to day such as cherry picking examples or not doing certain ablations because you'll know they


won't turn out well and all the things we generally do to get our papers accepted. He considers this


sort of a low key fraud indistinguishable from simple mistakes. And that's the reason we usually


let it slip. And of course, this whole procedure of being sort of a little bit dishonest in your


papers then gets into the broader culture and intensifies as more people need to publish papers


in the same conferences. He says worst of all because everybody is complicit in this subtle


fraud, nobody's willing to acknowledge its existence. Who would be such a hypocrite as to


condemn in others behavior that they can clearly see in themselves. And with large respect, he


actually does he calls out his own papers and claims that they are bull. And I have to say,


I can claim the same thing about my own papers for the most part. And it's often the case that


in a paper, you actually have a scientific contribution, there is something that may work


in certain situations. But in order to get it published, you have to present it in such a way


that is just absolutely unrealistic in how good it is and how absolutely zero criticisms against it


you can have and that it works in all situations at all times. So the author finishes with the call


to please commit more academic fraud because he argues that because the fraud is so blatant that


we can't ignore it. This is the only chance of the community to actually do something against


the widespread low key fraud. So once we pay attention to scientific malpractices, we have


a chance to weed it out and get to a better place. So I think this is not going to happen. I think


people will continue as is this is going on, as I said, more than you think the credibility of the


whole field will just slowly fade away because more than half of all papers published at conferences


have absolutely zero effect and zero scientific credibility. The author here points out that


readers of a paper have to become much more like reviewers questioning the paper analyzing it from


a critical perspective instead of simply taking for granted that if it was published in a peer


reviewed scientific conference, we can sort of get this as a seal of approval. And I fully agree.


In fact, I think we should abolish the peer review at the conference or at least make it


transparent. Absolutely surprised when people always call for more anonymity, more politics,


more in transparency in this process. Why not make everything open? Why not have everyone as a


collective decide on what's valuable and what's not? If you're worried that the big names will


get all the credit they already do. So I highly invite you to check out the article right here.


It's written in a fun way and it makes very good points. All right, this was it for this week's ML


news and no, this is not a weekly thing. This is not a regular thing. Stop telling me that this


stop telling me that this can be a regular thing. But I appreciate all the feedback we've got last


week. Thanks to all the viewers. I hope this helps tell me if you would like to see more of