millions of people would be happy and things would be good for a while because millions of people would have more money I know but a true redistribution of wealth has never led to more social justice in the long term not in any of the political systems that tried it it led to the impoverishment of these countries the landowner foot Christiana fund bechtolsheim sees higher taxes on the rich as dangerous as a philosopher I don't think much about this so-called rich tax for two reasons firstly if the terminology alone is stigmatizing and we


in Germany should avoid that and secondly the rich tax wouldn't do any good on the contrary it would cut into the backbone of the German economy because the typical German rich person is a media sighs business owner they make up the backbone of the German economy and if we want to destroy that we have no one to blame but ourselves the special issue is ready things have basically stayed the same the rich have a few billion more the richest 1% of Germans now has personal wealth worth a quarter of the country's

Germany: The discreet lives of the super rich | DW Documentary


hi there take a look at the following problem on the left right here so you have this quadruped and the goal is to have it walk forward or in any direction as far as possible now usually this is the domain of sort of reinforcement learning so you have inputs which is the sensors of the joints of the quadruped and you have outputs which is how much force you want to put on each of the legs and you have to somehow learn a policy to make it walk forward reinforcement learning does that by sort of trial and error using an

Meta-Learning through Hebbian Plasticity in Random Networks (Paper Explained)


environment to learn the policy directly however this paper does something different what it does is it learns a policy that is adaptive during training which basically means that at the beginning of each episode the policy in it is initialized randomly and by policy here we mean a policy network uh policy neural network which you can see at the bottom so that's initialized randomly and then during the episode depending on the input uh this network is changed and adapted in order to achieve high performance so even at test time


the network is started randomly and then adapted during the episode so this paper deals with this problem and tries to implement this sort of more biologically plausible way of learning a policy adapting to the environment and achieve ultimately good performance in this task and it has some nice property namely that it can deal with these things as you can see here front right leg damage front left leg damage but we'll get to that later but just so you know what's coming so the paper is called meta learning through hebbian plasticity


in random networks by elias najaro and sebastian rizi so we'll go through the paper what it does what evolutionary methods are really briefly which they use what hebbian plasticity is and the difference to classic reinforcement learning and then we'll look at the experiments and that's going to be it if you like content like this as always don't hesitate to subscribe and share it out and tell me what you think in the comments i still read all the comments so i am very interested in what you think about works like this and about the


video itself okay so they say lifelong learning and adaptability are two defining aspects of biological agents modern reinforcement learning approaches have shown significant progress in solving complex tasks however once training is concluded the found solutions are typically static and incapable of adapting to new information or perturbations so they contrast the two things here reinforcement learning as you know is very powerful in these domains but its goal is to learn a policy and then that policy is fixed and


it's specific to that particular problem however biological agents you know humans uh animals and so on they're able to adapt usually very very quickly they give some sort of examples right here like if a if an animal is born it almost immediately knows how to walk um so even if it has some sort of injury even if it has some sort of disability um usually the animal can walk uh pretty much instantly and that means it sort of adapts to the body that it is in sort of reconfigures itself on the fly and that's what we're going


to explore here so this isn't going to out compete uh rl anytime soon it's just a different way and a biologically more plausible way in order to do that so again they say we still don't know completely how biological brains learn and adapt so efficiently from experience it is believed that synaptic plasticity plays a prominent role in this process and that's why they are using these hebien learning rules in order to configure the network so let's contrast the two things for a second in reinforcement learning what you have is


a policy network now the policy network is a neural network that maps sensory inputs to actions okay so you have the observation goes in and out comes in action this is your policy network now during training in reinforcement learning what you do is you have some sort of environment okay this is the environment and you play this back and forth game with the environment and you try to improve this policy network right here as best as you can in order to achieve a high reward then during testing so this is train


then during testing you freeze you freeze this network right here so you freeze the network and then you simply play that game and you see how well it does okay so this gives you some sort of reward and that's going to be your testing reward and you know that can be generalization it can be to different environments and so on but the crucial part is that you in train you learn and then you freeze during test in this in this particular paper right here they do something different so let's call that the hebbian plasticity world


in the hebbian plasticity world again you have your environment and you play this game but you play the game in episodes and at the beginning of each episode you initialize this using some sort of distribution here a normal distribution you initialize the network and then you learn you adapt during the episode you adapt the network to have good performance okay so this thing right here these are the hebian rules so you update the network during the episode and then at the end of the episode you go back you initialize the network


again you start a new episode and you again adapt that randomly initialized network so what's actually learned here isn't the weight of the network what's learned during training is these rules that transform any randomly initialized network into a high performing network now of course you you might just object and say hey wait a minute i can just basically hard code the you know the optimal weights here into these hebian rules like my rules can simply you know not care about the input and simply output whatever good weights


there are and ultimately that would lead back to rl but as you will be able to see in the experiments they also have some videos provided that i invite you to watch you can really see that the network reconfigures itself first of all at the beginning it reconfigures itself to a good state but then also as the episode is progressing it continuously reconfigures itself depending on the input so this is the real power of these hebbian rules in that during the episode the network can continuously reconfigure itself


in order to achieve high rewards so it's not just that i can go from the random initialization to a good performing policy i can adapt that policy depending on what the input is so at test time in this habit world what we're going to do is again we are going to freeze the learning rules so you have to kind of rethink we're going to freeze the hebian rules but still we're going to randomly initialize our policy in each episode and then we're going to change that during the episode okay and then that's ultimately going to


give us our reward so that the thing that's learned is just something different here you learn the weights directly in the rl setting and in the heavy and plasticity setting you learn the rules to update the weights dynamically depending on the input this is a form of meta learning right it's not exactly but it is a form of meta learning so let's see what those hebbian rules are and you can as again you can see this right here during training so this is one episode and it always starts with these random networks at the beginning


and then you can see as you progress there is structure emerging and again i'll link to the videos and you can see that during the episode even this is changing and this is especially visible on their other example that they have here like this this car example so in this car example during the video you'll see that now there's a curve like this and then as imagine you're a driver like there is a kind of a left curve coming and you adjust your mental state let's say to say okay i don't know what's around the


curve i need to be ready to break and so on and then there is a straight piece coming and you'll be like well i i see everything you know i can focus on different things you cannot reconfigure your state in order to adapt to the the observation and that's exactly what you'll see in that video is that the weights are continuously updating not so much in these quarter pads to which we'll get later so these habian rules what do they look like these are biologically inspired rules and they say the following so this here is the delta w i j


and our perspective of policy networks is going to be that this is a neural network as we said and we'll just pick up one layer right here and there is going to be weights right here you know weights from all to all these are going to be fully connected networks and like this and there's going to be neuron i somewhere here and neuron j somewhere here okay so neuron i and neuron j are going to have a connection together this thing right here and there's going this the question is going to be how do we update that weight from one time step to


the next remembering the weights here are changed in each time step each time step during the episode we update the weights so how are they going to be updated let's contrast this first to classic reinforcement learning so in classic reinforcement learning we would keep these weights the same during the entire episode and then at the end of the episode right we keep those the same and at the end of the episode we'll get a reward and then we'll go back we'll look back and say how do we need to change the weights


such that in the next episode the reward will be higher and in again in classic reinforcement learning for example in policy gradient methods you will actually calculate a gradient with respect to these weights right here actually let's let's go into that later when we contrast evolutionary methods so the important part right here is that we change the weights in each time step so how do we change the weights of course we don't have access to the reward right in order to change the weights the reward is going to come into play


when we change the rules to change the weights but during the episode we don't have the reward at least we assume we only get kind of the reward at the end so we need a different uh method and the method is going to be the following right here the important things in this formula are going to be so how do we change the weights that's dependent on two quantities that appear during each time step o i and oj and these are going to be the outputs of neuron i and neuron j so how do we change the connection that's going to be dependent


on the output of neuron i which is here called the presynaptic output and the output of neuron j which is going to be the post synaptic output the rule the kind of mantra here is the fire together wire together means that if two neurons are active at the same time regularly then they probably should be connected together because they already correlate and you can see right here that there is a term in this formula that is o i times o j so this here is the correlation between or the covariance um or just the product


if if we're exact between these two neurons and if they are both active regularly then this quantity is going to be high and if they're both not active regularly that or if one is active and the other one isn't that quantity is going to be low and the a parameter here specifies how the weights are updated in response to this so the a b c d and eta parameters right here are these are the learned parameters these are going to be your learned rules to update the weights so these change once after once per learning step so


once per so after the episode is done you're going to change these capital constants right here including the eta which is the learning rate these things right here these are per step so this is each step gives you a different oi and oj and then you'll adjust the weight based on that you'll see that these constants here they are per weight so for each weight in this neural network we learn a separate rule of how to update that particular weight so the algorithm can it can basically decide for a particular weight you can


decide well if these two things fire together often i want to update my weight very heavily in response to that okay so if the a is very high that means the connection responds very thoroughly to when the two neurons fire together that is not the same as to say that connection should always be very strong it's dependent on the input so only when this quantity is high should the network or should the weight be updated and the a parameter modulates how well it's updated or how um how how strongly it's updated it can


also be negative it can be zero basically meaning that you know it doesn't matter if they fire together i don't want to update the weight this particular weight in response to that so you can see that you can learn these rules that can adapt to different inputs because all of the changes the delta here is dependent on the inputs so on the correlation but also on the different inputs themselves and then there is also a constant right here okay this it's as you can see it's a linear function of the inputs of the oi and oj


and their product so i hope this is clear that the these have been these habian rules you learn abcd and eta and that gives rise to an adaptive network that can change and reconfigure itself over the course of an episode depending on the inputs and one of the things right here and we'll get to how you actually learn the rules itself in a second but one of the things right here is very visible as i said in this first experiment where it reconfigures itself continuously but also in this experiment with this quadruped


right here so this quarter pad usually it's you know you simply walk in a direction that's your reward and rl is perfectly fine at this as well however this is a bit of a has a bit of a trick to it namely you are always in one of three situations either you have an undamaged quarter pad or its kind of left leg front left leg is damaged or its front right leg is damaged okay and you don't tell the you simply sample these situations uh uniformly and you don't tell the algorithm which situation it is in now if you look


at if you compare two methods one where you directly learn the weights you learn a fixed policy to solve you know this is one task right this is one task and all of these three things appear with equal probability so you have to learn one policy to make all of this work if you learn the weights directly and um you don't have a power like there's no doubt that like a powerful rl approach could deal with this task but if in this case if you just put a standard weight learner with this same number of the same size of policy as the hebian


they compare to if you put a weight learner on it it will not be able to solve this task satisfactorily what it will do is it will say well i need one set of rules that make me walk as far as possible as often as possible so if you can see at the table i'm already showing you the results right here the table right here if you have these static weights you can see that it's performing pretty well in two out of three situations right so it what it basically does it says okay um here is what where there's damage


what it does is it says i'm going to learn to walk with my left leg using my left front leg that means when i have no damage or damage to the right front leg i'm just fine and i'm just going to take the hit basically where i have damage to the left front leg because i'm it's just going to suck so they solved they solve this like walk more than 100 steps so that doesn't it since it can only learn a fixed policy it um basically discards the case where there's damage to the left front leg it takes that hit in order to be better in the other two


methods you can see it's outperforming the hebian rule in the other two methods but this shows you kind of the the difference and the power that these hebian rules or these generally neural neuroplasticity might have because the having one is perfectly capable of at least in part adapting to the different situations now you can see that is not symmetric also the hebian rules they learn to you know there's 860 and there's 440 of a thing that should actually be symmetric we do expect a drop when there's damage but


um it's not symmetric which means that also the hebian rules they kind of randomly focus on one over the other but at least they're able in some degree to adapt to both and that's because it depending on the input you know it has a rule in there that basically says well if the if the back left leg and the front right leg you know if they fire together then i want to um if they if they fire together the sensors that show me that they're moving if they fire together i'm going to wire them together because that's how i walk you know front


right back left and then the other way around and if that's not the case i'm not going to wire them together so that would be the situation where you have damage instead if they are not wired together i'm going to and can do this in the next layer of the neural network wire these other two things together you know if if the first thing is not the case i'm going to wire these other two things together to make up for that loss and there you can see there is kind of this logic built into the network now again i know you can do


this with learning a fixed policy you can achieve the same effects the point here is just to show that um given kind of the same size networks and so on there you that there might be there might be like a qualitative difference in certain situations again by no means this is meant to out compete rl or anything like this okay so we'll we went there now how are these rules actually learned and there we have to again make a distinction that is completely separate from the hebbian non-habian way okay so the heavy and non-habian


distinction was do we learn the weights of the policy network directly or do we learn the rules to update the weights now the question is whatever we learn how do we learn it and again we have to draw the distinction this time between i'm going to say classic or even though the terminology is not really correct classic rl and evolutionary methods okay so in classic rl what i would do is i would use my weights in order to obtain a reward and then i would update my weights so my delta w would be proportional to the gradient of w of the reward


okay so in the classic rl especially this is a policy gradient method right now so i use my policy my weights to get the reward and then i would calculate a gradient and you know usually the reward isn't differentiable so you have this uh reinforced trick in order to pull the reward out and you you can read all of this up if you look at policy gradient uh the basic policy gradient methods but this here tells me i need a gradient usually this is going to be the reward times the gradient of my fw of my input so


what this means is what this means is that if my reward is high then i i just want to know what do i need to do to make more of what i just did okay and the gradient ensures that for every single weight in your neural network you know what to do so the gradient means that i have an exact handle on how do i need to change this weight how do i need to change that weight how do i need to change this weight in order if the reward is high and because of this multiplication here i want to make more of what i just did


and the gradient tells me how if the reward is low on the other hand i want to make less of what i just did but also the gradient tells me how that can be achieved i simply go into the other direction than i would if the reward is high in evolutionary methods we don't have we don't do this gradient calculation okay now there can be advantages to not doing radian calculation sometimes back propagation simply isn't possible even if it is possible and this is maybe the case where we are now what we need to learn in our case is


these rules to update the rules and imagine you have an episode and that's kind of episode so you have step step step step and in each step these rules are applied right in each of these steps the rules are applied and at the end you get a reward so what you need to do is to back propagate that reward through all the steps and then through all the rules okay and that might be just computationally not feasible or the rules the rules right here are pretty um pretty easy but the rules might not be differentiable


you actually have the same problem in general in classic rl as well but you know you can cut off time steps and so on there are various hacks in any case there can be advantages to not having that gradient and evolutionary methods are a way to do that in evolutionary method usually you are don't train one agent you train a population of agents so you have a bunch of these uh neural network agents in here and the way you update the neural network agent is you simply let them run you know you let them run your app


the episode so this is your w one of them you let them run the episode they get a reward and then you can do multiple things so this depends on the evolutionary method so you can either pick out the best performing agent or you can update each agent according to some rule the goal here is simply to basically you always want to take your weights you want to add some noise to them and you want to see does it get better or worse if it gets better good if it gets worse not good okay the difference is without the


gradient you don't have a handle on how do you need to change each individual weight all you can do is basically random walk and observe what happens and if the random walk is you know turns out to be good you go more into that direction of that random walk so it's sort of a sort of a poor poor man's gradient method in these evolutionary methods again completely independent of what we learn you can use the evolution evolutionary method to learn the fixed weights and that's what actually what happens in the table i've shown you


uh below or you can use the other evolutionary method to learn the hebbian update rules as well you can use rl to learn the fixed weight or the update rules in this paper they use evolutionary methods to learn the hebian update rules and they compare mostly with using evolutionary methods to learn the fixed weights okay the exact evolutionary step they use right here is the following so ht here is going to be the thing that you learn you know as compared to w being the network weights h is going to be the hebian weights since


we learn the hebian weights so how they'll update um each agent is going to be they'll take the hebian weights and this this here is how you update right this is your delta h how do you update the heavy and weights well what you do is you you perform n random perturbations so i take my weights and i add noise i just add noise okay so i i'm here and i just make a bunch of versions of it and then i observe how well are these versions doing so how well are my random perturbations doing this is going to be the fitness


fi right here is going to be the fitness and then i'm just going to perform a weighted average so this is my weighted average of these new solutions okay so if this solution here did pretty well and this solution did pretty poorly i want to walk you know in this direction and then again i do the same thing here from here i do a bunch of perturbations and maybe this one did pretty well and this one did pretty poorly i want to walk in this direction and so on okay so that's how you you'll change the um you'll change weights or rules or


whatever you want in an evolutionary method it's you know it's pretty easy it's easier than reinforcement learning no back prop no nothing basically black box optimizer there are more complicated evolutionary methods but no we don't go into those here right now okay so again i've already shown you these results now i said these static weights are also with evolutionary method they also report what you would get with like a rl approach like ppo you would get kind of the same thing as they get um as they get here so oh sorry


this is not the same as the table yeah i was confused for for a second this here is for the car environment okay this is this vision based environment so with their method they get like an 870 rewards with the hebian based approach with the static weight but still evolutionary method they get a much lower reward in fact the hebbian based approach is about the same as you get here with an rl algorithm and as we said url algorithm more complicated and if you use like a if you use like a state-of-the-art rl algorithm not just


ppo you get a bit of a better performance but not that much if you look at if you look at the actual numbers so you know pretty cool to see that again this is not outperforming anything this is simply showing that um you can do that they do a number of experiments where they go in the episode and they kind of change stuff in the episode and one cool thing here is that they go and you know this is an episode so at the episode you start with a random network each time in this hebian setting and then pretty quickly the


rules adapt for a high performing right so it it starts to walk it reconfigures itself and starts to walk the reward here again it doesn't have access to that but we can measure it of course and then at this step a right here they simply go to the weights and zero them out so they just delete these weights right here and only 10 time steps later it has reconfigured itself as you can see right here in order to walk again so 10 time steps later reconfigures itself reconfigures itself and after a short while right here


it's back to its kind of original performance as you can see so that's i i'd say that's fairly um fairly impressive uh in this very short amount of time able to recover from such and such an intervention if you do this i mean of course if you do this to your policy network that's statically learned it's going to be garbage but i guess the fair comparison would be to delete the habian rules themselves and you know so it's not like it's not like this can adapt to new situations or something like this this is still


learned for particular environments right but the the point here is that you learn the rules and this is kind of a study on neuroplasticity now my question actually would be why this diagonal pattern appears and i have not seen a like a clear explanation um especially is this anti-diagonal pattern it's not so much here in the output layer right this is the output layer there are what 21 actions or so and this one is this this dimension um so not that much there but there seems to be this rule and this is not


the case at the beginning right you saw the beginning you saw at the beginning it was pretty random matrix so why why yeah here pretty random and then there's this diagonal pattern i don't know why if you know let me know i mean it's anti-diagonal maybe it it is actually diagonal and the forward the fully connected layer is just defined as something like wt times x and um but maybe this also depends on the random initialization but there is no inherent way why a particular neuron would you know care about sending


information to like the same height of neuron on the other side or is there i don't know i'm so is this a property of the evolutionary or of the learning rules it seems not because the learning rules don't depend on the position i'm genuinely confused about this and maybe you know maybe they've written it somewhere and i've just overlooked it though i they they do reference it they say oh there's this diagonal pattern appearing but i don't think they ever say why it is diagonal um okay i might just be i might just be


real dumb yeah so they also you know they do some more experiments they show for example that if you just have random hebbian coefficients then your algorithm just jumps around kind of um in in weight space around the zero point however if you actually learn these having coefficients as they do you have like this clear attractor here and you have these kind of oscillating curves uh when you know when when you do that and you can see here in the different situations where things are damaged and so on so all in all i think it's a pretty


interesting study and i think this neuroplasticity is it's a different way you know it's unclear to say if it will ever deliver the the performance that rl delivers but certainly there are situations where such plasticity is desired and if we can also combine this with greater generalization performance then you know we have agents that can quickly kind of reconfigure and a lot of work by these this kind of open-ended learning community also plays into these roles all in all pretty pretty cool uh non-standard way of doing things last


thing the broader impact statement uh every now and then we'll look at a broader impact statement since these are new just to get kind of an overview of what they look like so they say the ethics computer societal consequence of this book are hard to predict but likely similar to other work dealing with more adaptive agents and robots in particular by giving the robot stability to still function when injured could make it easier from them being deployed in areas that have both a positive and negative impact on society


okay well again this it's it's not really giving robots the ability to still function when they're injured i first i thought first i thought okay they train it when it's fully functioning but then they damage it during test time but as i understand it as i understand the paper they already train it with the damaged versions they just don't tell the algorithm in which version it is right now so um it's not the same as being able to work when injured unless you've specifically trained for it in this case again i could be wrong


about this yeah in the very long term robots that can adapt could help in industrial automation or help to care for the elderly on the other hand more adaptive robots could also be more easily used for military applications the approach presented is papers far from being deployed in these areas but is important to discuss its potential long-term consequences early on now okay so let's evaluate the broader impact statement let's well the first check to do is always to simply replace um whatever their method is with the word technology


okay so let's do that in the very long term technology could help in industrial automation or help to care for the elderly check on the other hand technology could also be more easily used for military application check the technology is far from being deployed in these areas okay i guess some technology isn't but advanced technology yeah so again the rule for broader impact statements seem to be you take whatever your method is and you go up until uh you find you know you're basically at technology or something equivalent uh


because no one actually i've never seen a broader impact statement that writes about the actual thing in the paper they always go up like one layer or two and then it basically regresses to technology even even though very few papers actually would be able to discuss their particular thing but you know um and that and then in terms of guidelines on broader impact statement this one is missing there's there's always this um the holy trifecta so the holy trifecta is you go like a you know like you're a you're a catholic uh you go with your


finger to your head chest left and right and you say technology good technology bad technology biased okay so if you want to write a broader impact statement go up the layers technology good bad bias and we're missing the bias here so that's you know i'm just following what these guidelines two broader impact statements are i don't make the rules i'm sorry the the hebbians make the rules apparently um i'm not having okay i've i hope you've enjoyed this paper and this video let me know what you think check out the


hi there today we'll look at big bird transformers for longer sequences by manil zaire and guru garuganesh at al of google research so this paper on a high level proposes to replace the quadratic attention mechanism in transformers by a mix of random attention windowed attention and selective global attention therefore achieving a complexity of linear memory requirement instead of quadratic memory requirement and as a result of that they can process longer sequences than traditional transformers like bert and achieve better results in some nlp

Big Bird: Transformers for Longer Sequences (Paper Explained)


tasks and they also evaluate on genomics tasks so we'll go through this paper a bit look a bit at the proof because they give a theoretical kind of guarantee that their random attention mechanism can still be touring complete and can still achieve the same things as a full attention mechanism but we'll also look at the drawbacks i sort of have mixed feelings about this paper and i think i'll voice my concerns as we go through here but first let's look at the paper let's look at the architecture and i think this is actually a pretty


cool paper for the empirical progression of the field to process longer sequences with transformers as always if you like content like this uh feel free to share it around uh leave a like and tell me in the comments what you think about the paper and about what i think whatever you just just uh go nuts all right so the basic uh the basic premise right here is that the transformers they've been pretty impactful especially in nlp so they say transformer based models such as bert have been one of the most successful


deep learning models for nlp unfortunately one of their core limitations is the quadratic dependency mainly in terms of memory on the sequence length due to their full attention mechanism so really briefly the full attention mechanism and i've done you know numerous videos about attention mechanism bert attention is all you need and so on so if you want a detailed explanation of what that is just go look up the corresponding videos but briefly what you'll have in nlp is a set of tokens a sequence of tokens as an input and you


want to transform them layer after layer into sort of a a higher order representation of that same sequence and for that you build these layers out of nodes and you have as many nodes usually as you have tokens in the sequence and the next set of so each token is represented by a vector at the beginning and each layer transforms this sequence as i said into sort of a higher level representation so you want the vector of this token right here um to be a better representation than the vector was right here and you do that by incorporating


information from all the other tokens into that particular vector now as i said this is called an attention mechanism and we don't actually have to go into how it works right here but you can see pretty clearly that if you want to do this for every token you need to have information routed from every token to every token like from here to here from here to here and so on and this is just one token and then you need to do it for this token and for this token and for this token so what you'll ultimately get if n is


your sequence length you'll get some n squared amount of computation and memory requirements for this so this is a problem and usually this means that you know this sequence length in bert this is limited to something like 512 tokens which is okay for some applications but if you want to summarize you know entire articles entire books even or do question answering with lots of context it's not really enough so people have been thinking about how to scale this input how to scale this and of course the main culprit is


this quadratic tension mechanism because if you you know double the 512 you need you know four times the amount of compute and memory so how does this paper go about reducing that quadratic dependency the goal right here is of course to get this to some o of n right because then as we double the input length we simply need to double the compute requirements and that would be fantastic and that's what this paper does and it does so without you know sacrificing the properties of the transformer so here's the architecture that big bird proposes


by the way big bird another character from sesame street i guess will continue the naming here after elmo and bert you know i i'm i'm waiting for the model that's the count um yeah that's going to be a fun model but so big bird basically has three different types of attention and here these are adjacency matrices in this attention mechanism so here is the input layer and the output layer is right here so that basically means that node i right here would be connected well sorry that's not a straight line would be


connected to this particular node and also to this particular node so we're now trying if we have node i right here we're now trying to not connect it to all of these nodes but we'll say we'll just select sum at random and then connect it to that okay this is what we call random attention and you can pretty clearly see if you connect each of the i nodes to r equals two to two random nodes then you don't have an n squared anymore but you'll have a like an o of r times n which you know if r is a constant is an o of n


attention mechanism okay so the main goal between the random attention mechanism is that for each query basically you select random tokens that you attend to and that random number is a fixed number that's not dependent on the sequence length and the paper is a little bit unclear about whether or not those random ones are the same for every sequence or are switched up or are the same for every layer or are switched up but they formulate all of this as sort of in sort of a graph in sort of a random graph so there they formulate the attention


mechanism in form of a graph so if we transform all of these nodes into a graph a full attention mechanism would mean that each graph each node is connected to each of the other nodes right fully connected graph i don't maybe that's it so that would be a full attention mechanism and then they say well if we just have random connections between these things then there are some theorems from graph theory that say that each random walk in this graph is going to um so this graph is going to mix pretty quickly so i can get from each node to


each other node by a random walk in a logarithmic time and this random walk which basically means that you go from here to here this would be one layer of the transformer and then if you want to go from here to here that would you would have to do that in the next layer so this formulation as a random graph leads me to believe that layer after layer the random attention pattern is going to be the same but also the formulation of the paper leads me to believe that the this random attention differs from sequence to sequence so


i believe what's happening is that they you know get a new sequence then they decide on this pattern right here once and then they use this layer after layer the same pattern again so you can see that um in the traditional attention information can basically throw flow from each of the nodes to each other node in one single step right because each node is connected to each other node you see this in the graph right here however if we only select a subset then you know it needs to if if i want to go from as i said from here to here then i


need to do it in two steps and therefore i need two layers and that's going to be the culprit of this method here and you know while it is mentioned in the paper it's sort of i feel at least that's my my assessment of this paper it's kind of swept under the rug a little bit i mean they do have a theorem that clearly says we can construct an example of a task that in the full attention setting can be solved with a single step so a single layer that in our random attention setting needs a lot of layers so a lot of steps


but you know the rest of the paper is sort of shaky on on this thing but nevertheless you can see how the random attention can if you have enough layers do the same information routing as the full attention okay however this is not a property of the random attention and we'll see this in the next thing right here so the next ingredient that this paper uses is window attention and you can see over here that big bird is ultimately going to be a combination of the three types of attention which will uh which we are looking at


here so window attention basically means that each each eye each token at the if position is going to attend to itself of course so here is i but it is also going to attend to its neighbors so here is i minus 1 and here is i plus 1. and this is a you know this is a window size w that you can that is a parameter but also it is a constant and therefore um you again go from n squared to w times n which you know is o of n if w is a constant and this might be familiar to you because we've already seen this in the


long former paper so i've made a video or base i think even two videos on the long former which used exactly the window attention in combination with the global attention and uh if you want to know more about that go watch these videos but the new thing in big bird right here is this re edition of the random attention again the the window here in is is has exactly the same properties as the random attention so you have instead of a fully connected graph you have a sparsely connected graph now if you have random attention the


sparsely connected graph is like like the one on the right but if you have a windowed attention you can it is kind of not randomly connected but each node is connected to its neighbors like this and you can also see that if i want to go from this node to this node right here i can't do it in one step but i can do it in two steps i go here and i go here so in the terms of the attention layers if i want to go from node one to node three i have to do it in two steps because each node is only connected to its neighbors


so the connection patterns would sort of look like this so i have to go from one to two and then in the next layer from two to three so the paper basically makes up for the lack of full attention by uh adding layers and you also might recognize this from a convolution operation like this basically because it is a convolution operation right in a convolution each node a only aggregates input from its neighbors for the next layer and then we know that as we go up the layers the de facto window that each node looks at


is going to be like a cone kind of like this so this is very similar to how a convolutional neural network works and the reasoning is very similar because the reasoning is well in a sentence the most important words for any given word are probably going to be its neighbors like the words around it and as you go up the layers you branch out more and more but ultimately the this neighborhood principle holds in nlp as well so again we already saw this in the long former but that's the reason behind the window attention and that's the second


ingredient and then the third ingredient is this global attention now the global attention uh is selected tokens that are so important and that's you know fixed by the developers that are so important that they are they are connected to everything else so for example in these transformers you often have what's you know this kind of cls token so this is a special token that you prepend to some piece of text and the output of this token is going to be your classification output because you don't want to bind your


classification if you need to classify the entire sequence you don't want to bind that decision to one particular word what you want to do is you want to have an extra token and that's this cls token that kind of aggregates information from all of this so layer after layer layer after layer you'll have so if we go here layer after layer we have this one special node and in each step every single other node is able to send information right here to this node and receive information from this node okay so now uh


as a result of this as you as you may be able to see every single uh every single path is kind of a maximum length of two because if i want to go from any node to any other node i can simply you know send information to this global node and then the global node in the next step can send information to whatever other node and that is a property that they use in their proof that this tension mechanism is as sort of as powerful as the classic full attention mechanism and we'll go through that in one second but first i


hope this was clear that this combination of random attention window attention and global attention is what is called big bird okay they have some engineering tricks that go along with this but in concept you can imagine big bird being long former plus these random attention right here and you know as an engineer as an nlp engineer that makes kind of total sense i you know i totally believe that a the introduction the addition of these random attention of these random attention patterns can absolutely help your classification


or whatever your nlp tasks because you know more attention better and i also am completely willing to believe that you know using the full attention matrix while it is of course more accurate it won't hurt too much to leave some of that attention away because essentially all the path lengths are just becoming too or even with the random attention are really short or logarithmic to route information from a node to some other node so the loss that you incur is kind of in a logarithmic scale in terms of performance


while the gain that you make is sort of in a in a quadratic or like a linear scale you go from quadratic to linear and that seems to me like a good empirical trade-off all right however the the proofs here the proof of um of how how these how these things are constructed are a little bit i don't know so what they do in the proof that this function can sort of a is a universal approximator people have already shown that full attention mechanisms are universal approximators um so they show here that this sparse


attention mechanism is also a universal approximator they make big use of star graphs what they say is okay if we have a star graph which is one node connected right here to every other node this is a star graph if we have a star graph we can achieve the same thing than with a full graph a full graph is where every node is connected to every other node but as i already said what they need for this is multiple layers of this star graph so and that has to do with the fact that if i want to route information i basically have to go via this


middle node right here and there's an additional complication because this middle node in our case right here is only one node i can't route information at the same t like i can't have this routing right here at the same time that i have this routing right here like going from here to here because i only have one middle node and i kind of this is not how the like this is very dumb math but uh maybe you have to imagine that there is one memory slot and you can only use that one memory slot at the same time for one of these things so essentially


what you'll have to do is you'll have to do the green thing first and then in the next step you'll have to do the blue thing second and then so these are now pairwise routing between nodes but ultimately what an attention mechanism does is it does everything to everything right in a single layer it routes information from all the nodes to all the other nodes and to achieve that so you need multiple rounds of this and it turns out that in the worst case you actually need n rounds of this so you know you trade off your you go from


n square to n uh memory and compute requirements in a single layer but in the worst case you need n layers to recover the the power of the full trend of the full transformer and that is the last one of their theoretical results right here so first they prove universal approximations and second they prove turing completeness these two properties have been proven for full attention mechanisms and third they prove that there are tasks where you actually do need n layers to solve them with their limited attention


um so you know i'm not sure but i i feel you can make any sort of polynomial uh algorithm into a linear algorithm like this like i have a i have like a cool sorting algorithm right so if this is my sequence that i want to sort what i can do is i can simply you know take a random subset of them uh like this this and this and then kind of go and and sort them and then put them like i send them to the to the global memory like this i sort them and then i put them back right and if i do this for enough if i do this for enough rounds okay you


know if i do this for enough rounds you know at the worst case i need n rounds to sort my or log n rounds if i do it smartly but you know in you know the single step here is uh the single step is just o of n so i have now an o of n sorting algorithm i you know i have my sort of a bit of wary to express things like that and um yeah but you know it is from an empirical standpoint i absolutely believe that this uh this is enough now my second quarrel right here is that if you look at the proof first of all what it


makes use is this star graph and the star graph corresponds to the global attention so that's not much to do with the random attention though they use the random intention in their proof but i at least believe that it would be possible with the global attention only and then the second thing is if you look at the parameters that they use for the um for the experiments and i've already set this in the long former video so in the long former video it turned out that if you look at how big this window attention


is it turns out that it you're still well you know the original bert attended to 512 tokens and then you look at the window and the window was still 512 tokens it's just that the global attention was even more so ultimately they ended up using more memory than the original bird and here if i look at the parameters of their um thing and they have multiple experiments right here and i believe this is the the base version so this is the base version they also have this large version but here this is the 12 layer um version and you can see they


have this block length and we'll get into the block length in one second but then you can see that their window size is three times the block length the number of random tokens is three times the block length and the number of global tokens is two times the block length so that results in eight times b so 8 times 64 is you know can i calculate this or am i stupid uh it's 512 yes actually calculated this before so this is 512 tokens so you know you you go from from bert that has 512 tokens and attends to 512 tokens to also


attending to 512 tokens of course the advantage here is that they now have 4 009 and 96 sequence length so they have the freedom to not attend to as many tokens as they have in the input length but you know to put it in perspective this here uses more memory and more compute on it on its face than bert because bert attends to as many tokens but has a smaller input sequence and you know i i there's sort of a thing where in order to make these sparse attention things work you have to go pretty pretty you know high in the number of things you


attend to you can leave away some but it's not like you can you know scale up orders of magnitude of your input sequence length so that's this promise of linear attention is sort of it's kind of fulfilled but not there yet the second thing i i would like to point out is that in a lot of cases the number of random tokens is actually set to zero so really making use i believe of these of the of the global of the number of global tokens so it's that seems a bit strange in that they continuously refer to their random


attention mechanism um but then in a lot of experiments they don't actually have a random attention mechanism i believe they have to do that because that's kind of what makes them different from the long former in principle but still um yeah so the last novelty let's say is an engineering novelty in that they now always consider not single for example they don't consider single random attention they always consider these in blocks and that's because our current hardware is really bad at sparse stuff really bad at


single indexing gathering single things so if you can do everything in blocks you basically get you get these blocks almost for free so it takes only marginally longer to retrieve this full two by two block right here than it would to retrieve the single uh instance right here of course that means you have you know four times you still use four times more memory but it is not four times slower than the original thing so you can use these blocks uh right here you can do it for the random attention you can do it for the window attention


as you can see here so you break this window pattern a little bit into blocks and that makes it a lot faster or that speeds up i get the speed up almost for free and then they make another approximation in that the way they do this windowing is and now let's just go really briefly so you can see right here that it would be very cumbersome to gather so what we need we're just going to focus this this dotted thing right here is a bit confusing so you want to attend to these things and these you can just get out with a matrix


slice really easy but then you want to attend to this kind of blocky thing right here from the window attention right like this thing and this is hard to get out because you'd have to kind of index each row individually and that's very slow so what they do there is this matrix roll operation where you can sort of roll the axis around so what you'll do is you'll take this thing right here and you put it to the left right here and you'll take for example this thing right here and you'll put it to the right or no like it's it's up and down but in


essence that's what you do and you can you can fold all of this blue stuff into a rectangular matrix if you know if you can see right here so you kind of roll this back roll this back roll this forward and you replace whatever is missing by these now this again gives you some inaccuracies because this block right here was never intended to be attended to and all of a sudden you see you have the k6 in here so it gives you a bit of inaccuracies at the edges of the sequence but you can take that you know you can take that hit for the


increased performance that you gain by now having a rectangular matrix tpus are really efficient at this not as efficient as this and then the only thing that's really slow is gathering these random blocks right here but also by having the same amount of random blocks per input token what you'll do is you'll end up with just one of these columns right here or you know r of these columns and that again gives you a rectangular matrix so this thing right here you can process very very efficiently using a tpu and you know the mistakes you make


are basically this thing right here and this thing right here because those weren't intended and are at the edges of the sequence so these were the the tricks of big bird to quickly uh summarize uh big bird is basically taking a transformer saying well why do we need all of this attention all of this full attention maybe we only need some of that and can already do a big job a good job especially you know considering the attention mechanism goes over multiple layers so we don't need a routing from each token to each token


we we can make up for not having a fully connected graph by simply running multiple layers so their sparsity is first of all you have this random attention which i believe changes from sequence to sequence but stays within or among the layers of the same sequence then you have the window attention with the reasoning so the reasoning behind the random attention is that if you have a randomly connected graph the path lengths are on average logarithmic so you can route information efficiently the reasoning behind the window


attention is that probably neighbor information is very important and that has been shown empirically and then the global attention the reasoning behind this is that some of the tokens that are fixed by the developers are so important that it it's very beneficial that each other node is connected to them and that they are connected to each other node the result of that is the big bird attention mechanism which is basically long former which already had these two plus the random attention this achieves a linear


linear complexity in terms of of memory and compute though linear has to be qualified a bit because it's modified by the window size by the number of random attention tokens by the number of global tokens and in practice often ends up being you know fairly large-ish and also the the theoretical guarantees now come with the fact that you need multiple layers in the worst case you need sequence length amount of layers which you know in the worst case would result right back into a quadratic requirement for memory


and compute they do some engineering some engineering tricks right here and their results are pretty good so the results in various tasks and we'll we'll look at some of the tasks right here so these are def set results using base size models for example where you can see they do outperform basic roberta models they outperform long former which may mean that the random attention is useful but you know in these things it also always may just mean that you've thrown more compute at it um at least i'm not really looking that


they outperform the models because as you can see right here if they compare to state of the art and you know granted these are models that have been trained specifically for these tasks and are you know crafted and engineered and big bird manages to big bird manages to hold itself against them in a lot of tasks and even gets state of the art on some what i'm more interested in is that it you know it can reach good numbers it doesn't necessarily have to be state of the art but it can reach good numbers which tells me that


okay probably the the empirical hit that i take by not having the full attention is you know is justifiable by the speed up and memory savings i do get um yeah especially when result when you see results mixed like this you know sometimes the other model is good and sometimes the big bird is good on different variations and so on i would not you know i would not make a big deal out of the fact that it is state of the art i get that the authors have to do that i would do so as well but you know um you know don't


don't think that this is the like the best thing now uh it's very probable they just thrown also a lot of compute at it what is cool is they do uh some genomics experiments so not only do they have nlp state of the art but also they go into genomics and experiment with data there don't want to go into that because you know ultimately it's another task and that we leave the papers about the architecture all right so that was big bird i hope you enjoyed this video and learned i learned something certainly if you want to check out the proofs


okay it's absolutely great to be here I am the official Kaggle representative kaggle comfy representative for this presentation or for this event we have a major Kaggle team meeting going on next week so very few people were able to attend and so I'm I'm happy to be here the event organizers actually contacted the cago leadership and they asked they asked could we get the the smartest Tagil employee to come give this presentation but unfortunately that person said no so so then the leadership said if they organizers ask the

Keeping Your Skills Fresh When Everything is Changing | by Walter Reade | Kaggle Days Dubai | Kaggle


leadership well maybe we can get the wisest Kaggle employee to come speak and unfortunately that person said no also so the organizers they got a little bit nervous and they thought oh maybe we can get the best-looking kaggle employee to come speak at this and at that point I was very uncomfortable turning down the organizers three times in a row so here I am actually the truth is none of that is true I work with an amazing team one of the things that kaggle values as a company is low ego and I think that is


really easy to do in this field because no matter how good we are there's always these super grandmasters who are so smart and they're so clever and they do so well and it's very easy for us to you know keep our egos in check so I'm going to talk about a topic that I think many of us probably struggle with and that is how do you keep up in a field that is constantly changing all the time right we know we need to keep our skills fresh but how do we do it and before I start I don't believe there's a right answer to


this so I'm gonna share what I believe is right and the kind of the strategy and methodology that I use and hopefully you'll be able to find a few things that are right for you so during the recent Kaggle de San Francisco event we had Francois Holly so he is the creator of the Charis deep learning framework and he spoke and he told an interesting story he joined Kaggle and entered his first competition and was competing and he wasn't doing very well so he thought well maybe kegels not for me he forgot about Kegel just didn't do anything for


a few weeks and then all of a sudden he got an email from Kaggle saying congratulations you've won most of us do not win the first Kaggle competition that we enter in fact most of us don't win any cattle competitions so it's very abnormal what are you shared most of us I think when we joined Kegel right some of us are very very skilled they come maybe with an advanced degree in data science or machine learning but a lot of us like myself come knowing almost nothing so so I'm gonna talk about some of the principles that will will help us


and again they're very applicable to me I really hope they transfer to many of you so I'm gonna start to talk about my education and my career and not because either one of those are particularly important but I really want you to understand where I was as an individual when I joined Kaggle in my first cattle competition so so that's me a long time ago I got my first computer in 1983 it was a Timex Sinclair 1000 you've never heard of it because they went out of business in in like a year it was the company Timex that makes the watches and


they wanted to get into computers it was an absolute disaster for them and that computer that I got into that by the way that's not a Timex Sinclair that's an Apple - that was my neighbor's that I used to go visit and and use their computer but my first computer had 2 kilobytes of RAM 2 kilobytes of RAM just to put things in perspective and so I taught myself basic I really enjoyed it right I wasn't necessarily very good but I really enjoyed it my high school taught a class in Pascal programming probably maybe not super familiar to a


lot of you I enjoyed the class so then I thought well what do I want to do for a career and it turned out that everyone I knew was going into computers and I was worried by the time I graduated from college there'd be a flood of talent and all of the jobs would be taken right of course that was foolish because the right the industry took off faster than the demand pool and still to this day there are people in computer science and there's demand for the talent so I thought well what should I study I chose Chemical Engineering and I chose it not


because I knew anything about Chemical Engineering I chose it because it sounded kind of cool Chemical Engineering well at least it sounded cool in 1987 so please don't judge me for that so when I was an undergrad I took a class in Fortran 77 I'm sure most of us have heard a Fortran and again I really liked it I went on to get a masters for part of my master's I used Fortran to write a program to model the combustion of a coal particle as it burns and I really enjoyed it so I went on to get a PhD in chemical engineering and this


time a hundred percent of my work was computational so I did direct numerical simulations of turbulent three three dimensional turbulent flow and the particle movement within that flow now interestingly enough I don't know if many of you know this but the Fortran originally you had to start in the seventh column right so space space space space space space and then you would start your statement and things like line numbers and whatnot could go in the first six but that was a throwback to the old punch card days and


with this was in the mid-90s they came out with Fortran 90 and Fortran 90 no longer needed you to have to worry about indentation which was really great right you could just write your program how you wanted and I thought it was fantastic how it unshackled me from having to worry about indentation now I also find it very ironic that in today's stage guido van rossum who invented python has reshackle me to a programming language that cares about indentation so one other point about this again just to give you an idea of how fast things


change so now we're a decade later we're in the mid 90s and I had an IBM workstation and I had received a $10,000 grant for equipment for my research and I spent that entire $10,000 to get a memory module for this workstation that was 256 megabytes of RAM which was you know a very large Ram back then but $10,000 for that so things change so I got my first job out of school and so everything up to them was great loving life loving programming loving science loving Chemical Engineering and then I got a job in an industry that uses


chemical engineering it was the paper industry and my job was to optimize air and drying systems for drying cardboard right we used cardboard you know they make it wet it's got to be dried and it was I can't say I really enjoyed it you'd have to go and so you know but you can see this is a person you'd have to go and stand up here and you'd have this long temperature measurement that you'd stick in there and measure the temperature of these dryer cans then you do the same thing with a humidity and you would do it on every single one of


those on the back in the front it was hot it was sweaty it was not what I went to college for nine years to do so I decided to upgrade my career and then I went into consumer products making diapers are nappies as they call in some part of the world and slightly more glamorous I suppose but still not necessarily giving me the career fulfillment that I had always dreamed of and by the way the technology that goes into making diapers is is actually pretty amazing it's just not glamorous work so now it's the early


2000s and I am absolutely bored out of my mind with my work it's just not exciting I I don't enjoy going to work and my company offered a distance education catalog and I was looking through there and they had a course on computational intelligence and that was three things it was genetic algorithms that was considered computational intelligence back then fuzzy logic which was super hyped up at about that time that I don't know does anyone use fuzzy logic anymore probably not and neural networks so I took this


course now this was pre Coursera days they actually physically mailed you DVDs and you would watch the lectures you know on your television you would do your homework you would email the instructor two weeks later he'd email you back your grades so that was the distance education that was back then I really found it interesting this idea so they taught a very early version of convolutional neural networks they didn't work well they were very narrow you know they're very shallow and we actually didn't program the so we're


still on windows so the instructor would he emailed us a precompiled binary we would run it on our Windows machine which was probably a single-core CPU or back then but even if it had more than one cards we're only using a single core but we would try to train this thing and and but the the problem was these little pop-can images I should have put a picture and I was blown away that a computer could recognize was it a diet coke was it a dr. pepper was it a sprite I was really fascinated by that the problem was is once I tried to apply


this technology to anything in the real world like so in in in consumer product manufacturing they're taking pictures of every that's being made to do some simple measurements and what and I thought well maybe we can use Ural networks for this could never get it to work so I said well neural networks interesting but not useful so so then fast forward another ten years I'm still bored out of my mind at work right but hey families got to eat and I stumble across this challenge it was the Merck molecular activity


challenge on a platform called kaggle it was fairly simple all you had to do is predict the molecular activity of these you know these molecules based on tens of thousands of molecular descriptors and I've got a PhD in chemical engineering I'm really good at data analysis top prize was $25,000 so I thought to myself this is gonna be the easiest $25,000 I've ever made in my life right I was literally picking out the car I was gonna buy so the person who won that competition does anyone know who that is it's Jeff Finn so his research lab won


the competition using deep neural networks and as Maria said I came in though there were two hundred and thirty six teams I came in actually 23rd but it was from the bottom I came a twenty third from the bottom and it wasn't because I wasn't trying I was literally trying my hardest and it was it was actually a wait but I mean it was actually it was very demoralizing but it was also a wake-up call because I realized there was all of these things that I had no idea that I didn't know so it was just was like window opened into


a whole new set of tools and technologies that I could learn fortunately another thing happened in 2012 so you probably know Andrew Eng is speaking here in 2012 Andrew hang it was a co-founder of Coursera and I found his course on machine learning and took that course and again I was blown away that here we have a course that would literally have costed thousands of dollars for me to take it's online it's for free so things were good for a while things were good because you could take that course there of course you could


read the few papers that came out Jeff Hanna came out with a few papers but suddenly there is a problem this is just a screenshot of Coursera today and there are more data science and machine learning classes that any person here can take even if it's full-time if you look at the number of machine learning archived papers that are published every year you can't keep up so on the one hand we live in an amazing time where you can learn the things you need to - like in my case change my career or become good at Kegel on the other hand


it's it's too much and you can't keep up and it's information overload so my question is how do you know get given them given the fact that you must everyone here must keep up their data science skills and the machine learning skills we have to keep up on the one hand on the other hand there's too much to learn so what do you do so the question is how do we keep up this is probably the most important point to understand is you don't you can't you can't keep up with everything new and machine learning and data science there


are people who are deep learning experts who can't even keep up in the deep learning field and once you recognize that it kind of then reframes your mind to say well if I'm not going to learn everything what can I learn and how do I learn it so I'm gonna give three guidelines okay on how you can continue learning - like if you're brand new how you start your learning journey if you're experienced how you continue and the three guidelines are be sensible be smart and be systematic now I was thinking about branding these the three


BS s of learning data science but I'm not sure if that's the best branding you can give me feedback if that's catchy enough so be sensible okay all right I crashed and burned on my first Kago competition and like I said there was an enormous amount of things I had to learn number one my computer was terrible was four years old had eight gigabytes of RAM and I realized if I want to compete so I started upgrading my computer more Ram Moore's better CPU more disk space more even more RAM better seat a new motherboard right so I started that race


that many of us do okay that got me kind of far I was only using MATLAB in some proprietary software me neither of which you can actually use to win a kaggle competition and at least at the time MATLAB wasn't the best tool so I realized I've got to learn Python so that was another big major learning step the second competition I entered I actually did okay I came in 34th out of 950 34 from the top this time and I use logistic regression not because it was the right tool it was literally the only algorithm I knew in machine learning so


I used it right there may have been veterans uh let's see I know who won like there's some people here I know one that I can't remember I know there's some grandmasters here that actually won that competition so then I learned you know you start seeing XG boost XG boost is winning everything so then you learn next G pose now uh deep learning I was sure that deep learning was a hype just like fuzzy logic and I avoided learning deep learning over time you know a year later I realized okay this is going to take over the world so I started


learning it well that required hardware GPU it required you know I was using Windows so I better go to one to Linux every time I made this shift I felt dumb again I felt stupid I felt like I don't know what I'm doing because it was a huge learning curve so when I say be sensible I want you to recognize that if you start feeling good about yourself and how good you are it's time to jump into something else and to kind of stretch yourself and feel a little bit intimidating again and that's perfectly normal and that's


gonna continue to happen III would say anyone who wants to be successful in this area has to get used to feeling insecure about what they know and what they don't know happens to me all the time not I'm I don't like when I say something and it's a mistake or I argue a position and it's wrong on the one hand on the other hand I'm actually glad because right it improves me and I get better so be sensible set your expectations to something that's reasonable so the next thing is be smart and what this means is you have to build


a learning plan for you if you go to github all these people say hey a step-by-step learning plan that might be okay it's probably not for you right and the reason is we're all in different places if you go through and somebody's learning plan that is a waste of time to advance not advanced enough you will not do well so be smart so here's what I recommend when you set a learning plan excuse me so say a learning plan say you have five hours a week to learn you all right you say all right every week I'm gonna learn and spend five hours


this is how I would recommend doing it number one and I was talking to Pavel earlier this is he's very big on this strengthen your fundamentals so that could be the fundamentals of linear algebra the fundamentals of statistics those things don't change for me most of the time I spend in there is learning Python deeper right learning the language I my favorite book to read and reread is called learning Python it's this thick it takes me forever to get through every time they get going new edition I start over again to refresh


those fundamentals the other piece of that is I spend a lot of time learning and relearning pandas every time there's a new addition I learn what's new and I also scan the API for things that oh maybe I didn't realize there was something cool there so strengthen the fundamentals of whatever you need to learn the second big chunk is your core work and your core tools so this is whatever you're doing for your project your problem you know the work you do at work for me is a kaggle data scientist a huge amount of that is domain knowledge so


somebody comes to us and says we want to host the competition on earthquake signals I've got to spend time learning about that so I can understand I can ask the right questions when Kaggle was acquired by Google all of a sudden we now and we using our local machines that now we're using GCP VMs right so learning kind of that system stuff I hate to bring this up but we constantly also try to learn the new ways that leakage can enter the data because there's so many thousands and thousands of ways insidious ways for leakage to go


and and trying to build our systems around how to how to minimize that that will be different from every one of you and then the last is spending time on the cutting edge now this is where most people make the biggest mistake when they try to keep up with the industry they spend way too much time because there's all this cool stuff there's a capsule networks for neural networks or you know the ordinary ordinary differential equation neural network stuff that was from the nerves conference so some really cool things


and I am NOT dissuading anyone from keeping up on those but if you spend the majority of your time learning those and and and trying to keep up the speed you will actually be sub optimizing what's critical for you there are many tools that are going to come out that you'll be able to use right out of the box like Bert the NLP tool that Google pre train model that Google released right that's something that is cutting-edge and you should be able to pull it right back into your you know your tool set so that's approximate you know I would


say you don't necessarily need to balance that on a week-to-week basis but over the long term try to to balance that one other point when it comes to learning new things I strongly believe that if you're not writing code and trying it out you're not actually learning it so I fall into this trap I watch a lot of YouTube videos Leyland icon comes out I like to watch as many of the Python videos as possible and unfortunately most of the time I'm not not actually implementing anything I watch it I go yeah that's a great oh


that's a great a great idea and then I promptly forget it so if you see something that's important make it a project and get it into your your repertoire so the last vs is be systematic and this is the secret to sustainability this is how you can do this in the long term and not two years from now 200 and I haven't kept up so what do I mean by be systematic I am a big fan of simple checklists and some of you may have read a many years ago I wrote a post mislay it was my most popular forum post I fangled or said


here's my standard work that I do for every competition and it literally is I update my Khanda environment I you know whatever so I do any environment updates I create a new repo I download the data I you know so it's just a very simple checklist of things I do so I use multiple machines so I make sure everything syncs out all those machines it wasn't anything that was complicated but the reason I use checklists is because it reduces cognitive friction when you're like oh especially with Kegel we're like oh it's really late I


really want to start this competition you don't want to have to think about it so having a checklist allows you to go boom boom boom boom boom and do it without having to think pilots use checklists pilots are some of the most skilled practitioners about what they do and the amount of training they have to go through they use checklists because it reduces cognitive friction if they have a headache if they're tired and make sure they do the appropriate things so so how how do you use a checklist or a system when you're learning for me I


have a list of things that I do every day to keep my learning up to date the first thing is super simple I have a bookmark bar for five different archives so archives that write the paper repository archiving topics fluid dynamics data and statistics vision machine vision and machine learning and earth sciences I open those up and I every morning literally before I do anything before I check email I check what those are so that's one of the things I do it's a habit it's sustainable the other thing I do is write any time that any of the PI data


the PyCon videos come out I make a list of all the ones I want to watch and I don't get to more than ten percent of them but I make the list and then I worked through them systematically till the next ones come out all right so if you want so as I mentioned there are so many courses it's actually intimidating where do you start what's the right course I have become a big fan of CagA learn not because it's put out by kaggle because I really actually agree with the philosophy what kaggle so if you think of a move if the time to make the


presentation to record the presentation to transcribe the presentation to host the presentation it's a very very very very long time scale what learned has done is say we're gonna take some very key important topics and we are going to let you do it in four hours and we're gonna focus on those key important things and get you pushing the buttons and learning these things what's nice about this there's no video actually one of the courses had videos but they're gonna get rid of them is they can freshen them isn't like as much as once


a week somebody will ask you a question and they'll realize oh we can make this better so they'll refresh the content so so for example if you if you look at this so it's kind of small but here's the current cago learn everything from python machine learning data visualization sequel micro challenges and the new one machine learning explained ability maybe you know those things but maybe you want a refresher it's really really fun and easy to go through so I'd recommend checking out one so here's like this one on


embeddings it's it's I went through it before I I had never got there I went Baba before you through it yesterday I thought oh yeah it makes a lot of sense so cago learned a great place if you ever feel lost go back to Cal go learn obviously there's tons of great stuff out there but if you're ever paralyzed about where you should go go to CAG alert this is an example right so they walk through with words they give you some code you run it it kind of lets you know if you were wrong it gives you hints on very the common errors and it's


fun it's interactive it's in kaggle kernels so you can access it anywhere and now that actually keeps track of your progress so you know if you stopped one halfway through it'll tell you which modules you've done all right the other thing that this this kind of surprised me if you go to youtube.com forward slash Kaggle there is an amazing amount of content and it's added to weekly everything from coffee chats to interviews with for example practitioners at Google brain all sorts of pretzeled live coding gets recorded


there so this is another great example of a place you can start there's probably too much content there for you to keep up on and that's okay right again give yourself the permission to skip things and focus along those lines there's a point I wanted to make every cago competition that I joined my primary goal was to learn something new so embeddings for example I'm not an NLP expert so if I join a cat go I did a playground who's spooky author so it was like three authors given a sentence predict which one so I did that with the


sole goal to learn NLP you may under one to learn okay finally I'm gonna learn convolutional neural networks or ell STM's or some sort of you know whatever you need to go so if you do that no matter where you place you will have one you will have had time well-spent so I want to close up a little bit with this quote it is by far my favorite quote it's a little bit intense so I want to explain it says the illiterate of the 21st century will not be those who cannot read and write but those who cannot learn unlearn and


relearn a hundred years ago so many people didn't know how to read or write and that prevented them from being in the workplace in today's world not just in data science and machine learning but almost any vocation you have to learn you have to be able to learn to unlearn and to relearn so that to me is the most important reasons for having yourself a smart plan that's sensible that's smart and that's sustainable so that you can learn whatever skills and do that five 10 20 30 years from now to keep up and have


whatever professional success that you want and with that thank you and I'll open it up to questions one second Ronnie Heflin let's see who has who has the mic area what I see a second period thank you oh that's a great question I I'm sure there are but I can't recommend any and the reason being is I just that's not something I looked at does if anyone here know of good data science for kids I you know I would say I don't know for sure but Khan Academy kha in the anatomy is one that has a lot of great content


but I don't notice they have data science No okay so sorry great question I'm gonna write that down and we'll follow up yes sir [Music] [Music] yeah it's a really question and and I actually don't believe it's the type of data I believe it's the volume of data and what I mean by that is if your company wants to do a very large data project and you have data in multiple legacy databases that sit in multiple organizations that can be extremely complex and hard to get my recommendation is always to start with the smallest viable scope so try to get


a little bit of all the data if you can or just get the data that seems the most reasonable and start doing model building on that the biggest mistake companies make and I've seen this many times is they say well we're gonna have a two-year project to get all of the data collected and then they get all the data collected and they don't have the signal that they need or only five percent of that data was important so I would say just like it was mentioned by grandmasters when your model building you go quick you go small you go fast


and then go from there I'd say the same same with data so then then you may say there's there's some very expensive data here that it's worth the time cleaning up it's worth the expense but then you'll find a date over here that's not worth the time okay the biggest mistake is companies say well let's clean up all of our data and that's just waste of time thank you question over here trick question take you back to your fluent that after navier-stokes yes and the reason is is because the navier-stokes is a you know


a three dimensional coupled nonlinear equation what the neural networks theoretically can approximate any function so neural networks can approximate the function so I just read a paper last week it was sent to me by a professor in Germany and so there's a there's a type of flow where you have a temperature differential and then you get these circulation patterns and if you look at if you ever seen a close-up of the surface of the Sun you see these like pockets that's what causes that and what they did is they did very very


high-resolution simulations of this and then they trained neural networks to identify so segments the the boundaries of this now they get the same results of these expensive simulations but it's orders of magnitude faster and there's a paper that came out a couple of years ago where I think Google was a co-author where like a smoke plume simulate that but then use neural networks to approximate that so now you can model those dynamics so you you you will see more and more of that but the answer is absolutely yes it is exciting work I'll


say another thing you're seeing more of this in chemical kinetics so chemical kinetics when you have multiple chemicals it's great it's it's stiff not nonlinear differential equations because you have very fast reactions very slow and people are now using neural networks if you have the system of chemicals we model probably react will see more than yes I am NOT a deep learning expert but I crush because our neighbors are noisy and we do do-dah like 1 million picture is just fine but do you never you are dead picture so great great


question so I'll answer that generally first so in general you want to pick the tool that's appropriate there's no reason to do a very very fancy 3d instead if you have a small amount of data and you can do logistic regression then it works fine so I'm a big fan of doing the simplest tool now with neural networks we have the advantage of transfer learning where Google has spent a month training a bank of GPUs on these large networks and so you can use those and actually I'm probably not a hundred but maybe a thousand images because you


have all of this pre train models so you know something like that I always say try it right if you have a thousand images kind of fine-tune a pre training model on on 500 see how well it does on the other 500 it's going to be very context dependent I think you know there's some image tasks that are extremely complicated that wouldn't work but there's some simple ones yeah that's a great question and you know to me to be very frank I am very bad at predicting what's next I've always seen what's happening and then act upon that you know there's a


lot of people that talk about these things and you know I don't pay a lot of attention to them to be fair i I just watch for tools that I can use in my work I will say this you know data science machine learning AI it's it's not going away in ten years it's not going to wait 20 years I think it's a fantastic field at the end so again is the people that are willing to keep up and whatever happens to be important and learn that I think are gonna do very well for themselves but again I just so I would be out if I knew the answer to


that I'd be on the main stage I think we have maybe time for one more question oh yes yes then little stars our afternoon break so we have like three after four minutes do you have any other questions [Music] yeah yeah that so if I understand your question is how do we help those who maybe are discouraged along the path yeah that's that's a hard question of you know so I I came to cattle with a PhD in chemical engineering in a good career and I remember the sense of being very intimidated to ask anything on the


forums right I felt I wasn't worthy I felt it was a stupid question that everyone knows it but me I think the more that we tell those stories the people that that's natural that's okay everyone goes through that I think the better right that's why I'm happy to tell my story how poorly I did I tell my story how frightening it was to ask the question on untangle but we can also do a good job I think so tango we try to monitor the forums and I can't say that I do this all the time but if I see somebody that you know they're due and


they're trying to say we I'm new to this you know I much as possible try to point them to hate just keep at it don't get discouraged so you know every one of you here can do that you don't have to feel like you're a grandmaster to give people encouragement so I really appreciated that question and regardless of where your level is I think you have the opportunity to do that and again that's what makes Cagle a wonderful community I love the cattle forums there's so much useful content people can enjoy each other a month ago is actually when I met


my first Aguilar I had never met somebody from Kaggle but I feel like I know them just by the online interaction so it's a wonderful place and we just have to convince people that they can do that at home and themselves thank you for them hey do you want to ask any more questions okay Pavel I can see your hand what is thank you but it will be the last question that's what your question my favorite game so probably he's maybe not as well known but a parent from Germany Mathias Mueller I liked him because I got ended


up with him he was always very helpful he was always very friendly he explained things that we easier though easy way for me to understand but but you know it to be fair I there's so many carers the grandmasters that I appreciate for all different reasons like like Abhishek I mean he's just he's just such an interesting character a funny guy as there has a lot to teach people yeah and I can go through the list but I'd say fair and fair and Mattias Mueller from Germany now in the Bay Area but actually maybe back in your baby is my favorite


hi there today we're looking at fast reinforcement learning with generalized policy updates by andre barretto shabo ho diana borsa david silver and doina preco so on high level this paper proposes a framework for reinforcement learning where you have many tasks at the same time and they propose a framework where they learn many policies at the same time that can or cannot correspond to these tasks and then their argument is that if you now have a new task that you haven't seen before you can easily construct a solution to

Fast reinforcement learning with generalized policy updates (Paper Explained)


that task from your old policies basically mixing what you learned about your old tasks and it's a pretty general framework and we're going to look at it in my opinion it's it's pretty cool for certain settings however i think it kind of breaks down the the more general you go which i guess is expected um of such a framework but uh it's as you can see it's kind of math heavy but we'll get into the examples and um what it's potentially useful for all right so that was it on a high level if you like content like this don't hesitate to


subscribe to the channel and share it out leave a like and tell me in the comments what you think i'm still reading all of them uh so i will see it cool let's dive in so they say the combination of reinforcement learning with deep learning is a promising approach to tackle important sequential decision making problems that are currently intractable well they're taking they're talking about you know things like um mostly these game playing ais like go and things like this so we're this combination of deep learning with


reinforcement learning has really shined or shun whatever one obstacle to overcome is the amount of data needed by learning systems of this type so again if you look at these systems like alphago they need a simulator and they need to collect enormous amounts of data um even more so with systems like the dota ai the openai5 dota or starcraft playing alpha star i think it's alpha star they need so many simulations in order to learn about the tasks because they always start from scratch in this article they say


we propose to address this issue through a divide and conquer approach we argue that complex decision problems can be naturally decomposed into multiple tasks that unfold in sequence or in parallel by associating each task with a reward function this problem decomposition can be seamlessly accommodated within the standard reinforcement learning formalism okay so what are they saying right here they are basically saying that if you have a task let's say you want to get whoopsie from here to here and that's very complicated


let's make it complicated super duper complicated you can basically subdivide that task into multiple subtasks right so here it's like left turn right turn go straight left turn go straight right turn and so on and each of these subtasks you can see the two right turns here might share a lot of common information there could also be tasks that are at the same time like you need to go forward and jump can be decomposed into going forward and to jump now they're saying is if each of these tasks now has its separate reward function in


the environment like for some reason the environment tells you this by the way is task task one and you're gonna get a positive reward if you do a right turn and this down here is task two the the left turn task and you're gonna get a positive reward if for that task so the entire task state can be decomposed into a vector so in our case here we have maybe a vector with three elements okay the three elements correspond to turn right go straight and turn left and now your this this right here is your reward vector so we're no longer talk in this


framework we're no longer talking about just a reward we're talking about a reward vector now each of these tasks is going to give you its own individual reward so let's say you're here and you're actually turning right this is going to give you a reward of one for this task but reward of zero for the other task okay so the environment will somehow tell you which tasks you you get reward for now there is a notion where you can map this back to a single number and that is the second thing they introduce here so the second thing they


introduce here is this thing they call w so w is going to be a mixing vector w is going to be a vector i will call w right here this is the reward vector w is going to be the vector that tells you your final reward so here we're going to do an inner product so we're going to transpose this and multiply by w and w mixes these rewards and comes up with your final reward right here so this this is maybe the reward vector this is the reward number how we're going to call this reward number so in this case w would have to look