by the way these this 14 times it's also it can be done because now you have more data right so you can also up the batch size then we trained a new efficient net l2 model with the efficient net l2 model as the teacher lastly we iterated again and used an unlabeled batch size of 28 times the label batch size the detailed result of the three iterations and so on okay so you can see


that it's a fairly complicated procedure but you can gain and gain and gain by simply up upping the um by simply upping the or iterating on this procedure and i think they have it somewhere here yes so as you can see if iteration one you train the efficient net l2 you start it with the the b7 you train the efficient at a2 with a batch size 14 times larger and

Self-training with Noisy Student improves ImageNet classification (Paper Explained)


you gain significantly right this gains about two percent over the original efficient net then you iterate again with the same batch size and you get uh like as a 5.5 improvement and you iterate again with an even larger batch size and you get a 0.3 improvement so there's diminishing returns but still you can see that you know the more with the introduction of noise with the


introduction of the larger model with the introduction of the larger batch size these are all things that help the student basically become better than the teacher all right so they do a bunch of other experiments so their main comparison is right here where they say look if we if even if we train the same model with this noisy student training we can make you know pretty large gains over the


model over the same model where we do not train it with this noisy student training so this really seems to help you know due to the noise due to the additional data they do a lot of ablation studies so that's pretty interesting and they also do these studies on this special imagenet data set for example imagenet c you can see that there are quite a bit of distortions right here i don't even see


if you can see it on this video but this is a swing so the swing right here is like something like this but you almost can't see it and you see that the bold on the left is always the prediction of their model while the thing on the right is the prediction of the original model so this model they claim is significantly more robust to these kinds of perturbations


and they do an analysis of this where they show yes in fact it is [Music] so i think we've already seen this at the beginning that the noisy student is significantly uh more robust to these perturbations and they also test this to adversarial perturbations so right here you can see that the original model drops pretty quickly as you increase the epsilon the epsilon


is kind of the strength of the adversarial perturbation and the noisy the original model drops very quickly to you know fairly low accuracy while as the [Music] noisy student training uh drops much much less quickly now this um is another testament to the fact that what you do i think what's happening is you have your data space right and you have your data points in


it now when you do the like normal data augmentation what you'll do is you not only force the model to predict those points correctly but you'll sort of make a bit of a cloud around them and you force the model to predict that cloud correctly now if you introduce more data and you do even more noise what you do is you'll make these clouds kind of larger and that means the model is more


robust to any sort of perturbations in these clouds right and and that means it's probably also going to be more robust to adversarial perturbations so that's sort of how you can think of this uh this introduction of noise uh to make it more generalizable how does this generalize better so if you think of this data point right here if i'm looking to generalize that means you know i have


this iid data set so probably my test data is going to be related to the training data so i might get a data point that's fairly close to that data point and generalizing means i classify it correctly now if this cloud is very small like it is here my decision boundary could be like here right and even though the tres test data set is fairly close to the original


training data point it's it won't it will be classified uh incorrectly however if my original cloud during training is larger you can see if i train a model it can maybe put the decision boundary here and then my test data point will be included in on that same side so that's kind of the idea behind generalizing better of course that's a vast simplification and also to say that


this here is an fgsm attack so this is kind of the weakest attack in the adversarial perturbation spectrum they do say under a stronger attack pgd which is a fairly strong attack with 10 iterations at epsilon equals 16. noisy student training improves efficient net l2's accuracy from 1.1 percent to 4.4 percent and not this um like you know 1.1 percent really means the


model is almost like dead this is lower this is like random performance and 4.4 is still a bit above random performance but um yeah you could probably you could probably get there by simply using any sort of noise in that case but still you can see that it is more robust to especially to natural distortions and therefore it generalizes better as i said they do quite a bit of drop


sorry not drop out ablation studies to figure out where exactly um the performance comes from and the answer is it pretty much comes from all the things that they've described so here you can see um the in the effect of that extra data set and you can see pretty much with that extra data set all the all the situations improve here you can see what do you what is


happening when you do not augment the student uh when you do not date to augment you can immediately see that the accuracy drops and then when you do not augment and also don't use these model noises then the performance drops again and lastly when you use the teacher but you noise the teacher you can see also here the performance is dropping from the original


um quite a bit so all of these things kind of contribute and they do much more ablations and they have listed their findings here so using a large teacher model with better performance leads to better results so you know as the original teacher you should use as good as possible a teacher model you can find second a large amount of unlabeled data is necessary for better performance


okay so if you want to do this you better get a large large amount of extra data because that's one thing that makes the student perform better soft pseudo labels work better than hard pseudo labels for out of domain data in certain cases fourth a large student model is important to enable the student to learn a more more powerful model okay so because usually this um


knowledge distillation is what it this this is usually called knowledge distillation if you use a teacher model to train a student model and it is often used when the student model is smaller than the teacher because you want to kind of become more efficient you from so the teacher is large you make the student small and you usually sacrifice some accuracy and here they say if you want to gain


some accuracy you need a large student model it can't be like a small one number five data balancing is useful for small models number six joint training on label data and unlabeled data outperforms the pipeline that first pre-trains with unlabeled data and then fine-tunes on labeled data so this is in contrast to like what people have done before in the self-supervised learning


and so on where it's always kind of pre-training then fine-tuning or in the in the transfer learning setting seven using a large ratio between unlabeled batch size and label batch size enables models to train longer on non-labeled data to it to achieve a higher accuracy okay we've already seen that they have used that and number eight training the student from scratch


is sometimes better than initializing the student with the teacher and the student initialized with the teachers still requires a large number of training epochs to perform well this is fairly interesting because it kind of alludes to the fact that the minima in weight space if so if this is of course the case if the student model is the same as the teacher model so in like iteration two or three


or what not um it means that you know in weight space if we look at you know you might want to start the student here and the minimum is right here and you might want to think that if i learn the same thing then the minima are fairly close together right so the the teacher's minima might be here and the student minima might be fairly close so it might be beneficial if i if i start not


over here but actually start at the teacher's minimum but this doesn't always seem to be the case and that is a fairly interesting observation because it kind of means that we're talking about different minima here we're talking about the student model learning different things and that's what we've discussed already the student model kind of learns to be robust and that's


probably a minimum that's fairly far away in weight space at least in in a sort of energy landscape weight space uh might be the case that it needs to actually overcome kind of a a hill here even though the minimum might be close there's lots of research in like how minima are distributed in these weight spaces which i don't want to go into right here but it is a fairly interesting


observation that it's not always helpful to initialize the teacher sorry the student at the teacher's optimum okay so this was the paper and you know this is this is the type of research where i do appreciate kind of the these large labs taking it on because they have the resources to do all of these ablations all of these different models cross them with these giant data sets


and so on which i guess university labs just would not have and this is a fairly um thorough paper really investigating which parts of the pipeline you know do something and which ones don't and usually i i'm fairly critical of pipelines that have like 50 billion tricks um because you never know where the improvement exactly is coming from but you can sort of mitigate that


hi there today we'll look at big bird transformers for longer sequences by manil zaire and guru garuganesh at al of google research so this paper on a high level proposes to replace the quadratic attention mechanism in transformers by a mix of random attention windowed attention and selective global attention therefore achieving a complexity of linear memory requirement instead of

Big Bird: Transformers for Longer Sequences (Paper Explained)


quadratic memory requirement and as a result of that they can process longer sequences than traditional transformers like bert and achieve better results in some nlp tasks and they also evaluate on genomics tasks so we'll go through this paper a bit look a bit at the proof because they give a theoretical kind of guarantee that their random attention mechanism can still


be touring complete and can still achieve the same things as a full attention mechanism but we'll also look at the drawbacks i sort of have mixed feelings about this paper and i think i'll voice my concerns as we go through here but first let's look at the paper let's look at the architecture and i think this is actually a pretty cool paper for the empirical progression of the field


to process longer sequences with transformers as always if you like content like this uh feel free to share it around uh leave a like and tell me in the comments what you think about the paper and about what i think whatever you just just uh go nuts all right so the basic uh the basic premise right here is that the transformers they've been pretty impactful especially in nlp so


they say transformer based models such as bert have been one of the most successful deep learning models for nlp unfortunately one of their core limitations is the quadratic dependency mainly in terms of memory on the sequence length due to their full attention mechanism so really briefly the full attention mechanism and i've done you know numerous videos about attention


mechanism bert attention is all you need and so on so if you want a detailed explanation of what that is just go look up the corresponding videos but briefly what you'll have in nlp is a set of tokens a sequence of tokens as an input and you want to transform them layer after layer into sort of a a higher order representation of that same sequence and for that you build


these layers out of nodes and you have as many nodes usually as you have tokens in the sequence and the next set of so each token is represented by a vector at the beginning and each layer transforms this sequence as i said into sort of a higher level representation so you want the vector of this token right here um to be a better representation than the vector


was right here and you do that by incorporating information from all the other tokens into that particular vector now as i said this is called an attention mechanism and we don't actually have to go into how it works right here but you can see pretty clearly that if you want to do this for every token you need to have information routed from every token to every token like from


here to here from here to here and so on and this is just one token and then you need to do it for this token and for this token and for this token so what you'll ultimately get if n is your sequence length you'll get some n squared amount of computation and memory requirements for this so this is a problem and usually this means that you know this sequence length in bert


this is limited to something like 512 tokens which is okay for some applications but if you want to summarize you know entire articles entire books even or do question answering with lots of context it's not really enough so people have been thinking about how to scale this input how to scale this and of course the main culprit is this quadratic tension mechanism because if you you


know double the 512 you need you know four times the amount of compute and memory so how does this paper go about reducing that quadratic dependency the goal right here is of course to get this to some o of n right because then as we double the input length we simply need to double the compute requirements and that would be fantastic and that's what this paper


does and it does so without you know sacrificing the properties of the transformer so here's the architecture that big bird proposes by the way big bird another character from sesame street i guess will continue the naming here after elmo and bert you know i i'm i'm waiting for the model that's the count um yeah that's going to be a fun model but so big bird basically has three different types


of attention and here these are adjacency matrices in this attention mechanism so here is the input layer and the output layer is right here so that basically means that node i right here would be connected well sorry that's not a straight line would be connected to this particular node and also to this particular node so we're now trying if we have node i right here we're now trying to


not connect it to all of these nodes but we'll say we'll just select sum at random and then connect it to that okay this is what we call random attention and you can pretty clearly see if you connect each of the i nodes to r equals two to two random nodes then you don't have an n squared anymore but you'll have a like an o of r times n which you know if r is a constant is


an o of n attention mechanism okay so the main goal between the random attention mechanism is that for each query basically you select random tokens that you attend to and that random number is a fixed number that's not dependent on the sequence length and the paper is a little bit unclear about whether or not those random ones are the same for every sequence or are switched up or


are the same for every layer or are switched up but they formulate all of this as sort of in sort of a graph in sort of a random graph so there they formulate the attention mechanism in form of a graph so if we transform all of these nodes into a graph a full attention mechanism would mean that each graph each node is connected to each of the other nodes right


fully connected graph i don't maybe that's it so that would be a full attention mechanism and then they say well if we just have random connections between these things then there are some theorems from graph theory that say that each random walk in this graph is going to um so this graph is going to mix pretty quickly so i can get from each node to each other node


by a random walk in a logarithmic time and this random walk which basically means that you go from here to here this would be one layer of the transformer and then if you want to go from here to here that would you would have to do that in the next layer so this formulation as a random graph leads me to believe that layer after layer the random attention pattern is going to be


the same but also the formulation of the paper leads me to believe that the this random attention differs from sequence to sequence so i believe what's happening is that they you know get a new sequence then they decide on this pattern right here once and then they use this layer after layer the same pattern again so you can see that um in the traditional attention information can


basically throw flow from each of the nodes to each other node in one single step right because each node is connected to each other node you see this in the graph right here however if we only select a subset then you know it needs to if if i want to go from as i said from here to here then i need to do it in two steps and therefore i need two layers and that's going to be the culprit of


this method here and you know while it is mentioned in the paper it's sort of i feel at least that's my my assessment of this paper it's kind of swept under the rug a little bit i mean they do have a theorem that clearly says we can construct an example of a task that in the full attention setting can be solved with a single step so a single layer that in our


random attention setting needs a lot of layers so a lot of steps but you know the rest of the paper is sort of shaky on on this thing but nevertheless you can see how the random attention can if you have enough layers do the same information routing as the full attention okay however this is not a property of the random attention and we'll see this in the next thing


right here so the next ingredient that this paper uses is window attention and you can see over here that big bird is ultimately going to be a combination of the three types of attention which will uh which we are looking at here so window attention basically means that each each eye each token at the if position is going to attend to itself of course so here is


i but it is also going to attend to its neighbors so here is i minus 1 and here is i plus 1. and this is a you know this is a window size w that you can that is a parameter but also it is a constant and therefore um you again go from n squared to w times n which you know is o of n if w is a constant and this might be familiar to you because we've already seen this in the


long former paper so i've made a video or base i think even two videos on the long former which used exactly the window attention in combination with the global attention and uh if you want to know more about that go watch these videos but the new thing in big bird right here is this re edition of the random attention again the the window here in is is has exactly the same properties


as the random attention so you have instead of a fully connected graph you have a sparsely connected graph now if you have random attention the sparsely connected graph is like like the one on the right but if you have a windowed attention you can it is kind of not randomly connected but each node is connected to its neighbors like this and you can also see that if i want to


go from this node to this node right here i can't do it in one step but i can do it in two steps i go here and i go here so in the terms of the attention layers if i want to go from node one to node three i have to do it in two steps because each node is only connected to its neighbors so the connection patterns would sort of look like this so i have to go from one to two and then in the next


layer from two to three so the paper basically makes up for the lack of full attention by uh adding layers and you also might recognize this from a convolution operation like this basically because it is a convolution operation right in a convolution each node a only aggregates input from its neighbors for the next layer and then we know that as we go up the layers the de facto window that


each node looks at is going to be like a cone kind of like this so this is very similar to how a convolutional neural network works and the reasoning is very similar because the reasoning is well in a sentence the most important words for any given word are probably going to be its neighbors like the words around it and as you go up the layers you branch out


more and more but ultimately the this neighborhood principle holds in nlp as well so again we already saw this in the long former but that's the reason behind the window attention and that's the second ingredient and then the third ingredient is this global attention now the global attention uh is selected tokens that are so important and that's you know fixed by the


developers that are so important that they are they are connected to everything else so for example in these transformers you often have what's you know this kind of cls token so this is a special token that you prepend to some piece of text and the output of this token is going to be your classification output because you don't want to bind your classification if you need to classify


the entire sequence you don't want to bind that decision to one particular word what you want to do is you want to have an extra token and that's this cls token that kind of aggregates information from all of this so layer after layer layer after layer you'll have so if we go here layer after layer we have this one special node and in each step every single other node is able to send


information right here to this node and receive information from this node okay so now uh as a result of this as you as you may be able to see every single uh every single path is kind of a maximum length of two because if i want to go from any node to any other node i can simply you know send information to this global node and then the global node in the next


step can send information to whatever other node and that is a property that they use in their proof that this tension mechanism is as sort of as powerful as the classic full attention mechanism and we'll go through that in one second but first i hope this was clear that this combination of random attention window attention and global attention is what is called big


bird okay they have some engineering tricks that go along with this but in concept you can imagine big bird being long former plus these random attention right here and you know as an engineer as an nlp engineer that makes kind of total sense i you know i totally believe that a the introduction the addition of these random attention of these random attention


patterns can absolutely help your classification or whatever your nlp tasks because you know more attention better and i also am completely willing to believe that you know using the full attention matrix while it is of course more accurate it won't hurt too much to leave some of that attention away because essentially all the path lengths are just becoming too or even with the random attention


are really short or logarithmic to route information from a node to some other node so the loss that you incur is kind of in a logarithmic scale in terms of performance while the gain that you make is sort of in a in a quadratic or like a linear scale you go from quadratic to linear and that seems to me like a good empirical trade-off all right however the the proofs here the


proof of um of how how these how these things are constructed are a little bit i don't know so what they do in the proof that this function can sort of a is a universal approximator people have already shown that full attention mechanisms are universal approximators um so they show here that this sparse attention mechanism is also a universal approximator they make big use of star


graphs what they say is okay if we have a star graph which is one node connected right here to every other node this is a star graph if we have a star graph we can achieve the same thing than with a full graph a full graph is where every node is connected to every other node but as i already said what they need for this is multiple layers of this star graph so


and that has to do with the fact that if i want to route information i basically have to go via this middle node right here and there's an additional complication because this middle node in our case right here is only one node i can't route information at the same t like i can't have this routing right here at the same time that i have this routing right here like


going from here to here because i only have one middle node and i kind of this is not how the like this is very dumb math but uh maybe you have to imagine that there is one memory slot and you can only use that one memory slot at the same time for one of these things so essentially what you'll have to do is you'll have to do the green thing first and then in the


next step you'll have to do the blue thing second and then so these are now pairwise routing between nodes but ultimately what an attention mechanism does is it does everything to everything right in a single layer it routes information from all the nodes to all the other nodes and to achieve that so you need multiple rounds of this and it turns out that in the worst case


you actually need n rounds of this so you know you trade off your you go from n square to n uh memory and compute requirements in a single layer but in the worst case you need n layers to recover the the power of the full trend of the full transformer and that is the last one of their theoretical results right here so first they prove universal approximations


and second they prove turing completeness these two properties have been proven for full attention mechanisms and third they prove that there are tasks where you actually do need n layers to solve them with their limited attention um so you know i'm not sure but i i feel you can make any sort of polynomial uh algorithm into a linear algorithm like this like i


have a i have like a cool sorting algorithm right so if this is my sequence that i want to sort what i can do is i can simply you know take a random subset of them uh like this this and this and then kind of go and and sort them and then put them like i send them to the to the global memory like this i sort them and then i put them back right and if i do this for enough


if i do this for enough rounds okay you know if i do this for enough rounds you know at the worst case i need n rounds to sort my or log n rounds if i do it smartly but you know in you know the single step here is uh the single step is just o of n so i have now an o of n sorting algorithm i you know i have my sort of a bit of wary to express things like that


and um yeah but you know it is from an empirical standpoint i absolutely believe that this uh this is enough now my second quarrel right here is that if you look at the proof first of all what it makes use is this star graph and the star graph corresponds to the global attention so that's not much to do with the random attention though they use the random intention in their proof but


i at least believe that it would be possible with the global attention only and then the second thing is if you look at the parameters that they use for the um for the experiments and i've already set this in the long former video so in the long former video it turned out that if you look at how big this window attention is it turns out that it you're still well you know the original


bert attended to 512 tokens and then you look at the window and the window was still 512 tokens it's just that the global attention was even more so ultimately they ended up using more memory than the original bird and here if i look at the parameters of their um thing and they have multiple experiments right here and i believe this is the the base version so this is the base


version they also have this large version but here this is the 12 layer um version and you can see they have this block length and we'll get into the block length in one second but then you can see that their window size is three times the block length the number of random tokens is three times the block length and the number of global tokens is two times the block


length so that results in eight times b so 8 times 64 is you know can i calculate this or am i stupid uh it's 512 yes actually calculated this before so this is 512 tokens so you know you you go from from bert that has 512 tokens and attends to 512 tokens to also attending to 512 tokens of course the advantage here is that they now have 4 009 and 96 sequence length so they have the


freedom to not attend to as many tokens as they have in the input length but you know to put it in perspective this here uses more memory and more compute on it on its face than bert because bert attends to as many tokens but has a smaller input sequence and you know i i there's sort of a thing where in order to make these sparse attention things work you have to go pretty pretty you


know high in the number of things you attend to you can leave away some but it's not like you can you know scale up orders of magnitude of your input sequence length so that's this promise of linear attention is sort of it's kind of fulfilled but not there yet the second thing i i would like to point out is that in a lot of cases the number of random tokens is actually set to zero


so really making use i believe of these of the of the global of the number of global tokens so it's that seems a bit strange in that they continuously refer to their random attention mechanism um but then in a lot of experiments they don't actually have a random attention mechanism i believe they have to do that because that's kind of what makes them different from the long


former in principle but still um yeah so the last novelty let's say is an engineering novelty in that they now always consider not single for example they don't consider single random attention they always consider these in blocks and that's because our current hardware is really bad at sparse stuff really bad at single indexing gathering single things so if you can do everything in blocks you


basically get you get these blocks almost for free so it takes only marginally longer to retrieve this full two by two block right here than it would to retrieve the single uh instance right here of course that means you have you know four times you still use four times more memory but it is not four times slower than the original thing so you can use these blocks uh right here


you can do it for the random attention you can do it for the window attention as you can see here so you break this window pattern a little bit into blocks and that makes it a lot faster or that speeds up i get the speed up almost for free and then they make another approximation in that the way they do this windowing is and now let's just go really briefly so


you can see right here that it would be very cumbersome to gather so what we need we're just going to focus this this dotted thing right here is a bit confusing so you want to attend to these things and these you can just get out with a matrix slice really easy but then you want to attend to this kind of blocky thing right here from the window attention right like this thing


and this is hard to get out because you'd have to kind of index each row individually and that's very slow so what they do there is this matrix roll operation where you can sort of roll the axis around so what you'll do is you'll take this thing right here and you put it to the left right here and you'll take for example this thing right here and you'll put it to the right or


no like it's it's up and down but in essence that's what you do and you can you can fold all of this blue stuff into a rectangular matrix if you know if you can see right here so you kind of roll this back roll this back roll this forward and you replace whatever is missing by these now this again gives you some inaccuracies because this block right here was never intended


to be attended to and all of a sudden you see you have the k6 in here so it gives you a bit of inaccuracies at the edges of the sequence but you can take that you know you can take that hit for the increased performance that you gain by now having a rectangular matrix tpus are really efficient at this not as efficient as this and then the only thing that's really slow is gathering these


random blocks right here but also by having the same amount of random blocks per input token what you'll do is you'll end up with just one of these columns right here or you know r of these columns and that again gives you a rectangular matrix so this thing right here you can process very very efficiently using a tpu and you know the mistakes you make are basically


this thing right here and this thing right here because those weren't intended and are at the edges of the sequence so these were the the tricks of big bird to quickly uh summarize uh big bird is basically taking a transformer saying well why do we need all of this attention all of this full attention maybe we only need some of that and can already do a big job a good job


especially you know considering the attention mechanism goes over multiple layers so we don't need a routing from each token to each token we we can make up for not having a fully connected graph by simply running multiple layers so their sparsity is first of all you have this random attention which i believe changes from sequence to sequence but stays within or among the layers of


the same sequence then you have the window attention with the reasoning so the reasoning behind the random attention is that if you have a randomly connected graph the path lengths are on average logarithmic so you can route information efficiently the reasoning behind the window attention is that probably neighbor information is very important and that has been shown


empirically and then the global attention the reasoning behind this is that some of the tokens that are fixed by the developers are so important that it it's very beneficial that each other node is connected to them and that they are connected to each other node the result of that is the big bird attention mechanism which is basically long former which already had these two


plus the random attention this achieves a linear linear complexity in terms of of memory and compute though linear has to be qualified a bit because it's modified by the window size by the number of random attention tokens by the number of global tokens and in practice often ends up being you know fairly large-ish and also the the theoretical guarantees now come with the fact that you need


multiple layers in the worst case you need sequence length amount of layers which you know in the worst case would result right back into a quadratic requirement for memory and compute they do some engineering some engineering tricks right here and their results are pretty good so the results in various tasks and we'll we'll look at some of the tasks right here


so these are def set results using base size models for example where you can see they do outperform basic roberta models they outperform long former which may mean that the random attention is useful but you know in these things it also always may just mean that you've thrown more compute at it um at least i'm not really looking that they outperform the models because as


you can see right here if they compare to state of the art and you know granted these are models that have been trained specifically for these tasks and are you know crafted and engineered and big bird manages to big bird manages to hold itself against them in a lot of tasks and even gets state of the art on some what i'm more interested in is that it you know it can reach good numbers it


doesn't necessarily have to be state of the art but it can reach good numbers which tells me that okay probably the the empirical hit that i take by not having the full attention is you know is justifiable by the speed up and memory savings i do get um yeah especially when result when you see results mixed like this you know sometimes the other model is good and sometimes the


big bird is good on different variations and so on i would not you know i would not make a big deal out of the fact that it is state of the art i get that the authors have to do that i would do so as well but you know um you know don't don't think that this is the like the best thing now uh it's very probable they just thrown also a lot of compute at it what is cool is


they do uh some genomics experiments so not only do they have nlp state of the art but also they go into genomics and experiment with data there don't want to go into that because you know ultimately it's another task and that we leave the papers about the architecture all right so that was big bird i hope you enjoyed this video and learned i learned something certainly


okay I'm sure many of you have already seen this because it was rather widely announced but the open AI team has announced a new model that produces pictures instead of text so as you can see right here on the Left you'll always see like a half a picture and on the right is the ground truth so they took this picture they simply cut the bottom half right here and then they let the model sort of

Image GPT: Generative Pretraining from Pixels (Paper Explained)


imagine what they cut away and what it comes up with is pretty cool I have to say like look at the birds like this is just awesome but the special thing about this isn't that it simply completes pictures the special thing about it is it does it one pixel by pixel so basically it goes at this pixel right here and asks ok what's that pixel and then what's that pixel and then what's


that pixel and so on so it is basically a like a language model but four pixels in that it goes over the images in order basically like this or like always from left to right left to right left to right and it has no clue of the spatial relations between the pixels it needs to learn that by itself as opposed to a convolutional neural network which is specifically designed such that if you


want to predict this pixel right here then it's specifically designed to say ok the most important information is probably around that pixel and then some like other important information is while around that pixel so cnn's are built with this in mind whereas this model right here which is also known as image GPT isn't doesn't have any of that it's simply a transformer model that goes over these


pixels one by one and we'll see how that's done there are some more examples right here particularly cool is the cat and you see that there is the beginning of this little white thing here which is this card and the completions of the model yes very interesting the model can of course as a language model can also sample by itself just random images you sample them once


through and this is what it comes up with so these are pretty good quality images for a model that just produces one pixel by one pixel now this idea of one pixel by pixel isn't new this has been around before but the investigation here is basically how much can we how far can we push these generative models for pre-training hi there this is Yannick from post-production I've


realized that I've forgotten to even read the name of the paper so it's called generative pre-training from pixels by March and Alec Radford round child jeff whoo he won't jus profile a dairy wall David Lewin and Ilya sutskever and since Henry AI labs has already made a video on this this video is going to be more of kind of a rumble rant about what I find interesting about


the paper and some thoughts about it rather than like a classic explanation I hope you still enjoy that so what you saw on the right wasn't even though this isn't the final result the supposed result this is simply the pre-training task it's fun to look at it but the actual object objective of the paper is the following what if we train we pre train on a large


data set to generate work good images like these or we to complete images like these and then we fine-tune on a classification task and the answer is here they say on C 410 we achieve 60 96.3% accuracy with a linear probe outperforming a super wide supervised the wide ResNet and the 99 cent accuracy with full fine tuning matching the top supervisor pre-trained


models an even larger model trained on a mixture of imagenet and web images is competitive with self supervised benchmarks on image net achieving 72 top one accuracy on a linear probe of our features so the goal here is that you have a data set that you want to trend like train a classifier on so usually you have a data set and the data set has images and you put them through like a


convolutional neural network and then you have to classify the image into one of I don't know how many classes on C for ten that's ten classes on image and it's a thousand and the data set is these images together with these labels now the idea of pre training is that you some where have a bigger data set that is sort of similar to the small data set but it's similar enough such that the


network could learn something so what you want to do first is your first one it take the large data set terrain this network right here and then in a second step fine-tune the network on this smaller data set and you sort of hope that what you learned from the large data set right here transfers over a little bit of knowledge you already have a little bit of knowledge and you can


make better use of the data that you have right here now the question is how do you do this pre training and of course this has a long tradition well long for maybe two or three years right now in the language community where people they pre trained these large models like we've just seen GP t3 or Bert was one of them they pre trained these large Transformer models on text


and then to fine-tune them on classification tasks for text and that's what this paper is doing right here they pre trained a transformer that is a GPT to scale model they pre train it on image generation and then they fine-tune it or transfer learn it to classification tasks and the point of the papers to say that like in text data in text data we have made


pretty good pretty good experiences with doing this with pre-training a generative model and then fine-tuning on a classification task while so far in images all we've ever done is we've pre-trained this pre training task he usually is a classification task or like a self supervised task with a contrastive loss or something like this what they're doing new is the generative


modeling in the pre as a pre training and again this isn't like entirely new but they show that if you throw a lot of computers at it and lots of data and a big model then that can work equally well to these self supervised tasks so their model as I said is pretty pretty simple they take an image and they unrolled the image now an fully unrolled image on let's say image net has 224


squared pixels and that times three right because you have three color channels that's too large even for an open a supercomputer so what they do is first they down scale the image so they down scale it's not as drastic as here where you just get a three by three image but they do down scale it to like a 32 by 32 or a 64 by 64 then they unroll it which simply means they go


through the image like this and make a sequence out of it because their models are naturally made for text sequences they simply put the image into a text sequence they further simplify this by reducing the three colour channels to a single one so they have their own color representation and basically yeah they reduce the three colour channels to one channel that simply indexes the color in


their color representation and they say still pretty good it's pretty faithful so ultimately they end up with like 32 squared length representation of their image and then they do one of two things they either do auto regressive generative pre-training which is the sort of GPT - style pre-training and the the idea here is that you always want to predict the next pixel of a sequence so


you can see right here that's the sequence that you are sorry that's the sequence that you input and you always want to predict what is the next pixel and in this case you've seen you see that we've already predicted everything here we've already predicted everything up to this red pixel so you want to know what's this next pixel this thing right here what's this going to be and the


diagram here basically shows you how the attention flow so every position in this transformer and if you don't know what a transformer is I haven't made a video about attention is all you need where these are explained but briefly every position here can sort of send information can send information only in one direction as to so you train all of these in parallel and when you predict


this pixel right here you only want information from whatever it was before that pixel otherwise the model could cheat right otherwise the model could simply learn to copy over the value but the attention pattern here is simply to show you that this is auto regressive and it's in one direction so you always want to predict the next pixel and then from all of this you want


to predict the next pixel and from all of this you want to predict the next pixel this is in contrast to this objective here that comes from Bert and I've also made a video on Bert what you do in Bert is you simply take that image and you cross a block out two of the pixels or many of the pixels and you simply ask your network to reconstruct those pixels okay and now you can see the


attention flows in all direction birth the B stands actually for a bi-directional so this is the contrast to the autoregressive pre training framework now the these two things have been applied in text both the autoregressive is usually it's easier to actually make it produce something like we saw producing these images because you can always just predict the next


pixel and then the next and then the next and then the next whereas in Beart it's a bit more unclear how you would produce things in a consistent manner because the predictions of these two pixels right here they are independent it's one forward pass and then both of these are predicted but other papers have tried to solve this like this not excel net I forget I forget its name it's something


with an X and yeah but but these are the two objectives they look at and it turns out they sort of trade off a bit they work equally well or a bit better and a bit worse depending on the task so once they have done this so they simply feed images and you will notice that you don't need any labels for this so what you'll do is simply input an image and then simply take away half of it like


this and then predict that pixel and then you want to predict that pixel and then you want to predict that pixel right that's all like you do with text and invert you simply input an image cross out pixels and then predict them so you don't need labels for this and that's why you can do it with this big data set and you can do it in an unsupervised fashion so you can just


crawl the internet for four images and just feed this in into there and it will sort of learn to produce these images now the question is if you produce if you learn to produce these images does that help you for classification and there they have two methods of of assessing this the bottom one here is the fine-tuning method where you simply so this is supposed to be the


representation you learn in the different layers of the network so this is supposed to be this thing right here what you'll do is you'll simply fine-tune that means you on top of this representation you add a classification head that has two outputs cat or dog and you train this entire network on your small data set that we discussed before so you train the entire network all of


the parameters this is called fine tuning in contrast to that what you can do is you can simply and this is the easy way you can simply add this classification head with two outputs and then only train this classification head and that is won't perform as well but it gives you sort of a better idea of how good is the representation that this network right here learned and on top of


that so if you spin this idea further you can actually go and do this at any intermediate layer right here so you can forward propagate until layer two right here and then here you add your classification head into the two into the two classes and you only train the classification head that being said you can also do this with fine tuning but in this case this is called a linear probe


and it is often used to assess how good the a representation in intermediate layers is whereas what it actually does is assessing how linearly classifiable a representation is which isn't the same as how useful or how informative but it is one way to to assess these things okay so these are the two things they assess alright so for as for data sets for C 410 they use like C for 10 and c


for 100 as data sets and the STL 10 and there you have to keep in mind the pre training is done on imagenet for those so that you pre train on imagenet without the labels and then you transfer learn or fine tune or or linear probe on these small data sets whereas later we're going to look at image net and they're the pre-training as I understand it is done on image net


itself but also a wider collection of a hundred million or so images from the web from the internet okay so as you can see right here this is what happens if you do this linear probing and you can see it works pretty well so you get like a ninety-five ninety-six percent accuracy with linear probes this is very powerful so it's not easy to get 96 percent on C for ten I mean current


state of the art is like ninety nine percent but still 96 percent is pretty good and this is the so the entire network there is this big giant network that you input your image into and then there is this one linear layer that does the classification and all of this right here has not been trained with classification in mind it simply has been trained to reproduce images it


hasn't even been trained on C for ten as far as I understand has been trained on image net so the the this is to stress how cool or how significant this result is basically that just a linear probe on top of that will give you such a good accuracy and the second thing that is obvious right here is this bottom axis is the layer so this is the layer where they attach the linear probe and usually


if you pre train a network with a classification task in mind so you pre train it with the labels or maybe even without the labels in a self supervised the way or something like this usually the last layer has the best representation for classification but here the special thing is that the intermediate layers in the middle have the best representation you can see that


representation quality in terms of linear probing falls off as they sort of it falls off as they go into higher layers and this is consistent across the datasets as you can see and the the idea here is or the way they interpret it is that if you have an image right here Dada Dada Dada and they you've blocked part of it so you've blocked this and this wrong way around this


so you've generated everything and now your task is to predict the next pixel right so you're you train to predict this next pixel right here and the idea is that as you put the image through the network what it will do is sort of since the first layers they're going to be if you're going to be similar to a CNN they're going to be doing some low level of feature transformation thing right


but also the last layers they're going to really care about what's the exact pixel that goes here right since it's their job to to do that they're going to care what color does it need to have you know what exact luminosity and so on how does it fit in with the previous pixels and so on where as so that's that's also good but it's not just low level information and consistency with other


pixels or something like this at some point if you want to generate consistent images and we saw that this model can generate consistent images at some point there needs to be some kind of a notion of the global information in the picture right such because the images are consistent throughout so there needs to be some some notion of what is in that image as a whole and that's the exact


information that we need for classification and the only way that could actually be is here in the middle since you know that's the place so the hypothesis is that the these models somehow learn a higher-level representation of global information somewhere in the middle before they then specify that information again down to predict the actual pixel and that's why


the best representations for classification are in the middle so this is one of the this is actually the interesting finding or one of the interesting findings of this paper means cool that they can reach a good accuracy but to recognize that maybe in these these generative models they have some intermediate stage where they represent the global information and that will


actually make the best representation okay the second cool thing right here is that you can see they have different sizes of models so the IGP TL I believe is something like sixty layers then this is like 48 layers and this is 32 layers we don't really so these are on the olive on the scale of GPT to either a little bigger or a little smaller it's not like a GPT three scale where you


need a ginormous supercomputer though they do do a lot of computation but it this still sort of fits within hardware of a standard size and not like exascale what's interesting right here is that you can see the larger models they reach a lower validation loss so here is the validation loss larger model if you train them on so these checkpoints here are always after the same amount of


steps the larger models do reach a lower validation loss right here as you can see so this is the large this is the medium this is the small and also you can see that on this axis is the linear probe accuracy so this is whenever you you go and you find the best intermediate layer for linear probing you probe it and you record the accuracy so you can see a general trend


as your validation loss goes down the linear probe accuracy goes up so there is a connection like it is in text models and text models there's a connection of the perplexity of your language model and the quality that of the representation you get for downstream tasks in this model it seems to be the exact same thing there is a connection between reaching lower


validation laws and reaching a higher performance on classification so that's one interesting thing the general trend - up to the upper right corner the other interesting and even arguably even more interesting thing is that if you look at the same validation loss so at this point all of these models have the same validation all's yet still the bigger model is better right you can see right


here the bigger model outperforms the smaller model even though they have the same validation loss on the image modeling task and this is also something that openly I in their in their text papers has stressed that the larger models they seem to be somehow more capable of forming good representations even you know even if they have the same loss so again this this could just be


sort of a training data remember training data remembering thing and when I set that in GPT three I didn't actually mean explicit remembering of training data I meant a kind of a fuzzy remembering of training data of I formulate that in the in the comments but I I feel a lot of people have misunderstood me there here I think it's a much harder harder to estimate what's going on also since


image pixels humans don't have a super good model of an on image pixels in their head as we have about text as you can see if you then fine tune so for now we've just do linear probing if you fine-tune these architectures then you reach like a 99% accuracy on C for ten which is on par with the best models that we have so G pipe is supervised pre trained on imagenet but also I guess


uses a bunch of data element ation while these image GPT it uses minimal data augmentation I think they simply random crop a little bit and that's about it so they also experiment around with this Bert objective so until now this was all the this was all this autoregressive objective and I feel the open hi people are a bit more of a fan of the autoregressive objective just given what


they've done so far in their papers and you can see here comparison of the two objectives on C for 10 and on image net again C for 10 is pre trained with image net and image net itself is pre trained with like a larger collection of images from the web all the pre training is done without labels now the blue is what you can reach with a linear probe and the orange is then on top of that what


you can reach by fine-tuning okay so no linear profile tuning oh I have to say that the fine tuning is always done at the end so even though the linear probe even though the linear probe can be attached anywhere in between and it's often useful to do that as we saw because the in between layers are the best they say they tried fine-tuning also in from in between but it always worked out


best whenever you fine-tune whenever you fine-tune you take actually the last layer so that kind of gives you an idea that the model is then it's sort of what seems to be important is this coming up with the higher level representation and then once you fine tune you're probably able to push that representation through to the end because of your training signal


but if you hadn't done the pre-training you wouldn't even have that higher level representation and then the signal I guess is not strong enough to back propagate through the whole model it would be very interesting if they investigate if they do this linear probe analysis again after they fine-tune the model that and to see if then still it is the intermediate layers that have the


best representation or if now the best representation in a linear probe sense shifted towards the end I'm gonna guess it's shifted towards the end but I sort of want to even see if the accuracy of the linear probe in the middle does it keep the same right so does the curve go like this this is the linear probe when you simply pre-trained right this is linear probe accuracy the


question would be does it change to be like this or does it change to be like this this is supposed to be the same at the end so basically does it stay as good as it is but simply get better at the end or does the representation like in this curve does the good representation now shift towards the end and leave the lower layer with even more capacity to do some low level stuff yeah


maybe they've done this I haven't seen it so and as you can see these Bert and autoregressive object if they sort of trade off so the birthda tends to do poorly in the linear probe setting but then it catches up during fine tuning in C for 10 almost being at the level of the autoregressive and in an image net actually outperforming it this this darker thing here it simply means that


you a ver across different maskings of Bert because I guess even in classification it's not entirely clear how to get a signal out of Bert because they don't do this CLS vector with Bert what they do for classification and linear probing and it's written up here they simply take the they do an average pooling I think they do an average pooling of the of all the representations of the sequence and


the last thing that I've also forgotten there's a lot of stuff when they fine tune while fine-tuning while fine-tuning the classification loss yields reasonable downstream performance we find empirically that the joint objective the generative objective and the classification objective works even better okay so even when you fine-tune with this model you have to keep the


generative modeling part the generative loss around and then it performs even more better more well whatever that word is so that's also something to think about I think this this paper right here it kind of lays down a lot of cool things that you can think about and it gives rise to a lot of hypotheses of how does this stuff work why does this stuff work I don't even think that the the


numbers are the most important thing it's mostly the fact of the effects and what does it mean okay so this was my take on it it was it's more kind of a my rant of what I find special about this paper then about the actual paper you can look at the paper their numbers are pretty good on imagenet they do not reach the same like SuperDuper performance as they do on c 410 and i


guess that's probably because they have to downscale the image net images way more than they have to downscale to see 410 images because those are of course only 32 by 32 so because they have to downscale so much they lose probably a lot of information and I would be interested to see if there is a way to involve convolution in this in all of this so to do the


downscaling that in a learned manner with convolutions or something I'm sure this has all been done already I'm just lazy to look it up yeah so I invite you to look at their blog post where they have these his samples they look pretty pretty funny and these full samples up here look fairly you know fairly cool for what it's trained to do and that it has no spatial awareness whatsoever it


Transcriber: Leonardo Silva Reviewer: Denise RQ So, we all have some behavior that we would like to change about ourselves. And we certainly all want to help someone else change their behavior in a positive way. So, maybe it's your kid, your spouse, your colleague. So I want to share some new research with you that I think reveals something really important about what gets people to change their behavior.

How to motivate yourself to change your behavior | Tali Sharot | TEDxCambridge


But before I do that, let's zoom in on one strategy that I think you probably use a lot. So, let's say you're trying to stop yourself from snacking. What do you tell yourself? Well, most people, in a monologue, will say, "Beware. You'll be fat." And if this was your kid, you would probably tell him that smoking kills and, by the way, he's in big, big trouble.


(Laughter) So, what we're trying to do here is we're trying to scare ourselves and others into changing their behavior. And it's not just us. Warnings and threats are really common in health campaigns, in policy. It's because we all share this deep-rooted belief that if you threaten people, if fear is induced, it will get them to act. And it seems like a really reasonable assumption,


except for the fact that the science shows that warnings have very limited impact on behavior. So, graphic images on cigarette packets, for example, do not deter smokers from smoking, and one study found that, after looking at those images, quitting actually became a lower priority for smokers. So, I'm not saying that warnings and threats never work, but what I'm saying is, on average, they seem to have a very limited impact.


And so, the question is: why? Why are we resistant to warnings? Well, if you think about animals, when you induce fear in an animal, the most common response you will see is freezing or fleeing; fighting, not as much. And so, humans are the same. So if something scares us, we tend to shut down and we try to eliminate the negative feelings. So, we might use rationalizations.


For example, you might tell yourself: "My grandpa smoked. He lived to be 90. So, I have really good genes and absolutely nothing to worry about." And this process can actually make you feel more resilient than you did before, which is why warnings sometimes have this boomerang effect. In other times, we simply put our head in the ground. (Laughter) Take the stock market for example.


Do you know when people pull their head out of the ground to look at their accounts -- not to make a transaction, just to log in to check their account? So, what you're seeing here, in black, is the S&P 500 over two years, and in gray, is the number of times that people logged in to their account just to check. And this is data from Karlsson, Loewenstein & Seppi,


it's control [data] for all the obvious confounds. So, what do we see? When the market is high, people log in all the time, because positive information makes you feel good, so you seek it out. And when the market is low, people avoid logging in, because negative information makes us feel bad, so we try to avoid it altogether. And all this is true as long as bad information can reasonably be avoided.


So, what you don't see here is what happened a few months later, in the financial collapse of 2008, when the market went drastically down and that was when people started logging in frantically, but it was a bit too late. So, you can think about it like this -- it's not just finance: In many different parts of our life, (Laughter) we have warning signs and bad behaviors now.


And they could potentially lead to all these bad outcomes later, but not necessarily so, because there are different routs from your present to your future, right? It can go this way, it can go that way. And, as time passes, you gather more and more information about where the wind is blowing. (Laughter) And, at any point, you can intervene and you could potentially change the outcome,


but that takes energy and you might tell yourself: "What's the point about worrying about something that might happen? It might not happen." Until we reach this point, at which time you do jump into action, but sometimes it's a little bit too late. So, we wanted to know, in my lab, what type of information does leak into people. So, we conducted an experiment


where we asked approximately 100 people to estimate the likelihood of 80 different negative events that might happen to them in the future. So, for example, I might ask you: "What is the likelihood that you'll suffer hearing loss in your future?" And let's say you think it's about 50%. Then, I give you the opinion of two different experts. So, expert A tells you:


"You know, for someone like you, I think it's only 40%." So, they give you a rosier view of your future. Expert B says: "You know, for someone like you, I actually think it's about 60%. It's worse." So, they give you a bleaker view of your future. What should you do? Well, you shouldn't change your beliefs, right? Wrong. What we find is that people tend to change their beliefs


towards a more desirable opinion. In other words, people listen to the positive information. Now, this study was conducted on college students, so you might say: "Well, college students are delusional, right? We all know that." (Laughter) And surely, as we grow older, we grow wiser. So we said: "OK, let's test that. Does this really generalize? Does it generalize to your kid, to your parent?


Does it generalize to your spouse?" And so, we tested people from the age of 10 until the age of 80, and the answer was yes. In all these age groups, people take in information they want to hear -- like someone telling you you're more attractive than you thought -- than information that they don't want to hear. And the ability to learn from good news remained quite stable throughout the life span,


but the ability to learn from bad news, that changes as you age. So, what we found was that kids and teenagers were the worse at learning from bad news, and the ability became better and better as people aged. But then, around the age of 40, around midlife, it started deteriorating again. So, what this means is that the most vulnerable populations, kids and teenagers on the one hand, and the elderly on the other hand,


they're the least likely to accurately learn from warnings. But what you can see here is that it doesn't matter what age you are. You can be 20, 30, 40, 50 or 60; everyone takes in information they want to hear more than information that they don't. And so, we end up with a view like this of ourselves. (Laughter) Our mistake as teachers, as mentors, as employers


is that, instead of working with this positive image that people so effortfully maintain, we try and put a clear mirror in front of them. We tell them: "You know, the image is just going to get worse and worse and worse." And it doesn't work. It doesn't work because the brain will frantically try to distort the image, using Photoshop and fancy lenses, until it gets the image it's happy with.


But what would happen if we went along with how our brain works and not against it? Take handwashing, for example. We all know that handwashing is the number one way to prevent the spread of disease, and this is really important in hospitals. So, in a hospital here in the United States, a camera was installed to see how often medical staff do, in fact, sanitize their hands


before and after entering a patient's room. Now, the medical staff knew a camera was installed. Nevertheless, only one in ten washed their hands before and after entering a patient's room. But then, an intervention was introduced: an electronic board that told the medical staff how well they were doing. Every time you washed your hands, the numbers went up on the screen


and it showed you your rate of your current shift and the rate of the weekly staff. And what happened? Boom. Compliance raised to 90%, which is absolutely amazing. And the research staff were amazed as well, and they made sure to replicate it in another division in the hospital. Again, the same results. So, why does this intervention work so well? It works well


because, instead of using warnings about bad things that can happen in the future, like disease, it uses three principles that we know really drive your mind and your behavior. Let me explain. The first one is social incentives. In the hospital study, the medical staff could see what other people were doing. They can see the rates of the shift, the rate of the week.


We're social people, we really care what other people are doing, we want to do the same and we want to do it better. This is an image from a study that we conducted, led by PhD student Micah Edelson, and what it's showing you is a signal in the emotional center of your brain when you hear about the opinion of others. And what we found was that this signal can predict


how likely you are to conform at a later time, how likely you are to change your behavior. So, the British government are using this principle to get people to pay taxes on time. In an old letter that they sent to people who "forgot" to pay taxes on time, they simply stressed how important it was pay taxes, and that didn't help. Then, they added one sentence,


and that sentence said: "Nine out of ten people in Britain pay their taxes on time." And that one sentence enhanced compliance within that group by 15%, and it's thought to bring into the British government 5.6 billion pounds. So, highlighting what other people are doing is a really strong incentive. The other principle is immediate rewards. So, every time the staff washed their hand,


they could see the numbers go up on the board and it made them feel good. And knowing that in advance made them do something that they, otherwise, may not want to do. Now, this works because we value immediate rewards, rewards that we can get now, more than rewards that we can get in the future. And people tend to think it's because we don't care about the future,


but that's completely wrong, we all care about our future, right? We want to be happy and healthy in the future, we want to be successful, but the future is so far away. I mean, maybe you'll behave badly now and you'll be fine in the future, and maybe you'll be altogether dead. (Laughter) So, the here-and-now you would rather have that tangible drink, that tangible T-bone,


rather than something that's uncertain in the future. If you think about it, it's not altogether irrational, right? You're choosing something sure now rather than something that is unsure in the future. But what will happen if you reward people now for doing actions that are good for them in the future? Studies show that giving people immediate rewards make them more likely to quit smoking,


more likely to start exercising, and this effect lasts for at least six months, because not smoking becomes associated with a reward, and exercising becomes associated with a reward, and it becomes a habit, it becomes a lifestyle. So, we can reward ourselves and others now for behaving in ways that are good for us in the future and that's a way for us to bridge the temporal gap.


And the third principle is progress monitoring. So, the electronic board focused the medical staff attention on improving their performance. This is an image from a study that we conducted, that shows you brain activity suggestive of efficient coding of positive information about the future. And what we found was that the brain does a really good job at this,


but it doesn't do such a good job at processing negative information about the future. So, what does this mean? It means that, if you're trying to get people's attention, you might want to highlight the progress, not the decline. So, for example, if you take that kid with the cigarette, you might want to tell them: "You know, if you stop smoking, you'll become better at sports."


Highlight the progress, not the decline. Now, before I sum up, let me just share this small anecdote with you. A few weeks ago, I got home and I found this bill on my fridge. And was really surprised because there's never any bills on my fridge. So, I was wondering why my husband decided to put that on our fridge. And so, looking at the bill, I could see that what this bill was trying to do


is get me to be more efficient with my electricity use. And how was it doing it? Social incentives, immediate rewards and progress monitoring. Let me show you. Here are the social incentives. In gray is the energy use on the average energy use of people in my neighborhood. And in blue is my energy use, and in green is the most efficient neighbor. And my reaction to this was --


my immediate reaction was: "I'm a little bit better than average" (Laughter) -- a tiny bit, but still... and my husband had exactly the same reaction -- and "I want to get to the green bar." And then, I got a smiley face. That was my immediate reward and it was telling me, "You're doing good," and it made me want to put this on my fridge. (Laughter) And although I have this one smiley face,


I can see an opportunity there to get two smiley faces. (Laughter) So, there's an opportunity for progress and it's showing me my progress throughout the year, how my energy use changes throughout the year. And the last thing this bill gave me: it gave me a sense of control. So, it gave me a sense of I was in control of my use of electricity. And that is a really important thing,


if you try to get people to change their behavior, because the brain is constantly trying to seek ways to control its environment. It's one of the principles of what the brain is actually doing. And so, giving people a sense of control is a really important motivator. OK. So, what am I not saying? I'm not saying that we do not need to communicate risks, and I'm not saying that there's one-solution-fits-all,


but I am saying that, if we want to motivate change, we might want to rethink how we do it, because fear, the fear of losing your health, the fear of losing money, induces inaction, while the thrill of a gain induces action. And so, to change behavior in ourselves and in others, we may want to try these positive strategies rather than threats, which really capitalize on the human tendency


well I'm honored to be here and my hat is my favorite hat I think it makes me look rather handsome this hat is made from a mushroom called Amadou Amadou is a birch polypore and a Madhu is a hardwood conch and this mushroom is responsible for human survival not too long ago there's no doubt that we all came from Africa we went north we discovered something new called winter

Mushrooms as Medicine with Paul Stamets at Exponential Medicine


oops this mushroom allowed for the portability of fire moreover you can hollow this mushroom out put embers of fire inside and carry fire for days and the fire keeper of our clans thousands of years ago were absolutely critical for the clans survival well this mushroom has other properties and when you boil this mushroom it delaminates and becomes mycelium a fabric and since


some ladies in Transylvania have kept this tradition alive so this threat of knowledge has carried forth over thousands of years and so many threads of knowledge have been interrupted because of famine disease and war well this mushroom is first described by Hippocrates in 450 BC II as an anti-inflammatory as well as for Carter izing wounds another mushroom I brought


mushroom friend of mine also is a polypore wood conch and this is a Garak on a gerakan is the longest living mushroom in the world rose exclusively in the old-growth forests now presently only known from Northern California Oregon Washington and British Columbia in a sky island or two in Central Europe it was described by ascribe ease in the very first materia medica as Alexei reom


at Langham vitam the elixir of long life and it was suggested thousands of years ago as a treatment against consumption later to be known as tuberculosis so I'm going to take you on a journey and I'm going to take a radical left turn halfway through this talk and I'm going to present some data that has never been shown any to anyone else outside of my research team so I am


honored to be representing triple a s on June 9th this year I was awarded as the invention ambassador this is great it's like a first audience that knows what Triple A s is so I don't have to explain that but it's a huge honor and I grew up in a small town in Ohio and my brother John with my got me into science and he went on to Yale my brother Bill one on the Cornell and we had this incredible


laboratory in the basement which they would not let me have access to but they went off to college and I suddenly had this fully equipped laboratory including the the radio from the aircraft carrier the intrepid my father was on it and after world war ii he got the radio so I was listening to all sorts of things behind the Iron Curtain I was just having a fabulous time so my dream was


always to live in the country and be a scientist and have my own scientific laboratory well on June 9th I got that award my brother John we were competitive and there's you know like brothers are you love him 80% of the time and 20% of time they kind of piss you off and so John really never respected you know my interest in my ecology what is this mushroom stuff but


so when Triple A yes gave me this award it was highly vetted and I said Wow you know I can I get this is exciting I can tell my older brother now John I got this award so I called him he didn't answer and then I emailed him and that was a day they discovered his body John I died from cardiac arrest standing up and I just want to tell all of you you have brothers and sisters that bug you


you know think about the good times and how our life is so precious and so short so this talk is dedicated to my brother John who first got me into science so my main theme is biodiversity as biosecurity this is we live in I live in Washington State in the southern regions of the Puget Sound and I want to point out the largest organism in the world is a mycelial mat 2,200 acres in size over


two thousand years old and it's one cell wall thick surrounded by hundreds of millions of microbes per gram of soil we have several skin layers to protect us from infection the mycelium has one and yet achieves the largest mass of any organ in the world how is that possible well it's possible because it the is involved in I own his own microbiome it selects beneficial bacteria that it


works in concert with and the mycelium has based on a network like design the mycelium is digested nutrients externally we share more common ancestor with fungi than we do with any other Kingdom six hundred and fifty million years ago we split from fungi and there is an announcement a new super Kingdom it's been published called a pice the canta that joins Animalia and fungi


together we exhale carbon dioxide we inhale oxygen and the fungi are able to stream nuclei to their tips and because of epigenesis the ability to adapt to change this is one of the few organisms that actually benefits from disruption and so when these mats are disrupted the fork the us streaming of nuclei epigenesis comes into play reassortment nuclei at the end tips it codes for new


genes for new enzymes acids to capture new food and then the information becomes back channeled into the mice illegal network so using the epidemic properties of mycelium i think is a way of the future of medicine so the mycelium is a lot more pervasive than most people realize virtually 90% of all plants have mycorrhizae fungi has now been determined that these microbial networks


and the mycelium communicates across landscapes in between plants and indeed all plants are part fungi so any research on botanical medicine the contribution of the end of it funds that are associated inside this plants needs to be taken into account because the conferring medicinal properties may be welcoming from the end of it expunged I as opposed to the plant by itself


these my cellular networks stream across landscapes and I have these epiphanies and I believe habitats have immune systems and the mice ileal networks are the foundation of the food web that's joining us all together now here is something that's I grow lots of mycelium twenty to thirty thousand kilos a week we have a small company sixty seven employees and this is differently


frankly just not fair that I can tell you this in fifteen seconds which took me thirty years to discover the problem with mcgroin mycelium in laboratories is immunologically naive it's grown in pure culture when you throw it out into the ground all these organisms consume it well we've soaked woodchips or a straw under water salt water or fresh water for two weeks in aerobic interwebz


become predominant then we take this out and we drain off the water and then the oxygen becomes a sterilizer the anaerobes are largely killed simply our robes are in there and then the mycelium then becomes immunologically educated this is a profoundly powerful mycelium it's got an immune system and resident within this mycelium is the enormous amounts of bacteria we did next-gen


sequencing here and this is a color heat map a thousand full difference in the relative abundance of different genre of bacteria to different mushroom species selected out whole different constellations of bacteria this enables the mycelium to cell set up guilds and have commensal mutualistic organisms that i can combine with that allows it to conquer such large habitats well we


all know that we have cancer 41% of us will get cancer 21% of us will die from it but did you know still that 73% of all against our cancer drugs have their origins and natural products we grow about 500 different species turkey tails featured here or when the best described and studied medicinal mushrooms in the world we received a 2.2 million dollar breast


cancer clinical grant from the nih for phase one breast cancer and the results of the studies have been published and on a dose-dependent basis well prior to radiation your immune system is as active and then when the turkey tail mushrooms eight capsules per day are consumed there is an upregulation of natural killer cells and then post radiation most of you know the immune


system is damaged it and then it has to recover and then on a dose-dependent basis two weeks and then four weeks the immune system kicks into gear natural killer cells are enhanced dramatically and also cytotoxic T cells look at the significance value here and so the immune system is activated by the consumption of these mushrooms and there's TLR tlr4 receptors I don't want


to get into that right now but we've identified seven different distinct pathways of the immune systems activated than the consumption of these mushrooms now this became deeply personal to me in june 2009 when my 83 year old mother called me up she's a charismatic Christian she's not seen a doctor since 1968 she called me a mrs. Paul I'm scared I didn't recognize


her voice she was shaking I said what's wrong and she said my right breast is five times the size of my left I have six angry lymph nodes dark and swollen on my right side I said I couldn't believe it why didn't you tell me sooner and so I rushed her to Swedish breast cancer clinic in Seattle and then we got the worst news after the second visit the oncologist said she should have been


seen two years earlier the cancer is inoperable they could not do a mastectomy because of her age they couldn't give a radiation therapy because of the same reason because the likelihood of infection and so the oncologist tried to make best of it saying you live a long life and we kept on asking how long how long how long and she said you lucky if you had three


months the tumor is erupting out of her breast across the Meridian invaded her sternum and when it went into her liver so we had the circle meeting many of you have had this we planned for her funeral she chose a pink dress that bought the cheapest coffin that she could find because she was going to Jesus there's a lot of Tears and then on the third visit the oncologist said you know if your


immune system could kick in patty you might be able to beat this and so she said you know there's these turkey tail mushroom study that's on a misty romantical school University of Minnesota Medical you might want to take start taking turkey tail mushrooms well that's my mother said well that's what my son was talking about but she had to hear it from a doctor right so my


mother started taking turkey tail she was on tax haul briefly had a horrific reaction refused to take it and she was ten then she's worth taking Herceptin a wonderful drug well that was in June of 2009 and I'm happy to say my mother she crossed the five year disease free period she's totally cancer-free this led to then a study saying well maybe turkey tail mushrooms can enhance


Herceptin now the good news that my mother survived but then she told me something her oncologist told her of the 50 women who joined at Herceptin program and listed in a new program in Ellensburg Washington where she enlisted of the 50 women 48 of them have died my mother was the only one taking turkey tail with Herceptin so how did this is interesting on multiple levels


she's been written up as a best case outcome several medical journals she that no chem brain no nausea no loss of appetite so she's happy and she her acumen has come back she's smarter now than were quick with her wit than I've ever ever seen so then a series of other articles came out this past year turkey tail enhances the microbiome specifically of lactobacillus and


Bifidobacterium while suppressing and anti flattery bacteria so this is extremely interesting because this speaks to the fact that when we grow the much of mycelium in pure culture we do see a resident mutualistic population of bacteria which may we at first thought were contaminants but now we understand better that they're part of the of the microbiome of the fungi when we


split from fungi 650 million years ago we chose the relevance circulating our food in a gastric in a sack and a stomach basically digested nutrients within the mycelium went externally well just as we have a microbiome within us the mycelium is selected a micro biome also Muta listicle e to its advantage I'm very interested in the viral to cancer connection there are seven


identified viruses or probably a lot more that caused cancer then Fred hutch medical school called me up and said we have a very interesting case from the Merkel cell carcinoma one of the most deadly cancers of the world only ten people have ever been reported to it have ever recovered from it and I call it the NIMH hypothesis dr. paul NIMH MD PhD at Fred hutch and they had this


patient he started taking a seven species mushroom blend and this is immune evasion and then after taking the mushrooms there's no chemotherapy no radiation therapy nothing can be done for these patients and then after taking the seven mushroom species blend he had spontaneous recovery and he is alive today so we think that it D cloaks cancers for discovery by the immune


system we don't know exactly how it does it but we've seen this over and over again your immune systems activated and your mutant cells can discover receptor sites in the stroma of tumors this could go abroad and be useful for addressing lots of solid tumors as an adjunct therapy so this case also was written up in the medical journals and then Hayling Lou and I submitted an application to


NIH to standardize the methodology for analyzing medicinal mushroom products it was amazed or so and unfortunately the sequester occurred and we did not get funded for this but it's a great paper that I'd be glad to provide to anyone who likes to see it the Bison team produces extracellular metabolites and in these metabolite droplets are all sorts of interesting compounds I was


working with a bio shield program of the US Defense Department directly after 9/11 we submitted over 700 samples and this is the said the samples of mushrooms in particular a gerakan reishi and chaga and this is the selectivity index and the viruses h5n1 h3n2 h1n1 and the rip of ayran is being the positive control the selectivity index is an indication of antiviral activity our


extracts were diluted from the mycelium 100 to 1 and this is the selectivity index of the diluted extracts that were much far more powerful than the pharmaceutical control well then we were we did by our guided fractionation of the University Mississippi this is a school pharmacy and we've identified a group of sterols this one has been unreported in the literature I've given


it the name phony top straw so we have our first eight api's here that are active against in this case poxviruses now that's coming on the same mcgarrick on extract that was active against flu viruses but we sent these structures to to st. Jude University in order to st. Jude hospital in order for tests against HIV and when they were totally inactive which suggests that there's more than


one antiviral API that's present within these these mushroom extracts NIH called us three times in the past month we've submitted now ten of these structures they potential api's for testing against Ebola and a wide number of other viruses so resident when these mushrooms are very interesting complex molecules that were beginning to discover so after ten years I finally received a patent


universality of opinion by the patent extent patent examiner's and it took a long time to get the patent but I was happy to see that vector in Russia published an article two years ago authenticating that a gerakan is highly active against flu viruses this article was published yesterday so is there people are catching up but it's great that other researchers are


authenticating that which we had discovered now working with a gerakan instructor scott franzblau who is the director of the tuberculosis research institute at the university of chicago we started doing experimentation and he started using our mycelium and we did by organic fraction we found a new active anti-tb set of molecules chlorinated chlorinated coumarins now this is interesting to me


this mushroom has a dual activity against viruses and bacteria very few medicines do that the majority people who die from viral pneumonia actually die from bacterial pneumonia and so to have something as a nutraceutical that can be broad-based against multiple viruses multiple bacteria I think is medically extremely interesting my wife and I spend a lot of time in the old-growth forest the force


used to be resplendent around the world and now we are facing a radical change and in our ecosystems through deforestation so the composition in the ecology of the forests have changed and 70% of the soils are composed of microbial mass of which 40% of the mouse's fungal but because of our practices of logging and harvesting and creating monoculture which in repetitive


Harbor planting of trees leads to premature decline disease vectors spread the diameter of trees become smaller and you lose that that plurality of biodiversity of Ages of trees and their associated organisms we have really changed the face of this planet so I'm going to now take a radical left turn so this is the case now imagine hundreds of millions of years our ancestors and


other organisms in the ecosystem have been used these resplendent for us and now we've deforested much of the planet and the deforestation continues at an incredible clip we've now entered 6x the sixth greatest extinction event on the life of this planet and we're losing about 30,000 species per year of 8.3 million species on this planet that means that a hundred years


we'll lose more than 30% of the biodiversity on this planet this is this is an all-hands-on-deck moment so a friend of mine came to me he said Paul I do a lot of work with antibiotic genic fungi controlling insects and says can you help the bees and then whole foods provide this very interesting graphic here's your dairy choices with bees and there's your dairy choices without bees bees


liberate pollinators 30% of the food in the grocery store is direct result of pollens it and pollination 70% is indirect and the President Obama came out with a presidential memorandum and there is a we call it click on a hex efekta there is like six different converging stressors on the ecosystem deforestation is one the bees now don't have the ecosystem that it's evolved to


to draw from as part of this menu it's banquet of food well then because of the pollution and one dozen speakers mentioned his blood was analyzed he has a thousand different xenobiotic toxins president is blood unprecedented a theater of evolution mites are carrying viruses and then you have of the the fact that the bees are being trucked hundreds of miles into almond orchards


in the middle of the desert in California in January and February this is totally unnatural so the bees fly out and we see bees around a flower that's the last seven or ten days of a life the bees flap their wings into the wings are shredded and the bees then with a colony collapse disorder the bees leave the Beehive and they just don't come back they just suddenly


disappear now it's a very complicated set of stressors but just like there's colony collapse disorder I suggest to you that we are facing cultural collapse disorder this is a proverbial canary in the coal mine so I had some very strange events in my life a lot more that I can tell you but but I was growing mushroom mycelium in my garden this is 1984 and I went out to my garden


and this is the mushroom beds and I went Wow what's going on here I look very closely and bees had come to my mushroom bed move the wood chips away and started sucking on my mycelium I went what is going on from day to night for 40 days for 40 days a direct stream of bees from my beehives to my mycelium back and forth all day long the mycelium shrunk from about 8 or 10 inches to about 3 inches


well I noted this in one of my books and Harrow Smith magazine virtually everybody ignored me I got one beekeeper from Canada wrote me well maybe that's why do they go to sawdust piles so ok I put that in the back of my mind and then a friend of mine said you know what can you do to help the bees and I thought well you know I had this very weird experience in my garden 1984 so here's


dusty in the old-growth forest and bear scratch trees well we used to have a lot of bears but the tip industry put a bounty on them killed the Bears and only in the past 20 years we've come out with research knowing and finding out the Bears bring salmon carcasses up on the banks and returning sea phosphorus from the ocean into the roots of the trees which is a limiting nutrient for tree


growth so the industry totally got it backwards Bears help trees grow well the Bears scratched the trees and dusty and I are hiking in the old-growth forest and the South Fork of the hoe and we go around the corner and dusty sees this bear scratch BAM the bear scratched the trees the best bear scratch I've ever seen but that's why I photographed it now looked into this and wow the timid


industry says the bear scratched the trees and it causes a mushroom to form which is related to a Garrett con so we went back two years later there's that bear scratch ok so think about this and the bear scratch the trees then resin comes out and bees go after the resins and they get propolis which is a very strong antimicrobial and they used for patching up you know spaces in the


Beehive so the red belted polypore sure enough was growing out of that tree with a bear scratch that we saw so in a sense the timber industry is correct this is a parasitic fungus that kills the trees and then grows Sacre physically well also interesting the mycelium breaks down pesticide herbicide and fungicides okay so that's another box another experience I had the garden


now hiking the old-growth forest a bear scratch I looked into the you know why the timber industry was trying to kill the bears then this article comes comes out these are all very very recent and it turns out that fungi produced the P chimeric acid related to the chloral chlorinated coumarins that Scott Franklin our fund their active against tuberculosis by the way and it turns out