The specialist models would be such models that have this exact task in mind, and therefore they don't care about other tasks.


Big Transfer (BiT): General Visual Representation Learning (Paper Explained)

They outperform some of the specialist models, but not all of them.


So this is not the new state of the art in everything, but it is in this transfer learning regime.


And I think even more important, if you see on the left, this is in the small label regime.


So here you have something like 100 or 25 or 10 or even five labels per class.


And if you take five labels per class for C410, this model, so of course you have to pre-train it first on the big data set,


but just taking five labels per class, you still get like 94% accuracy on C410.


And that's pretty good.


That is pretty impressive, especially if you compare it to this baseline model here, which is a ResNet pre-trained on just the ImageNet data set.


So that really shows you the power of pre-training with full data.


So one thing they say is that in their big data set, in their 300 million images,


they make sure to remove all the images that then appear in the downstream tasks, right?


Because otherwise it is fairly conceivable that this database here is just scraped from the internet.


And of course, these tasks are often, like C410, are also scraped from the internet and also ImageNet.


And it is entirely conceivable, of course, that the test data is already here.


And they say we remove images, but I think they just remove exact duplicates.


So it could still be that, you know, someone has taken ImageNet and then kind of recoded it into another color scheme or whatnot,


or just compressed it a bit more, and then they find these images on the web.


So it's a little shaky, this whole thing, because these data sets might just be part of one another.


But, you know, given the results, I do generally believe the improvements here.


But yeah. So I guess what we need is like people to actually go out with cameras and shoot new pictures for a new test set.


But in any case, let's dive into how to pre-train something like this.


So they divide their findings up in two parts, how to pre-train and how to fine tune.


So how to transfer to downstream tasks.


And the methods they find are surprisingly easy.


They say there is two components to pre-training.


The first component is scale.


So you have to have a lot of data and a lot of models.


And that is a pretty important recognition.


So down here, they have this ablation where they scale up the model and scale up the data.


So look at this, for example.


You can see here you have the different data sets to pre-train on.


So this is the small data set, the medium data set and the large data set.


So in this direction, you have data set size.


Then here you have accuracy, not labeled.


Again, I guess we can understand accuracy.


That's fine.


I'm nitpicky now.


And there we have the different models.


Now, the larger the dot here is, the larger the model architecture.


And you can see within the individual bins, the larger the model, the better performance


you usually get.


But as you can see, like here, this improvement in the large models isn't as much as when


you have much data.


And you can also see by the slope of the line here, the larger amounts of data help more


when you have larger models.


So only scaling up the data is not as effective as scaling up the data and the model at the


same time.


And in some cases, like in this small architecture here, it actually hurts to incorporate more


data, at least they say that.


And you can also see that here.


And here, it just doesn't help as much anymore if you incorporate more data.


So if your model is too small, you can't handle the big data.


Of course, there are weird effects like here, the performance goes down and then up with


the larger data.


So this might actually be an effect of the images in these data sets being somewhat qualitatively


different, also with respect to the task that you are training for.


But in general, it holds that you need a combination of data set size and model size to go up.


And this, I think, might be an indication of where we are in Belkin's double descent


curve.


And if you look at, there's the researcher, Mikhail Belkin, and others people also research


in this area, they have this sort of empirical finding and hypothesis, sort of, that if you


plot a graph, and here is the number of parameters when the relation to the data set size.


So number of parameters in relation to size of data.


And here is your validation loss.


Then what happens as you have very little parameters, you can add more parameters to


your model to get better validation loss, right?


This is, you know, we get a better model.


And we train that and we get better.


And then at some point, you'll start to overfit, you know, we've all learned this in our general


machine learning course.


And there is a point here, the what is called the interpolation threshold, where you have,


this is one, so the number of parameters is equal to the number of data points, which


is just interpolating your training data, sorry, the data point here, that's train.


But then the discovery sort of is that this comes down again, and it stays down.


So as you go up in number of data points, sorry, number of parameters with the same


data set, you're perfectly fitting the training data set, you're past the number of data points


in your in your model, but still, your validation loss comes down.


And there's various hypotheses why this could happen.


And here, we find ourselves maybe in this sort of situation where if you have a model


right here, and you want to scale it, you want to add more data, you can't just keep


the model constant.


Because if you add more data, that will shift you to the left here, because you add more


data, but you keep the mod, the number of parameters the same.


So this number will shift to the left and you actually go up in your validation loss.


So maybe this is actually what's happening right here.


The fact that the model is too small.


This is just a hypothesis by me.


So if you want to up your number of data points, you also have to up your number of parameters.


And that will keep it going.


And maybe these models here are more on this side of this interpolation threshold.


And the models where it doesn't happen might be more over here.


Though that is a big thing to assume.


Maybe not.


Now that I think about it, since they have even more parameters here, they would be even


more here somewhere.


So maybe you add a bunch of data, it's just not as bad.


There might be some weird interactions here.


Like this.


Who knows?


Let's just skip this.


In any case, the message here is you need more model and more data at the same time.


All right, then there is a second message, a second recipe for pre training.


There we are.


The second method is group normalization and weight standardization.


So they criticize batch norm.


Batch norm has of course been used a lot.


That is where if you have a batch of data, and you put it, so these are all data points,


you put it through your layers, and it has some intermediate representation.


What you want to do is you want to calculate sort of the mean and variance of your data


in each of the features, and then make it such that it's nice mean one and standard


deviation.


So mean zero and standard deviation of one.


That is called batch norm.


But of course, it is dependent on your batch size.


So it is dependent on how many data points you have, because that's how well you can


estimate these mean and variance parameters.


And what people do nowadays is they take these batches, and they group them into different


groups.


And they distribute those groups on to many, many machines, which is called data parallelism,


especially with TPUs nowadays, you can just distribute everything to so many TPUs.


I believe they say they distribute to something like 500 TPUs, which and so they have a batch


size of I think 4000.


And they distribute to 500 TPUs.


So that leaves them with eight, eight samples per batch.


So this is eight.


And eight is just not very good for batch norm.


And if you have to, if you want to circumvent that you need to in each layer globally sync


with all of the other workers, your batch norm parameters and that slows you down.


So people have gone around this using what they call group normalization and weight standardization.


So these two techniques of weight standardization is a is a addition to group normalization.


They don't require the other samples in the batch, they work on a per sample basis and


they normalize the features within groups of each channel.


So the group normalization groups together different features within a sample and then


normalizes across that and the weight standardization is a bit like standardizing the features,


but it standardizes the weights to be of a normal distribution.


And just suffice to say, these are standard techniques that you can build in and they


allow you to not have to synchronize constantly between your workers at the training time,


which makes everything a lot faster.


And also not a problem that you just have eight samples per worker.


Alright, so that's what they do.


They do large data, large models, and group normalization with weight standardization.


That's how they pre train and then how do they fine tune, they say they have a rule


to select hyper parameters, they call the bit hyper rule.


And that's just sort of a formula of how you have one hyper parameter.


So you have one, I guess it's a hyper hyper parameter, and that hyper hyper parameter


you run through their rule.


And the rule will tell you what each of the hyper parameters should be.


So it's like a lookup table basically, it's oh, you set this one number and we give you


the rest of the hyper parameters.


And that one rule works pretty well.


So you only have to find for fine tuning, you only have to grid search over one hyper


parameter.


It's not really grid anymore, is it?


And then they basically decide on the training schedule length, resolution and whether to


do mix up regularization.


Mix up is a technique that can help when you have very little data and what it does is


it interpolates between data points and also trains on kind of like data points from half


this class and half that class, just to make more data available.


But they all have this packed into this rule.


And they of course, the exact settings of this rule are presented.


So you can look it up.


Then they have a data pre-processing, resize the image to a square, crop out smaller random


square, randomly horizontally flip the image at training time.


So they basically describe a standard training protocol here.


Don't want to go mix it too up, up too much.


The only thing they say is surprisingly we do not use any form, any of the following


forms of regularization during downstream tuning, weight decay to zero, weight decay


to initial parameters or dropout.


I think they only use weight decay during pre-training and that's it.


So let's look at some of the graphs.


We've already seen some.


Here is where they pretty much outperform the generalist, these generalist models on


all of these tasks, including this visual task adaptation benchmark.


I've made a video about this.


This is a benchmark that includes 19 different visual tasks from all over the place.


And they have significant improvement here as you can see.


They do not always outperform these specialist models.


But as you can see, they outperform, for example, this on the flowers data set.


And they come pretty close.


And here you can also see how much they improve when pre-training on the larger data set.


So far people have basically pre-trained on this ImageNet data set.


And now that they pre-trained on the larger one, of course, they gain a lot of performance.


And the largest one isn't even in this table.


So what I finally want to look at is this visual task adaptation benchmark.


This consists of 19 tasks and they're divided into natural tasks, which are kind of natural


images and then specialized tasks, which are, let's say, the medical images, not really


natural, and then structured tasks.


And the structured tasks isn't simply labeling or locating something.


It is tasks where you have to maybe reason about something.


So let's say there is an image and there is a cup right here and there is a glass right


here.


And the question is what's to the left of the glass.


And there's a bunch of other stuff around here.


And you have to say the cup.


So it sort of requires a structured understanding of the image.


And you can see the main performance boost here comes in the natural images, which is


to be expected.


So you only get what you feed in.


And this 300 million image data set, I'm pretty sure that's just a web scrape of photos or


mainly photos.


So the main improvement you're going to get is on pictures that are similar to that, as


we said at the beginning.


And these natural tasks have images like that.


And you can see that the model here improves extremely in that category, improves slightly


in this specialized thing, and only improves a little bit in the structured tasks.


So this, as I said, is to be expected.


Just know if you use this model, know what is in there.


You have to know what it does, what it does well.


It does well on natural images that are similar to what it was pre trained on.


Okay, so they do have some analysis here.


And we've already went to most of them.


I find this to be pretty, pretty impressive.


So they say, when they apply the standard computational budget of ImageNet pre training,


when they scale up to the larger data set, it seems detrimental.


As you can see right here, the performance actually goes down when you go to the larger


data set.


Only if you train longer, then your improves.


The axis labeling is just amazing here.


Standard long, longer.


Oh, how long you trained for?


Longer.


Thanks.


But I guess the point is taken that the, you have to invest more computation along with


your bigger model and bigger data set.


Sorry, it's the same model, but the bigger data set.


And they also make some other points here that if you, for example, if you decrease


your learning rate too early or set your weight decay parameter different wrongly, that also


hurts you.


So on the right here, you see a smaller weight decay initially looks better.


So initially, you're higher.


But through the training, you end up at a worse place than a higher setting right here.


And I mean, they make a big point out of this, but who's to say that someone else doesn't


come with like 10 times longer training and figures out that ultimately you start off


like this and then maybe goes up super high.


So to me, this, the lessons learned here is pretty much that there's always a way to get


more performance out of more compute.


And probably, there is a way to schedule all of these things because that's combined with


decaying learning rate and so on.


There's probably a way to schedule these things with the current, with this particular method


that would end up somewhere here.


We just haven't found it yet because it's so complex.


I would guess that is the case.


Here they make an interesting point that if you decay the learning rate too early, then


you also end up at a worse place.


So this dashed researcher here, the ha ha, the noob.


So after eight GPU weeks, which come on, what is that?


Eight GPU weeks.


That's just eight GPUs for a week.


That's nothing, nothing.


It looks like this, right?


It looks fairly flat.


And this researcher now decides to decay the learning rate and that results in this thing


here.


Decays the learning rate here, here, and here.


Sorry, not here.


So decays the learning rate here and then it flattens out again and then decays the learning


rate again.


Ends up at this level.


Yet if you train for longer, you can see right here, if you look over eight months, you can


see that there is a slight upward trend still and it hasn't converged yet.


And you can, if you decrease the learning rate only later and always wait for this to


fully converge, then you will end up at a better place right here above 70.


Again, who's to say that if I just wait here, there isn't a slight upward trend.


If I wait for eight GPU years or eight GPU solar system births, then there might be even


a better point to decay, finally decay the learning rate and then go up.


I mean, again, this researcher here only takes 0.5 million steps where you take two million.


So that's the first point.


The second point is ImageNet or visual state of the art research is now officially out


of the hands of academia.


This is it.


If you see things like, if you see a paper dissing on people that only wait eight GPU


weeks to decrease their learning rate for the first time and advocating that you should,


you know, at least wait until eight GPU months.


Maybe they wait twice as long.


It's over.


That's it.


Yeah.


Bye bye.


Maybe, maybe, you know, you want to do some theory or something.


What I find interesting is the mistakes.


So since on CIFAR-10, they reach like 99.4%, there's only a handful of mistakes that they're


still making because it's not that large of a dataset.


And they do classify it.


So red in particular, I think, means red is the ground truth label is correct.


But green is the machine is correct and the ground truth label is wrong.


And you can see there is a fair number of green things here, right?


So the model says ship and the label says cat and the model says bird and the label


says cat.


Clearly, this, this would be one weird cat.


So it gets to the point where you also have to expect these errors to be in the training


set.


So it could just be that the model here doesn't necessarily even make those mistakes, but


it's just somewhat consistent with the training set in making the mistakes.


And also here on ImageNet, they have selected ones where, you know, the model says notebook,


but it's actually laptop and the model says mouse, but it's actually space bar.


You know, the model says Alp and it's ski.


So or here the model, the model says candle, but it's a, this is a dishwasher.


What?


And so you see that the, the, the types of mistakes here, we get to very quirky, very


fine grained points in these models.


Last thing I want to show, I have never seen these ImageNet 21k images.


These are just funky.


Like look at that.


So here's the state of the art previously, I think said triceratops and the new model


now says, bit L says starfish.


Good job bit L. Wow.


Probably the correct label would just be weird.


And this, no.


Okay, I don't want to rag on this too much.


This is a cool paper.


I believe this will be the new starting point for a lot of practitioners in when they do


visual tasks.


I always, as always, invite you to check out the paper, subscribe to the channel, leave


a like, leave a comment if you want.


I do read them usually.


Bitcoin Protocol: Proof of Work shown in Python Code

What is going on everybody and welcome to another Python tutorial where you're going


to be shown how to create your own proof of work system as well as seeing first hand how


and why proof of work works.


So this video is both for people who are following along with my Bitcoin protocol series as well


as people who are interested in possibly enabling proof of work within their own website.


The idea of proof of work is to inhibit spam and DDoS attacks for the most part.


What you're going to do is you're going to present the computer that's making a request


on your server with a computational problem or challenge where you're basically defining


the difficulty of that problem and then you ask the requesting computer to solve it.


Now the beauty of this and a proper proof of work system is that while it takes a significant


amount of CPU power to solve the problem it takes a minuscule amount of processing power


to actually validate the answer that you're given.


Generally even with a simplistic cache cache or proof of work system it's at least 100


times easier to validate that the computer got the correct answer compared to how hard


it was to actually get that answer.


So if you have Python 2.7 feel free to follow along.


If you don't have Python 2.7 I highly encourage you guys to go ahead and head over to python.org.


In fact I'll just show you guys real quickly.


So this is the official website python.org and head there go to download and scroll down


and download the proper installer for Python 2.7.


I'll probably show the same thing for Python 2.3 or 3.3 rather eventually but for now we'll


use Python 2.7.


Once you have that downloaded and installed the installation process is extremely simple.


All you'll want to do is come over to your start bar and just move the screen over basically.


Hit that go to all programs find yourself Python 2.7 click that click on idle.


That will bring up a version of you know it's like Python's command line basically.


And then what you're going to want to do is go to file new file click on that you'll get


this and then just go ahead and immediately save this wherever you want.


I'm going to put mine on the desktop however and call it whatever you want but be sure


to end the like first of all save it as all files and then here I'm just going to save


mine as hashcachexample.py hit save and now you're ready to start.


Okay so the structure of something like hashcache's proof of work was described at least introduced


to you in the previous video but I'm going to go ahead and summarize it again.


So what ends up happening is the server is going to pose a challenge to that requesting


computer and the challenge is usually just a random string.


So then the requesting computer is tasked with finding a solution to that string that


when appended to that string and added you know to the end of that string it's then hashed


and it's supposed to produce the hash.


We'll just have a certain length of zero bits at the start of the hash.


This is purely by chance and the solution is just random basically so the best method


of discovery actually turns out to just be brute force.


So that's what we're going to do here as well as actually visually see what it's supposed


to even look like since I'm sure a lot of people still don't understand like what do


you mean by zero bits at the beginning of the hash blah blah blah.


So here we'll actually see what the deal is.


So the first thing that we're going to want to do is we're going to import a couple things.


The first thing we want to import is string and we want to import string so we can actually


make string text.


So normally if you wanted to make text you would just type it out but since we're trying


to generate random text we want to first take string and then since we do want to make it


random we're going to import random as well.


So our first challenge that we need to do is actually we need to create something that


just makes a random string.


So that's the first thing that we're going to do.


So we'll come back to the rest of the imports we need in a little bit and first we need


a challenge string.


Now the challenge string can be really anything so we're just going to use something real


simple and basic and I'm just going to say example underscore challenge so we're just


defining variable here equals oops not double equals just equals and we're going to make


this 9kz this this this this you don't have to copy my challenge string by the way your


results are probably going to be different than my results anyways.


Okay that's a challenge string.


Now we have a challenge string now we need to make our program be able to you know just


add something random to the end of that so we're going to say define generation and this


is going to be our generation function that just kind of generates an attempt and so first


we need let's see we're going to say this will have a couple variables first we'll just


say challenge and that's just going to be example challenge and for now we'll say size


and then we're going to say size is 25.


The reason why we're doing that is first of all we want to know you know like what's the


challenge that we're given and then also the size is just how big do we want our random


you know answer attempt to be.


So the next thing we want to do is we're going to say answer equals and then basically we're


going to start with a empty string and then we're going to say dot join so basically we're


appending to the end of this and we're going to join random dot choice so it's going to


choose something completely random between what and it's a random choice of string dot


a s c i i so ASCII characters and these are going to be lowercase examples then we're


going to say that plus string dot ASCII again only this one is going to be upper case and


then finally plus string dot digits so any number and then what we're going to go ahead


and do is say plus or not plus sorry and then for x in range of size so basically what this


is going to do it's a little challenging to anybody that's new but basically it's going


to join a random choosing of either an upper case or either a lowercase character an upper


case character or a digit and it's going to do this for x in range of size and basically


what this is saying is for x for whatever however many numbers are in size so in our


case it's 25 so for x in range of 25 it's going to do this 25 times so it's just going


to give us a just at least a pseudo random answer so that's our answer now the attempt


here will be equal to challenge and then plus answer so this is just going to append it


so just to show you guys let's first run print we'll print the challenge and then after that


we'll print the answer so the answer will be just totally random and then we'll come


down here and I'll just call generation we don't have to put anything in here because


both variables we've given a default so it's not a big deal so generation we'll save and


run that and so we can see here that it you know it repeated our example challenge right


here and then here we have an absolutely random well not a truly random but pretty darn random


generation sequence so if we ran it again you would see that the first sequence is obviously


identical because the challenge doesn't change but then the sequence changes so we've got


a completely different sequence here so that's the first step right and then our attempt


is challenge plus answer so we could print attempt here save and run that and now we


actually have an attempt you know that is the challenge plus and then right here well


here starts the answer so now what we want to do let me close out of this is this function


needs to really stop printing out stuff so I'm going to get rid of this because eventually


we're going to make this generation run like a bunch so we don't really want all this printing


and now what we actually want it to do is we want it to return the attempt and we also


need it to go ahead and return the answer because if it does indeed produce what we're


looking for we want to know what that answer was because that's what that's what we have


to return to the server so we do that now we need to if you recall to find out whether


or not this is correct or not we have to make a hash out of it and see if for example it


starts with three zeros okay so to do hash we're going to import hashlib this is just


the hash library so then what we're going to do is we no longer want to call generation


we're all set there and for now I'll just delete these I think just so we can save some


space alright now what we're going to do is we're going to say sha hash equals hashlib.sha256


so what we're just saying here is we're defining variable and we're just saying that that is


equal to sha256 done so now what we need to do is we want to find out if this attempt


is correct so we're going to come down here and we're going to say define test attempt


and in here first we're going to say attempt and answer equals generation empty so this


return so basically attempt and answer equals whatever this is right here so you're going


to run this function and return attempt and answer as variables and now what we want to


do is we're going to say sha hash dot update and what we want to update this hash is attempt


and then now what we want to do is say our solution is equal to sha hash dot hex digest


and finally for now let's go ahead and just print the solution okay so then what we'll


do is I'll come over here and just to show you guys a quick example of what happens like


for example if we ran this right now we'll run test attempt and we have the very first


attempt here let's see print and that's our first attempt but if we oh we could actually


just run it again so let's do this and it's our second attempt we could do it again and


again this one at least started with a zero but as you can see none of these so far let's


say our goal was three zeros none of them hit it yet so now let's make just a quick


look we could do something like this you could say for x in while use x range but if you're


in python 3 it would just be range x range zero and a hundred what do we want to do we'll


just run test attempt just to see here so let's save and run that and so here we have


a bunch of attempts now I'm just looking with my eyeballs here and I see a lot that start


with zero but I don't see any that are have two zeros so now what we want to do is we


need to kind of change our function a little bit here and what we want to do now is instead


of doing this we'll say if solution dot starts with and let's say we're only look let's say


we're looking for three zeros if that is the case then let's go ahead and print that solution


so we'll close this and then we'll actually allow for let's say ten thousand and let's


see if any of them fall under that so we'll save we'll run that and we'll run we need


to let's see it well this should have already run the ten thousand so instead let's do a


little few more here save and run that there we go so now we got a few in a much larger


number we're still getting some so now it's only spitting out ones that have three zeros


and I just saw one go by that actually had four zeros but as you can see it took us one


million attempts to start getting some and then if we added maybe another zero and actually


let me just stop this one from running and so we've had another zero let's run it again


and okay we're actually starting to get some let's add another zero then not bad not bad


so as you can see it's obviously slowing down though at the pace that it's actually you


know finding these hashes that start with zero this is the next one's taking forever


I found these two pretty quick and now it's like nope nothing else so the next thing that


it only found two so obviously so to find even a nut like five zeros in a list of a


million attempts it only found two so two in a million right so pretty bad odds so anyways


we'll close out of this and now what we actually want to go ahead and do is we want to kind


of switch up this attempt test attempt a little bit so what I'm going to do is let's say first


let's just get rid of this so instead of having it be like a million what we can do instead


is we'll just delete that and under test attempt I'm going to say we're going to make a variable


called found and found is going to be equal to false for now and then also well we'll


get back to the other thing in a second so now I'm just going to delete this so found


equals false and now we're going to say wow found equals false then we're going to do


these things so I'm just going to tab these over and then if solution.starts with and


again so we're going to make this an infinite loop so it doesn't really matter here it'll


never run out but we'll do five just so the video will end at some point if it starts


with five zeros then let's print solution and then we need to go ahead and say found


equals true and when we come out eventually this function would have to return an answer


so for now let's just print the answer we don't really need to return it because we're


not going to use it for now so we can save that and now we need to go ahead and let's


call test attempt here save and run it so it's going to run run run run run until it


finds something or the hash that starts with at least five zeros so I'm going to pause


it until it finds something okay that took like I don't even know like a minute or two


minutes or something so here we have the hash that starts with five zeros and this is the


answer to the challenge that produced this hash so now just out of curiosity and interest


we can come up to the top here and let's import time import time and then we can come down


here and we'll say start equals time dot time and then what we'll come down here and says


if the challenge is found let's say how long did this take so we'll say time took equals


time dot time minus start so that'll just do the math in seconds basically and so we'll


print the solution and then we'll go ahead and print time took and we'll say time took


here so now we can save and run it and in fact well we could we probably should change


let's just make it four zeros so it'll find one quicker alright so this one took only


three seconds to find one now let's go ahead and run it again and this one took about another


you know three ish seconds and let's go ahead and run it again one more time and this one


took about two seconds so anyway if you were running say a website and you wanted to protect


against DDoS attacks for example you know a typical mail abuser would send maybe ten


thousand emails a minute well if you can only process like on my computer would be able


to send ten thousand emails a minute and my internet would allow it so I could get away


with that but if the processing time per email is costing my computer three seconds well


obviously now I can only send like twenty a minute right so it severely cuts me down


but then what happens is when we find this answer we go ahead and we're going to come


back to that the actual server that we're making a request on and we say okay we got


your challenge here's our answer and then all the server has to do is run that and say


okay well here's our challenge here's the answer that person gave us and they try it


and either it works and access is allowed or it doesn't work and access is denied and


you'll see you know are you a person that kind of stuff so that's just kind of like


the basics to it now as you can see these are taking somewhere between two and three


seconds but if we as you remove the zeros I mean it'll come this one came within you


know point oh five seconds let's run it again point oh three seconds point oh four seconds


ish point one seven seconds and so on so as you add zeros it gets a little harder and


that's why Bitcoin is a little bit different because wow we actually found this one in


one that was really quick let's do that again earlier it took us like a minute to


find so it's kind of funny the odds of that are pretty rare I thought but no it's finding


them pretty quick now anyways so after I you know coded all this I realized that maybe


this was a little more challenging for a brand new person than I thought but do feel free


to like pause it or something and just copy the code and but basically all that's happening


is this is where you know we're creating the sample string this is where we're testing


it making a hash out of it and finding out whether or not it starts with zeros and basically


you can see here that at the more zeros that's required the more processing time is required


in the more you're literally paying in CPU so it's pretty neat stuff to be honest now


I leave you guys with this with hashcash in mind see hashcash was never intended I mean


maybe it was but it was it it so far has not been intended to actually make money for anybody


it's basically meant you know hashcash just means the requester has to pay and that's


it and usually that CPU power is pure waste so with hashcash in mind and a simple you


know monetization of CPU power in the context of a proof of work system we now have a new


way to monetize websites because while proof of work here in this scenario it was just


kind of wasted you know we have actually a system today where this exact job is actually


a profitable job see traditionally websites have used hashcash only in the sense that


the cash was just something that was expended and the CPU power was just totally put to


waste but now we can actually utilize a system like hashcash be paid in hash answers or even


just attempts where we use these attempts to actually support the Bitcoin network for


example so not only can we use this to monetize a website we can also protect against DDoS


attacks so while doing these things we're supporting the Bitcoin network and you know


like more valuable parts of your websites could be slightly more costly than other parts


you could pretty much just have an infinite range in pricing no chargebacks instant payment


no advertisement no middleman I mean wow so you know you could either charge people like


obviously in this case the typical case of hashcash the question is you know please you


know produce a hash or the produce an answer to my challenge that produce a SHA-1 hash


that begins with three zeros okay so in that case you're just producing one hash now in


the Bitcoin network the hashes are extremely difficult to achieve so in this sense you


know you could require them to at least produce one hash that you know starts with three zeros


but the next thing that you could do is you could say not only do I want that one hash


that starts with three zeros I want another hundred hashes just give me another hundred


hashes and if any of those hashes happen to start with let's say 40 zeros thank you very


much for my free 25 Bitcoin so anyway so you can use this to actually monetize a website


and be paid in hashes that actually do have a real market value on the Bitcoin network


so a lot of interesting things can be taken from that and so I'll just leave you guys


there to ponder that so hopefully you guys enjoyed hopefully some of you guys are all


wide-eyed right now with aspiration as always thanks for watching thanks for all the support


[ML News] AI models that write code (Copilot, CodeWhisperer, Pangu-Coder, etc.)

GitHub Copilot is now available to all developers while a big open source community is leaving it


behind. But not only GitHub, but also Google and Amazon are jumping into the game of AI


assisted source code generation. Welcome to ML News. Welcome to ML News. Today we talk all about


models that generate source code and that assist developers in writing source code. The GitHub blog


released a post last month saying GitHub Copilot is generally available to all developers. Copilot


is obviously the product by GitHub based on open AI codecs model that suggests source code completions


to you based on a large language model that's been trained on all of public GitHub repositories. This


is I have to say a really cool product. I was part of the closed beta and it was a game changer,


especially if you write any sort of boilerplate code, this thing will just write an entire


function for you, it will write your tests, it will write your doc strings, it will write your


assertions and your error messages. It's just very, very good for a specific subset of programming.


But nevertheless, that subset is making a lot of difference in a lot of people's lives. So the


product now is out of this beta and is available to all developers for a price. So it's 10 bucks a


month or 100 a year, which I feel is reasonable. If you are a programmer by profession, this thing


is potentially going to make you a lot more productive than the 10 bucks a month. It is free


for verified open source projects and for verified students. Now this is AI news and not necessarily


and not always AI shilling. So GitHub has not been without controversy. Obviously, we have reported


on this GitHub has been trained on a lot of code, including open source code, including code that


has been licensed under various copy left licenses with the intention that whatever products are made


from that code are also free and available to the community. These copy left licenses, such as the


GPL are specifically made such that no company can just grab that code and then resell it as a


product because it's based on the work of a lot of unpaid volunteers. Essentially, co pilot is


doing exactly that it's taking a lot of code that's publicly accessible yet licensed under


such licenses taking it in training a large language model on it and then selling that to


you as a product. Now this is a legal gray area. For example, you as a programmer are perfectly


entitled to go look at a piece of code even if it's under the GPL and learn from that piece of


code and then implement that same algorithm in your own way in your own code. That is not a


violation of copyright is a different story if that algorithm is patented. But in terms of


copyright and copy left, you're perfectly fine doing that. So it's perfectly reasonable to say


that training a large language model on that code that then sort of takes bits and pieces learns


from it and then synthesizes its own version from what it learned is a lot like a human doing that


same thing. However, obviously it being automated and it being you know, cranked up to 11 in size


and speed and it then being sold to all the developers out there might be a different story.


And that's why the register writes open source body quits GitHub urges you to do the same. This


article is about the software freedom conservancy. This is a nonprofit focused on free and open


source software. And they are arguing that GitHub is essentially using your work to build its own


proprietary system, namely GitHub Copilot and GitHub itself. Remember, the source code of the


GitHub website isn't public. So your work as an open source developer essentially goes into GitHub


as a product. And that's exactly what a lot of these open source people don't want. So the


software freedom conservancy has released a blog post called give up GitHub, the time has come in


which they detail that not only they are leaving GitHub, but they tell you to do the same and they


are announcing a plan and support structures from them to support people to get away from GitHub and


to move to more open source friendly alternatives. Specifically, obviously, the biggest impact is


going to make to move the source code hosting away from GitHub to some other place be that either a


cloud hosted provider or a self hosted something. Now while I recognize that the idea kind of makes


sense, if those things are important to you, it seems like a bit useless and pointless, like just


as no license is stopping GitHub from scraping its own repositories. If you put your source code


on your website, nothing stopping GitHub from just scraping that it's the same deal a human is allowed


to look at it, learn from it and then reimplement it. So as a language model, at least for now,


so it seems like the real path forward here would be a legal one in which there could be a license


that explicitly states that no training on this data of any sort is allowed, which essentially


might amount to just a patent. But I don't know, I'm not a lawyer. So I don't know what can even be


done in these kinds of situations. And the boundaries between humans and language models


and code assist and whatnot get extremely murky. Copilot is an insanely useful product. And GitHub


has been a absolutely great place for most of open source in the last many, many years. And


obviously, as with a lot of free products, there's got to be a way to make money around that. Now,


sure, there are various business models around open source, but I'd rather pay for Copilot than


seeing an ad every time I want to clone a Git repo. So there are a lot of questions in the air


right here. What's also interesting is that they give you this snippet that they encourage you to


put into your readme if you can't move away from GitHub just now saying we are using GitHub under


protest. This project is currently hosted on GitHub. We are deeply concerned about using a


proprietary system like GitHub to develop our FOSS project. Any use of this project code by


GitHub Copilot past or present is done without our permission. We do not consent to get ups use


of this project's code in Copilot. Yeah, it's about as effective as the if you are not the


intended recipient of this message, delete this email right now. It does nothing. I mean, it's


obviously there to raise awareness. But still, I don't see how even moving away from GitHub will


solve the larger issues around this topic. But let me know what you think in the comments. Be


happy to hear your opinions. Google released a blog post called ml enhanced code completion


improves developer productivity. This is about an internal study that they have done where they


augmented their own code completion engine, which is based on very classical code completion, such


as what variable names exist, what functions exist, yada, yada. And they augmented that with


ml based code completion, such as Copilot. So they experimented with various flavors,


such as single line completion, multi line completion, or simply ranking the outputs of


the semantic engine that they already had by using a machine learning model. This all is based on a


language model architecture. Notably, it only has point 5 billion parameters. So a tiny model in


current standards, but they say this is due to latency requirements. So that makes a lot of sense.


Google has deployed this internally to their developers, and they found a great increase in


efficiency of programming compared to a control group. Now, while it's really cool that a big


company can just run these experiments internally on their people, it must suck to be in the control


group. Like this is the latest and greatest tech and you know, your company internally only has


access to it. And then you're like, bam, you're in control group, I'm sorry for you control groupers,


I hope you get access soon. So this blog post here claims that just under 3% of all new code


that's added to the Google code base is code that has been accepted by recommendation from a machine


learning engine. There's a 6% reduction in coding iteration duration, there's a 7% reduction in


context switches such as moving away from the IDE to go look something up and they have about a 25%


5% acceptance rate, which is how often a suggestion pops up versus how often you accept


that suggestion. These numbers look a little bit different for multi line suggestions, but still


very encouraging. Now while this is really cool, as I said, it's only available Google internally


currently, it also has been trained on their internal code base, which is huge, we're left to


see whether or not that or something like this is going to be available to the general public anytime


soon. As we saw with copilot, there is definitely money to be made with ml supported code completion,


but Google might just be happy with the increase in productivity of their own workforce. And that's


going to make them a lot of money by itself. There's a new paper called language models can


teach themselves to program better. Now this is a little bit different from code completion as it


deals with programming puzzles, specifically programming puzzles that are formulated as tests


in programming languages. So the general structure is that the problem is posed as a function f that


takes one parameter and checks the validity of that parameter. Somehow, you can specify a lot of


things as taking a solution and then verifying it. I mean, I guess you can specify any sort of problem


in that way. And then the solution to that would be a function called g right here, g gets access


to the source code of f and is then supposed to write code that returns something that's then fed


into f that's going to make f true bit more complicated example is down here. So f will


accept an x and check if that x is a palindrome. Now there can be more arguments right here, for


example, the length of that palindrome, and g does get access to these arguments as well, but still


the same principle g is going to get access to the source code of f is can analyze it as much


as it wants and then has to come up with its own source code that makes f go true. So the problem


f here is in fact, the finding of a palindrome with exactly n copies of each of a given list


of substring. And so you can see right here that the solution is you simply take n of each you join


them and then you add the reverse to it. I guess that wouldn't work if either of the arguments here


are themselves a palindrome because then technically that string would also appear in


that part right here. Or if like the cross here like the cross boundary, well, you see it gets


arbitrarily complex, but you get the point. These are illustrative examples. So there is a training


set, but it only contains 155 puzzles authored by humans. And the trick here is that not only use AI


to solve these puzzles, but you actually use it to generate more of them. So we have lots of open


source models and closed source models such as codecs that can generate source code that are


pre trained on source code. So the paper prompts these models with a bunch of prefixes from the


training set. So here you see that's just the problems, not the solutions. And then the models


are tasked to come up with more problems. The next step you use the same language models or


different ones to actually solve those generated problems. And you give them a bit of time so they


can explore a bunch of options which you can automatically verify. Now that leaves you with


a large set of automatically created but programmatically verified synthetic puzzles


on which you can then fine tune that language model and start from the top. So you can use the


same language model potentially multiple times to come up with new problems, new solutions to them,


verify all of that and then retrain these models again. Now as far as I understand the paper only


does one cycle of this and already observes a huge boost, especially on the verified examples. So when


they make sure that he generated problems and solutions actually, you know, match and work and


return true. In that case, there seems to be a big boost if you retrain these language models. So you


can see right here, variant of GPT Neo solves only about 7.5% of the test puzzles when just


tasked like that. But if you go through all of the steps, it solves 38.2% of all these puzzles. Now


there are several issues right here. Obviously, information theoretically, you can't just conjure


out information out of nothing. So whatever these models know, you know, you essentially just feed


that back to them with the step in between actually verifying the code. But given that


they've been trained on public code, and a lot of that presumably runs, especially if it's kind of


filtered for more higher quality training data, then that check shouldn't be too much of a barrier.


So it seems like if we just prompted these models better, we could probably get them to solve a lot


more of these programming puzzles, since the knowledge seems to be somewhere in there. And


also there's the other issue that these programming puzzles, you know, humans came up with them and so


on, they might not be on GitHub themselves. So deduplication is obviously necessary. But


deduplication might not be enough as kind of like the solutions to the problems themselves might be


in some way somewhere on GitHub, like in the training data of these models. And that way,


if you just prompt them in that direction, there might be some effect right there. I don't know,


but it is definitely a cool result. And it seems like if we can pair these models correctly prompt


them correctly, and then use additional resources, such as these external verification procedure in


order to enhance the training data in order to just make it better, less noisy, more to the point


of what we want, that could be a good way forward to get these large models to do what we want. And


it might be an alternative to coming up with smart prompts that just kind of work somehow like the


let's think about it step by step trick, like it would be nice if we had a more systematic way of


getting these models to do what we want. And I think this paper is a step in that direction.


Okay, so Amazon joins the ring of ml powered code completion with its code whisperer product. Now,


much like copilot, this is a model that generates source code, and you can subscribe to it,


it integrates with your ID and then you can try it out, you can let it complete source code and


suggest stuff. Now it's a little bit different in that they not only want to do completion,


but they also claim to do security scans in your code. And it's apparently specifically good at


interacting with AWS API. They claim it's trained on open source code, but also on Amazon internal


code. Now for now, this product is closed, there's a waitlist, you can put your name on there,


no guarantee. But it's interesting to see that yet another company is sort of hopping on this


ml based code completion thing. There's another new paper out of Huawei called Pangu coder program


synthesis with function level language modeling. This is a system based on the Pangu alpha


architecture, which is a Chinese large language model and is much like codecs fine tuned on code.


Now there are a few notable differences. For example, this paper focuses on solving the human


eval data set challenge in the end, which is a Python challenge where you get a description of


what a function should do. And then you should generate that function, you also get a bunch of


unit tests, it is kind of like stuff that we've seen before. But it's also different. The


architecture here is nothing special. It is a decoder only language model that is first trained


on on just source code in general, and then fine tuned more and more towards this challenge. One


interesting thing is that as they progress, they pay attention to the quality of data, which seems


to be quite important in these code completion models. So they verify the abstract syntax tree


of Python files. And then as an intermediate step before they actually go to the data set,


which is remember human descriptions plus the function body that you're supposed to generate,


they do take the doc strings of functions that are of appropriate length as an intermediate,


like as a proxy task. So they view the doc string as the description, and then they generate the


function body from that seems pretty straightforward. And obviously, there is lots of


suspicions that things like copilot are training at least in part on similar things. Now they do


have a bunch of other improvements and technical nuances over which I don't want to go in here. But


all of this results in models that are smaller than other code generation or other coding


competition models yet improve upon their performance, which is pretty cool. So if


you're interested, check out the paper, I'll link it in the description.


And just a few helpful things for this week. Quaterion is a blazing fast framework for


fine tuning similarity learning models. So the specific focus here is on fine tuning these


models in a very fast and data efficient way with small data, I should say potentially small data,


obviously, you can use large data, but it is possible with small data. This is built on top


of pytorch lightning. So it's quite accessible and user friendly. torch dim is a project out of


pytorch. It's in preview, but it introduces named tensors. So named tensors are a concept of first


class dimensions in tensors and things like pytorch. Now the idea here is that instead of


you having to remember that the first dimension is the batch dimension and then always address with


a zero and just keep that in mind is that you address dimensions specifically. So this introduces


a dim type a type for dimension, for example, batch and then you can simply use that batch dimension


in order to index tensors. This isn't a speed up in runtime or anything like this. It just makes


code a whole lot more reasonable and a lot less prone to error. The mosaic ml composer library


now has automated gradient accumulation. So they claim that composer lets users seamlessly change


GPU types and number of GPUs without having to worry about batch size. CUDA out of memory errors


are a thing of the past. I'm not going to believe that I'm sorry, even if you solve every single


problem that we know of CUDA out of memory errors will stay with us until the eventual downfall of


civilization in the year 2089. But apart from that with the trainer of composer, you can simply tell


it to gradient accumulate automatically gradient accumulation is a concept where you don't pass


the full mini batch, you only pass part of it, which I guess is then called a mini mini batch.


So the full mini batch, if you wanted to run it, you propagate it and computing the gradient would


blow your memory because you're training a transformer that's just too big for your GPU


at that batch size. So you can propagate just you know, a few samples or even one sample,


you can propagate it and then essentially store those gradients and propagate the next thing and


then accumulate those gradients in place until you've passed the entire mini batch and only at


the end of passing all the individual samples or sub parts, you will then do the gradient update


step to your weights. This is a known trick. So essentially, you're training behaves as if you


were to use the large batch size. And we know that large batch sizes are important for some of the


current models, especially the large ones. So it behaves like you train with a large batch size,


but you can run it on hardware that can only handle a smaller batch size. Now the trade off


here is time though you use the amount of forward passes in time that you split your mini batch into


but it's better than not being able to run it at all. And this library does it automatically. And


lastly, m map ninja will store your training files as memory map files, which makes training iteration


or evaluation any sort of iteration over these files a lot faster. So here the read me says,


when do I use it use it whenever you want to store a sequence of non pi arrays of varying shapes that


you are going to read from at random positions very often. So the problem here is that if you


have a file on disk with a lot of stuff in it, and you want to read at random positions, then very


often the operating system makes you scan that file either from the beginning or from some


intermediate large chunk barrier and that can be very cumbersome. So memory mapping is a way of


speeding that up and this library handles it transparently for you. Alright, that was already


it for this episode of ML news. Let me know what you think about AI models that code and everything


When BERT Plays the Lottery, All Tickets Are Winning (Paper Explained)

Hi there, today we're looking at when Burt plays the lottery all tickets are winning by Sai Krasana, Anna Rogers and Anna Rumshisky.


So a high-level overview of this paper is the following. The paper basically looks at Burt in terms of the lottery ticket hypothesis.


And it says that if you fine-tune Burt on different downstream tasks, then the lottery ticket winners you're going to find are different between the tasks.


And also the claim all tickets are winning refers to the fact that if you remove the winning tickets, then you can still train the rest to relatively good performance.


Therefore all tickets are winning, not just the sub-network.


So that was the high-level overview for those of you who just want to be interested if you want to continue watching this video.


If you do like videos like this, consider sharing, liking, subscribing, telling your mother, father, brother and friends about it.


Alright, let's dive in.


So Burt, it is a language model. Basically if you don't know what Burt is, I've done a video on Burt, but really quickly.


What you can do with Burt is you can take a sentence, something like hello there, and you can put it through this multi-layer neural network.


And what you'll get out is basically an embedding of that sentence.


So a vector embedding of it. We'll make it really easy.


And what is usually done is this is pre-trained on a task called masked language modeling.


This is unsupervised training.


And then you take this embedding and you fine-tune.


Basically you put on a classifier head and basically say let's take sentiment classification.


So you have two output classes and you want to say is this sentence I put in positive or negative sentiment.


So you would train this classifier by basically taking this part from the pre-trained masked language modeling and then training the part here that does the sentiment classification.


You would sort of add that on top and then fine-tune the entire network to solve this task.


So that is basically Burt fine-tuning on different tasks.


And there is this benchmark called GLUE where it has a number of tasks.


In this case I think they look at nine tasks of GLUE.


It has nine of these tasks.


One example is the sentiment classification and it basically gets a score for each one.


And thereby you can sort of estimate how good your language model is by how well it is performing on each of those individual tasks.


But the notable difference here to let's say a computer vision like an ImageNet classifier is the fact that first it is pre-trained.


This part here is pre-trained on a large corpus and second there are different downstream tasks that you fine-tune on.


So the second part that is important is the lottery ticket hypothesis.


So I've also done a video on the lottery ticket hypothesis.


And if you very quickly what the lottery ticket hypothesis is the following.


So let's say you have an image classifier and I have a bunch of layers in my neural network.


I'm going to draw them like this.


And at the end I can classify it into like 10 different or a thousand different classes, whatever.


And the input here is an image.


So my neural network is going to have weights.


So every one of these neurons is connected to each other.


This can be a convolutional network or a MLP.


So all is connected to all and as well here, right?


Everything's connected to pretty much everything.


So we know first of all, we know that we can train these big networks to relatively good accuracy.


And then second of all, so first we can train them.


Second, we know we can prune them after training.


What pruning means is the fact that after I have trained such a thing,


I can then go and I can figure out which of these connections that I have learned are the important ones.


And maybe I'll say, ah, these here, actually these five, I don't need more than those.


Let's actually connect them to the end.


These seven or so.


I don't need all of the other ones.


I just need those and I can pretty much get the same accuracy as the full network.


Now, the important part here is that you can only do pruning after you've trained a network.


If you try to prune at the beginning, it doesn't work.


So what the lottery ticket hypothesis says is basically, so how does training work?


First, you have your parameters.


Let's tell them as a list.


So each of these weights is an entry in the list here.


First, you initialize these randomly.


Then through training, you get to your train state, right?


You get each of the ones into your, these are now trained.


And in the train state, you can select the ones you think are important.


The lottery ticket hypothesis says if I take those that are important and basically go back to the beginning,


like here, here and here, and I basically roll them back to that state that they were in when they were initialized.


So I put the same random number there that I got at initialization.


I can then make a network where I only have those.


I can train that network and I can get a good accuracy.


So this basically wasn't possible in the pruning framework because we said we can't prune,


we can only prune after training because only then do we know which ones are the important ones.


In fact, the lottery ticket hypothesis or the paper shows that you can train the smaller neural network from the beginning.


But the catch, of course, is you have to know which ones those are and you have to know what value to set them on at the beginning.


And you only know that after you've trained the full network.


But still, it kind of gives the it tells you that you don't need all these connections for training.


You basically only need so many connections such that somewhere in there, there's going to be the good ones.


And if you knew the good ones from the beginning, you could just train those and then only train the smaller sub network.


So that's the lottery ticket hypothesis.


Naturally, these connections here, this small network is called the winning a winning ticket,


because if you knew what it was, you could basically train a much smaller network and reach the same accuracy.


So this paper looks at BERT in terms of this lottery ticket hypothesis.


Now, it's a bit more complicated than just in these feed forward networks because BERT is not a feed forward network.


BERT is a transformer.


So what does that mean?


A transformer consists of many layers and each layer, let's expand the layer here.


So each layer consists of let's go over there, need some space.


So again, we have our layers of BERT and it goes signal goes like this.


So each layer consists of two things.


First of all, of many attention heads.


That's called I'm going to draw these as blocks right here.


So for let's call them for attention heads.


Individual attention heads are all parameterized by individual matrices.


And then on top of that, there is an MLP.


So this is the multi layer perceptron.


This is one basically a feed forward layer residual actually.


And so there is a skip connection right here.


And these are the attention heads.


OK, and then the next layer would again have the same structure for these and then one of these and so on with the skip connection.


All right.


So the pruning in BERT is different than pruning in the feed forward or convolutional networks that we looked at.


Pruning in BERT to what this paper looks at is pruning either an entire attention head like this.


So kind of leaving out in the entire head away, which is this is an entire matrix is not a single weight.


This is many, many weights or even more drastically leaving away an entire MLP and basically only relying on the skip connection.


All right. So this you have these two these two things you can do.


You can leave away heads or you can leave away entire MLP.


So you can combine these things in some way.


Right. So the notable difference here to the lottery ticket hypothesis pruning is the fact that here over up here what we prune are connections.


So prune connections, individual connections, individual connections.


And here we prune entire modules.


Now, this is a in my opinion, this is a qualitative difference, a very large qualitative difference, actually.


Why? Why would you do this?


So this paper basically doesn't invent this kind of pruning.


They go after already existing literature. So what's the advantage in pruning modules?


Well, you have to see what's the advantage in pruning per se.


So in pruning, what you're trying to do is obtain a smaller network that gives you the same accuracy, but that you can run faster.


Right. You that uses less memory and you can you can run it faster.


And if you prune like this, like we did in the lottery ticket hypothesis paper, you don't really gain anything.


Because if you have a matrix, if you have a matrix, matrix multiplier, I have two matrices and I multiply them right here.


If I if I cut out one weight here or one weight here, it doesn't help me because I have GPUs and those will parallelize these matrix procedures.


And it doesn't really help me because we don't really have good hardware for sparse or matrix, matrix multiplication or matrix, matrix multiplication with holes or things like this.


So it almost gains me nothing.


The lottery ticket hypothesis paper is very much a kind of more of a scientific curiosity paper.


And once we have sparse matrix multiply hardware, which I think it already exists but is not super widely distributed.


Once we have that, we will be able to make use of this. Whereas the people that prune BERT.


So these are more, let's say, industry people.


If you prune an entire module, well, that's an entire matrix that falls away.


So I have to I can basically save an entire matrix, matrix multiply in the forward pass here on the and the backward pass.


Well, OK, I don't prune during training, but I can basically save an entire matrix multiply here by pruning an entire module.


So I'm not sure if if I were an author and I say I want to look at BERT in terms of the lottery ticket hypothesis, I would find a way I would go away from this literature and find a way to also just prune here individual weights.


It's not going to be faster, but the lottery ticket investigations aren't supposed to be faster.


It's supposed to tell you something about the nature of the things you're investigating.


And of course, how you do this is simply by masking.


You simply force these entries to be zero and therefore you don't have forward signal, you don't have gradient.


Interestingly, they actually do the masking, but they do it on the whole entire module level.


OK, so this was BERT and the lottery ticket hypothesis and the all tickets are winning.


We're going to investigate later.


Let's see what they say in the abstract.


Say much of the recent success in NLP is due to the large transformer based models such as BERT.


They say, however, these models have been shown to be reducible to a smaller number of self attention heads and layers.


So this would be pruning.


We consider this phenomenon from the perspective of the lottery ticket hypothesis.


For fine tuned BERT, we show that here's the contributions.


A. It is possible to find a sub network of elements that achieves performance comparable with that of the full model.


So basically, this is the pruning objective, right?


You want to to prune it such that the performance holds.


And in terms of the lottery ticket hypothesis, you want to prune, reset to the beginning and then also and then train again.


And that will give you in actually in the lottery ticket hypothesis.


You can gain performance if you prune by a certain amount.


In this case here, they always lose performance.


But yeah.


So second of all, similarly sized sub networks sampled from the rest of the model.


So the non winning ticket perform worse.


So if you just read if you just prune away the good parts, then the bad parts perform worse, of course.


However, the bad sub networks can be fine tuned separately to achieve only slightly worse performance than the good ones,


indicating that most weights in the pre trained BERT are potentially useful.


So this is interesting.


If they be fine tuned separately, this is exactly what the lottery ticket hypothesis is doing, right?


It's basically fine tuning only a sub part of the network.


And here they say, even if we take the parts of the network that have low scores for pruning and we retrain those, then we can achieve a good performance.


OK, so they further they say, we also show that the good sub networks vary considerably across glue tasks.


This is this benchmark opening up the possibilities to learn what knowledge BERT actually uses at inference time.


All right. So there this is the overview of the paper.


So a last thing to say, which I've already kind of alluded to, is the fact that in the original lottery ticket hypothesis, as I said,


you had a graph and you had some sort of here was 100 percent accuracy and here was how much you prune.


Of course, you start at 100 percent if you prune nothing.


But then as you prune, the interesting thing is it kind of goes up and then it goes down.


Right. So this is the first thing here. It goes up to a certain amount if you if you don't prune.


And in the original lottery ticket hypothesis here, somewhere here would be 50 percent of the network, I think.


And then once you go down, let's say here to 90 percent of performance, you are at something like five percent of the network size or three percent.


So you can prune away most things and still be like extremely, extremely powerful.


Now, we're going to see what these essentially what these people do here is here is 100 percent and they simply prune until they reach 90 percent.


So we don't necessarily know what happens in the middle.


We just know they start here and somehow they get to 90 percent and what they end up with is something like 50 percent of the network still remaining.


So again, see the qualitative difference here between the five percent of the lottery tickets in the original paper and the 50 ish or so percent or considerable amount more in this paper right here.


And I'm pretty sure that is due to the fact that they prune entire modules here.


So they don't prune on a fine grained enough level to investigate these phenomenon, because, as I said, we don't know.


But I'm pretty sure this just goes down here and not up first. So qualitatively, it seems different.


All right. So here they introduce what they do.


Again, BERT is made up of these attention heads and MLPs.


The MLPs have a skip connection, as you can see here, and the attention head attention layers are basically made up each of n of these attention heads.


What they will do is they'll they will look at 12 layer networks.


Each layer will have 12 of these attention heads and one of the MLPs.


So you have in total 144 heads and 12 MLP layers.


The way they determine which ones to prune is pretty easy.


In front of each attention head and in front of each MLP, they put one of these binary variables right here.


These variables can take values zero or one.


If it's if they're zero, the layers or the head is basically inactive, no propagation. If they're one, they're active.


And they determine what value to set them to by computing important scores, basically determining how important is a head or a layer for the network.


And that's pretty simple. You simply take the gradient of the loss.


I think they go after this paper right here. That's a post or post following.


You derive the loss by the by these variables right here, and therefore you get these important scores.


And then you can simply prune the layers with the lowest important scores because that means that the gradient with respect to them is the smallest.


That means your loss changes the least if you were to leave them away.


OK, so. They here determine their their pruning strategy.


Their constraint here is, as I said, 90 percent of the performance of the full model, so they train the full model, fine tune the full model on this task.


And then they they set themselves a budget of 90 percent and they simply prune until the model reaches 90 percent.


Once it goes lower, they stop. OK, so they have three methods of pruning.


One is heads only where they only cut away these these attention heads.


As I said, there are 144 of them. They have the pruning strategy of MLPs only where they only prune the MLPs, leave all the attention heads alone.


And they have this heads and MLPs. They say we compute head and MLP important scores in a single backward pass, pruning 10 percent heads and one MLP with the smallest scores until the performance on the dev set is within 90 percent.


OK, then we continue pruning heads alone and then MLPs alone.


And this, I guess, until again, they are no longer in the 90 percent. So until they reach their budget.


So this is a combined strategy. This strategy results in a larger number of total components pruned within our performance threshold.


So this is the this is the thing we should focus on. Right. Because in pruning, the name of the game is how much can you take away and still be within your budget?


And this strategy seems to be the viable strategy here. So.


Our last thing here is fine tuning. So the other difference between this paper and the lottery ticket hypothesis is that we said that in the original paper here, these are randomly initialized weights.


Like you train a class for an image net or something, you start from randomly initialized weights and the lottery ticket papers, they all they all kind of presuppose random initializations.


Whereas BERT, when you do the same thing for BERT, these are not random initializations.


We said in BERT, what you usually do is you train the the encoder part here, you pre-train with masked language modeling first and then second, you train the entire thing.


Let's get the color here. Second, you train the entire thing. You fine tune the entire thing.


So if we talk about initializations in the BERT task, then the initialization would be at this point right here after the masked language modeling would be the initialization.


So the weights are not random. The weights are actually pre-trained on the masked language modeling task, which is also a qualitative difference and sort of lets us inspect.


So the authors say that since we train with masked language modeling and people sort of claim that masked language modeling learned something about the language, we can now investigate kind of which attention heads, which modules in BERT are encoding which parts of the language.


And this is going to be interesting once we look at which heads, which attention heads and which modules survive in the individual tasks, we can sort of compare tasks across each other by seeing which of the heads they share in their winning tickets.


All right. So they produce these graphs here. These are sort of one of the central graphs here.


And the way to read this is on the left side here, you have the layer. You have the layer index.


And on the X axis, you simply have the index of the head. There are 144 boxes here. Each one corresponds to one of the attention heads.


The top number is always the mean number of glue tasks that this head survived in. So what they do is they take the pre-trained BERT, they fine tune it on these nine tasks.


And for each of the nine tasks, they determine the winning tickets.


And the number here says how many in how many of these nine tasks is this particular attention head a part of the winning ticket.


Now they repeat it for different random seats. That's why you have floating point numbers. And the lower part is the standard deviation across that.


So you can see quite a number of heads make it into a lot of these tasks. So you can say this part, this thing right here, red on red.


This head right here survives in seven out of the nine tasks. So it should probably encode something fairly substantial about language that is shared across these seven tasks.


You can see some of the heads like this one here doesn't survive in almost any task, which basically means that it's, you know, that one's not really super important for these tasks.


It might have been, you know, important for the pre-training, but not for these particular tasks.


What's interesting, so what you can see is that the mean or median or so is like three, four or five.


And that means that a lot of the heads are sort of somewhat important for some of the tasks.


And you can see the qualitative difference. If this were the like original lottery ticket paper, most of these numbers would be at zero because the lottery ticket size is just so much smaller.


But here you can directly see that you are going to retain a large number of things in your network in order to get 90 percent of the performance.


And that's probably because you prune entire modules again.


So they have this for two variants here.


The first for this strategy of masking heads only and the right one is for masking heads and MLPs and the same here on the bottom. These are the same numbers, but not for attention heads, but for MLP layers.


And you see again, this is masking MLPs only. This is masking heads and MLPs.


And if you compare the two, you see that, for example, this here and this here are substantially darker, which means more of this stuff survives.


Now, we can't really. It seems like here, for example, there is it's darker than here.


So on the right side, more stuff survives, but also you have more things to prune, right?


You can prune the heads and the MLPs. And they claim before that the masking heads and MLPs strategy results in more things being pruned, which doesn't really isn't really congruent with here generally more things surviving.


But it could be because of the fact maybe the sum of the two is still lower than the sum of each individual's individual thing here, though it doesn't really look like it.


So I'm a bit confused about this, but I'm just going to assume that the sum of the two is is is lower.


Does that make sense if both are dark? Well, it shouldn't be the sum. It should be the sum of this plus a completely dark this in terms of masking heads only or vice versa versus the sum of those two.


So that should be the measure. But it just seems a bit a bit doesn't work out too much.


But OK, that's what they say. So by the way, if the authors are here, you have a cut off.


Yeah, this is annoying. This is like trying to get lot to do things and it doesn't comply.


All right. So what you can another thing you can see in the authors point out here is that if you mask heads and MLPs, you sort of shift more things to the back of the network, to the higher up layers.


And they reason now because you also mask the heads. Basically, they can't do as much work.


So you so the heads would be masked somewhere here. So all that work is going to shift upon the MLPs and mostly to the back of the network, because this thing here cannot take over work.


That's this attention head here is now not performing anymore because it was pruned because the signal travels this way.


So the authors kind of interpret these results right here. And I think the most important thing to see is simply the variance of things.


So most heads are actually important for at least two or three tasks and no head is important for all the tasks consistently.


I think that's the take home message right here.


OK, and they contrast this to previous research that has basically said this experiment follows up on a study by this that showed that only a few transformer heads in machine translation task did the heavy lifting, while the rest could be pruned.


And this paper similarly showed that most of BERT self-attention head in MLI task could be pruned and that the good heads were mostly shared between the MLI matched and mismatched.


And they basically say, yeah, that's correct, but that is only within one task.


If you go beyond, if you go to several tasks, then the heads that are important differ quite a bit.


OK, so let's continue and go here.


They ask how task independent or the good sub-networks.


And they basically look at these kinds of graphs right here, which are pretty interesting.


So we've got this. This is heads shared between tasks.


So what this measures is these are the different tasks in the glue benchmark.


And they basically look at each task, look at its winning lottery ticket and look at which heads survive in the winning ticket.


And then they put that here on the diagonal.


So if in QNLI a head survives, it gets a one here.


And if it doesn't survive, it gets a zero.


So on average, 85 out of the 144 heads survive, right?


85 heads survive. That's pretty, as I said, this is somewhat like over 50 percent of the network.


It's entirely different than the original lottery ticket hypothesis paper.


So 85 percent, not percent, 85 of the 144 heads survive.


Now they look at the other tasks. So for QNLI, they would look at maybe MNLI task here and ask which of the heads that survived in QNLI also survives in MNLI.


So that gets you the shared heads. And again, the lower number is a standard deviation.


So 62 heads survive in QNLI and MNLI.


And the authors here are sort of arguing that from these sort of numbers, you should be able to see which of the tasks share different linguistic knowledge.


So different linguistic knowledge could be relevant for different tasks.


But if some tasks share a lot of the attention heads that survive in the winning tickets, that basically means that the model is using that information that is in that head for both tasks.


This could be good in that you say, oh, yeah, these tasks really are used similar linguistic features.


Or it could be something that you don't expect.


And then you might be able to investigate maybe the model is doing something shady here because it really shouldn't. You know, these tasks don't really have much in common.


So they do this for the heads and the MLPs here.


Now, if you ask why the WNLI here has a bunch of zeros, that's because it's a wonky task.


And basically the best thing you can do is predict the most frequent class.


So you can prove just about anything away on these MLPs because they have the skip connections.


You don't need them to predict the most frequent class.


What I want to go into is the following statement right here.


So note that figure one, so the figure before, shows very few heads or MLPs that are universally useless.


Only seven heads that survived in less than two tasks.


86% of heads and 67% of MLPs survive in two to seven tasks with relatively high standard deviation.


They say this means that the good subnetworks for different tasks have relatively little in common.


Right. So they make this sort of statement again here that these the good subnetworks have little in common.


And it might seem like that for the figure initially.


But if you look at this figure, it actually shows something pretty interesting, I think.


So if you look at a number, let's say, for example, this here, this 74.


And I haven't actually tried. Yeah, let's look at the 74 and this 7 this here.


So let's look at these tasks Q, Q, P and R, T, E.


OK, so if you look at QQP and RTE, you could see that these are tasks that already they don't have a lot of heads in common.


Right. And you might be able to say, well, if what they're saying is true, that the tasks share relatively little, you would expect them to be relatively independent.


But if I look at this 78 here, it means that 78 out of 144 heads survive.


And here it means that 74 out of 144 heads survive.


So if I now would think that, OK, generally for different tasks, things are different.


How many heads would I expect there to be surviving in both if the tasks are independent?


So that's these two things multiplied, right, times 144.


So I can scratch this here and the 7 times 7 is whatever, 49.


Let's go 7 times 8 about this. So that's 5, 6.


Do I need to get out the calculator? I want to do this calculator.


Shaka bean. I do this the right way. OK, I hope you can see that.


So that's 78 times 74 divided by 144. Did I do it right?


Probably did it wrong. I did. I. 78 times 74 divided by 144.


All right, so that's 40 heads. And you see that there's 43 heads.


And I've actually gone through a bunch of these numbers before, not these ones, but generally the shared number of heads is higher than what one would expect if you assume that the tasks are independent.


And I'm sort of missing sort of an analysis of that here because that I find to be a pretty interesting finding of these things.


And sort of I mean, I get the fact that they say based on the graphics up here that the tasks are sort of seem to be relatively independent with respect to the heads that survive.


Of course, relatively independent is a relative term, but sort of an investigation into why we see considerable dependence between tasks here in terms of that.


So these numbers are always over the what you would assume for independence. That seems to be pretty interesting.


All right, so they say they here go into this figure two and this pairwise comparison and they analyze a couple of the different tasks here and what you would expect.


And I don't want to go too much into these tasks because honestly, I also don't know all of these tasks.


I don't know which tasks should share a lot of things, which ones shouldn't, but it is a good way, like it is a very smart way to investigate if the model really learns similar tasks to use similar information.


All right, the last thing they do right here is the good and the bad subnetworks in BERT fine tuning.


So they say our final experiment puts the above evidence of good subnetworks in BERT fine tuned from the perspective of lottery ticket hypothesis, which predicts that the lucky subnetworks can be retrained from scratch to match the performance of the full network.


To test this hypothesis, we experiment with the following subnetworks. So that means I wasn't really sure when I read it the first time, but now I'm fairly sure that all of the results so far were just pruning and maybe not retraining.


So just sort of doing the pruning thing and not doing this lottery ticket retraining, which shouldn't make a lot of the difference as we're going to see, but just for the understanding, because it seems like pruning and retraining doesn't do that much for the winning tickets, as you'll see right now.


But yeah, so now they actually retrain from scratch. So good networks, the elements selected from the full model by importance scores as described in the previous section.


So here they're going to evaluate these good networks. First of all, they're going to evaluate them pruned and they're going to evaluate them retrained in the lottery ticket style.


Okay, then they're also going to evaluate bad subnetworks, the elements sampled from those that did not survive the pruning plus a random sample of elements with high importance score, such so as to match the size of the good subnetworks.


So because the good subnetworks are 50% or more of the network, they're going to sample from the things that did not survive, so from the bad ones, and they plus a random sample of the good subnetworks to just match the size.


Okay, so we would expect these to perform maybe worse, but maybe we can also train them to achieve good performance. And then they investigate bad subnetworks, simple inversion of the good subnetworks.


So these would be just anything but the good.


They are 5 to 18% smaller in size than the sampled bad subnetworks, but they do not contain any elements with high importance scores.


And they say okay for all of them, they evaluate their performance on all tasks, simply after pruning and with fine tuning the same subnetwork with the same random seeds, and with the rest of the model masks.


So this is really what the lottery ticket hypothesis does except they of course mask entire modules and not individual weights. And here you can see the general results.


The general results look like something like this. This is a typical example. So this is the...


Let's go out. Oh yeah, this here is simply the dumb classifier that always tells the, the highest probability class. This is the, like this is sort of the the idiot's baseline. Okay.


This here is the full model.


This here is the good pruned and this here is the good after it's retrained again. Okay. So you see by retraining you can basically gain.


Now the original lottery ticket, this would sometimes even go up here depending on how much was pruned.


But you can see that there is a slight gain after you retrain the pruned part.


Okay. And the other thing to note here is that you don't lose much. Basically, you, you only drop a little bit by pruning, which that's what makes it the good part.


You only drop a little bit. However, if you have the bad part, which are these, and let's say the good plus bad.


These are the bad plus some of the good ones. You see that the performance drops pretty heavily, almost to the baseline of the most frequent class.


And also here. So I would actually, I would go with this one right here if because that's just the bad ones.


You see the performance drops considerably, but then, and that's what the authors claim is pretty interesting. If you retrain that part, the bad part, so to say, you can achieve sort of a very comparable performance to what you can achieve with the good parts.


And this appears to be true for most of the results right here.


There are some outliers like this one, but there the score is also, so this is the Matthews correlation and not the accuracy. So the score is a bit different there.


But you can see here the good plus bad also gets a fairly high accuracy.


So the authors claim this is pretty surprising, which I guess it is if you look at this. But what I want to do is I actually want, I have asked the author of the lottery ticket hypothesis this question.


And this is from our machine learning street talk with Jonathan Frankel. And this is another channel that I am I'm a part of. And I would like to show you this right here when I asked him this question.


This question from Reddit in Nemo asks, Suppose you try to construct a lottery ticket by taking all the weights that were not part of a winning ticket and retraining from those will that model be unable to learn the task, or might there be another winning ticket hiding among them, or one that wasn't originally used.


This is actually the most common question I get by people who read the original paper. And I hope that by answering it here in a public forum I can answer it once and for all.


The challenge in doing this experiment is let's take the MNIST example. So suppose that we find a winning ticket on MNIST. It's going to be about 3% of the original size of the network.


So that means that if you remove it you've still got 97% of the weights left. And so my guess is that if you were to train those 97% of weights, you'll get to the same accuracy as you got with 100% of weights because you've barely pruned the network at all.


You could randomly prune by 3% and it wouldn't affect it. And then you could go and find another lottery ticket that's mutually exclusive with the first. You still have 94% of the weights.


So you could probably iterate this for a very long time. You could probably this way find 10, 15 lottery tickets like this, maybe more, that are all mutually exclusive and still leave you with a remaining residual that is capable of training to full accuracy.


So the challenge with this experiment is that the lottery tickets are small, which is great, but it means that whatever is left is large enough that I'm sure there's another lottery ticket in there and another lottery ticket in there and so on and so on and so on.


So it's an interesting idea in principle, but once you kind of look at the sizes of things, you've still got so much over parameterization left that I think you just find more lottery tickets. You can even probably, I'm guessing, swap out one weight from a lottery ticket with another weight and it wouldn't matter or swap out a handful of weights.


And so, combinatorially, the number of lottery tickets is massive and we're just finding one.


Right, so as you saw, this is kind of the most common question that Jonathan gets here.


And as you can see, the difference here, of course, is that our original tickets are already sort of 50% of the network. So what's left is only 50%. So this is substantially different.


Now, two things I have to remark here. First of all, because we are pruning modules and not individual weights for the good one, it's the reason that we do get these big winning tickets, right?


But also what I think is happening is that because we are pruning these entire modules, we're actually not fine grained enough. So that means every time we eliminate a module, we actually kill some good ones and some bad ones.


And so in here, I'm going to guess there are some good ones and there are some bad ones. But since we can only kill entire modules, we simply kill the one that on average has the most good ones.


But I'm guessing that in the thing we kill, there are simply there are, sorry, we kill the one that has on average the least good ones. But there are still some good weights in there.


And if you believe the original lottery ticket hypothesis, that means that these actually these very few weights in those modules can still train to full accuracy.


And actually, what these authors claim is surprising in light of the original lottery ticket hypothesis, I think if you look at it from the perspective of the actual hypothesis, which considers individual weight and a very small subset of them,


the original hypothesis would pretty much predict that you could train something where you pruned away a bunch of modules entirely. Or you could train these bad modules because they are still going to contain a small sized lottery ticket that is going to be responsible for the good performance.


So that's the kind of the first thing. And the second thing, in general, you heard Jonathan, I don't think that is actually even a question of the size of the tickets.


Nothing in the original hypothesis forbids the non-winning ticket from also being trained to good accuracy. It simply says something about the winning ticket. It doesn't say anything about the non-winning ticket.


So those are the two comments. And I think the question and the investigation, even though it is interesting, I think is sort of maybe not thought through, at least in the perspective of what they go for here.


I mean, the result is very interesting. But again, I think they claim the original hypothesis would sort of say these are the bad parts and you couldn't train them. And then they say it's surprising that you can.