take much processing. So the Bayes algorithm is pretty simple. You're going to have basically


Naive Bayes - Natural Language Processing With Python and NLTK p.13

the posterior and that's just like you know the likelihood equals and it's going to be


prior occurrences times likelihood and then all of that is going to be divided by the


current evidence and that will be the likelihood of something to be positive and then you would


do the same thing for something to be negative. And that's basically all it's done. So it's


a quick algorithm. If you want to know more about that algorithm, I highly suggest you


just go Google it and read like I don't know the Wikipedia page and maybe some articles


on it and figure out more how it works but it really is actually quite the basic algorithm


as far as learning algorithms are concerned. But it's again not necessarily the best algorithm,


it is just scalable and it's really easy to understand. So once we've gotten to this point,


now we're ready to actually create the classifier that we want to use because we've got a training


set, we've got a testing set and now we're ready. So first of all, we have NLTK already


imported so we'll just continue using that and we're going to say our classifier, classifier


is going to equal NLTK and we're going to use the naive Bayes classifier algorithm.


We're going to dot train and we're going to train against our training set. I'll just


copy that and paste it. And now, first of all, what we can do is we can do a couple


of things. First of all, we can check accuracy right away. So we've trained it, now we can


test it. So we can just say print and we'll say naive Bayes algo accuracy colon comma


and then now we can use NLTK dot classify dot accuracy and we can run that and what


you, the parameters that you pass through here are going to be what classifier are you


using and what's the set that you want to test against, that's testing set and then


we'll just multiply the answer by 100 so it's in a percentage and we'll even denote this


as accuracy percent. Okay, so then, so we can do that but we'll also add one more thing


to it. We're going to say classifier dot show underscore most underscore informative underscore


features and we'll show the 15 most informative features and let's go ahead and save and run


that. Let me just check to make sure I don't need major syntax errors. I don't think so,


so let's save and run it and we'll talk about this as we're waiting. So this will print,


we don't need to print this anymore, let's stop that from happening. So that's going


to hog up a bunch of space real quick but anyway, now it won't. So this trains it, this


runs the accuracy and what this is going to do is tell us like the most popular words


on both sides and whether or not they tend to be positive or tend to be negative. So


it looks like it's probably updating now. Right, so first of all our Naive Bayes algorithm


accuracy percent we're given here is 89%. Now I will add that depending on your samples


and actually let's go check and make sure we didn't screw anything up. This is actually


a higher number than I normally see with the Bayes algorithm. You might find that yours


is as low as like 60 or maybe even lower. Sometimes I've seen it hit 59 and stuff while


I was setting this up. So just understand that this is highly volatile and don't be


like ashamed if you didn't hit 89. You probably didn't do anything wrong. Anyway, so now


we can look down here and we can see okay what are the most common words. I can't believe


3000 is one of them. I'm actually kind of shocked. I want to go check and make sure.


It is a little eerie that we use 3000 here but that shouldn't make any difference at


all. Anyway, so we can see here one of the more important, the most important one is


this word slip and it appears 11.6 times more often than it does in a negative review. 3000


is a negative one. I'm trying to think of 3000 like why would 3000 be one. Anyway, then


you've got symbol, seamless, idiotic makes a lot of sense. This is an unimaginative.


Okay, these ones make a lot of sense. Idiotic appears 8.5 times more often than it does


in a positive review. Unimaginative, same thing, 7.7 and so on. So we can actually end


up using these later on down the road to change the algorithm slightly but as you can see


none of these contain grammar and that's why grammar just didn't matter. We have such a


large data set. We have 3000 most common words that grammar is of course not on here because


it isn't an informative feature. So we can actually kind of use that to our advantage


and since I'm talking let's rerun this one more time before I cut this tutorial off.


I'll be really surprised if we get another high 80s return so hopefully we'll get something


a little more reasonable because it's just uncommon that you're going to hit that much


accuracy but anyway so we'll just wait for this and then we'll talk about this. So yes,


it's not 100% like I said it's not as accurate as 89. 89 is unlikely. Anyway, we'll talk


about how to improve the accuracy of this or at least improve the reliability because


one of those two is really important. It's nice to have both but we definitely want to


have reliability over random high accuracy points. So anyways, that's it for this tutorial.


If you have any questions or comments please feel free to leave them below. Otherwise,


as always, thanks for watching. Thanks for all the support and subscriptions and until


TransGAN: Two Transformers Can Make One Strong GAN (Machine Learning Research Paper Explained)

Hi there, today we'll look at TransGAN, two transformers can make one strong GAN by Yifan


Qian, Shiyu Cheng and Cheng Yang Wang.


So in this paper, the authors attempt to make a generative adversarial network, a GAN, out


of only transformers.


So far, attention or transformer-like things have been used in GANs, but they've always


had some component of convolutions in there.


This paper attempts to do generator and discriminator, just using transformers.


They discuss what is needed to do that, how they built the architecture.


And there are a couple of training tricks that make this work and actually make this


competitive to current state of the art architectures.


So the biggest data set they tackle is Celeb A, which is 64 by 64 pixels.


But you know, due to their numbers suggest you can scale this much larger.


The model is called TransGAN.


I don't know if this is a bit of an unfortunate naming.


I guess the question is which bathroom do the TransGAN go to?


I don't know.


In any case, let's dive into the paper, let's check it out.


If you like content like this, share it out, leave a like and tell me what you think in


the comments.


So the paper is fairly straightforward.


Actually, there is code available.


So definitely check that out.


I'll link that of course, in the description.


The paper is fairly straightforward and answers one question.


Can we build a strong GAN completely free of convolutions?


So usually in GANs, you have convolutions both in the generator and the discriminator.


And their goal is to just replace that using transformers.


As we say, there are contributions, there are three, the model architecture.


So the discriminator, as we're going to see is a vision transformer.


Like we saw before, the generator is also a transformer that is interlaced with upsampling.


Then training technique, they do discuss that you do need three things specifically.


So you do need data augmentation, you need multitask co-training for the generator, and


you need a localized initialization for the self attention in order to make this work.


And then they reach a GAN.


So their model, their biggest model TransGAN-XL reaches very competitive FID scores, and also


very competitive inception scores.


Wait, this is FID, here is the inception score.


The IS score is a bit of a misnomer too.


I mean, the S is already score, but you know, who, it's okay.


So first architecture, the architecture is fairly straightforward.


So for a GAN, you need a discriminator and a generator.


Now the discriminator, as I already said here, that is the exact model from VIT and I've


done video about it.


The paper is called a picture is worth 16 by 16 pixels or something like this.


I don't exactly remember, but you can you can check you can definitely find that it


is a transformer based image classifier.


So what you do with an image, so here you see an example image, this image of the dog,


what you would see if you were to feed this into the discriminator, of course, the discriminator


gets the output from the generator, but also real data, you would, you would unroll that


picture into these kind of sub pixels, as you can see right here, but not into full


pixels, but into kind of the super pixels.


So every one of those super pixels will then be unrolled.


This is this flattening operation right here into a single vector.


And that then is like a word in a sentence, okay, so that this picture here just becomes


a series of vectors, and then you can simply apply your regular transformer architecture.


So every patch becomes a vector, like a word embedding, and then you just go ahead and


you put a transformer encoder.


So this is very much like BERT, for example.


It is a similar architecture, as you say, you can go look at this paper.


And at the end, you simply classify whether it's real or fake.


You do have to add position encodings, because, you know, lacking the convolutions, the transformer


has no idea where in the picture a given given thing appears, because it is not a sequential


architecture, it's actually a set transformation architecture.


So you do need to add positional encodings.


But in general, this has been shown to work quite well in things like ImageNet classification.


On the generator side, it is very similar, but you know, a little bit different.


So here, what you need to achieve are, of course, are these 32 by 32 by three pixel


image, right, that's at the end, you need to achieve that.


Now you can't just go the reverse from over here and somehow try to predict these patches,


because that I guess that is just too, you know, if you predict these patches as such,


like independent patches from each other, the borders would never match up.


In a discriminator, this is not, does not matter, because you don't need to construct


the image, you simply need to classify it.


But if you need to generate images, it's, you know, it doesn't look good if you have


these borders here where things don't match up.


So you will actually need to produce an image that is in the size that you require.


So in this case, yeah, 32 by 32.


And of course, three color channels.


So the way they achieve it is by this upsampling architecture.


The problem with transformers, of course, is they do require quite a bit of memory,


and also compute, because the attention mechanism basically connects every single token with


every single other token in each transformation.


In this case, they connect every pixel to every other pixel.


Now, if you were to do this for many, many layers, that is going to be you know, 32 squared


in this case, memory requirements, pretty quickly, you will run into problems.


So what they do is they have intrinsic upscaling of their dimensions.


What does that mean?


So at the beginning, you have like some some noise input, and you have a little MLP generating


the initial sequence.


Now, the initial sequence is going to be eight by eight by number of channels, you can see


there are also position encodings right here.


So your noise generator essentially creates an eight by eight grid, okay.


Let's say for the sake of argument, we create a two by two grid instead of an eight by eight


with a number of channels.


So here is the number of channels to the back, you want to unroll those into four vectors


of the these channels.


It's 1234, you get the idea, and then that you feed into the transformer.


So now you have four tokens or here 64 tokens in that case, but in our case, four tokens


that you feed to the transformer.


So right now, at this stage, this is like a sentence with four different words.


So you run that through m layers of the transformer.


And then at some point, you decide, okay, now it's time to do upscaling.


And the upscaling in the upscaling, you take that those four words, so you take that two


by two image that you have right here with the C channels, and you generate somehow from


it, and we're going to look at I'm going to draw this over here.


So you generate somehow an image that is double the density in pixels.


So this is now a four by four image, but it has less channels.


So the way they save memory is that they start out with many channels, but very, very coarse


resolution and progressively as they go up the layers, they up sample so that they have


more resolution, but less channels, okay.


And the exact so this is this is very much like, like the convolutional GANs do.


So like, they would start out with a very coarse image grid, and then they do some kind


of up sampling, some kind of strided pooling, and so on, in order to reach higher, higher


pixel densities.


And with the higher pixel densities, they often decrease the number of channels.


So you get a trade off between the density and the kind of depth of information.


At the end, they end up with their target resolution and a number of channels.


And then they feed that through a small they feed each individually through a small linear


projection in order to project that to the three channels.


So that's how they end up with three channels.


So how exactly does this up sampling work?


By the way, I hope you can you can see the the whole pipeline now, right, you start out


by this is this is sort of noise generated.


This is what is derived from the noise.


And then the input is just transformed, transformed, transformed, up sampled, transformed some


more up sampled, transformed some more until it is at the target resolution.


Thereby, in the lower layers, you have lots of information depth, not much resolution


in the higher layer, you have lots of resolution, but not that much information depth anymore.


So the computations higher up might be more localized, they might be more to do with the


exact kind of the exact details of that particular patch in the image, right, all of these things


are representative of patches, especially in the down scaled, like this pixel right


here is representative of all the pixels that are going to be generated out of it.


So of this one, one layer higher, and of course, one even one layer higher, it's going to be


of its own four by four pixel grid.


So the computation you do down here on this pixel will affect all of these pixels later.


The way they do the up sampling is by this pixel shuffle algorithm that they have from


this paper right here.


And I'll link to that, of course, as well.


So this is a paper that was, as I understand it, originally derived for convolutions.


And they'd asked, how can we do sort of convolutional operation on high resolution images without


having to do the compute for high resolution images.


And they figured out that if they had, if they had a high resolution image, they can


sort of represent they can rearrange a high resolution image into a smaller resolution


image with more channels.


So here, you see you have they call this r squared number of channels.


So this number here is r squared.


And they can sort of unroll this image into this one.


And they do that by treating these things here, maybe you can see this is a repeating


pattern as sort of super pixels and see that.


So one of these super pixels is going to be one column here.


Alright, so this this way, so you're going to up sample by having lots of channels here,


doing the computation on as if they were lots of channel in a low resolution image.


And then you up sample by just unrolling the channels locally.


So by treating each of these things as just, you know, one super pixel with the elements


of the channels being the, you know, kind of the different pixels in the neighborhood.


So you want to unroll that.


And then after that, you continue with your processing with putting this through the next


layers until you up sample it again by unrolling some more channels.


I hope that's clear, so you're going to start out with a lot of channels because each time


you unroll, you're going to lose some of them, you're going to trade off some of the channels,


channel depth for more resolution.


Alright, so here you can see, every time they up sample the resolution by two, they need


to divide the channels by four, because you need to up sample by two in the width and


in the height direction.


Actually it's not even necessary.


You can totally, you can totally choose this because in the attention block, as you can


see here, sorry, in the transformer block, you have this part, which is the attention


mechanism.


And then you also have this part right here, especially this MLP here, it takes in each


token of these, it takes that after it, you know, it goes through the attention after


the whole thing goes through the attention, each of the tokens is fed separately through


the MLP.


So the MLP there is, it's actually not necessary that the output dimension of the MLP is the


same as the input dimension, except for this skip connection right here.


Now if this skip connection, like in ResNet had some sort of a linear projection as well,


then you could totally think of changing the dimensions here.


But I'm not even sure if you do the projection, isn't this just the same as the MLP with if


you feed each individually?


Maybe there's no point in having the skip connection at all.


In any case, you could probably get around that, you know, that requirement to have this


exact number of channels.


Nevertheless, that's what they do.


So the generator is actually manageable memory wise, because it does this this trade off,


as it progresses up, it generates an actual grid in the resolution of the image in with


the required channels being a projection of the final channels here out of the transformer,


then it's fed into the discriminator, the discriminator immediately divides the image


into patches, interprets each as sort of a token embedding, and then simply it adds positional


encodings and then simply uses a transformer, like like BERT.


And at the end, you have this CLS token like you have in BERT.


And that classifies real or fake, you can back prop through the whole architecture.


And that's again, for you.


So that was the architecture part.


And now, so they do they do initial they do a lot of good ablations, where they say, Okay,


what if we what if so we have a generator and the discriminator?


What if we have kind of this auto GAN is what they is one of the things they compare with.


So what if we do that?


And then what if we just replace the generator with the transformer?


What if we just replace the discriminator?


So they find out that they can, they can replace the generator just fine.


And that even gives, you know, gives competitive performance, as soon as they, you know, transfer


the discriminator to a to a transformer, that drops in performance.


So in order to really make this work, they need some more tricks.


They have three tricks.


The first trick is data augmentation.


They say data augmentation is crucial for trans GAN.


And the type of data augmentation they do is also from a paper for data augmentation


for GANs.


It's this right here, differentiable augmentation for data efficient training.


So the whole point is that your data augmentation, so the augmentation T right here is a differentiable


function.


So data augmentation is things like cropping, or changing the brightness, color jitter,


rotating and so on.


So as long as that's a differentiable operation, you can use this technique right here, where


you back prop through the augmentation, you can see right here in the generator update,


you actually back prop.


So the back propagation happens through the T function.


And therefore, you get much better signal, plus you get all the benefits of data augmentation.


And the point they make in the trans GAN paper here is that given that transformers don't


have this convolution, they don't have this locality bias built into their architecture,


they need a lot more data.


And we know that transformers they work well, if you have an abundant amount of data.


And you can sort of get around having lots of data a little bit by using data augmentation.


So they argue that data augmentation, it works for all GANs, but it helps a lot more in these


transformer based GANs, because the transformers benefit better from having lots of data.


Again, the story about transformers is pretty clear, I think, if you have lots of data,


they tend to work well, because they're just a more general architecture.


So here you can see, in the different GANs, you can see that the augmentation, which is


when the checkmark here is, it helps sometimes you can see not always sometimes here it does


fairly well, but here in the trans GAN, you can see that adding data augmentation drastically


improves the results and already gets these GANs into the ballpark of the state of the


art.


Not yet there, there's still a big difference, but it gets it, you know, gets them gets them


in, in like target distance.


So the second trick they have is this co training with the self supervised auxiliary task and


specifically they do super resolution.


So so we're gonna write this.


So this here, this, it's a super resolution task, right, super resolution.


And what they mean by this is simply they in, in addition to the whole GAN training,


right, so here, you have the data set, data set, I know beautiful.


So the discriminator over here, the D, it gets images from the GAN, as you can see right


here, and it also gets images from the data set, right.


And that's your main GAN loss.


So here you have the discriminator loss, you back propagate that through the GAN, you update


all the parameters.


What you also do is you take data set images, you put them here as a target.


So this is the target for the GAN.


So the GAN needs to output something.


And what does it get as an input, it gets this thing, but scaled down.


So I'm gonna say this big picture goes to small picture.


So you take pictures from your data set, and you deliberately downsample them, you deliberately,


you might even add some noise or something, but I guess they simply do lower resolution.


So LR means low resolution, and then the task of the GAN is from the low resolution input,


predict, like it needs to predict the high resolution image.


This is a, it's completely different pipeline than usually, because it actually gets the


small thing, the small real image as an input.


The GAN usually never, the generator usually never sees real data, right.


Now it gets a small resolution.


This is not the same image that goes to the discriminator.


By the way, I think at least, this is just a different thing you can also do, you mix


into your batches of, you know, noise GAN samples with this loss, you simply also mix


things, you mix this loss right here, the super resolution loss.


So you have this loss, and then you have the loss from the super resolution, and you simply


add them with a parameter to, you know, trade off one or the other.


And this helps the generator to, so given a low resolution image, these stages here


will have to learn to sort of up sample realistic looking images from lower resolution images.


And that's what you sort of expect this GAN to do.


So it makes sense that this is a good auxiliary task.


And this turns out to help quite a bit.


So as you can see, right here, here, they have it with data augmentation.


And if you add this task here, it, you know, the scores improve again by a bit.


And then the last trick they have is to also do this locality aware initialization for


self attention.


And you can see that again, pushes the scores.


So what is this last trick?


In this last trick, they say, look, the convolution, it seems to be a pretty good prior for images


after all, right?


That's why I mean, that's why CNNs are so effective.


It seems to be a good prior to look locally, like to have local features.


But of course, the transformers, they are more powerful.


And eventually, they want to look at the whole picture.


But maybe it makes sense to first teach them that local things matter.


And once they're at a certain quality level, we can kind of let them look at other pixels


in the image.


So what they do is they handcraft a schedule.


And so over the course of training, have this gradually increasing receptive field.


So in early training, they simply say, you're only allowed to look at your immediate neighborhood.


So each super pixel right here, remember, this is in a downscaled world sometimes during


in the generator, you're only you're only allowed to look at this at the immediate neighbor.


So we introduce a mask that says it here, by which each query is only allowed to interact


with its local neighbors that are not masked.


Okay, and then say different from previous methods during training, we gradually reduce


the mask until diminishing it, eventually self attention is fully global.


So at first, they say, you know, in the in the transformer layer, you have, you have


the you have the keys down here, they have a series of keys.


And you have a series of queries from the individual tokens.


And they say for a particular token, you're only allowed to look at your immediate neighbors.


As if you aggregate information.


And then later, they say, Okay, now training.


So this only look at this, and you can only look at your immediate neighbors, and so on.


And later in training, they say, Okay, now you've sort of learned well, you are now allowed


to also gather information from kind of further out, until at the end of training, the all


the queries are allowed to look at all the keys.


I'm sure that if you engineer this smartly, this is local attention, right?


This is known as local attention.


And you can also make a bunch of, you know, speed ups, probably in early training here,


you can see right here in early stage, only immediate neighbors.


In middle stage, they sort of widen the circle of where you're allowed to look.


And in the final stage, each query is actually allowed to do the full attention.


So when I saw this, I was like, Okay, here, I'm told, we're going to build a GAN absolutely


without convolutions.


All we're going to replace with is kind of an a linear operation that is applied over


the whole image in a fashion that it only gets to look at its neighbors, right?


It's totally not a convolution.


It's just a linear operation that is applied equally across the image while only looking


at your immediate neighbors.


I'm so glad we're building GANs without convolutions.


convolutions are for losers.


We're all for locally applied linear transformations over the whole image that only can look at


their immediate neighbors.


So yeah, no, I mean, you get the point.


This is this is essentially an attentionized version of a convolution.


But within with training as training progresses, they do release that constraint.


This is simply to help the GAN do training, though I am fairly convinced what you you


wouldn't maybe have to do this as a fixed schedule, right?


This is like a fixed schedule.


I say, okay, you know, at you're allowed to look at this many neighbors and then after


this many steps, this, this and so on.


I'm fairly convinced you could somehow formulate this maybe as a two player game, right?


But like, like another GAN thing, or maybe Yeah, maybe another GAN thing or sort of an


self play thing, where the one player tries to sort of get the most information out of


the neighborhood, and the other player tries to sort of constrain that player.


And but it only has a certain amount of budget and so on.


I'm not sure.


I mean, but you could probably do something smarter than simply a fixed schedule that


is adaptive to the difficulty of the task.


And you would also in turn lose a bunch of hyper parameters that you need to build this,


this schedule over here.


All right, the last thing they do after all the tricks is of course, what everyone does


best with transformers.


And that's just scaling that thing up to many layers, many dimensionalities.


And I don't know if they do a lot more data, probably not in this case.


But if you had more data would also work better.


And thereby, they do reach, you know, scores that are state of the art or at least very


competitive with state of the art.


So they're TransGAN XL model, as you can see here, for example, on CIFAR-10, they do reach


very competitive scores beaten only by Stalgan V2.


They also reach very good or state of the art scores on other datasets here on STL-10.


They are the best.


So there is a, it's cool.


By the way, it's nice to see papers going back to kind of the 64 by 64 images because


we're so used to these super duper high resolution GANs now.


This reminds me of old times.


So the paper as a whole is pretty cool.


It's actually pretty straightforward.


As I said, they develop an architecture that works that is actually computable with this


kind of upsampling and the pixel shuffle channel reduction as they go along the VIT discriminator,


then they present three tricks to make that work.


It's data augmentation, it's super resolution task as a code training task.


And it's this locality aware initialization for the attention with the decreasing with


the schedule over training.


And finally, they scale that model up.


And that gives them pretty, pretty well performing GAN.


And it's only made of, so it has no convolutions.


Their goal isn't to use only transformers, their goal is actually to use no convolutions.


Yeah, that was it for me.


Tell me what you think in the comments.


And I invite you to check out the paper and the code.


Building Search Word Database - Data Analysis of Google Tracking Takeout p.1

What's going on everybody and welcome to a video where we're going to be going through some of our Google takeout data


If you're not familiar with Google takeout it is a way you can go to Google like your


Google profile and download all of the data that Google has on you at least through Google services


So if you want to do that it can take a while to actually get the archive


But if you log in you can go there's like a blue. You know account top right click on that


and then on the left hand side there's data and personalization you can click on that and


Then from there scroll down just a little bit, and then you can download your data


And then you're going to pick the things that you want now you might go through this list


And you might think you know what those things mean, but you probably don't


So just get it all so you get a good idea of all the things that Google has on you because you would actually be


Kind of surprised or at least I was like there were some things that I thought I understood like I thought I knew like


What kind of data Google had on me?


But I didn't and so it's just a good idea. Just grab it all just so you have a full


Idea of all the information that they have so it's even like little stuff like


They've got like purchase history for example even though you might not shop on Google


But Google parses your emails and from parsing your emails


They've got purchase history from like confirmation emails and hotel booking emails all that kind of stuff


They're just extracting that by parsing your emails


So stuff like that you you might not realize they have


So anyway definitely I think just get it all


But mainly we're going to be focusing on the my activity section at least to start


But later on if you guys want to suggest stuff that you want to see us go over feel free so we might do other stuff


So finally you'll create your archive and this will take like at least for me


It took a few hours, and you'll get like a confirmation email and stuff like that like for security


I can't think of a much more scary email to receive them


You've requested your Google takeout data


And if you weren't expecting that email so anyway cool


So that's how you get the data


And then once you get it you'll get like a ticket like you'll extract it


And you'll have a takeout directory kind of like this this is not the full one for me


But just some of the stuff. I wanted to point out so this is that purchases and reservations thing I was talking about


Location history this is you tracked everywhere you go based on your phone, but not just everywhere you go


It's not just coordinates. It's are you walking are you staying still are you in a car?


I didn't think they were doing that, but they are and


Based on that you can you can then extrapolate information so based on that


You know where the person because like where where are they sleeping at night basically? That's probably their home


Where do they go every day Monday through Friday?


Same similar times that's probably their work. You know or school or something like that


And then in based on the coordinates they can pretty easily find out like what's at those coordinates and stuff like that


So anyway like I remember the first one of the times


I moved I went to the same store like twice in a row or something and Google just assumed


That's where I worked it was really weird


But my phone was like you need to go to work soon one day, and I was like what are you talking about?


I don't have a job so anyway


Yeah, so we're interested in my activity inside of here. There's stuff like Android again


One of the things I didn't think they tracked was like every app you open look. They just track it


all right cool and


As time goes on like all this stuff like on its own or alone doesn't really sound that nefarious


But like then you start realizing like all the stuff adds together to basically be your whole life, and it's kind of creepy


So anyway on that note we're going to be going through our entire search history and hopefully


You know first of all just as a forewarning


You guys don't get to comment on my search history unless you post your search history


So no making fun of me so anyway. This is what it looks like it's an HTML file


Which is kind of weird because other stuff like for example the location data. Let's just pull up that


That's a straight-up, JSON file, so I'm not really sure why this one is an HTML file and like some stuff is JSON and stuff


I don't really know but anyway


So we want we actually in this case we're going to have to parse this HTML file


Which is kind of a pain in the butt versus like the location history like getting through that is like a breeze you just import


JSON and you go


And in this case you could use beautiful soup or something, but we're just going to do some stupid splits


I think I think that's the way to go


So the first thing we're going to do is just something really really basic


We're just going to run through all our search queries split by word and then just start looking at what are the most frequent words


And then what we can do is you know oh you can do that overall, but this is like five years of search history


so then you could do like a


Daily moving one-year window so like every day along the way


What was the previous year's worth of search history and what are the most common top ten words for example?


And then and that should probably give us like a good overall like macro understanding of major


Interests of ours and major life changes and stuff like that


And then we could go smaller like month a month window or a week window and figure out


You know micro things that were going on just at that time so anyways pretty cool


So that's the plan later we could do way more advanced stuff like do


Like word vectors and get general concepts that you know I'm interested in or you are or whatever


We can do lots of really cool stuff one of the other things that I saw that I absolutely have to try is


They also have all of your Google assistant


Data so your the actual translation is mapped to the audio


Which is basically a text-to-speech data set waiting to happen so and also a speech-to-text


So pretty cool


And then we could do a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a


Pretty cool, but mainly so you can create a


text-to-speech


But this sounds like you also take note Google could do the same


Great so anyway. Let's jump in


So first of all I'm just gonna make a new a new folder. I'm gonna call this


G data I'm gonna put take out into G data, and then I am going to


Figure out why my mouse isn't showing up hopefully oh, I can't I just can't see my mouse okay cool


All right, and then what I'm gonna do is file. I'm gonna save and


I am gonna put that in desktop G data, and then for now


I'm just gonna call this I don't know search database stop pie search


Cool awesome, so that's all we're gonna. Do we're just gonna parse through this save the times


We're gonna convert those times to Unix time because the date stamp is gonna be kind of a pain to work with


We're gonna split by word and then save that into our database and scratch all of our itches


Okay, let's get started so first of all we're gonna need imports


in


Port sqlite 3 for the database we're going to front well actually probably in this file


We don't need the collections. I was gonna bring that in


Let's go ahead and bring we're gonna use from TQDM import TQ TQDM


That's cute


and then we're gonna do import date you till dot parser and


We're not gonna need JSON. We're gonna still scratch it. She's I'm sorry guys. I got like allergies or something


I


Don't know well. We'll just get started so first of all let's get these search activity activity


Location so I'll just do take out my activity


search


Take that copy pasta slash my activity dot


HTML don't know if these backslashes will actually create a problem in this case, but I'm gonna just fix those manually real quick


Then what we're gonna after this we're gonna create our table or actually first


We need to make the database which is actually super simple with sqlite you just connect


And if it doesn't exist boom it's created, so let's do that so we're gonna say


Con equals sqlite 3 dot connect and we're gonna save this as my life dot database


And then C equals con dot cursor awesome


define make


Table now what we're gonna. Do is see dot execute


And we're gonna create table if not exists the table will be called words and


Basically words is gonna contain I think we'll still will have a primary key why not ID


Integer and that will be a primary key


ID we're gonna be Unix that will be a real we could also go with integer, but I'm gonna say real


And then the actual word itself which will be a text type


Awesome okay, so that's our table


Can I please?


What's what's working


Indentation canate I'm so glad that this thing is like that's that's violating pep 8 no tabs allowed


Let's fix that really quick and then using spaces fix this stupid idiot


Awesome okay, so make table, so let's go ahead and we'll run


Make


Table we don't actually need to run that now though and now what we want to do is work on search data so define search


data


What we want to do here the first thing is we're gonna split by word


But we don't want all the words like a the and these are words


We don't care about those in the NLP community are called stop words these are just words


we don't we just want to kind of toss them out so the first thing I'm going to try is stop word list and


List nLTK is list of English stop words heck yeah, that is not the format. I wanted


But up here we go. Here's a list


And you want the above list as an array here you go. Yeah, thank you because the other guy obviously isn't a programmer union


This looks like a great list. I'm taking it boom


Copy nice big long list, so we're going to say stop words equals bang


um


It's kind of a this is not the greatest list of stop words


thorough


Has meaning wonder also has meaning seriously has meaning


I think I'll use this list, but


It depends on your depends on what you're trying to do. I mean some of these are actually meaningful words


I'm not sure you'd want to toss them out


In our case. I think it will be fine


But it's interesting some of these. I'm just not sure I agree


But the show must go on so now we want to do is we want to open oh


We've already closed it, but we want to open that HTML file


We want to split for searched for I got to open it because I got a figure. Oh wait is this it nice


We got a split for


because we don't care about this we care about searched for so we want to split for that and


Then we want to parse out each word in that link


So pretty easy task so search data stop words so now we want to do is open that file


So what we're going to say is with open


the search activity with open search activity with the intention to read as


f


Contents will be equal to f dot read then we want to split those contents and iterate over those contents


So we're going to say is for item in


contents


dot split


And we want to split by searched for with a capital s


For item and kind of okay cool well


Let's just print item print item and then we'll break because we're going to do some development here


We don't want to continue. Oh well. We made the table, but we probably didn't do search data. We did not search


Data let's try one more time


error


Carmap blah blah blah blah blah okay fine. We'll open it with some encoding


Wish this was somehow easier to get back over here


Here


Encoding equals and we'll make that utf adles, and we'll see what happens beautiful not that beautiful, but it's what we wanted


oh


So when we split by search for actually the the zeroth element. We don't actually care for so let's do one


colon try one more time


Beautiful exactly what we were hoping for so now


We're going to do some pretty ugly splitting here, so I'm going to say


search


underscore


string equals item dot split by


That that escape that there


And then it'll be the first if dot split by the closing link tag here


Split by boom and in that case won't the zeroth


So now let me print


search string


Beautiful just beautiful now we want to parse out the date. Where is date?


Here so it should be the first break


Someone's gonna be like this is so ugly in the comments. Whatever bro


date equals item dot split


Break


You did this so stupid look you could have done it this way, okay?


split


Div and then zeroth here's a regex string this would work so much better, okay? I'm done


Print and then date and let's just stop printing item. Let's just see how that looks


Beautiful beautiful now we want to convert that date to a unix time so we can use it as the program as we are


so date


util dot parser dot


parse


the date dot


timestamp


That's not gonna do anything for us


D equals


Date rather than date we'll just print D print the D


Looks good


a TZ name EDT identified but not understood past TZ infos argument


What if I don't want it like I don't really care about EDT


What is how do I what's TZ infos can I just TZ infos date you till?


I don't know what the heck that's about


TZ infos TZ infos additional time zone names, okay, cuz EDT. I don't even know is that the same. That's probably the same as


EST like Eastern I don't know what the D stands for Eastern date time zone


I don't know what the D stands for Eastern date time. I don't know


So this is probably the Delta from what do you call it?


UCT I don't know


I'm just not worried about it right now. This is working, so I'm just gonna skip it


like a real programmer


Yeah, the other thing you could just parse up to like the last four or you could pass TZ infos or


Replace like what if I did this?


replace


EDT with EST what happened then


Okay, you don't even understand EST okay, well


I don't care to cater to you, bro. Okay, so we've got the search string. We've got the time now


We can insert the words into a database, so now what we're gonna say is for


For w in search string dot and we're gonna split by a space


Cool if w not in stop words


What do we want to do we want to do is?


What do we want to do we want to see dot execute execute?


Insert


In to insert into words oh, man. I'm gonna flop here I


I think it's like this words. I'm gonna try this Unix word


But I can't remember if it's like this and then it's like to I'm gonna check don't worry


That that and then I want to say it'll be a comma tuple and then


W no no Unix is first whoops oh gosh. Um


This is going south fast so


DW and now I'm beginning to wonder maybe you don't have a comma tuple


And now I'm beginning to wonder maybe you don't have a comma there


That's looking good. What do you guys think I think I made it


Let's see


Let me check


This is the best programming research resource if you've never heard of it, it's a phenomenal


Dynamically inserting into a database


Dang who is that guy he's good-looking


I


Didn't do values dang it. I was close. I was close. I deserve partial credit


Values so like that and then


Question mark question mark comma tuple is that about right?


Close comma tuple yep, I was so close everybody


Okay


I started cool break I


Think that's it y'all so let me get rid of that break and before we forget you need two things


We need a con commit and we need a conduct close at the very end


Now this is not too much data like our full data set is 80 megabytes


So there's no way this is gonna be bigger than 80. It's probably like less than 10


so


Committing at the very end is totally fine


Many times you might not want to commit once at the very end


Like you probably want to commit in batches


But in this case the entirety is smaller than a batch that I would commit as so we'll just commit at the very end


No big deal


We're gonna comment that out and we are gonna TQ DM this


Operation here, so we see where we are percentage wise along the way and finally


We are going to try and accept because we are probably gonna hit an error somewhere


And I don't want this whole thing to flop because of one stupid error, so we're gonna say try


Because like some of this stuff you've got like visited and then searched for so that's kind of awkward


And that's probably gonna cause an error, and there's probably other things too, so we're gonna try


accept exception as e print what


What no no what's going on


What is going on?


What oh now you want to fix okay, whatever print?


String II look at this thing is just throwing a fit. What is your problem?


Linter


Are we good I think we're good okay, so let's run this


G data


CMD


Looks like we actually probably cuz we already ran it so let me just delete that and listen it shouldn't be a problem


But I forget why we had to rerun anyway Python search database dot pod


We can do it here. We go. We got a progress bar. It's looking great


And we're going quick


I just for the record if you want to look at your database like I'm going to and this is done


I can't really see anything yet because we haven't committed anything, but I'm using like it's like sqlite database viewer or something


open with


DB browser for sqlite just for the record


We're gonna wait for this oh, it's done


3.4 megabytes


That's really small okay browse data. This could be risky everybody about to show the world


I could always blur it and editing I guess then I'd have to edit


okay, so


Cool looks like it works. We got the unique ID a unix timestamp, and then the actual words


I'm not going to scroll too far so I can be dangerous


All right, so we are ready for


The next tutorial which is where we will


begin to build the images that will represent the frames in our video of


Our search histories over time that's going to be pretty cool


Shout out to my most recent channel members Rodrigo Klawe and why you in Christiansen and


Kunal thank you guys very much without support from people like you I wouldn't be able to play with Google data like this I mean


It's pretty awesome


So thank you guys very much for your support, and I will see the rest of you guys


I


Fast and Slow Learning of Recurrent Independent Mechanisms (Machine Learning Paper Explained)

Hi there, today we're looking at fast and slow learning of recurrent independent mechanisms


by Kanika Madan, Rosemary Nankö, Anirudh Goyal, Bernard Schilkopf and Josha Benjo.


So this paper on a high level proposes an update to a previous paper which was about


recurrent independent mechanisms.


And the update it proposes is to learn the individual parameters of the different subsystems


that comprise recurrent independent mechanisms at different timescales.


The idea behind recurrent independent mechanisms is that you have sub modules in a reinforcement


learning agent that specialize on different sub tasks that the agent has to do.


And then you have sort of higher level modules which are attention based modules that select


those sub modules and decide how they communicate with each other.


As I said, this paper here builds on that and proposes to learn these higher level parameters


at different timescales than the lower level parameters such that the higher level units


can generalize to multiple tasks and this helps you in environments where you have to


do multiple tasks.


So we're going to go over this paper and we're mostly also going to go over what recurrent


independent mechanisms are.


And as I already said, this paper didn't doesn't introduce recurrent independent mechanisms.


That's a previous paper by it has some overlap in authors.


So keep this in mind as we go through it.


If you're specifically interested in recurrent independent mechanisms, I invite you to go


read the previous paper.


We'll go over both RIMs and the update to it.


In the end, this paper demonstrates that by decoupling the learning, you get benefits


in environments where this structure of you know, multitask, multi objective is given.


I can generalize to unseen tasks pretty well.


And on the other hand, I think for what this paper does right here for the fact that it


simply proposes this this update, I don't, I don't think it does enough to demonstrate


really, that this is something worthwhile, or it doesn't analyze it enough, I feel.


And they also call this what they're doing meta learning, which I, I don't really agree


to call this meta learning, but you'll see for yourself, we'll go over the paper.


And yeah, bear with me.


So as always, if you like content like this, don't hesitate to share it out and tell all


your friends about it.


And tell me what you think in the comments.


They say in the abstract right here, decomposing knowledge into interchangeable pieces promises


a generalization advantage.


When there are changes in distribution, a learning agent interacting with its environment


is likely to be faced with situations requiring novel combinations of existing pieces of knowledge.


So the hypothesis here is that if you are in an environment that has sort of different


tasks inside of it, that that where the environment itself changes, so your objective changes


as well, then it might be helpful to recombine old knowledge.


And the situation you have to have in mind with this paper is one of their core environments


here is sort of a grid world environment.


And the grid world environment is simply have this grid, and the agent occupies one cell


right here.


Maybe the agent is here.


And the agent can sort of move around here and do different actions.


And there, there's gonna be different things in this environment.


So maybe there's like a key right here, this is a key.


And maybe there's like a door over here.


And the agent will get an instruction.


Now the instruction in this environment might be get the key and go to then go to the door,


then go to the door.


Okay, so this might be the instruction.


It might actually always be the same instruction in this particular environment.


But if you change the key, and you change the door, where they are, that's already like


different tasks, or it's not, it's not the same environment all the time, you can also


vary the size of these environments pretty easily.


So all these tasks, these different tasks, they share some underlying structure, which


is there's always kind of this world, and there's a key and there is a door.


And there might be a wall right here.


So they all share this structure.


However, what exactly you have to do differs from episode to episode.


You can also imagine that there is maybe I don't know, maybe there's like an orange here.


So there's an orange right here.


And then the text instruction will say, get or go, go eat the orange.


So now the agent has to ignore the key and the door and go to the orange, right.


And additionally, so you can modulate this a lot.


Additionally, you can say, okay, the agent maybe only sees its surrounding, maybe like


this, right.


So the agent only sees whatever is in front of it and a little bit to the side.


So it needs to sort of turn around and explore.


There's lots of variations.


The important thing is that there's an environment that has some kind of over overarching structure.


And there's different tasks and each episode is sort of a new task that the agent needs


to solve.


Now, what happens if the agent here is implemented in as in classic reinforcement or deep reinforcement


learning as one big box, like one neural network, and then you perform your episodes and you


update the neural network, the parameters of the neural network according to your reward.


If you solve one task, you will update according to that task, right.


So if you solve the key, the key door task, let's call it that, then your neural network,


all the parameters will be updated with respect to that task, right.


The way you train a neural network is that you change the parameters such that your loss


decreases.


So you train your neural network to solve that task as well as possible.


But now the task changes, right, then all of a sudden, it's get the orange.


Now all of a sudden, this doesn't give you a reward anymore, right.


And now the orange gives you a reward.


So all the parameters you're going to change in order to serve this new task, you know,


finding the orange.


By the way, this is supposed to be like a little light spec.


I'm terrible at this.


I'm absolutely terrible at this.


It's like an orange donut.


But you get what I mean this in general in the fields of like lifelong learning and multitask


learning and so on.


This is known as catastrophic forgetting, catastrophic forgetting.


I don't even know why I bother to write.


No one can read anyway.


So there is lots of work in preventing catastrophic forgetting in these types of situations.


And the way that this or the previous paper, the recurrent independent mechanisms proposed


to do that is, let's not implement our agent as one big box.


Rather, let's implement it as a collection of like little sub modules.


And these little sub modules, they focus on individual sub tasks.


Okay, so a sub tasks might be fine, go to somewhere, okay, with the somewhere being


a parameter that's then taken from the instructions, or maybe one one parameter specifically for


recognizing the orange.


So now and the other one is for recognizing the key.


Now if the instructions they go to the key, the module that is recognizing the key might


become active.


And the module that is that is for going somewhere might become active and the combination of


the two might then get you to the key.


So in each time step, the idea is let's only activate a sub part of these modules, not


all of them at the same time.


And now only these modules will be active because they are relevant for the current


tasks.


And then only these modules will receive a learning signal and not the other modules


okay, the other modules will stay fixed for that particular for that particular step on


in time.


And this makes sense if you if you think about it, right?


If your module isn't relevant for the task, then it shouldn't receive a learning update.


And that's how you try to prevent catastrophic forgetting.


So if this here, this module down here, remembers to read or can recognize the orange, and right


now you're trying to find the key and get to the door, then if you don't, if you do


update that module, it will be in service of the goal of finding the key and getting


to the door.


So it will forget the orange.


However, if you decide no, this module isn't relevant for the current task, and then you


prevent an update to it, then it won't forget the orange, it will only come into life once


the task is actually about the orange.


And then of course, you want the learning signal.


So that's the idea right here to prevent catastrophic forgetting, I do have my doubts that that


is, is so like that that scales to because the combinatorics of catastrophic forgetting


are rather large, and therefore, but, you know, depending on how you factor the independent


things you need to do, it, it is a good idea.


Okay, so that's the core idea.


It is that instead of having this one box, you have a lot of small boxes.


And now you do this, right?


These reinforcement learning problems, they're often implemented as like recurrent networks.


And it's not a, it's not by chance that this thing is called recurrent independent mechanisms.


Because each of these little boxes, like the big box would be is a recurrent neural network.


So the way that these things work is that you have your different your inputs, which


is frame by frame by frame, right?


And the input goes through some sort of an encoder into a hidden state.


And you do have your hidden state that's from so the hidden state that the agent itself


carries, this is kind of its internal memory.


And you use the input frame of the game.


So this is frame one, this is frame two, this is frame three, use the input frame and your


own hidden state to produce the next hidden state.


And you can easily implement this with some sort of an LSTM, right?


And then use that and that to produce the next hidden state.


So that's the normal way of how things are done.


Now in the so that's if you just have like an LSTM controller.


Now if you have a recurrent independent mechanism controller, then your hidden state will be


sort of a it will consist of many hidden states.


So the hidden state itself will be a collection of hidden states, right?


And so these are supposed to be little vectors.


And then the input comes in here, and then only a subset is selected.


So maybe this one and this one are selected.


Now the way that this works is I shouldn't even draw one circle here, I should actually


draw four circles.


So you have four LSTM controllers, and only two of them are selected.


I'm going to tell you how they're selected in a second.


Actually I'm going to tell you right now, probably that's better.


So what what you do is you not let's let's do that after.


So you select two, you deactivate the other two.


And the way you produce your next hidden state is sorry is simply you copy over the hidden


states of the deactivated modules.


So you just copy those over.


So they remain and you would update the hidden states of the modules that you selected.


So only those modules are active.


All right.


So now, yeah, so that's that's that.


And there is also a communication step at the end.


We'll go into that here because here's the diagram.


So down here, you see what I've just told you, this is the system.


Okay, you have to imagine there is the last frame right here, there is the next frame


down here, the frame and also the so that's the observation and the instruction, they


go through some sort of an encoder, which would also be the same encoder up here and


down there.


Then there is the hidden state which is here in blue, so these are the independent mechanisms.


Wait, that's the wrong blue.


So we have in this case for four independent mechanisms, those would actually carry over


over time, the state, the internal state of the agent, right.


And then at each time step, you have an output of a value head and a policy head.


The method they use right here is proximal policy optimization, as far as I understand


it.


This is a variant on actor critic method.


If you don't know about deep reinforcement learning or proximal policy optimization or


actor critic methods or why we need value and policy heads, I invite you to go look


that up that it's fairly simple.


It's very basic algorithm, where you can do reinforcement learning, you can calculate


a loss and then you can back propagate to these either to the encoder, and also to the


to the parameters in the recurrent cells here.


Okay, so how do we decide which modules are activated and which ones aren't.


And that goes through an attention mechanism.


And that's what they call here input attention.


So input attention is the following.


You have your input, okay.


And you do have the encoder for the input, which is like maybe a some concoction, some


alchemic concoction of neural network, right, that gives you a vector like an embedding


of the input.


Now you go to your little modules, each of them will have a hidden state already.


And they get to do attention to that input.


So the input will emit keys and queries.


Now you can do this in multiple heads.


But ultimately, let's do one vector.


Okay, so here is a key.


Sorry, it will emit keys and values.


Okay, there is a key, and it will also emit the value, we can we can just get we can just


do like say the value is the input itself, if we do not have a if we don't have multiple


heads.


So ultimately, they emit keys and values and every single one of the mechanisms emits some


sort of a query.


So in essence, the input outputs a descriptor for what it contains, right?


That's how you have to think about attention.


And the the each of the mechanisms outputs a query for what they would like to see.


So they get to look and at their hidden state.


And they get to decide what kind of information would I like to read from the input or what?


It's more like a filter, what kind of input is relevant to me.


So the mechanism that cares about the orange, it would output probably a query for saying,


is there something orangey in the input either in the instructions or in the picture?


Is there like something about an orange there?


And the the one that cares about the key would obviously say, well, is there something about


the key in there, but you can also imagine more abstract things.


And then the attention is computed via inner product.


And you can see here, it's those two mechanisms that are closest in inner product to the key.


And then only those two get get selected for this particular time step.


And those get eliminated, not eliminated, but only the two on the right get to update


the hidden state.


As you can see right here, the ones that are not selected, they the hidden state is simply


carried over.


Whereas the ones that are selected, they actually get to do computation and update their hidden


state.


Now, at the end of the update of the hidden state, there is a communication step.


So these are not fully independent, they do get to communicate with each other.


And so they here they have a new hidden state.


And here they have an old hidden state.


And now we get to communicate with each other.


And again, the way this works is that every single one of them processes the input actually.


So the input goes through all of them.


And all of these emit again, a query and sorry, a key of them emit a vector saying, you know,


what did I get out of this input, even the ones that were not selected, they emit some


sort of information.


And the ones that were activated, they get to emit a query for what they would like to


see of the other modules.


And that's how you get the intercommunication, right?


That's how you get to like higher order independent mechanisms.


So you could actually get a mechanism for going somewhere.


And then that mechanism would query sort of another mechanism that says, well, where do


I need to go and the other mechanism that was like, well, I I know where to go, because


the instruction said, find an orange, and I'm the orange module.


So I located the orange.


So they get to communicate to to each other.


So that there's going to be attention based communication, where the active modules read


from both the other active modules and the inactive modules.


And then you go to the next step and you repeat and then the next step, it could be that different


modules are activated, right?


So these two attention mechanisms, the first one called the input attention, that selects


the active modules, and then the second one called the communication attention that says


how the different how the different modules communicate with each other.


Those are sort of the higher level modules that control the flow of information of the


lower level modules.


And now, in the recurrent independent mechanisms paper, this, as I understand, is just learned


end to end.


Okay.


Now this paper comes into action and says, Wait a minute shouldn't like if, if we have


the same environment, but different tasks, okay, so here you see individual episodes,


and these individual episodes are comprised of a couple of time steps.


Now, they say, if we want to learn these little modules such that they share knowledge, like


they learn the independent things, and they can be recombined in different ways across


the tasks.


Shouldn't we sort of when we learn the individual modules, yes, we do the what they call fast


update, we do the classic RL, where we learn maybe frame by frame or from short sequences


within an episode.


Okay, so if you know the goal, then let's learn the little pieces that make the goal


happen.


But in order to learn to select the pieces, you should look across different spans across


different episodes.


So that's what they call the slow update right here.


So they propose to learn these meta parameters or what they call them the communication parameters


in a slower fashion, feeding in longer episodes.


And here you can see it even spans across the different tasks.


And the idea here is that the these slower parameters, they consider longer time spans,


they see multiple tasks at the same time.


And they learn how to select the different modules, depending on the current input the


current task.


And yeah, so by seeing different variants of that, in a single episodes, they get to


they get to know the differences and the commonalities between tasks.


Now that is a high goal.


So here, my first problem is they call these like meta sequences.


And yes, okay, they are meta sequences, but I disagree that that is meta learning.


So what they ultimately do is here is algorithm one.


So they randomly initialize the parameters of the they randomly initialize the parameters


of the attention units.


And here the the little mechanism units, they randomly initialize them.


By the way, the also the the policy parameters are part of the meta unit parameters.


And the value head parameters are then part of the attention parameters.


They're not actually part of these modules, but they're learned also in different timescales.


So the policy is learned fast, and the value is learned slow.


That's just because feelings.


So while not done, we sample a batch, a batch of tasks.


And then for each task, we sample a trajectory.


And then we learn the modules, the mechanisms in the fashion, right, we keep the attention


parameters constant.


That doesn't mean we always select the same module.


The attention parameters being constant means that the way the queries and the keys are


generated from the input that remains fixed.


But it's still going to be differently selected modules from from from time to time.


It's just that the way in which we select which ones are active aren't updated from


time step to time step.


And keeping that fixed, we learn the individual little things.


We learn the mechanisms in a very classic fashion.


So you can see right here, these are individual episodes, okay?


The loss function is the proximal policy optimization loss, a very classic with like an entropy


term and so on.


They have it somewhere here.


So this is a very classic PPO loss.


This thing right here, you have this clip loss for the policy you can see here is the


so here is you have the probability ratio, which is sort of like the policy parameter.


This is the current policy.


This is the old policy.


And then you have the value function loss, and then you have an entropy parameter loss.


So quite a standard loss for reinforcement learning.


And you learn that from individual episodes, and you update the parameters of the mechanisms,


as we said, right?


So you only activate the modules that are currently that are selected by the attention,


and the back propagation would reflect that.


In then in the second step, you sample again trajectories from tasks, but then instead


of keeping the tasks and the episodes separate, you now concatenate all of them into what


they call meta sequences.


And then you update your attention parameters using those meta sequences while keeping the


mechanisms constant, right?


So in the first step, you learn, you know, given sort of the activation policy of the


mechanisms, how should the mechanisms behave in order to achieve good reward, right?


So how you know, how they're selected remains constant.


So they, they just get selected, and then they're, they're meant to maximize the reward.


So any any mechanism here, you know, when they're selected, they're just being like,


okay, what do I need to do to solve the current problem?


And if they are selected in a consistent mechanism, that will cause them to specialize, right?


If one is always selected, when the orange thing is in the input, it will sort of start


to specialize in these kinds of tasks.


And in the other step, the mechanisms are kept constant.


So you have the little sub modules that can achieve or can can can do certain sub tasks.


And now you're trying to select the best ones of them.


So you're trying to train the attention mechanism, how do you facilitate the selection and communication


between the these given fixed mechanisms, such that the reward is the highest.


So in this two step fashion, the little mechanisms get better at the tasks they're tasked with,


which causes them to to specialize if they're selected correctly.


And then the selection itself is updated, which in turn, makes the learning signal for


the mechanisms better, and then better mechanisms make the learning signal for the selection


better, and so on.


You can imagine that this two step process is sort of, you know, kind of swinging itself


up bootstrapping itself up to very, very good interlocking pieces of things.


Okay, in the experiments that looks fairly promising.


You can see often see so they probably you can't see the blue one is vanilla, which is


sort of an LSTM controller.


The green ones is the recurrent independent mechanism one, while the red one, I don't


have red here, I have orange.


Red one is this new two step approach.


It's not always the case.


And reinforcement learning is quite tricky.


But this being largely the same authors, I guess they do at least have a good comparison


to recurrent independent mechanisms.


Though I have to say this is measured in frames.


So how many frames did you consume?


And that is an important thing because sample efficiency is important.


But also given how complicated this scheme is, I wonder if this is slower or faster than


just training both things at the same time, like the recurrent independent mechanisms


did.


Okay, so again, the difference between this and the last paper is simply that they, they


proposed this two step process where you have one step here, and another step here, instead


of learning these two things jointly.


And they do so deliberately in environments where you have multiple tasks given.


So you know, like, it's another lesson in, hey, you know, you need to evaluate on the


things where you are really, really meant to be good at, and you need to evaluate in


the quantity that you're meant to be good at.


I'm not sure if time here would show the same plots if you had like in the x axis as time


or computation or anything like this, it might very well be.


So they demonstrate that they do, you know, a lot of have a lot of success with this,


they demonstrate that if they train on, let's say, small environments, that they call difficult


environments, that the metarims, that's their system, the modular is the old paper and vanilla


is the base implementation, they demonstrate that even though they all get to fairly good


success rate and reward on the difficult problems, if you make it zero shot, more difficult,


so you increase the size of the problem with without ever having trained on the bigger


problem.


So you make that room a lot bigger for finding the key, the these meta, what they call meta


rims, they generalize a lot better than the other ones, right, you can see right here,


the other ones largely fail, and they claim their system generalizes a lot better.


So reinforcement learning, experimental results are very, very tricky, right, you can you've


already seen sort of the just the, the bars here, the error bars up here.


And that's after a long probably experimentation, maybe, and also selecting the right metrics


and so on.


Here we don't even get bars.


And here, it's, it's quite tricky, because not only do, for example, the vanilla ones,


generalized worse, they also start at a worse point, right?


So they start at much less reward.


And maybe that's responsible for them not generalizing so well, if you were to actually


push like point nine, five to point nine, seven doesn't seem much.


But if you look, it's like, almost half the error, right?


So like, if the maximum reward is one, then this gets, you know, five less than the maximum


reward, and this only gets three less, this is quite a reduction.


Maybe that's the reason why it zero shot transfers to the more difficult environment.


Also here, the modular ones, which you have to remember is the exact same architecture


as the meta learned ones.


They don't even have good success in these tasks.


So the hypothesis of this paper here is that if you learn all these things at the same


time, you will still be subject to catastrophic forgetting in these environments where you


have multiple tasks, right?


By learning the high level parameters in a slower way, in a first of all, in an independent


way.


Second of all, in a in a way where they see a longer sequences of things.


And I do believe also, and this is also a bit unclear, I also do believe they do less


update steps, maybe not.


No, I think it's just that their, their steps that they consider the time steps they consider


are four times more than the time steps that the individual that the learning here considers.


So line six has some number of steps, n number of steps, and line nine here considers four


times n, the number of steps.


Okay, so they consider longer time scales.


If you want some other numbers, they always have five of these.


So they always have five, which is what they call little n, and of the five, there are


always k equals three active.


So there are always three or five things active at any given point in time.


And that is a bit of a different problem I have here.


You know, to their contribution is, let's learn these higher level parameter independently,


and in a more slow fashion.


That's the contribution, right?


Not the recurrent independent mechanisms, the the separation.


Now, I would expect there to be a lot more investigation into what exactly this separation


and slower learning is doing.


They do have some ablations right here.


But not many most ablations are about the recurrent independent mechanisms itself.


So for example, here, they compare k equals three and two, and they show look across the


episode, different modules become active as time progresses, which gives you an indication


that yes, in fact, the different modules do specialize in different things, which is cool,


right?


That is not a property of the separation.


That's a property of recurrent independent mechanisms.


And here, again, they the ablation they do here is different k, so different number of


sub modules being active.


And you can see that if all the modules are active all the time, you have the pink curve,


which is quite bad.


And if only some modules are active here, like k equals three, you get a much better


performance.


Now, I would expect that, that you actually try to go to k equals one or something like


this to show maybe there's an optimal subset and so on.


But again, this is a property of recurrent independent mechanisms, only here where they


say, shorter meta episode.


So here they say, what if we do the same thing that works well, but we make this meta episode


shorter.


And then you can see that the curve here, it also it sort of follows the trajectory


of the of the worst baseline.


Now that is one thing right where they make they don't say how much shorter they make


it, they just say, we make it shorter.


And that hurts.


I mean, okay.


Here they analyze the value function, which is cool.


You can sort of see that the value function reacts to different things in the environment.


Again, that is not a that is not a property of what they're doing.


And here, choice of attention, this is ablation choice of attention parameters as slow parameters.


So they say, now, let's do a different thing.


Let's actually flip.


Let's learn the attention parameters in a fast way.


And the meta parameters in sorry, the mechanism parameters in a slow way.


And that's what they call meta flip.


And here they show they show that that performs worse.


Okay, so the the top one here is the meta what they propose.


And the bottom one here is the flipped one where they learn the other parameters slow


and the attention parameters fast.


And again, okay, that's a that's a thing, right?


But it's it's not so much worse, honestly, like, and at some point, they say, Well, it's


somewhat worse.


And in the text, some they say that is did not perform very well right here, this did


not perform very well.


And I disagree a bit like it performed.


Okay, like it's certainly better than the than the vanilla one, it looks like it may


be at the same as the vanilla one.


It doesn't seem super duper bad.


And I just don't think this is since this paper is about adding this thing, the addition


of this thing, and the sort of, you know, how much that contributes and what exactly


of the thing makes the algorithm stronger.


I don't think that's explored enough in this paper, I think too much space is wasted on


exploring like the value function and which modules are active, which we already know


from the recurrent independent mechanisms, right?


There are in fact, two things going on, right?


There is the slowness, there is the fact of, hey, let's learn one set of parameters more


slowly than another set of parameters.


That's one thing.


And the other thing is, hey, let's decouple learning the two parameters.


Now the decoupling actually is what I think makes it not meta.


This is simply decoupling, this is not meta learning, as far as I'm concerned.


This is not learning to learn or anything like this.


It's simply that we have two different things, and we learn them at two different times.


This is very much like, you know, the in the beginning of GANs, you have whatever your


generator and your discriminator.


And here and here you have your, your data set.


And here you have your binary classification.


And here you have your latent vector.


So these, this is a basic drawing of a GAN.


And what people used to do, at least at the beginning, before we realized how we can stabilize


GAN training is they did these independently, they said, I'm going to do one step, learning


the discriminator, and then I'm going to do another step, learning the generator, instead


of updating them both at the same time.


And at the beginning, we even did things like, hey, let's learn the generator for five steps.


And let's learn the discriminator only for one step, once we get to the discriminator.


So it is exactly the same thing.


That was not meta learning.


This is simply the fact that if you have a system where the parameters are sort of entangled


with each other, like the discriminator, depends on the output of another system, which itself


has parameters.


And if you change everything at the same time, that can get you into trouble that can get


you into instability.


And therefore, it might be a good idea to separate these.


And if one system is sort of stronger than the other system, it might also be effective


to learn these at different time scales, is nothing sort of to do with meta learning.


And it's two different things, right?


This time scale and the separation are two different things.


And yeah, these are not entangled here.


And they also compare with what they call slow LR, they say, well, in order to compare,


what we can also do is we can simply learn the parameters of the attention and the mechanisms


at the same time.


But we can give the we can give the attention, simply a lower learning rate.


Like we divide the instead of dividing the number of steps by four, we divide the learning


rate by four.


And they say show that doesn't work.


And I mean, it's not a surprise that doesn't work.


That is absolutely not the same thing, right?


It's and I'm not even sure what it's supposed to show.


I guess it's supposed to show that that you need the separation.


Instead the slowness itself isn't a thing.


But I don't think you even if the slowness was a thing, it is not that you can simply


replace the number of steps by a smaller learning rate.


Yeah, in any case, but it is it is at least like some kind of experiment that that shows


something about the system, right?


What I would expect from an experiment like this is Yeah, here again, like what the modules


are learning, which is cool.


Like it's cool that you show, look, this module is learning this, this one is active when


that happens, and so on.


And we can ablate the winner modules.


So what they do is they take the modules that are selected, and then they randomly drop


out some of them.


And they discover, well, the more we drop out, the less well it works.


Wow.


But there's no investigation into, okay, what is the effect of learning one thing more slowly?


How much is the effect?


Can we modulate that?


Can we set the number of slow steps equal to five to six to 10 to 20?


You know, can we can we discuss how long these meta episodes need to be like here is just


like shorter, okay.


But there's no indication like how long do they need to be?


What's a good length?


Then give us give us like the time penalty that we incur here, not only the frames, right?


What's what's the time penalty?


Might there be already something good about simply separating the updates?


You know, like all of this kind of stuff is not really explored in this paper.


So again, there is really cool parts about this paper.


It makes sense to separate these two because you have an interdependent system reinforcement


learning is brittle enough already.


And it really seems to help against this catastrophic forgetting.


However, for the fact that this paper simply adds this two step approach.


I don't think it does enough to show what they're doing and to show the reasons of why


what they're doing works works.


And also I object to this being called meta learning.


So that is my opinion.


Please tell me your opinion.


This was a bit more ranty than I usually do.


But I hope you're still here.


And I'll see you next time.


Remote Access with SSH and Remote Desktop - Raspberry Pi and Python tutorials p.3

What's going on everybody and welcome to another Raspberry Pi tutorial. In this


tutorial what we're going to be talking about is accessing your Raspberry Pi


remotely from your local network. You can access your Raspberry Pi from anywhere


in the world you can set that up but that has a lot less to do with the


Raspberry Pi itself and more to do with your actual network and that's much more


advanced tutorial we're not going to be getting into that. So there's two major ways to


connect to your Raspberry Pi one is via some sort of remote desktop and the


other is via just SSH which is secure shell. Both have their own good benefits


to them so you would do like a remote desktop if you have any sort of UI or if


you've got video that you want to show or you just don't you want to actually


just share the keyboard and mouse on some computer between the Raspberry Pi


and all that you don't have to hook it up every time. So the first thing that we


need to do to remotely access the Pi is make sure we're connected in some way to


the local network you come up here to your Wi-Fi click on that you can't see


it actually because my face is in the way. There you can choose your Wi-Fi make sure


you're connected or you can connect you know physically with an Ethernet cable or


whatever. Now once you've got that go ahead and do control alt and T and that


should open up a your terminal now I want to make it a little larger so


everybody can see and hopefully it's not too large to see what we get. Not bad. So


so now we're going to go ahead and do is you can do ifconfig and that will give


you a bunch of information if you are connected physically it would probably


your internet IP address would be up here but I'm not connected physically I'm


actually connected via wireless LAN so that's down here and what you're looking


for is your INET address my address is 192.168.0.103


okay so we have that information now what we want to do is see if I can get


to the bottom of this I've clearly made this too big let's go down a little bit


once we've done that once we have that we could in theory connect to the


Raspberry Pi but it's not going to let us because the SSH server is disabled by


default now on the Raspberry Pi. If you have an older Raspberry Pi actually I suppose


it would come with the installation so like your actual operating system so it


used to be the case that SSH was always just enabled by default but as more and


more people are using Raspberry Pis or using them in their businesses and all


that it has become a better idea to not do that because if people got open Wi-Fi


they just need to find the Raspberry Pi's IP address they know the default


login and password so that can be problematic so generally if you do have


the SSH server enabled and you're using the default username and password it's


going to keep warning you by the way you're doing this and stupid anyway


we're going to open up the Raspi config so do sudo maybe hmm I'm still not on


screen I don't think let's see if I can do this I think we're just typing off


screen yeah so what you want to do is sudo raspi-config hit enter and this is


the configuration for your Raspberry Pi you can do a whole bunch of stuff you can


change the password you can change your hostname you can new boot options or


interested in interfacing options let's go ahead and enter there and we want to


enable SSH so go ahead and enter here and would you like the SSH server to be


enabled yes boom done okay so now SSH is enabled and we can connect so at this


point we actually can connect via SSH now if you aren't familiar with SSH I


will hit finish basically on Windows you're going to need something like


putty so P U T T Y you can go to putty.org download that and you can connect


that way or if you're on Macintosh or Linux it's as simple as opening up your


terminal and doing SSH username which would be pi at whatever your IP address


is in my case it was 192.168.0.103 so you could do that and connect now if


you're on Windows it's a little more challenging oops I've already got this


one's already up but basically let's do new session you'd come over here and


you'd type in the IP address so 192.168.0.103 hit enter this will pop up


asks you to log in I'll log in as pi password is raspberry by default


connected and boom and as you can see SSH is enabled and the default password


for pi has not been changed blah blah and the next time we log in to the


actual GUI desktop we'll also have another warning there telling us that


we're probably making a big mistake so okay so that's how you can SSH into the


Raspberry Pi once you do that it's basically the command line as you can


see and so now we can do all kinds of stuff here once you've done that like I


was saying before there are going to be times where you want a GUI desktop for


whatever reason so you can also do a remote desktop now for this on Windows


and Mac you probably the easiest is just to download remote desktop by Microsoft


I think Mac also has a remote desktop software I'm not sure and then on Linux


it's GR desktop so GUI remote desktop GR desktop so that would be GR desktop like


so otherwise if you're on Mac or Windows it's virtual or I'm sorry it's remote


desktop by Microsoft Microsoft okay if you're on Windows you should already


have it now in order to get there what we need to do is install some sort of


remote desktop client on the Raspberry Pi I choose XRDP if you have already


tried to install XRDP and it failed for whatever reason in the past it was


easier now it's harder for some reason because you already have a VNC server


you might need to run sudo apt get remove XRDP VNC for server and type


VNC server if you've already tried to install you might need to run that


command just to remove everything and start clean otherwise if you are clean


you should be able to just do sudo apt get install type VNC server server now


I already have it so this should be done and then you can do sudo apt get install


XRDP if for whatever reason something gets when we when you go to connect and


it doesn't work for whatever reason it hangs or whatever go ahead and run that


thing to just clear everything and try again so once you've got XRDP installed


you're good to go you you could restart the service XRDP but it should be


enabled by default once you install now on Windows what you can do is open


remote desktop and in here you can type the IP address I'm actually one of three


hit connect yes and then here's your like your login screen now you can type


PI raspberry hit enter and if you hang here that's probably when you'll want to


do your you know remove everything and try again otherwise here we are in the


remote desktop now because I already have a remote desktop it's just showing


me like a copy of this other one because this is also a remote desktop connection


but normally you won't share it like so this is not on the screen on the actual


Raspberry Pi and I can even turn it on and show you that that is not showing


you know this stuff here and if you are on a remote desktop and you're trying to


show something from the Raspberry Pi camera by default it's actually going to


show on that screen I'm not positive if you unplug the HDMI if it would work but


anyway no big deal but just know sometimes it can be kind of screwy


anyway that's it for the remote desktop and remote connection SSH all that fun


stuff if you're having a problem you got questions whatever feel free to leave


them below otherwise I'll see you in the next tutorial where we're gonna be


talking about actual terminal navigation and commands and all that kind of stuff


State-of-Art-Reviewing: A Radical Proposal to Improve Scientific Publication

Alright, hi everyone. Today we're looking at state-of-the-art reviewing a radical


proposal to improve scientific publication. So this has been on my mind


for a while. The review process for modern science, especially machine


learning, is just broken. I've spoken numerous times about the fact that we


need to replace it with a better system and now Samuel Albany et al. have


actually come up with such a system and we're going to explore it today. I am a


big fan of this work and I'm 100% on board with this. So they basically say


peer review forms the backbone of modern scientific manuscript evaluation. If you


don't know what peer review is in machine learning right now, if you have


some genius idea, so here is your idea. That's a light bulb by the way. You write


it up into an eight page PDF. Yes, it must be PDF and yes, it must be eight pages.


You submit it to a conference which is some kind of... so you submit it to be


accepted in a conference proceeding and if the conference organizers, of course,


they're just a bunch of people, they can't review these 1,000 million


applications that come by themselves. So what they do is they recruit experts


which are called peers. So peers are other people, these are called peers and


they have usually written up their own papers and they can critique each other's


paper and they decide what gets accepted and what doesn't. Now I've spoken


numerous times of how this is super noisy right now. They're way, way... they're not


enough peers, they're not experienced enough, so whether or not your particular


idea gets accepted is extremely dependent on probability, on a coin flip


usually and it's just overloaded and just makes no sense and they ask the


same question. Is this fit for purpose in 2020 and we need to replace it and I


support. So they... you can already see they kind of want to automate this


away with their state-of-the-art review score and the score will be an out of


ten score that can be integrated into something like archive and, you know,


display right away. So they have some requirements to this new system. What


should it be done? It should have the ability to scale. Very important, our


current review system doesn't have this, right? Our current review system relies


on other humans reviewing your paper and that means that the reviewers need to


scale with the amount of papers, which just isn't the case currently. So a new


review system must have the ability to scale, right? And then, you know,


automating the reviews away or scaling it up in a distributed fashion does this.


Speed, yes, because right now if I submit my manuscript for review it takes them


months to review it and our science progress is faster than that. So a speedy,


more speedy version of peer review is definitely required. And then


consistency and this is the most shocking part, right? There is a the


Grand 2014 NeurIPS experiment which concluded that 57% of papers accepted by


one committee were rejected by another committee and vice versa. Reviewing the


exact same paper, different committees came to completely different conclusions


to an astounding degree. So basically you're flipping a coin of whether or not


your paper gets accepted or not and I think this is just not acceptable and so


they proposed these three things, speed, scale, consistency and their new method


certainly has this. Now let's jump down here where they introduce this


state-of-the-art reviewing, SOAR. So they say, okay, the quality of a scientific


work can be judged along three axes efficacy, significance and novelty. So


there are these three pillars, right? Efficacy, which means is kind of how


effective is your work in achieving the goal in machine learning that's usually


to train some good classifier or something like this. Then the other one,


sorry, is significance, right? Significance is how relevant is what you've


done to the field, right? And the third one is novelty. So you know, in your


scientific work should be an original contribution to the knowledge of


mankind and therefore it should be novel, right? So the more of these three


things you have, of course, the higher your score should be and here in the


middle is where the highest scores should be. So imagine this is kind of a


landscape and so you want to grade papers along these three axes. Now they


have a pretty good method of assessing these in an automated fashion. So first


of all, assessing efficacy. So efficacy, they say, is best assessed by determining


if the proposed method achieves a new state of the art. I think that's not


really, I don't think you can really doubt this. I mean this is


kind of the gold standard of whether your paper is effective, is whether or


not it achieves state of the art. I mean, it might be a bit of a controversial


opinion, but if a paper doesn't achieve a state of the art, it's, you know, why?


Why do you even care? Like no one cares. So from, they say, from an implementation


perspective, they can use, they can kind of abuse a fact of the


current research environment is that you don't actually have to review this


yourself, but the authors themselves can be relied upon to state this repeatedly


in the text, right? And this is important. So the authors will state that


they have state of the art many, many times in the text if they have actually


achieved it. If they haven't achieved it or aren't so sure about it, they probably


won't repeat it as many times. This is, it can be, can kind of abuse now to


distribute it. Basically, you don't imagine now these, all of these


reviewers, they don't have to do this work anymore. They can just


distribute to all the authors of their own papers, right? Because the authors in


the text, by the way, the text is structured. It's kind of an NLP approach to


reviewing. Kind of NLP mixed with game theory, right? So the other authors


themselves, if they have state of the art, you have to do some stemming and stuff,


but they will put that into the text a lot. So it's a bit controversial, but the