benefit. In order to train gopher they also collect a new data set which they call massive


[ML News] DeepMind builds Gopher | Google builds GLaM | Suicide capsule uses AI to check access

text. It's a collection of large English language text data sets from multiple sources, web


pages, books, news articles and code. So not only do the authors confirm that more text


is a good thing, but they also confirm in their studies in their analysis that very


much the quality of the input text is just as important as the amount of input text.


So cleaning the data and also sampling the data according to its quality makes a big


difference in these models. The author's note we provide a holistic analysis of the training


data set and the models behavior covering the intersection of model scale with bias


and toxicity. Now I have to say something like bias and toxicity is given a pretty big


weight in this paper. I don't know why because it's an investigation into many many things


of these large language models. And I personally don't see bias and toxicity being like a specifically


bad problem that specifically needs to be highlighted. It's not like we don't have enough


problems on our hands with the 151 other problems. But for some reason DeepMind chooses to highlight


this one. The blog post also briefly goes into the main results which were already mentioned


in this short summary. But as you can see right here, Gopher often beats GPT-3. However,


it's still behind human experts in most tasks. And when it comes to things like scientific


and mathematical reasoning, it actually just as GPT-3 does performs pretty poorly and purpose


built systems to do mathematical reasoning, even though they are still lagging behind


human experts are much better than something like Gopher or GPT-3. I think this is to be


expected as just sort of picking up from language you learn a lot of things like a lot of factual


knowledge about the world and a lot of things that people say and stories they tell and


so on. Yet for something like mathematical reasoning, it is not as much a language input


thing, it is much more an algorithm that you have to sort of practice over and over and


someone needs to show you how to do it and specifically, essentially program your brain


to do an algorithm. Now, I do believe there's evidence that large language models in principle


can do these things. But what I'm saying is that if you simply feed a large language model,


a lot of data from the internet is going to pick up on like common sense facts a lot more


easily than on mathematical reasoning, because I doubt there's many websites that say, you


know, look here is how you do step by step logical inference. So the model essentially


would have to pick it up through what amounts to reinforcement learning, whereas common


facts about the world, they can just recite from some website. So is it the lack of appropriate


training data? Or is the model architecture simply incapable of performing logical reasoning,


I believe the community is quite split on this point. And it'd be interesting to hear


what you think. The second paper is called ethical and social risks of harm from language


models and is an investigation a bit of a survey of different areas of risk about these


language models. The abstract says the paper outlines six specific risk areas, discrimination,


exclusion and toxicity, information hazards, misinformation, harms, malicious uses, human


computer interaction harms and automation access and environmental harms. The most interesting


paper though is the last paper it's called improving language models by retrieving from


trillions of tokens. There is a special blog post to go along with the paper if you want


a shorter, more condensed version. But in essence, this is a language model. It's called


retro that not only does it produce language, but as it produces language, it is able to


go to a database of things that it can retrieve. So in this database, you can put all of Wikipedia


here they say GitHub books, news and so on, essentially whatever you would usually train


on. So your training corpus, you also make it indexable via a lookup index, then as you


train your language model in each step of producing the next token, what you do is you


take the current input or whatever you've produced so far, you go to that database,


you retrieve the nearest neighbors of whatever your input is so far. And these nearest neighbors


you retrieve with something like pre trained BERT embedding model, I guess you could also


do some TF IDF things. So you want to get the sort of closest neighbors out of the training


data set or the whatever database you have, and then you provide those to the language


model as additional reference to take from the paper introduces a special chunked attention


model such that it can actually refer to these individual passages that the retrieval step


takes out without having the quadratic memory blow up of attention. And as you can see,


it interleaves self attention layers like in a regular transformer language model with


these cross attention layers that now attend to the retrieved things from the database.


The result is pretty astounding, as they say, they can achieve sort of the performance of


these large language models while having much, much less parameters. And it seems what's


happening here is that we always used to think that for these large language models, you


had to scale the data up so they know more stuff or can do more things. But in concordance


with scaling up the data, you also had to scale up the model because what we do during


training is kind of we take the data and we sort of embed the data into the weights of


this neural network by training it right. The reason GPT-3 knows so much is because


we've baked all of this knowledge into the weight somewhere. So GPT-3 not only has the


rules of how to produce language, but also sort of the knowledge that it will produce


all in its weights. So we always used to scale data and model size and compute at the same


time. Now it seems possible. And that's what this research shows that you can in fact,


take some of that data and sort of decouple it from the model size and the compute that


you put in by supplying it at essentially inference time. So now the language model


can be much more focused on how do I need to construct language, it may have a little


bit of knowledge in there, but it can always look up more knowledge at inference time and


use sort of that to produce the output. The paper goes into more details about the architecture,


the chunked attention mechanism and much more stuff. But what's also pretty cool is that


you can if you want take just this transformer, this language model and use it as a regular


language model by not retrieving anything and that seems to work okay ish. So even if


the model cannot retrieve something, it's still able to give good outputs, not perfect,


not the best, but good. And conversely, it also seems to be quite easy to take a pre


trained language model and augment it by such a retrieval mechanism. So to what they call


retro fit it, which is a wordplay because their models called retro. So this is like,


this is like a dad joke that's been in the making for, you know, nine months or so. So


I hope I hope you enjoy this moment where you can say, look, we retrofit the model.


But it is pretty cool, though, you can take a language model that's been pre trained and


with a bit of fine tuning, it seems you can make it use this this retrieval mechanism


and therefore you can supply it with much more data that has been trained on this can


also be a method to keep these models up to date because you know, the training data set


gets older by the day by definition. And instead of retraining, you might be able in the future


to just switch out the retrieval database and therefore keep the models outputs up to


date. All in all pretty cool. If you are interested, check out the blog post the papers and deep


mind no affiliation. Leandro Von Vera has a blog post on the hugging face blog called


training code parrot from scratch where he goes in detail in through how you can train


your own model that is like github's copilot. So it takes your code and it suggests what


next code you want to write. Now copilot by itself is an amazing system. And obviously


there's there's a lot of engineering behind it, there is way more parameters than you


could ever train. But if you want to train a small model from scratch or from a checkpoint,


this is an excellent insight into how this is done. So it goes through everything getting


the data, cleaning the data, training a tokenizer for code, actually training the model, evaluating


it and everything, it shows you how to do some optimizations, like how you can make


everything a bit more efficient by concatenating different samples. So you always fill out


the context shows you what you need to pay attention to when cleaning the data sets turns


out on GitHub, very, very many files are actually duplicated. And that really hurts training


performance goes through hyper parameters, it goes through data parallelism and optimizing


your training code. And it's just super detailed. So here you can see, for example, the comparison


of the accuracies and the code parrot models, even though they're quite small, they do actually


get some significant ish performance. Now it's nowhere near open AI is codex model,


which is the model powering GitHub's copilot, supposedly, but it still you know, does something


and that's pretty cool. So here you can see an example of this. So the prompt is a function


definition called is even that returns true if a value is an even number. And then the


model is asked to set up a unit test for is even. And as you can see right here, the completion


that is given not only is it the correct name has a good doc string, but also it actually


tests the function in question. And it doesn't really, you know, get what it's supposed to


do. But still, the structure is sort of already there. So you could, you know, just assert


like false right here. But as we know, these models really shine when it comes to like


knowing how to handle API's of some libraries and so on. Because supposedly, these libraries


either themselves are on GitHub, or there are many code projects that already use these


libraries. So the models would essentially know how to use the libraries and what functions


to call and so on. Here you can see that the model is perfectly able to build a BERT classifier.


I guess, you know, this is also a bit of a shill for hugging face because it just takes


two lines of code with their code base, but still models pretty cool. So if you're interested,


definitely give this blog post a read. There's a paper out of MIT called learning to see


by looking at noise. And this paper questions the paradigm of pre training on data by switching


to pre training on noise, and they actually get some pretty decent results. They do investigate


different styles of noise. So there is procedurally generated noise, statistical noise, there


is initialized style, and so non trained style GANs, where you simply forward pass data and


what comes out you take as training images. And there is also feature visualization procedures


of trained models. Now here you can see in dark the actual pre trained models on real


images. And you can see that the models that have been pre trained on noise aren't that


far behind. Especially interesting is that style GAN models just initialized randomly


and then forward propagated give pretty decent results. Now these results are on pre training


on a data set, and then linearly adapting these models to image net, which is obviously


not the most performant thing to do, but it gives sort of a baseline. Also interesting


is that apparently Minecraft images also do quite well. There's much more to this paper,


including feature visualizations, evaluations, and so on. If you're interested, paper code


and data sets are available. DeepMind has another blog post called simulating matter


on the quantum scale with AI. Now I have tried reading through this paper and even through


the blog post. And honestly, I have no clue of anything quantum like quantum chemistry,


anything like this, this is just beyond me. But this paper deals with the prediction of


where electrons are in a molecule. So it turns out you don't actually need to track the individual


electrons, you just sort of need to track the density function of where any electron


could be at any time. And in order to predict that various approximations and heuristics


are used. And turns out that if you use machine learning and a little bit very clever data


engineering and feature engineering, then you can come up with a system that outperforms


any of these previous systems. Now again, the papers been published in science, I have


no clue what any of this means if you do and if you're interested, go check it out. Google


AI publishes a blog post called more efficient in context learning with glam. This goes along


with a paper called glam efficient scaling of language models with mixture of experts.


This is a model that is over a trillion parameters in size. Now, this is a sparse model. So it


is not directly comparable to whatever the 175 billion parameters of GPT three, which


is a dense model. So in a sparse model, what you do is that in the feed forward layer of


the transformer layers, you would not activate all of the feed forward layer for every token,


but you would route the tokens to one of many what are called experts. So these models are


generally called mixture of expert models. So the idea is that you have this gating layer


and the gating layer decides which of the experts become activated. This results in


each token only activating a small part of the network, which makes it way more energy


efficient to actually forward propagate at inference time also makes it faster. And with


the current hardware and algorithm optimizations that the Google AI team has put in here, it


does require more flops at training time, because it trains on a way larger data set


than current dense models. However, it does require actually less electricity. And that's


pretty cool. I guess it's a little bit that you're trying to find some kind of a metric


where you're better than anyone else. But I do find it cool that both at inference time


and in terms of training energy consumed, this is actually the preferable model. Now


it is huge and you need a huge architecture to train it. But I think that counts for all


of the models currently, they do have a lot of investigations into comparing dense and


sparse models. And they do generally find that the sparse models outperform the dense


models given the same amount of training tokens and their final model outperforms GPT three


on a number of natural language tasks. So pretty cool if you're interested, check out


the paper. Colin Raffel releases a call to build models like we build open source software.


This is a blog post with a general appeal to the community where he first lists a bunch


of the advantages of open source software versus closed source software and a bunch


of features of open source development such as version control, submitting patches and


pull requests merging semantic versioning, compatibilities and so on. And then he tries


to make analogies to how we could develop models. So at the end, he has this paragraph


right here where he details how a potential future could look. So this says researchers


at Salomon University decide to train a new language model called clamp, they have limited


access to computational resources. So they are only able to train the model for enough


time to attain reasonable performance on a few downstream tasks. After fine tuning,


they set up a framework for testing the models fine tuned performance on a suite of downstream


tasks and release version 1.0.0 of the model to the world. Later a different group of researchers


at the University of Duxville make use of their computing cluster to perform additional


training use a training method that only updates a few of the models parameters so that they


can cheaply communicate the proposed changes back to clamps maintainers. The new models


performance is rapidly verified on the task suite thanks to the ability to reuse updates


from previous fine tuning run. However, it turns out that the Fidmore Foundation has


also been performing additional training in parallel. Fortunately, the updates by each


organization can be merged and they are included in a new release of clamp in version 1.0.1


and it goes on. So this tries to make a bunch of these analogies and I have to say some


of them are pretty accurate and would be like nice to have especially sort of this collaborative


development of models a you release a checkpoint someone else improves upon it, you sort of


merge this together and so on you raise like a pull request on a model but some of these


are a little bit more shady like you would only update a small part of the model because


that makes it cheap to communicate. Usually the communication overhead is in like distributed


training where you need to communicate 1000s and 1000s of time that's when it matters.


But when I train a new model and I like raise a pull request, I don't think it matters whether


I have 40 or 60 gigabytes of weights that I want to merge into the different model also


sort of this notion of backwards compatibility, I think is a little different in real software


versus versus models. And the only true example Colin gives here is that the model would still


take the same inputs and give the same outputs but that honestly that has nothing to do with


machine learning that is again like that is a regress to to actual software engineering


right that that would be using our old systems for software engineering and in between somewhere


is a model. So it might be a bit of a sort of forced analogy at some places but I do


think it's pretty cool. And I do think new paradigms of how we develop models together,


especially as opposed to a few companies internally developing these huge models just in silos


and then selling them via API's but a few things are in the way most notably the very


very often requirement to train things end to end which sort of makes this whole you


know modularity among models a bit tricky. If you want to read the whole blog post feel


free to check it out. Ernest David releases a paper on archive called deep learning and


mathematical intuition a review of Davies et al 2021. This is a response to deep minds


paper about using deep learning and fundamental math. Now, ML news has reported on this with


our outside reporter Marcus bedding last week and this paper kind of criticizes the hype


around this this math paper now fair to say this paper has been kind of overblown in pop


culture like oh AI solves math and whatnot. I mean my own thumbnail was a clickbait for


exactly this but I just want to draw attention to the abstract here in the not theory result


the role of deep learning was small and the conventional statistical analysis probably


would have sufficed in the representation theory result. The role of deal is much larger


however is not very different in kind from what has been done in experimental mathematics


for decades. Moreover, it is not clear whether the distinctive features of deep learning


that make it useful here will apply across a wide range of mathematical problems. Finally,


I argue that the deep learning here guides human intuition is unhelpful and misleading


what the deep learning does primarily does does primarily does is to mark many possible


conjectures as false and a few others as possibly worthy of study. I don't think deep mind has


actually said anything else. Like just the amount of salt in this abstract is I haven't


actually read the paper so the paper could be totally sane and reasonable but it the


salt here is I can taste the salt through the internet but I'm sorry if a conventional


statistical analysis would probably have sufficed then why didn't you do a conventional statistical


analysis why aren't you going out and doing conventional statistical analyses getting


more fundamental theorems or more results in mathematics I wouldn't that be like a better


use of your time no I'm obviously like it is important to also criticize in academia


I think that that is a healthy part of the ecosystem but let's be honest this paper has


mostly been overhyped by media and the paper itself is actually stated fairly accurately


what the contribution of deep learning was so I doubt that an academic paper is the correct


refutation to media hype I think that refutation has to actually just come from other media


but if you're interested in a more sober analysis and maybe a little bit of salt give this paper


a read okay some helpful things for this week transformers has a new release with lots of


updates version 4.13.0 is out and has a lot of new models such as segformer image GPT


Berta v3 and the trainer now supports bfloat 16 numbers excellent cohere AI releases a


really really nice basic introduction to prompt engineering where they show how to engineer


prompts for very different tasks and what has generally worked in the past to give good


outputs of these language models that you can query using in context learning check


it out they not only have posts on prompt engineering itself but also how to handle


temperature or how to set top k and top p variables and so on excellent not really machine


learning thing but GitHub improves its code search I have been previously not so happy


with github's code search and they have a bunch of updates bunch of keywords you can


use bunch of filters and reg exes and so on and I'm quite happy about that so I thought


I'd share it with you hugging face introduces the data measurements tool it's an interactive


toolkit for looking at data sets this is a tool to do some basic investigation into data


sets like show summary statistics drill down into some distributions like word count distributions


see if there's anything off if there's anything over or under sampled look at associations


between words and samples and so on and the goal is I think to also make this into a tool


where you can create new data sets pretty easily the data measurements tool like everything


else is available on the hugging face hub as a space very similar Microsoft releases


a responsible AI dashboard that has various tools to analyze the outputs of your models


and whether or not they conform to some standards where the most mistakes are made and really


drill down into performance issues so here are a few things it supports error analysis


model interpretability data explorer model statistics counterfactual analysis causal


inference what if questions and more this is important especially for practitioners that


are trying to actually build real products and need to diagnose various failure cases


that might not necessarily be covered in the training data Sasha rush releases mini torch


this is a tutorial ish book ish thing where he goes through a building torch from scratch


or something like torch so in this tutorial you'll learn about mathematical operations


how you can build up a system that does auto differentiation how you can build up a tensor


class yourself how you make everything more efficient and so on and there is a github


repo to go along with this if you just want to skip to the end or if you want to follow


along excellent the pandas tutor is an introductory tool to pandas that lets you understand how


pandas transforms your data so in here you'd put your pandas command your your python code


that operates on pandas data frames and it would show you line by line what happens to


your data so here is a data set of dogs if I go down you can see it recognizes the first


operation is filtering by a boolean mask and it shows me exactly what's happening in my


data frame with a nice visualization and even a little bit of animation the second line


is a sort so it shows me what thing it sorts by shows me where every data point is going


then there's a group by and finally a median which are visualized using colors and again


a bunch of arrows they do have more visualizations than just arrows and colors but this is just


an example if you're new to pandas and try to understand what a given piece of code does


or try to debug some kind of a bug that you have this might be a nice place to look you


know is a search engine that given a description gives you an appropriate anime to look at


I am not a big watcher of anime but if you are this might be just the tool for you though


if you are a big fan you probably already know all of them so you know but it's a cool


project the author describes in detail in how this went about there's a lot of analysis


of the data set the code is available there is a collab where you can try it out so here


is an anime where the main character is very smart but no one knows about it you can set


a slider for curiosity and you get various suggestions the US Bureau of Reclamation


has a competition where you have to predict how much water is released from snowpack so


this is a really important measurement because during the winter snow falls into the Rockies


and then during the spring and summer it melts off and provides all the fresh water to essentially


the western part of the US mainly and predicting where how much snow is and how much of it


is going to melt is very crucial to planning ahead there is actually $500,000 to win right


here this is split up so the overall winner gets 150k but if you are also the best in


various regions you can collect prize money from each of the regions and there's also


prize money for the best report so yay reddit user arno walks in ski writes the story about


creating an alpha zero like solution for playing ultimate tic-tac-toe in the browser this user


did not know anything about web development when they started and it has resulted in a


website where you can actually play this game now I didn't I didn't even know what what


this game was but it's a very interesting game so you play tic-tac-toe but it's it's


sort of a super grid superimposed and your opponent will be able to play in the sub grid


of sort of the cell you select right here so if I select this cell the opponent will


be able to play in this cell the next move so you kind of need to plan ahead and then


if you win let's just let's just screw up horribly right here let the opponent kind


of win again in this cell right so if the opponent wins down there then it's not over


but you sort of have to not only win the small games you have to win like the super games


this this is just for a human this is crazy and this user has developed a and sort of


an alpha zero like AI for this and the development is really nicely documented so if you want


to give it a try or if you want to follow sort of the development of this check it out


NL Augmenter is a framework for task-sensitive natural language augmentation and as you can


see it has a bunch of authors I'm reporting this because I've previously shouted out this


project and I think it's a pretty cool initiative the paper has collected augmentations natural


language augmentations from all users and anyone who submitted one is an author on the


paper now whether authorship is meant for that I don't know but you know if the foundation


model team can do it then certainly this is justified the final library of NL Augmenter


is available on github and as far as I know still being extended very cool and lastly


there is a collection of 33 psychology related data sets user yumque writes on reddit this


is by the website open psychometrics and if you are interested in psychometrics and learning


from that data this might be just the opportunity for you Swiss info writes sarco suicide capsule


hopes to enter Switzerland now this seems horrifying by itself but it was actually more


horrifying initially there is a long fact check along editorial note that the article


was changed it originally said this already passed legal review and that it works with


various organizations within Switzerland which is not the case the capsule wants to enter


the Swiss market and is currently in the process of entering the market as you know in Switzerland


assisted suicide by choice is legal and there are organizations that sort of consult with


you and you have to justify to them why you want to go through with a suicide usually


it's because you're terminally ill and you don't want to cause your family more trouble


than needed as far as I know they do have a pretty high bar for when they will actually


go through with the procedure this company seeks to replace with the capsule here's a


description the person will get into the capsule and lie down is very comfortable oh gee yeah


thanks is very comfortable they will be asked a number of questions and when they have answered


they may press the button inside the capsule activating the mechanism in their own time


at that point the oxygen will just be reduced and you'll fall asleep and die like I have


no trouble with the method of dying right but they say our aim is to develop an artificial


intelligence screening system to establish the person's mental capacity naturally there


is a lot of skepticism especially on the part of psychiatrists yeah you think but our original


conceptual idea is that the person would do an online test and receive a code to access


the Sarko oh wow so right after I take the online test for what's your cheese type I


can also take the online test to get into the suicide machine I mean I have to say it


is a tricky subject right because you want to give people this opportunity but also if


if you think that there's an easy way to sort of assess consent and mental state it is also


a big underestimation of how for example depression works and what it actually does to you and


your mental state so even though you might be sort of conscious and legally allowed to


make decisions it is still very very tricky now I'm generally of the opinion that in principle


in principle it might be possible that an AI system might be on par with a psychiatrist


in assessing said mental state but I don't think we're gonna be there like right now


or in the near future but who knows maybe you'll end up in one of these pun intended


and lastly TechCrunch writes Synthesia raises 50 million US dollars to leverage synthetic


avatars for corporate training and more Synthesia is a company that creates these virtual avatars


so here is the three step process select your AI presenter type in your script and get your


video excellent now I'm absolutely for not actually needing to portray a human face anymore


with this like either you hire an actor or someone company internal needs to do it and


then their face is somewhere recorded and so on so I can totally see why this is appealing


ironically the little chat that pop like who who who makes these chats who thinks these


chats are a good idea like I've never ever ever entered anything into a chat that pops


up on a website ironically the person in the chat as you can see is one of the one of the


avatars so the company goes full meta right here in that the salesperson selling you the


virtual avatars is a virtual salesperson excellent now of course these virtual avatars are useful


in certain situations though it does seem a little bit dystopian it also does seems


that other industry notably the adult industry might profit quite a bit more from them but


who knows maybe there will be sort of a lash back and the desire for real humanity and


actual imperfection and the most desirable actors will be ones with scars and no makeup


and dirt and disformed faces and anything and everything that shows that they are not


AI created though I have my doubts about that all right this is it for ML news thank you


so much for listening watching please check out weights and biases thank you so much for


XCiT: Cross-Covariance Image Transformers (Facebook AI Machine Learning Research Paper Explained)

Hello there, today we'll look at excite cross covariance image transformers by Facebook AI,


Indria and Sobon University. So in this paper, the authors propose a kind of a transpose of an


attention mechanism. So instead of the attention working across tokens and tokens attending to


other tokens, now the it is the features or the channels attending to other channels. And in a


matter across the entire sequence that you input, this means there is no longer a quadratic


complexity in the length of the input sequence. And this supposedly works particularly well for


image data. So these are akin to the vision transformers that work on patches in patched


images, and they reach comparable good performance on things like image net classification,


self supervised learning, but also dense prediction, like segmentation and so on. So


we're going to look into this paper, it is, it is kind of weird to how to think about this. And so


the idea is pretty simple. But I think it's kind of weird. And it the question is, to me a little


bit, can this still be called a transformer in the way that it operates? Because, as it seems to me,


after reading the paper, and I think they also mentioned this during the paper, it is more like


a convnet, honestly, that just kind of has one dynamic part in it. So one of the convolutions is


a dynamic convolutions. But we'll see. And, you know, this could be a good architecture for future


image for future image processing. So here they say, let me grab my yellow. Following tremendous


success in NLP transformers have recently shown much promise for computer vision. Okay, so the


self attention operation underlying transformers yields global interactions between all tokens,


ie words or image patches, and enables flexible modeling of image data beyond the local interactions


of convolutions. This flexibility comes with a quadratic complexity in time and memory,


hindering application to long sequences and high resolution images. So this is the problem,


transformers, good attention mechanism, powerful. However, there is a quadratic complexity in time


and memory in terms of the sequence length. And that's why we can't apply it to long sequences or


high resolution images. They say we propose a transposed version of self attention that operates


across feature channels rather than tokens, okay, where the interactions are based on the cross


covariance matrix between keys and queries. The resulting cross covariance attention has linear


complexity in the number of tokens allows efficient processing of high resolution images, yada, yada,


yada. Okay, so and then they propose a an entire architecture built upon the XCA the cross


covariance attention, which they call excite. So that's the cross covariance image transformer. It


says it combines the accuracy of conventional transformers with the seal ability of convolutional


architectures, sorry, scalability. We validate the effectiveness by reporting excellent results on


multiple benchmarks, including self supervised image classification on ImageNet object detection,


instance segmentation, yada, yada, yada, they're super good. Okay. So what is this new kind of


attention? This is the main graphic in the paper. And on the left, you can see how the whole attention


looks. So this would be the whole model is consistent of these excite layers. So you'd have


sort of input tokens down here, and then you have L of these excite blocks. And at the end, you'd


have whatever a classification layer or a segmentation layer or something like this. But in


in our case, this here is what would be a self attention but followed by a feedforward network.


And you can see that the cell it's essentially the same, the feedforward network is still here.


But the self attention block has been replaced by these two blocks. And the bottom one is this cross


covariance attention, which does attention pretty much like you're used to, there's a there's a tiny


difference. I said the idea here is pretty simple. In the in the mathematical way, it's just a bit


weird to think about it. So on the top, you have the classic self attention that is used throughout


transformers currently. And on the bottom, you have this new proposed cross covariance attention.


And you might notice that the only thing that is different, if you look at the at the pictures,


is that the green and the orange matrix here are skipped. So for that, we dive a little bit into


what attention does regular usually. So I think I've drawn this picture about 1000 times, but


forgive me if I do it one more time. Okay. So every we have, let's say we have a series of


tokens like this one here. And this can be word word embeddings in language, but this can be image


patches in images. So the way vision transformers work is, it's prohibitively large to process each


pixel individually. So what they do is they take the image and they put it into patches. And now


each patch becomes sort of one of these tokens, okay, as opposed to convolutional networks,


which can actually work on these high resolutions directly by applying only the local convolution


operation. So these are sequence elements of whatever form, and every of the one of these


sequence elements exposes a query vector. So the query vector is a vector that's supposed to tell


sort of what it wants to know about the other sequence elements. And then also each one exposes


a key vector. So the key vector tells a little bit like what's contained in the in this token. So the


way this is routed is that the query each query is compared to each key, and then the information


is routed according to which ones have the largest inner product. For example, the next


representation of this token right here, we need to look at its at its query, and we need to compare


it to all the keys that we find. So in this case, only this key right here matches. So we would


expect that a lot of the connection between those two is very strong. Ultimately, what you're going


to do in here, in here, you're going to build up a fully connected layer, right, everything's


connected to everything with different strengths. But the strength of the connection is dynamic,


the strength of the connection is determined by the by the attention mechanism, rather than fully


learned. Okay. So, so an MLP would be a fully learned connection matrix, which is fixed. However,


an attention matrix is a dynamic connection matrix. In this case, in the cross covariance attention,


we do something very similar, but we have to think a bit differently. So now here, what we have is,


essentially, we have vectors. Let's represent these token things as vectors. And let's have


three, no, we have five data points. And they all have four dimensions, we'll leave away query and


key and so on. So what what you do is, you don't watch the tokens as a sequence. However, you watch


the channels as the sequence. So this here is now one element, this is one element, this is one


element. And this is one element. So you'd have to somehow trans Can I rotate this? I cannot. Yeah,


I cannot rotate it. You just imagine in your mind this rotated now each channel exposes a query. And


then each channel exposes a key. And now the information is routed not between sequences of


not between from token to token, but from channel to channel. So essentially, you look across the


entire sequence in the first channel, and you decide, okay, what kind of information is in this


first feature across the entire sequence. And you can see kind of how that makes sense. So with the


self attention, you can see that, you know, a token in a, in a picture, it might be an eye, so a patch,


a patch might contain a part of an eye, right? And then another patch might contain a part of a mouth


right here, okay, there's a tooth. And it would be important if these two things could communicate


with each other, because that would give a hint that there might be a face in the image. In this


framing, we look across, we look across all of the things, right? And maybe the first channel is responsible for recognizing


I like structures anywhere in the image right across all the patches. So this could be like the channel that is kind of like, I think there's an eye somewhere. And then this here could be the channel that says, I think there's like, a mouth somewhere in the image. And you can also see


it's valuable if those two things communicate, it comes away from this localization aspect, and more towards communicating across the entire sequence, what kind of features there are. Now, it's not directly the channels that expose this, of course, if you think it's also not, you know, directly the tokens that are compared here. So if you think of your data matrix x as a big matrix, and this big matrix


has is n by D, somehow, not somehow, but exactly. So you have n data points. And every data point has an embedding of size D, maybe D is four here. So we have n vectors, each has four entries, what you would do in the self attention is you would transpose this, like so. And what you would obtain would be a matrix of size


D ID. But not until in between, you multiplied with sorry, you multiplied with the keys and the value matrices. So the way the self attention formula works is that you first multiply x by a, they have the formula somewhere here on the comparison. So what you do is if this


is x, you multiply this by a matrix that is learned, that gives you the queries, and then you multiply x also with the you multiply x with the matrix that is supposed to give you the keys, and then you transpose this, and then that is your self attention. So it becomes something x w q w k transposed x transposed.


So you can see the how the information flows is modulated by these learned parameters here. And that gives you the self attention matrix. So essentially, you will have a transformation matrix right here. Let's say that's D by D for simplicity. And that is, you don't want to compare the tokens directly, but you want to compare sort of a function of the tokens. So we have that, then you have the key


weight matrix, which is also D by D. And then you have this thing right here. So you can see that gives you an n by n matrix, ultimately, which tells you how much every single data point is connected or attending to how to which other data point. Okay, so this is this routing table we saw up here.


Ultimately, this matrix right here is this matrix right here. And that's how it comes to be. So what do you do with this matrix famously, right, you take this, you do the softmax of your x w w x, like this, and you multiply it by the so called values, and the values are nothing else than again, you multiply some sort of weight matrix,


multiply some sort of weight matrix with your data. So do I have this correctly right here? Yeah, I guess so you have this, and you multiply this, you have the softmax of this, you multiply your, again, your data matrix by some sort of other function.


But essentially, this here are the values, and you decide how to mix the values of each of the tokens to get the next tokens. So from the point of view of one token, in the output layer, you decide how should I aggregate across the values of the input layer. That's what the attention gives you. Now, if we look at cross attention,


sorry, if you knew all this, but it's now we contrast this with cross attention. So what we do in cross attention is, we again have our data matrix like so.


But what we do is we, again, we multiply by queries and keys by these matrices. But now we do it differently, we do it. So first,


now I need to replace this up here. So why is it green?


Orange, wow, I didn't know you could do that. This is freaky. All right, I'm done now. Thanks. So we again multiply this here. But we multiply by the other thing from the left, like this.


So it's the same data, the same matrices. But now they're multiplied in a different order, which means that as you can see right here, this is no longer the matrix of inner products being computed here.


This is in fact, I guess the matrix of outer products. And coincidentally, the matrix of outer products is probably smaller than the matrix of inner products, because the dimensionality here, d is smaller. I have made.


Yes, okay. So you can see here, this is d by d. This is d by n, this is n by d. And then this is d by d. So the resulting matrix is going to be a d by d matrix, not an n by n matrix, which means that right here, we aggregate across the sequence. Okay, so the information of where things are going to be is going to be the same as the information of the product.


So where things are is in the sequence gets lost. And is aggregated across. And this here directly, this here is the, if this were centered, it's the covariance matrix, but I think they call it the cross covariance matrix.


Or, yeah, because it's not centered, but essentially, it is the covariance matrix of the mini batch you have right here, not of the mini batch, sorry, it's the covariance matrix across the tokens in a single data point.


So this matrix here essentially tells you how you need to aggregate the channels for in order to go to the next layer. So this again is multiplied by the values. And as we said before, the values are just a linear function. But again, here, this is now multiplied from, this is now multiplied from the left and not from the right.


So again, we have our data right here. And we have our this, by the way, I didn't label it before this is VW, sorry, WV, another learned function that gives you the values. Okay, so this here are the values.


And this here tells you how you how one channel tends to the other. So every token here goes through this process independently, okay. So for every token, it's essentially every token by itself goes now through this process of aggregating features from the other channels in the token.


So very much this is like a one by one convolution, okay, with this here being the convolutional kernel. So usually, I guess the convolutional kernel is represented differently, because you also want to represent it in in space.


But essentially, this tells you how you aggregate information across channels in this one single token. So every single token goes through this map. That is, first of all, the learned map, but then the dynamically constructed map.


So this is very much a dynamic one by one convolution, where the convolutional kernel is dependent on the entire sequence. But there is no information mixing, there is no information sharing across tokens anywhere here, except implicitly, because of course, the weights in this kernel are dependent on the entire sequence of the sequence.


So it's dependent on the entire sequence up here, but not explicitly. So once we have the kernel, once we have the how we aggregate across the channels, every token only aggregates across its own channels, okay, so the information doesn't get spread across the across the image or whatnot across the sequence, like in the self attention.


Because that's why I'm saying I'm not even sure this is a transformer, because so far, it's just a dynamic one by one convolution. The second layer, sorry, the third layer here is a feed forward network. And this is exactly the same as this right here.


So the except in the feed forward network, again, every token goes by itself, and reconfigures itself according to some channel mutation, according to some one by one convolution.


However, the feed forward network is a learned, a learned transformation, and not a dynamic one. So the XCA transformation is a dynamically, so it's learned, but the dynamic production is learned. And the feed forward network is just learned directly with a direct weight matrix.


So essentially, these are two feed forward layers here, except one is dynamic. And then the only other thing they have here is this local patch interaction.


And what is this? This is essentially a convolution, not essentially, it is exactly a convolution. So if you think of this of this sequence of tokens.


The first step is we aggregate across all the tokens, right, then we come up with a transformation, and then every token goes through this transformation by itself.


So that's the that's the first layer we just discussed.


Then there is a convolution. And the convolution is just a local patch interaction, they call it, but it's essentially a convolution. So it's a convolutional kernel that slides across the sequence.


And yeah, gives you sort of the next sequence. So for example, this token right here, it, it will be able so it's convolutional kernel reaches this, this and this one.


Okay, and this is not an attention mechanism. This is just a classic convolutional kernel. And it is even depth separated.


So this goes only within the same feature channel. So if you think again of our data matrix, here, with the feature channels, the convolutional kernel would be something like aggregating over this, and just you just slide it everywhere, you slide it.


So it's depth wise, separable, and you slide it across the image right here.


So the good thing here is that this gives you the interaction between tokens, even if only local, but it doesn't add a lot to the parameters, because if it's depth wise separable, right, it's very few parameters and actually also very few.


There's not much compute and memory overhead. But again, this is a convolution. So the first step is a convolution. The second step is a convolution, and like an explicit convolution.


And the third step, the feed forward one, again is kind of like a convolution. So there, you have a box, much like here, except you don't come up with the box dynamically, you simply learn the box.


And then every token goes by itself through the box. Okay, independent of all the other tokens. And that's how you get the next layer.


So this is it. It's a dynamic convolution, followed by a real convolution, followed by a, it's a dynamic one by one convolution, followed by a real depth wise separable, but not one by one bigger convolution, actual convolution.


And then it's followed by a feed forward layer, which again, is kind of like a one by one convolution.


So that's the idea behind this.


Now, is it good or bad? Or, you know, independent of whether this should be called a transformer? Because, you know, if I think of a transformer, I do think of an attention mechanism. And the core of the attention mechanism is this information routing between elements of the sequence, right?


Just because you transpose it and call it attention doesn't, I mean, it's kind of like an attention mechanism in that it contains a softmax. And it contains like keys and queries.


But yeah, then just because then you call it attention, and then that becomes a transformer. I'm not super sure.


Yeah, maybe, you know, are we now calling everything that has dynamic weights, a transformer? I don't know. I guess we have to come to terms with the the terminology right here of this.


However, this appears to work quite well.


So here they say, these are the contributions right here. So they include cross covariance attention, it includes a it provides a transposed alternative to conventional self attention, instead of channels instead of tokens, yada, yada, yada, it tends to fixed number of channels irrespective of the number of tokens, okay, they're more robust to change as an image resolution, which is also a good thing, right?


So you can do variable size images. And they say, for image classification, we demonstrate that our models are on par with state of the art vision transformers from for using multiple model sizes, they reach good accuracy on ImageNet.


They can do dense prediction tasks, and they can do self supervised learning, using something like dyno. And I've made a video about dyno. And if you so if you use the back the x side backbone with dyno, it works apparently pretty, pretty well.


So cool. This raises a number of questions, right? So it raises kind of more, I'd say more theoretical question to explain what's going on in here, because there is an intrinsic connection between the two kinds of attention, right?


So they're not just random and look the same. But there's actually a discussion in the paper right here about the relationship between gram and covariance matrices here. So you can transform one into the other other.


And also the the eigen spectrums are related, not only related, but actually equivalent. So they say the nonzero part of the eigen spectrum of the gram and covariance matrix are equivalent. And the eigen vectors can be computed in terms of each other.


So there's an intrinsic connection between the two things, even though conceptually, they're very, very different. And I think to to go ahead and really kind of explain which one is good in which situations, why we do what and so on.


There's not even a difference that is still to be seen. The second thing is that if this actually really works as they advertise, and you know, with recognitions of things like MLP mixer and so on, it seems like it's, it's not even important how you do it, as long as you kind of shuffle information around a little bit.


And then you kind of do feed forward layers mixed with shuffling information around a little bit in some way. And this all appears to be kind of performing on par with each other.


Now, we have seen a trend to go away from we got a new state of the art to more like we perform on par with. So you never know how much you know how much trial and error and engineering went into this to actually make it perform on par with.


And then lastly, yeah, this is interesting, because as you can see right here, this model can handle, for example, different image resolutions, and it does scale linearly with the image resolution.


So the GPU memory consumption, you can see right here is even better than something like a ResNet 50. Right. And that's, that's pretty, pretty impressive. Though, on the engineering side, there are a number of things that apparently you have to do when you do these things.


So one is like L2 normalizing correctly. And without that, it breaks down. Temperature scaling is another thing. So they have a learned temperature parameter right here, as you can see, without which the performance degrades a little bit too.


And there are there's another thing, this block diagonal cross covariance tension. So not even they don't even attend from all channels to all channels. So this matrix I've shown you before, they actually do this block diagonally.


So only like the first two channels can attend to each other. And the last two channels can attend to each other. They compared this to something like group normalization, that also has success, only normalizing groups of channels together.


So it seems like to me, this is my opinion, it seems like this is much more a, a never a better evolution on the on conv nets, then it is anything much related to transformers.


So because also the same kind of things help right here. And yeah, making it more local gives you better performance and so on. The fact that there's no info, no long range information exchanged, it really seems like an evolution on the on the conv net.


So I'm not really sure what to think of this other than that, I would love to see this kind of architecture on other tasks, such as language, because again, it being essentially a conv net also makes it really astute to working on images here, you can see, by the way, the attention maps of the classification layer, which look super duper clean, I guess.


So they say heads are sensitive to similar pictures within the same or across images. Yeah, so I would be interested to see this in other tasks than than images.


To really see it's, let's say it's transformer like properties. Though I'm not Yeah, maybe we can start a hashtag, leave transformers alone or something, I don't know, we will have to all decide what a transformer really is.


In terms of performance, of course, these models, they perform fairly well, as you can see right here, though there are some trade offs you can see right here in terms of in terms of number of parameters, if you compare them to models of the similar size parameters, these large ones right here, they do often have more, more flops.


As you can, as you can see right here, though, you can also modify this, you can modify the resolution, and they exist in smaller versions, which means larger patches.


Sometimes the performance is better by a little bit. So here you can see it like it outperforms a little bit. I think it's a good thing that people say more like we perform on par with then touting the point one better performance as kind of state of the art in their sub classification.


So you also see self supervised learning, it performs pretty, pretty decently. And down there, you can also see, I think, they don't have pictures. So there's object detection, instance segmentation, and so on.


They do ablation studies, where they figure out that, for example, removing this XCA layer drops their performance significantly. So this really seems to be the key ingredient to this, even though it's kind of just quote unquote, a dynamic one by one convolution, but this seems to be the key ingredient to the workhorse.


Also, this local patch interaction, like the actual convolution, it drops the accuracy, but not by that much. But not by as much as removing the cross the cross covariance attention layer.


And you can see that without the L2 normalization, it just completely fails, which, you know, is interesting that. So, yeah, maybe as a lesson for future architectures, if you're looking to build a new architecture, and you see it just fails, probably one out of 200 current tricks that we know might make it converge and actually perform better than other models.


So, who knows? Who knows?


Okay, so this model, it looks like, yeah, it looks like a good thing to try. My last criticism here is that they always use patches. So, at the beginning, they tout, oh, what we do is we do, you know, we can, we can, we don't depend on the sequence length, this quadratic complexity, yada, yada, yada, so on.


You know, we say right here, high resolution images are prohibitive, yet they still use patches. And I get the idea behind using image patches.


But it seems like if you are able to process the full resolution images, then the lowest patch size, why should it be eight by eight? I think here, I think the lowest patch size they have is eight by eight, if I'm not mistaken.


Yeah, so this here, it means I think 24 layers, patches of size eight, like, isn't it possible now that we have the fully, like linear complexity in the number of tokens to actually go full resolution on these things?


Though maybe, maybe they did, and I just didn't see that in here. But it seems this usage of patches themselves is a bit questionable if you have a model that is able to go to high resolutions.


Or maybe they just want to put their parameters somewhere else entirely possible. Alright, so I invite you to check out this paper and check out the experimental results. If you're interested in that.


It's all fairly, fairly well documented, there is a long appendix that details even more things and more experimental results. There is pseudocode, pytorch style. And yeah, there is even some some more queries and key visualizations.


Object Oriented Programming in js - JavaScript Programming Basics p.5

What's going on everybody and welcome to part 5 of the javascript basics tutorial in this tutorial


We are going to be getting into object-oriented programming


So object-oriented programming is a specific programming paradigm and with javascript. There's actually many programming paradigms


It's a multi paradigm programming language


So you do things also like functional programming and others


I'm just going to do object-oriented programming. It makes the most sense to me. I've tried functional programming


I just can't so in functional programming is not to be confused with writing functions like pretty much everything


We've done up to this point is script


So we write some variables. We're using functions just to make our lives easier


But we're not doing like functional programming here


So anyway, so why might we want to use one of the paradigms?


So with anything just like with editors and programming languages and significant others


You want to try a few and go with the one that you like the best so the one that everybody else likes isn't


necessarily the one for you


So with object-oriented programming or any of these other ones the reason why we want to start doing this is it's


In general we want to be able to abstract things out and build on top of things


So in the case of blob here in order for us to have


Let's say we like in my case. I want to have many blobs, okay?


This is fine for the one blob, but when it comes time where we want to have maybe 10 15 30 blobs


This becomes a lot more challenging because now we you know we could


assign some sort of


What do they call it?


And I think it's called object literal in JavaScript, so it's like a dictionary in Python


But it's like a key to value representation


You know you you could do something like that and track them, but that's gonna be so messy and silly


For what we actually want to do so with object-oriented programming we define this object and this object has attributes and


methods and the attributes are like descriptions right like like


What's the size of the blob? Where is the blob located? What's its color?


Those would be the attributes and then a method might be move


Okay, so doing things right that's what we're gonna use methods for


So that's object-oriented programming and then basically we just define this blob object, and then we create variables


That are a blob object


And then it just becomes so much more simple to expand and grow and build on top of this blob


Object and manage many blobs this way. I'm not saying it can't be done purely with writing functions the way we've done it


I can think of a few ways to do it. It's just probably not the best way


And then also if you make classes and stuff your code just becomes much more usable either by yourself in the future or for others


Okay, shut the heck up. Let's go somebody will post a timestamp tutorial starts at 15 minutes in okay, so


So I'm actually gonna leave the function here for now


Just because we're gonna copy pasta some of these things so


To start our definition of our class we literally just say class


And then we say the class name of what it is and then it's some curly-do-dads


And then we're good to go now the class is gonna contain


Most of the lines will be methods most likely


Most of the lines will be methods most likely but the first method is called your construct


Constructor method now this is the one that if when you define a very you define your object if you wanted to be able to


Pass certain values these are where you're gonna pass them so you'll notice I didn't put you know parentheses here


It's not because I forgot it's because actually they go here so in your constructor method


We're gonna pass color in size


And then the code to the method so it looks a lot like a function


Now the only big difference is basically this constructor method anytime you define a blob object the constructor method will be run


So in this case anything we put in here is gonna be run now generally what you actually put in here are the attributes


So we're gonna have the attributes like X and well. I don't know why I didn't


anyway X and Y and


Color in size these are the things that are gonna go in here now in order to make these attributes


accessible across the entire


Objects we actually lead them with a this I


Don't it's not a keyword. I don't think I'm not really sure what to call this word some JavaScript expert


That's watching these tutorials for some reason can comment below


Anyway, we lead them with this and it just refers to this object


So we're just gonna say this dot X this dot Y this dot color and this dot the loss


This dot size. It's like self in Python so now we're gonna say is this dot X


equals


And in this case we're just gonna start these randomly so I'm gonna say math dot random


Which is just a built-in and now math dot random generates a random a pseudo random number between zero and


one so we can treat that like it's a percentage and just say math dot random times


canvas dot width


Boom done, and then we're gonna do the exact same thing with Y


Except rather than the width that would be the height


Then this dot color is equal to the color that got passed in those parameters this dot size same story there


Let's do our semicolons because we're good little JavaScript errs


so now


what we want to do is


We're gonna define the X change in Y chain, so we're gonna say this dot X change equals


We'll say one for now this dot Y change is also equal to one great


I think at that point we've got all of the


Initial methods that we want. I'm sorry


Initial attributes that we want and now we're gonna write our methods


so coming down here I


Kind of want to point out older


so sometimes


I'll save it for the end. Hopefully in the end. I'll remember so I'll probably finish this tutorial and at the end


I'm gonna bring up how old old classes were defined


So stick around if you want to look at that because the problem is with JavaScript


There's so much like outdated JavaScript that just keeps getting recycled, and I'm sure that somewhere in here


I'm making that sin as well


I'm doing my best to like write the new ES6 stuff, but I'm sure I'm screwing that up so


Anyway, so call me out on it if I do


But anyway, I will show that at the end how you might see other people using classes because it's completely it looks different than this


You'd be like wait. What's this so anyway moving along the next regular method that we're gonna make is the move


method and


Move will not take any parameters for now, but later it might and in fact


Let me just zoom in a little bit so everybody can see


Hopefully that's better


Can everybody in the back see that?


Okay, so now


So so move is basically gonna be this code here, so actually I'm just gonna cut that


Bop on up here pasta that


Fix the indenting


Boom I went too far. Oh, no shouldn't that oh dear I


Thought you could do


Closing or opening square brackets, and that would shift back over anyway. I don't know


Okay, so now to access this value


This is why we use this so now we can actually even though these are what appear to be two totally separate methods


There are meant separate methods in the same of the same object so or the same class so we can actually reference them


but ins from inside


different methods


So actually I just need to say this dot X


This dot X change this dot X and then we're just going to continue doing that here here in here


This is super error prone, but I think we'll make it


Boom cool, so now we have our move method completed, so now the next thing


My dog is currently dreaming, and I think he's barking at things sorry


So we're gonna come down here, and I'm actually gonna add another method called draw and


I'm a little undecided is where whether draw should actually go inside of here or not


We might actually make draw a separate thing entirely and maybe make it put it just in a function or something


Like that to draw all the blob objects the blob Jaxx


But I'm gonna put it in here for now


Just because I want more than one method honestly


So to do this we're gonna copy everything except for clear rect because we don't want every time we draw any blob


We don't want to clear that entire canvas


so


Posted that up in there and in this case. This would be this dot X


this dot Y this would be this dot size, and then this would be this dot color and


At this point we're actually done with the blob function, so I'm gonna delete the blob function and in fact


I'm gonna delete set interval as well


Because we're not going to use that we will use that in the future, but for now. We don't need that


and instead


What I would like to do is interact with


our blob Jaxx so first of all to define


This it's it's kind of like a data type, so I don't think we've done a raise yet. I can't remember


But anytime you make this new object it will be like variable like let or const


The name of it, and then it's a new object type right and then you pass the parameters


So for for us that's const and then I'm gonna call this new blob Jaxx


That's const and then I'm gonna call this new blob, so that's just the variable name and that's equal


to a new


blob


object and then the parameters we're gonna pass are green and


25 because it is that's the color and the size that we want so if we come up here


Remember that was passed through the constructor method


so


Now I'm just going to say new blob dot draw


And that should draw it to our page because we've defined the canvas so let's go ahead and run this see if we hit any


Errors nope there's our blob beautiful, so the next thing


I'm just we can actually open up the console so it's usually like f12, and then you go into console now. This is


Unbearably tiny I wish I could make this larger for everybody, but I don't think if I scroll in I don't think that does anything


Yeah, let's see if there's a quick font size here


Tab size key bindings


No oh dear


Lovely I'm on a VPS


I really wish there was a font size, but there's not so just listen and follow along


so what we're gonna do is is


So we've drawn it now what we can do is actually interact with it in the console even you could do this in your code


As well, but we just reference that new blob, so it's all lowercase except for the B in blob


Dot and then we can do move


Semi colon enter that blob is moved now. We could actually draw it so new


capital B blob dot draw open and close parentheses semi colon


Boom you should have seen it update on my page, but probably on yours as well


So now we could just like up arrow twice up arrow twice


And just keep doing that and we can see that the blob is slowly moving


And that's how we're interacting with it in the console so pretty cool now


I


Think that's probably everything I want to cover for before we get into


Making many blobs because the whole reason we were doing this is so we could have many blobs


but in order to have many blobs we need to cover for loops and


somehow


Pro tutorials syntax guy has gotten us into object oriented programming in a basic series


before


for loops


Huh?


Good job, so we need to use for loops, and that's a different concept


And I don't really feel like covering that here, so I still have I still want to show you guys old classes


And I'm going to do that uno momento


but first shout out to my most recent channel members backwards are and


Michael Michelle


Berardi I


Don't know tell me in the comments is that Michael or Michelle if you're watching okay?


so


What I'd like to do now is just talk for a moment


About how the old classes looked because they're kind of silly so so in the past


It would be something like this it would be like function


Blob I might be screwing some of this up


But it was basically like this and then you would have your init method, and they would just call it


In it and actually I want to say it was actually an embedded function, so it'd be like function in it and then


Some stuff and then sometimes people would just end it with open and closed parentheses


And that would run it immediately, but then other times you would see people would call in it right after their init method I


Don't think that was ever actually


Object-oriented programming, but they would also use this the this keyword


Or I'm sorry I mean it was object-oriented programming, but I don't really know and on the back end how javascript actually ran things


But just so you know you might see something like this. I know I've seen it around


I'm honest. I see this more often than I see the classes


And that seems to be true with a lot of


Es6 like I don't I don't know why is it the case that like like the first time I saw let


Being used and this I'm pretty sure was when I started using tensor flow dot jS


And so I don't know. I don't know why it's it's almost like I thought the difference between python 2 and 3


That like chasm was bad


But it seems like this chasm is worse because you got both people using it


And there's no like update that forces anybody to use one or the other I


Don't know anyways


Yeah, super confusing for someone like me first coming into here, and it's like wait


Which what do I use and like if you google how to do object-oriented programming in?


Javascript you're often going to find not this way you're going to find the old way so anyway if you ever do google stuff with


Javascript I highly recommend


ending it with


Anyways, goodbye dog is leaving, so I guess that's the end of the tutorial people are leaving the class so


Thanks for watching everybody and in the next tutorial


We will cover a couple of for loops and just kind of iterating over some blobjects making many blobs


And just making a really satisfying


iMAML: Meta-Learning with Implicit Gradients (Paper Explained)

Hi there, today we're looking at meta-learning with implicit gradients by Arvind Rajeshwaran,


Chelsea Finn, Shamka Kodd and Sergei Levine.


So this paper deals with the task of meta-learning.


Now if you don't know what meta-learning is, let me quickly introduce the term.


So in meta-learning you assume you have some sort of a distribution of tasks ahead.


So let's make some examples.


For example, task one could be you get an image, you have a data set of images and you


want to classify them into cats or dogs.


And you know you have a little data set with labeled images and you can train, test, split


that and that's one task.


Now task two is going to be, again you have a small data set of different images but let's


just all make image examples here but you want to locate the pedestrian.


So you want to locate the human in the image.


Where is the human?


And the task three could be again a small database of tasks, sorry of images and in


each of the image you want to visually question answer.


You want to, or let's say you want to point out there is a ground, there is a tree and


there is a question about it.


Yeah let's take visual question answering right, which give you yes or no questions,


something like this.


Now let's just say you have to segment, you have to segment that image.


So down here would be ground, you have to segment the ground.


Okay so these are all image tasks right, these are all perfectly fine independent tasks,


for each one you have a small data set.


Now sometimes these data sets are very very small such that you cannot really train a


state of the art model on them.


For example if you have medical images, oftentimes the labels of these are very hard to get.


I mean there's privacy concerns and then you know doctors have to look at it to produce


the images, costs money and so on.


So it's not like you have a lot of images and you could profit from more images.


So one method that people come up with is called transfer learning.


So in transfer learning you say I have this giant database of images, let's say this is


ImageNet right, I have this giant database ImageNet with labeled images.


What I can do is I can use this to train a neural network.


These are the bunch of layers of neural network and I can train the neural network on this


big database and get parameters theta right.


These are the parameters of the neural network and then I will basically adapt these parameters


to each task individually.


So in task one, sorry about that, I would then take these parameters as an input to


the neural network.


I would initialize the neural network with these parameters and then I would use this


training set in order to fine tune, this is what's called fine tuning, to the task specific


parameters here phi, so phi one because it's task one.


For task two I would also take these as a starting point in order to train its neural


network to fine tune it on this bounding box task in order to obtain the parameters for


task two right.


So you can see that there's a pre-training stage here to obtain good initial parameters


and then we adapt these initial parameters for each task separately in the fine tuning


stage.


So this is one way we can do it.


Another way we can do it is called multi-task learning.


What we do in multi-task learning is we say well see probably a neural network that can


segment the grounds is also pretty good at doing bounding boxes like it will use some


of the same features.


So can't we just kind of pool together these data sets into one bigger data set and then


train on like if it's an image from task one we'll train on the loss of task one and if


it's an image from task two we'll train on the loss of task two and so on but we'll sort


of use the same neural network basis we just have kind of different heads on top of them


and we'll basically so this is called multi-task learning have one shared neural network with


different outputs for the different tasks and basically counting on the fact that you


can sort of learn from one task what's useful in the other.


Now this is a good method to combine the tasks and to basically share data information but


it will also limit you because you now have to trade off between the tasks like this neural


network right here this joint encoder will never be able to fully geared to one task


because it also has to you know it has to perform for the other tasks as well.


So you kind of limit yourself in your top-out accuracy now maybe the regularization effect


is good.


So these are two methods the first is called transfer learning here on the left and the


other is called multi-task learning.


Now meta-learning goes a different direction meta-learning is like transfer learning but


it says well what if we don't have this giant data set right here.


What if we find a way to learn these initial parameters right.


So what we'll do is we'll start out with a guess a guess of good initial parameters let's


call that theta zero and now we have all of these three tasks take theta zero and run


their run their fine-tuning right to come up with with their own parameters starting at theta zero.


So this is Phi one started from theta zero and we'll also give it to task two and to


task three and each of these tasks is going to train on its own training data set and


then evaluate on its own validation data set and then report back a number.


So we do this for every task and every task basically trains this runs through the validation


data set reports back a generalization error and then we know once we get all the information


from all the tasks we know how good were these initial parameters right.


We get a measure of how easy is it for the tasks to adapt these initial parameters to


their own data set and then we somehow need to figure out a way okay these parameters


were on average 81% good.


Can we come up with a better set of initial parameters theta one in some way can we somehow


find a better set of initial parameters such that it is easier for the tasks to adapt these


initial parameters and even more so there could be task four which we are not seeing


during this training phase right this is kind of our so these up here could be our train


training tasks and this down here could be our validation task.


So basically we're trying to come up with a set of initial parameters that if a new


task comes along and it takes this thing as its initial parameters it will be able to


adapt very quickly these initial parameters to its own data set and most importantly it


can do that it will result in a much better model than had the task just trained on its


own small data set from scratch.


So our task is in meta learning is basically to come up with a learning procedure to generate


or to iteratively generate better and better and better initial parameters right and what


better way to do this than using gradient descent.


So this is this is meta learning using gradient descent is is a is the core of this paper


basically now what do you need for gradient descent so if you want to try to go from one


task to the next using SGD or even GD gradient descent you need to come up with a gradient


now why is this a problem in this case so this is in this figure so this meta learning


using gradients was done in this technique called MAML and essentially what you can see


here is that if you are you have this set of initial parameters this is your current


best guess of these good initial parameters and you want to come up with a gradient of


how to get an even better set now this gradient here is indicated by this arrow so you don't


let's imagine you don't know the gradient yet you want to come up with a gradient so


what you'll need to do is you'll basically have to compute the loss function and you


have to differentiate that loss function with respect to your parameters that's down here


what the description of the orange arrow so the your loss function of your meta parameters


is going to be the average or the sum of the loss functions across all of the different


tasks individually so the gradient is going to be the sum of the gradients of these loss


functions with respect to your original parameter now this is the difference right usually we


differentiate with respect to the parameters that we input into the loss function but not


here what we input into the loss function is what is at the end of the task adapting


to its own parameters so this thing we input here is a function of our initial parameters


but it's not our initial parameters so what we have is initial parameters we give them


to a task we wrote this task runs SGD for k steps it runs it for a number of steps comes


up with the adapted version to its own problem that goes into a loss function right so the


this this this thing here are the is the neural network that finally determines the loss function


so if we want to back propagate we can back propagate this loss function right through


the neural network the F the neural network is right here is parameterized by these things


we can back propagate through that but then we'll have to back propagate through the optimization


procedure that was used to derive these things so that's the the problem right here you can


see this here you start out with the initial parameters and let's say you give them to


task one task one is going to take these as initialization and then run SGD so maybe it


will perturb these parameters to come up with here Phi one these parameters these parameters


are the adapted version for task one and then at the end of task one you use these characterize


a neural network and you can calculate a gradient so how would you need to update these parameters


in order to make the loss go down and the neural network or sorry the computation will


maybe result well you need to go up a bit well this is too strong it will maybe say


you need to go into this direction right here with respect to these parameters but now the


question is how do you have to adjust your initial parameters such that your your final


parameters will go into that direction and that's not really clear you could make a guess


right you could make a guess and say well if my initial parameters just go up a bit


maybe the optimization procedure will just you know sort of look the same but shifted


up here so something like this and then I will end up here but that's not guaranteed


like this is a super nonlinear procedure that you're running it through this SGD thing and


it will basically it's an iterative recursive procedure so it will sort of accumulate its


own nonlinear errors and that's why what you have to do is basically you forward propagate


through SGD and then you have to back propagate this gradient right here you have to back


propagate this through the entire SGD optimization procedure and that is computationally very


expensive because if you have to compute the loss once here for a neural network you have


to forward pass once and the back propagation will cost as much as the forward propagation


or maybe twice as much constant number but if you run k steps of SGD you basically have


to trace back those k steps via back propagating it through each step so basically this is


k times a back propagation step and then computationally that's just not feasible for more than very


few steps and so you can only ever do very few steps you basically accumulate your nonlinear


error because gradient descent is a linear procedure and then you get some estimation


of the gradient at the end now if you do that for all of your tasks then finally you can


decide so maybe for a task one here the result will be in order to make this go up a bit


you need to shift this a bit to up and the right right because then the gradient descent


will kind of sort of end up here and you do this for all tasks and you average the gradient


like here then you can come up with a final gradient for your inner sorry for your outer


model for your initial parameters so this is a big load of computation that MAML does


here now there is a naive approximation and this is exactly what we said at the beginning


right this first order MAML is the guess that if we want to go up a bit at the end here


why don't we shift the beginning up a bit right and so the first order MAML would just


result in basically looking at the gradients at the end and sort of aggregating them right


here and then coming up with a gradient but this is very inaccurate and generally doesn't


work well because you have to understand how your end gradient is connected to your initial


gradient because this is very nonlinear you can't just basically transfer it over now


implicit MAML this paper right here circumvents that it circumvents the step to have to explicitly


back propagate this gradient along the forward pass but it's still able to come up with an


expression for how the final gradient relates to the initial gradient so this is quite cool


and we're in this video I would basically like to explore how this comes about and why


this comes about we won't go through all the theory and the proofs but I would like you


to understand that this comes about by basically them imposing a quadratic regularizer and


therefore this quadratic regularizer makes a very it kind of gives rise to a very strong


connection between this final gradient and this initial gradient so they can basically


transform one into the other and therefore they can compute the initial gradient in a


closed form setting or at least in theory all right this was this now let's go into


the problem formulation as they see it the entire problem formulation is you want to


find these best meta learning parameters and they call this the outer level so on an outer


level you want to run gradient descent to find the best meta learning parameters to


minimize this function f right here now what is f f is the average of the validation loss


function so here this is the loss function on the test sets of the individual tasks and


the neural network that we evaluate on the test set is the neural network that is trained


the alg here is is that is a training algorithm is trained on the training set of that particular


task while starting from these parameters theta now you see there is no perhaps no no


dependence on the task here there is no I down here for these meta parameters because


these are always the same right that's the crucial point all the tasks start from the


same initial parameters then they optimize on their own training data set and then they


evaluate on their own test data set and that will give you the loss for that particular


task and your goal is to find the meta parameters such that this function here the average loss


that results from this procedure is minimal okay so where they say right here the inner


level is this algorithm component so the algorithm starts from these from these meta parameters


and runs gradient steps on the training data set loss now this is just the first step right


here of this procedure of course in the next step these are going to be replaced by the


by the Phi that by the Phi I that results from the previous step so the first step is


run on the meta parameters and then subsequently the task specific parameters are updated the


important thing here is that this doesn't need to be gradient descent actually with


their method because their method doesn't need to back propagate through the optimization


you can think of any inner optimization procedure that you want it can be like a black box solver


whatever you want I'm going to be it is interesting to see how this is going to affect something


like reinforcement learning and so on this might have already happened and I have not


looked up this so the crucial part of the paper I think is right here and it's sort


of like you know section 2.2 but I would I would want to point out that I think this


is the crucial part this is why the method works ultimately why it's in why it's able


to build this implicit gradient so the section is called proximal regularization in the inner


level and we'll go through this with a bit of detail to have sufficient learning in the


inner level while also avoiding overfitting ALG that's the inner optimization procedure


needs to incorporate some form of regularization right so their their their goal here or their


point here is that if especially if these individual tasks have small training data


set you need to have some kind of protection against overfitting and that and that and


they say since mammal uses a small number of gradient steps this corresponds to early


stopping and can be interpreted as a form of regularization and Bayesian prior so mammal


is this this previous method this basic method that back propagates through the optimization


procedure and since it does this since it back propagates through the optimization procedure


it's computationally limited to only run very few forward optimization steps because it


then has to back propagate through each one right needs to store each one so it's computationally


limited so by necessity it uses only a small number of gradient steps and therefore this


is kind of early stopping and we know to prevent overfitting one thing you can do is to stop


before your training accuracy reaches full zero and you can stop earlier than that ideally


at a point when your validation accuracy reaches the low point but they say basically this


this limited number of steps is a form of regularization now of course in in this new


method we we don't have this constraint anymore we can run our inner optimization to super


convergence and therefore we we don't have this implicit regularizer anymore and we'll


have to make up for that they say in cases like ill-conditioned optimization landscapes


and medium shot learning we may want to take many gradient steps which poses two challenges


for mammal first we need to store and differentiate through the long optimization path of alg


which imposes a considerable computation and memory burden right that's what we said second


the dependence of the model parameters Phi I on the meta parameters shrinks and vanishes


as the number of gradient steps in al grows making meta learning difficult so what they're


saying is if you optimize your inner optimization algorithm to the very end then it is not very


it's its dependence and especially for gradient descent it's linear dependence on the initial


parameters so on the meta parameters shrinks the more optimization steps you do because


the more and more you're going to basically forget about your initialization move away


from that and move to a local optimum from which you could reach you know from many many


different initializations so that's still a question whether that's happening but this


that's the idea here right if you're at the local optimum you could have reached that


from any sort of point and there's going to be very little information about the end of


the procedure from the beginning and therefore if you want to calculate the gradient that's


going to be like super inaccurate and they say to overcome these limitations so they


they solve these two things in one right here we consider a more explicitly regularized


algorithm so what they'll say is they'll say we don't just want to optimize this inner


objective so this would be here so the we don't just want to find the minimum of this


inner loss function that's one goal we have but the other goal we have is to stay close


to the initial parameters and that's where this regularizer in comes in here so this


basically says we we want with our parameters here that we are optimizing we want to find


them such that they minimize this loss function you know really minimize it find the best


point but also with a trade-off of lambda we want to stay close to the to the initial


parameters that we started from right this is this is the initial parameters and the


closeness is measured in the L2 norm so it's a quadratic regularizer on how close you are


you might know from you know initial supervised learning or something that sometimes you do


something like plus lambda times the L2 norm of the weights so this would be called weight


regularization weight decay L2 normalization something like this where you regularize your


weights such that you stay close to the zero point right implicitly in this there is a


minus zero given but here you want to stay close to your initial parameters so the inner


optimization is no longer just minimizing the loss on the training data set the inner


optimization is that and with ALG star we denote if so if ALG has a star we say that


the this is this is referring not to the procedure of the algorithm but to the minimum that the


algorithm has found right so ALG star means the algorithm has optimized this inner procedure


to its minimum which is a balance of the training loss of that task and staying close to the


original parameters and this I think this they say that here the proximal regularization


term in equation three encourages Phi i to remain close to theta thereby retaining a


strong dependence throughout this is why their method works and we're going to see in the


math right soon how exactly this this how exactly they're able through this to establish


this implicit gradient correspondence so they formulate their entire algorithm as follows


we want to find the best meta learning parameters by minimizing the function f where f is the


average losses and L here now that's the test set loss the average validation loss for each


task of the parameters that the inner optimization procedure finds when it runs to its optimum


now this you can see here this is already different from the original mammal the original


mammal was simply running it for a number of steps and now we're really running it to


the optimum at least in the ideal case what does the inner optimization algorithm do the


Alex star here minimizes this function G right now G has two arguments G as G has these parameters


which are the local parameters and these are the meta parameters and we only optimize the


local parameters right we take these as initial and then fine-tune them where the function


G is defined as the training loss of the local parameters plus this closeness regularizer


okay cool now the question of course is how does that lead to gradient descent so ultimately


we want to minimize this function f right here so we have we're going to have to do


something like d f by d theta right to in order to run gradient descent we need to calculate


this gradient because we need to do gradient descent so what's that going to be that's


going to be of course since f is a is this one over M up here right the gradient simply


distributes over that sum now it's the gradient or the sorry the derivative of each of these


inner loss functions let's go with this Al star I theta that's basically what you have


right here okay so in order to take the gradient of f we need to be able to take the we need


to be able to derive these loss functions and you can see right here theta is not the


argument to the loss function theta is the argument to this inner procedure so by the


chain rule right now this gives us this so the chain rule says we derive the outer thing


with respect to its input that's this part right here the gradient of the loss with respect


to the neural network and that thing here that's the Al star so we need to gradient


of the loss function with respect to the end parameters of the optimization procedure now


that's easy that's we know how to do that that is the or that is the so if you remember


the drawing at the beginning this gradient is the end arrow right here this is easy this


is one backward propagation this is regular supervised learning backprop right you have


parameters of a neural network a gradient for the loss function cool the hard part is


to have the derivative of the algorithm itself with respect to the to the meta parameters


so this is going to be this here is going to be a vector it's the gradient with respect


to these parameters and this is going to be a matrix and the matrix will relate basically


one dimension each dimension of this vector sorry this is going to be a product between


this thing and this thing and it will result in this thing so the left thing is the gradient


we want this is the derivative of the entire thing that's this now the right thing is the


gradient at the end of the optimization procedure and this matrix here relates the individual


dimension of this end gradient to the gradient that we want right this is a matrix that relates


the two in a linear fashion this is what we're looking how if how do we need to change the


initial parameters in order to change the end parameters by a certain in a certain way