match this blinded hash right here. So it should be able the server should be able to decrypt the


How Apple scans your phone (and how to evade it) - NeuralHash CSAM Detection Algorithm Explained

payload using that thing, right? Because it was, in fact, encrypted with this. So it should also


be able to be possible to be decrypted with this, you actually don't need. So this is only a


conceptual thing, right? So this is what's happening, you take the neural hash, you compute


the blinded hash for the neural hash, you can do that. And if you are able to decrypt the payload,


that means that that the neural hash here actually resulted in this blinded hash here, whereas


if it was just kind of a random neural hash, the H prime will not give you the same blinded hash


as is here, as you used to encrypt. And therefore, you won't be able to decrypt the payload. Now,


I was a bit hesitant when I when I saw this, because, you know, this is a this is a database,


right? And the security here, you know, it's a good idea, but the security appears to rely on the


size of that database, right? Because, um, sure, if this is like a giant database, you know, you


have no chance of selecting the correct blinded hash from from here, like, all of this works. But


let's say this is only like 100 rows, right? And we know the client used one of the blinded hashes


in the database to encrypt their payload, like they had to they do this procedure where they


look up the blinded hash, and they encrypt the payload with that. So there's a limited set of


keys that the client could have used to encrypt the payload. So what keeps the server from simply


trying all of them? I don't know that honestly, like, I think we're just relying on the fact that


this database is so large that the server can't try them all. But that means it must be something


like exponentially large, which I don't think is happening. Maybe I'm missing something here. Maybe


there is some additional thing. But I would guess, you know, if I'm Apple, and I really want to know


what's in the payload, I just go through all of this database. And I just use all that because


the key needs to be one of those things, right? Maybe I'm mistaken right here. But, you know,


that's, I guess that's the thing. So this works. If you assume the server cannot just try all the


blinded hashes, if you if you assume that, you know, the server, the only choice it has is to


actually determine the blinded hash via H prime. And try to decrypt because only if in fact, this


is the image that led to the creation of this blinded hash at this row in the first place, the


this will actually match and the server will be able to decrypt otherwise not. Okay, so this is


the first thing, this is the private set intersection, the client doesn't learn which objects matched,


right, it just always uploads the neural hash and the payload for every image. And the server is


only able to decrypt if there was in fact a match, and it learns nothing about the images for where


there wasn't a match. So this this fulfills our requirements. The next requirements is with respect


to what's called threshold secret sharing. So this is private sec set intersection. The next thing


that Apple wants is we don't they only want to know about you if you know if you've matched like


five times or more. And that's, that's a technique called threshold secret sharing. And what we're


going to do is we in fact are going to do two different levels of encryption. So remember, I


said in this payload, there is the image, we put the image in there. This means if any of these


matches the Apple gets to look at the image. So we're not going to do that. In fact, we're going to


make it a little bit more complicated, we'll put like a little box into a box, you see this here,


there's first encryption layer and second encryption layer. So the first encryption layer is


going to be as we have it right now. But the second encryption layer is inside the first


encryption layer. So even if there is a match and Apple can decrypt the payload and look at the


payload, the payload itself won't help. And that is it's a pretty simple technique. In fact, there


is a way in which you can create a key. So in draw a key right here, a key in in cryptography,


in in cryptography, and you can shard it or make shares out of it. So what you can do is you can


derive many, many shares as many as you want with the property that you can only decrypt whatever


message I encrypt, if you have at least, let's say three of them. So if you have any three of those,


then you'll be able to combine the three and decrypt the message that I encrypted. If you have


less than three, then you're not able to. So we're going to encrypt. So inside this payload,


we're going to encrypt the actual image information one more time with this key. And then for every


payload we send, we only going to put one share of that key inside. So remember, whenever the neural


hash of the image matches, which is up here, the server is able to decrypt this outer layer.


So they will learn one share of the key. That means if you know, five of my images matched,


the server was able to decrypt five of the shares. And then it has enough to decrypt all of the


images. So now repeat this box here. Repeat this box many times like one, two, let's do three,


right? Repeat this box many times the cryptographic header up here, there is a box inside that can be


decrypted when any of the ones match. And then inside there is a share of the key. And a little


box that you can only decrypt with the key with the payload inside of it. So once if if if only


two things match, right, Apple doesn't have access to this in their box, let's say only to these two


inner boxes, it cannot look at any of the images. But if three match, Apple has access to three


of the inner boxes, which means it has three keys. And then it can go and decrypt not only


the last one, but it can in fact decrypt all of the previous matches as well. So at that point,


Apple will learn about all of the in thus far encrypted payloads. So we have both Apple can


never decrypt anything if the neural hash doesn't match. And Apple can only decrypt things when the


neural hash match neural hash matches whenever they enough matches have been made. There is a


last thing in that. Yeah, so they display this in in various ways. There's a last thing in this.


There's a last set here, where they generate synthetic match vouchers, because now, you know,


let's say they can still see how many vouchers match, okay. So they do these synthetic vouchers


in order to confuse themselves. So the devices will actually every now and then send dummy data.


So they're called synthetic vouchers differ from real vouchers in the following ways. The underlying


image information is substituted by dummy data. The secret share of inner key is substituted by


a random share that is totally independent of the inner encryption key. And the cryptographic header


and the outer encryption key are chosen to always result in a match on the server. So you upload


security vouchers that always result in a match. But the key share on the inside is the match


side won't do anything because it's just like a random, random bit of numbers. So whenever you


exceed the threshold, Apple will attempt to decrypt because it thinks it has enough shares. But if


some of those things are synthetic shares, then it won't be able to. And this seems like this seems


like a hurdle. This seems like it just makes introduces more noise. But this is exactly the


goal, right? So Apple can never, if it just knows the number of matches, it says, well, we don't


have enough matches yet to decrypt this person's account, it can never exactly tell how many


matches of those are real. Because as long as they can decrypt anything, they have no idea if these


vouchers are real or fake, right? And even if they like if they even if they have enough, like


initially, before they have enough real ones, let's say this is a fake one, they can't tell


which one is fake, they can only say, well, one of them is fake. Yeah, we need more. Okay, so


there's, as you can see, there's a lot of mechanism where the engineers here made deliberate choices


to limit their own abilities. I'm going to guess they did this out of you know, if you were,


let's put that here. You know, if you're designing an algorithm like this, it's already hard enough


to get the public to accept this. And they did, I think they did a pretty good job mitigating


whatever they could, in order to say, look, here's how we're going to design it, we're going to


maximally preserve user privacy in while still be able to do what we're doing. And this would all


be good except, except this issue I mentioned here, you know, this would all be good, weren't it for


the pesky pesky deep learning. So where are the problems in the system? As I see it? Where was


this diagram here? So the problem in the system? No, here? No, here. The problem in the system


are at the first of all, let's talk about this database. So you have a database that Apple


presumably gets from this government institute. Well, sorry for scrolling around my devices.


So presumably, Apple gets this thing from here, right? Um, cool. You know, as long as that's the


case, and as long as that database contains really images that are of child abuse, we're all we're


all okay. However, this database is probably going to be quite guarded access to it is going to be


limited. As I said, it's not even clear that Apple gets access to it. I mean, they, they probably do


themselves a favor if they don't need access to it, they just send the neural network to the


organization or to the to the government agency and say, please compute the neural hashes and send


the hashes to us, we want nothing to do with this data whatsoever. That, you know, Apple be smart


doing that. That also means though, there are there's very tight control on that database. And


not a lot of people are allowed to go and access the database. Good thing in principle, bad thing,


if you think it in a different way, namely, what I can do is I can, if I am the government, one of


the few government officials that's actually allowed to interact with this database, I can


insert a new thing. Now, if I'm a good, good bureaucrat, I'll insert new child abuse material,


because I want to find the people that share it. However, I can insert anything, right? And, you


know, there is an algorithm, if I insert something, blinding step, yada, yada, yada, no one actually


knows what's in the database, right. And then at the other end, equal some something will go bing,


bing, bing, bing, bing, if that's actually on a phone of someone. So that this gives me as a


government, this gives me a general mechanism, like I have to have to control Apple a little bit


if Apple actually does the matching, but it's not even said it could be that Apple just forwards the


decrypted information to the government. But you know, at the end, I have an algorithm, I insert


anything into this database, any picture, but this is going to be this is this is just pictures is


just the start, right? The they're going to widen this to all kinds of things. So I insert anything


into the database. And, you know, a second, a minute, an hour, a week later, I'm going to get


big red lights for any single phone for any single iPhone that has that thing on their iCloud. This


is the potential for abuse of this is is enormous, right? If I'm a political party, I want to find my


opposition. I just insert something into this database that I know is only likely on phones


where my opposition is maybe I confiscated one of the phones and I just entered the stuff into the


database. And then right after that, all the all the people that are part of the opposition of the


rebellion of whatnot, light up and I know exactly who these people are, right? So the Yeah, the


potential for abuse for whoever controls the database is huge, because of the nature of the


material, but also because it's a, you know, a government agency, we are not going to be able to


check whether the things in the database are actually what they claim they are. So Jen, like


really big red flag for me there. Second of all, the image part, in order to compute the neural


hash on the device, and we saw this up here, this is computed on device, client device computes the


neural hash of the image. Now, in order to do that, I need to have the neural network on my device.


So I have an image here, I put it through the neural network, I get out a vector. Okay, very


standard neural network stuff. That's what that's what they do. They input stuff, they output


vectors or whatnot. We there are things they're known as as as adversarial attacks. And adversarial


attacks can be run on technically any machine learning system, but it's really easy to do that.


But it's really easy if you actually have access to the model, which you would if this is on your


device, right. So what I can do with an adversarial attack is, I can remember when we said, even if two


images are really close, they're only maybe you I cropped them a little bit, the neural hash should


be the same. This is true for, let's say, random distortions, distortions that happen naturally,


or anything you can think of. However, there are techniques called adversarial attacks, where you


can specifically engineer the distortions such that the distortion to the image is minimal, like I


only change a few pixels by a little bit, humans won't even notice it. But the output here will


change drastically. Okay. So if I have access to the network, and also have, like if I have access


to the LSH hyperplanes, that it's really, really, really easy to create an adversarial attack that


will switch the output just into a different bucket. This is this is insanely easy, right.


And people that, okay, these might not be the smartest people that share this kind of stuff and


and upload them to iCloud. But one of them will come up with this idea and have a bit of a software


engineering background. So if if you have a phone with root access, you could even, you know,


install software that just automatically whatever picture you have, it automatically puts some


adversarial perturbation on it, such that the output is switched to a different bucket. As


Apple says, if you if your image is legit, the probability that they'll, they'll match you is


really small, which means most of these buckets are safe. So whatever you have to do, you just


switch the bucket to some other bucket, you're going to be just fine. So it's quite easy to evade


this, right? This is not like all this engineering afterwards, all of the private said in a crypto


that he the yada yada yada. This is all cool. But this relies on the fact that this neural hash,


it's doing what it's advertised to do, which it is for normal images, but in the face of adversarial


attacks, it is not. Now, there is a second thing in that I can if I can make two vectors be far


apart when they should be close together, I can make two vectors be close together when they


should be far apart, right? So if I have an image, and it would give me, let's say this vector,


but I know this vector is a bad vector, right? This vector is illegal material vector, what I can


technically do is I can make an adversarial perturbation that shifts this to that. And so that


it ends up in the same bucket, while only changing the image a little bit. Now, this is a bit more


complicated, because it requires me to actually obtain this bad vector, which I think the the


general the way they hash everything, and so on. The only way of doing that is I would actually


have to up, I would have to obtain an image that I'm relatively sure is in one of these databases,


and then not get caught myself. And in order to derive this vector right here, which, you know,


don't like this is this is an illegal step in itself, right? But if, if you're able to do that,


and then you're able to essentially frame people, so you can derive images that just look right,


this this looks like I can take any image and do this, it looks like just a normal image,


but it's perturbed in such a way that it matches with one of these illegal vectors,


that will be sent to Apple and so on. And now it depends if you really trust that everything here


is manually reviewed or not. Yeah, again, the the potential here for for abuses is big. And if you


now think of the fact that people who share this kind of material are probably going to employ some


kind of these evasion techniques, like I presented here, some kind of these adversarial attack based


evasion techniques, then, you know, it's the system is quite easy to evade. Yet, the potential


for abuse, as we saw down here with you know, who gets to do put what in the database, and the,


I would say less less important, but still present danger of people framing people, which also


necessitates a failure of the manual review. All together, it the picture of whether this is a,


a, you know, a desirable system to implement becomes less clear. So if I understood this


correctly, I would be quite worried here. And I would like, you know, if I would like to see a


world and I don't want to say I would advise I would not advise but I would like to see a world


where every single person in the world does does technique one right here to any image they have


on their phone, right? It's like, if only one person uses encryption on the internet, like


that's suspicious. But if everyone does it, you know, we're all you know, it allows bad people


to do bad things. Yes, because that's encrypted. But the ultimate safety for everyone is is better.


And you know, we'll have to look for other techniques to catch the to catch the people


sharing this this material. Yeah, so that that is kind of my, my, my take here. Yeah, I would


say, yeah, I won't be doing this though. I don't have iCloud. So yeah, it's it's going to be it's


going to be interesting to see what's going to happen. In on top of all of this, in a general


more meta meta layer, we're about to see a step of where where the company essentially, you know,


scan every image on your phone as I explained, but it goes into the direction of Hey, you know,


whatever you do with our stuff, we were going to essentially look at it even if this algorithm we


can't but it is an expansion of the power of these companies, which is also worrisome by itself.


Make of that as you will. This is already too long. Thanks so much for listening. If you like


this, leave a like, subscribe. You know, if you have better ideas, I'm more than happy to


REALM: Retrieval-Augmented Language Model Pre-Training (Paper Explained)

What's the angle of an equilateral triangle? So if your high school math isn't fresh in your head,


you might be forgiven for not knowing this. But what do people do when they want to find out the


answer to that question? Of course, the standard way nowadays is to go to search engine like Google,


type in the question, find some website that contains the answer, and then sort of read that


website and answer the question from there. Now, the goal of this paper here is to do the same


thing, but in a machine way. So the machine would see this question right here, and it would be able,


it will be able to get additional textual knowledge from a corpus and consult that and


then at the end, come up with the answer, which is 60 degrees right here. This type of this type


of task is called open question answering. So like open qA or qA. And the distinction here


between this and the previous kind of tasks that were often called question answering is that


usually in question answering, you simply have a question, and then you have either no help at all,


so the model just has to answer the question. And things like GPT-3 demonstrated that that is


actually something that's possible if you have a large enough model, or much more common, you would


provide the question and then one document. And you would sort of guarantee that the answer is


somewhere in this particular document. So even though the task was called question answering,


it was more like, it was more a machine reading task, because you knew, okay, all I have to do


is I have to find the answer somewhere in the document to this particular question. So the task


was more kind of a pattern matching sort of approach. Here, the it's really the task really


comes close to what humans understand as question answering, namely, you get a question, you want an


answer. And it's open in the sense that you can the machine can go with the question to like a


search engine, I have no clue how to draw a globe to a search engine, get multiple documents that


would help it kind of rank them and so on. It's basically able to use a search engine and then


answer the question from there. So that's what we're going to look at today. There has been a


lot of work. I'm not not saying this task is new, there has been a lot of work in open domain


question answering. And this is one of the latest incarnations of it. The paper is called realm or


realm. I'm really not sure how to pronounce this the word would be called realm, I guess. It's


retrieval augmented language model pre training by Kelvin Gu, Kenton Lee, Zora tongue, Panopong


puzzle pot and Ming yi chung. So the paper is first and foremost about a pre training method,


as you can see, right in the title. So the entire system that's presented here has sort of been


explored in papers before like other papers have already done this, or we retrieve other documents.


And in this particular case, as you'll see, the documents are retrieved using inner product search


through a pre embedded through corpus, which is usually Wikipedia. So you'll see all of this. The


new thing about this paper just to make this clear is the way that the pre training works


for these systems. And we'll we're going to look at the entire architecture, but just you know,


such such that you're aware of what's really coming from here and what's gathered from what's


kind of conglomerated from what worked so far. So the improvements here are pretty stunning that


they achieve with this new pre training method, which is pretty cool considering that it's,


you know, the new thing is a pre training method. So we'll look at this, we'll look at the


architecture, the pre training method, the kind of hacks that you need to get it to work. And finally,


the results. As always, if you enjoy content like this, don't hesitate to share it out,


subscribe if you are not already. And with that, let's jump in. So the abstract says


that language model pre training has been shown to capture a surprising amount of world knowledge


crucial for NLP tasks such as question answering. And here again, we do we say question answering


is kind of the broad category of anywhere where you have to answer a textual question. So what


do they mean by world knowledge? What they mean by world knowledge, they mean something like


the question that we considered, what's the angle of an equilateral triangle, you can't from the


question itself, you can't answer the you can't answer the question is not like a little math


question where you just have to do the correct calculations or so on, or, or which one is the


longest words of the following words, it really is additional knowledge that you had to have learned


somewhere. So that's what we call world knowledge. And the fact that an equilateral triangle has 60


degree angles, you need to have picked that up from somewhere. Now, if you are GPT three, then


what you have done is you've taken this giant corpus, right? And you just did language modeling


on it, and that gives you GPT three. Now, that means since GPT three is so huge, that means that


all the world knowledge that is contained in this corpus is baked into the model and can be sort of


parsed out with good querying. So if you provide a correct query, you can sort of parse out what's


in the weights of the model. But it's very intransparently, it's very intransparent in


the weights of the models baked together with the language modeling. They're, they are criticizing,


not criticizing, but sort of arguing against this right here. They say, however, this knowledge is


stored implicitly in the parameters of a neural network, requiring ever larger networks to cover


more facts. To capture knowledge in a more modular and interpretable way, we augment language model


pre training with a latent knowledge retriever, which allows the model to retrieve and attend


over documents from a large corpus such as Wikipedia. And sorry, used during pre training,


fine tuning, and inference. For the first time, we show how to pre train such a knowledge retriever


in an unsupervised manner, using masked language modeling as a learning signal and back propagating


through a retrieval step that considers millions of documents. Okay, so that there's that there's


a lot of information here. So first of all, what they want to say is they want to say that in such


a corpus, there are two kinds of knowledge, right there is there is language. And there is this


world knowledge, okay. And they want to make this sort of separate. So they want to have a model


that can go to the corpus, retrieve documents, and then use those documents. So whereas previously,


the world knowledge has been joined with the language model, they want to sever this connection,


say we want a model where we can simply teach it to go look for information, we can teach it to go


search for things, and then the searched things will inform its answering of the question. Okay.


So that's what that's what these systems are trying to achieve. And we saw that before in the


we saw that before in the diagram. So they say we augment language model pre training with a latent


knowledge retriever, which allows the model to retrieve and attend over documents from a large


corpus. And also they use this masked language modeling as a pre training as a learning signal,


and back propagating through the retrieval step. Now, this is the interesting part right here. So


what you'll have is you'll have a question. And we can actually look at this diagram right here.


So the pre training is going to be masked language modeling, okay. Ultimately, what you want to do


is what we looked at before, ultimately, what you want to do is question answering. So this thing


right here, where the input is a query, and then you want to retrieve documents, and then you want


to join them. And let's actually draw this up. So you have a query. And you want to retrieve


documents, how do you do that, you train an embedding for the query, which is usually a,


you know, a BERT model. Like that's the fashionable thing to do. If you don't know what BERT is, I've


made a video about BERT. But basically, BERT can take a piece of text, and then it will output


a vector or multiple vectors for it. In this case, we just need one single vector for the entire


query. Okay. And then you have a bunch of documents in your corpus. So in your corpus right here,


you have z one, z two, and so on. What you want to do is you want to embed all of those. So you


want to have B of z one and B of z two. Okay, you want to embed all of those documents. And then


you want to compare these embeddings. And the you want to retrieve the document that's most


relevant for your question, right? If your question is about equilateral triangles, the


angle in them, then there's probably going to be like a Wikipedia article of triangles or


equilateral triangle specifically. So this corpus right here, we're going to consider this to be


Wikipedia. Now, ultimately, especially like a company like Google would like this to be the


entire internet. But for the these tasks for the academic tasks, this is often a limited


corpus. And then the data sets are also made such that they can often be answered with that limited


corpus. But in essence, this could be the entire internet. But for now, it's Wikipedia. So we want


to embed every single document in Wikipedia and then compare them using the inner product. So you


train your model to first of all, take this corpus, and then assign each member of the corpus a


vector. So this could be z one, this could be z two, this could be z three, and so on. And you want


to train it in such a way that if you have a query, then the query will be very close in inner


product space to the to the document that's relevant. So the query might be your question


about the angles. And the document right here might be the document about triangles. Okay, and


this document might be the document about, I don't know, England. And this one right here might be


the document about, I don't know, weightlifting, I have no idea. Like just random Wikipedia


documents. Okay, so you want them, you want them to be let's let's, you know, let's draw a little


dumbbell right here. So you want you want the other documents to be far apart from the query. So you


train two things, you train this model right here, which is the embedding of the corpus, okay. And


you train this model right here, which is the embedding of the query, these are two separate


models. And then you want the inner product between the two to be small, to be large, whenever


the document is relevant for answering the query. And you want them to be far apart whenever it is


not. Now, the question is, of course, how do you know, how do you know when it is relevant when it


is not because you have to have some training signal right here, right? You, you have to basically


know in advance which documents are relevant and you don't. So they start out with this masked


language model pre training, which we see up here. The masked language model pre training


does the following. So this is unsupervised, you take some string, right? Like this one,


the, and then you mask out a token, this comes straight from BERT, you mask out a token. And then


your goal is simply to reproduce that token. Okay. So if we were in BERT, you would forget about all


of this, you would simply try to predict what the mask token is. But here we say, well, we allow the


model to use additional context in order to read to fill in the blank. And you can see already how


this is going to help later. But okay, so we take this sentence, and we allow it to retrieve documents.


And maybe the document retrieved is this one right here, the pyramidion on top allows for


less material higher up the pyramid. And then you concatenate the input, sorry, the input is this


right here, with the mask token, as you can see here, you concatenate that with together with


this thing, which is this thing right here. And then you train a different model to take this as


an input and tell you what the mask token is. Now, if the retriever is good, then this model has a


pretty easy job. Because here you see at the top, something is at the top. And here you see the


pyramidion is on top, then it becomes fairly, fairly easy. Hey, the question again, of course,


is how do you teach the retriever to do well. And this is somewhat of a of a loop. So


informally, formally, the knowledge retriever right here is going to we're going to model


this distribution as a joint distribution. Sorry, this is I this is down here. Alright, so here,


the central formula is this, what you want is a model that takes in a question, or in pre training


a masked string, and it produces the answer or in pre training, this is going to be the


mask token. So this is going to be the question. And this is the answer or this is going to be


the masked string. And this is going to be the token that has been masked from the string. Okay.


Now, you're saying I can decompose this probability distribution into the following probability


distribution. And here, we take z as a latent variable, usually, but here z is the document,


okay. So what we want is a model that takes in your question, and a document


that is relevant for answering the question. And from that it produces the answer.


And in order to fill our probability distribution, we have to have this other model that takes in the


question and outputs a document. Okay, so this here is the retriever. And this here is going to


be the answer. And in order to make this the valid probability distribution, you need to


marginalize over all of the documents in your corpus. So now you can see how you train this,


you simply retrieve all of you train this model here to predict which documents are relevant to


a certain degree in a back propagatable way. So in a continuous fashion, assign each document a


probability to be relevant for answering this particular question. And then you take each of


the documents and answer the question, why from it, and you marginalize over all the documents


in your near data set. And then you get a final probability and all of this is completely


differentiable. The problem, of course, is that, especially in this paper here, there are like 13


million documents, so you won't be able to train very far according to that. So let's look at the


individual parts. First of all, this knowledge retriever, the knowledge retriever model is a


model that will take in a question and a document and tell you how likely that how relevant that


document is for this particular question. And this, as you can see, is defined as a probability


distribution, specifically here, this exponential distribution of F. And what is F? We've already


seen F is simply the inner product between the embedding and of the question and the document.


So that's the kind of thing we drew before, where the document is supposed to be have a high inner


product with the query that it is relevant to, and a low with all the other queries. Now, since they


cannot take all of the documents, what they do is simply they go in. So at the beginning, you know,


if let's say you're somewhere during training, right, and you have this index built up of all


of the documents, what you'll do is you'll go, you'll project your query into this space and you


retrieve the couple of documents that are closest to the query, okay. And, and you only use those.


So you sample a few documents, this is the same thing that we do in you know, contrastive pre


training and so on. It's just taken here to the the retrieval mode. So you don't marginalize over


all documents, because that would be computationally too hard, you simply marginalize over all the


documents that have a reasonably high inner product with the query that you're considering.


Why does that make sense? Because if you look at any other, like this one here, the inner product


is going to be almost zero. So the inner product with the query is going to be almost zero. So it


does not contribute at all to this probability right here, which also means that the gradient


is going to be fairly small. Now, even though the gradient is fairly small, it can still be that


you haven't learned something good yet. And actually, the document would be pretty relevant


for that query. And because you never use it to train, you will, you know, you will never ever


recover it, because you don't ever use it to train, there's no gradient flowing to it, and so on. So


you're sort of relying on this being sort of self organizing, like over time, it you know, these


these turn out to not really be relevant, because you've learned something stupid, and then your


query embedding, either would change and change the query, maybe during training, change the query


more towards the direction of the relevant documents, or the relevant documents themselves


would sort of shift and push each other around and so on. So you're kind of relying on effects


like this, but there's definitely a death spiral that can go on. So they make a they make a address


they address this right here. And yeah, they address this right here.


Here, the key computational challenge is that marginal probability py of x, which is this one


involves a summation over all documents in the knowledge corp z, we approximate this instead by


summing over the top k documents with the highest probability under this retrieval step. This is


reasonable if most documents have near zero probability. Even with this approximation,


we still need an efficient way to find the top k documents. Note that the ordering of documents is


the same as under the relevance score, okay, which is an inner product, thus, we can employ


maximum inner product search algorithms to find the approximate top k documents using a running


time storage space that scale sublinearly with the number of documents. So there are these


algorithms to do maximum inner product search, which you can use to find the top k documents.


To employ these algorithms, we must pre compute the embedding. So all the embedding of the


documents in the corpus, you must pre compute them for every z and construct an efficient


search index over these embeddings. So this now becomes very much like a search engine,


where you have to have your corpus and you have to build an index in order to find things fast


in there. Like it looks easy in our 2d examples, but to find maximum inner products in high


dimensional spaces actually very challenging task. However, this data structure will no longer be


consistent with p with this retrieval thing, right? Because as we train it, our index is going to be


old. So as we train it, our index might change. But if we only build it once, then that's of no


use. If the parameters of the embedding are later update, hence the search index goes stale after


every gradient update on theta. Our solution is to refresh the index by asynchronously re embedding


and re indexing all the documents every several hundred training steps. And they have a drawing


of this right here. So they have two different jobs. The trainer here trains updates itself


using the old index. So an index for a couple of 100 steps, then every couple of 100 steps,


it sends over its new weights. And the index builder builds a new index using these new weights,


right. And then the process starts again, these can run in parallel, as you can imagine. So as


soon as the index builder is done, it sends over the new index retrieves the new parameters and


starts again, building an index, because ideally, you want to rebuild the index after every single


step. But of course, that's going to waste too much time as well. So that was the retriever step.


The actual answer step is fairly, fairly easy. So once you've retrieved good documents,


right, now you don't need, as we said, you don't need all the documents where we are.


Right here, you don't need, we're not going to do this with all the documents anymore,


we'll simply retrieve the most relevant documents. Because that's going to approximate


this some fairly well, the answer here. That's pretty simple. That's going to be just a BERT


model that takes in z and x. Okay, so this is going to be another BERT model that's going to take in


the retrieved document and the question. And it's going to output y. How does that look?


In case of the masked language model, we've already seen it, right? You simply would input,


where is it? You simply would input the concatenation of the two with the mask, as you


can see right here, and then the output is going to classify at the classification task. So in the


case of BERT, you have your query right here as text, and then you have your document z right here,


and there somewhere would be a mask token, you would put BERT on top of that,


everything together. And then at the position of the mask token, you would do a classification


across all of your vocabulary, right and see which word is most likely. And that's how you


train that and evaluate that. If you are in the fine tuning mode, then you don't have masks


anymore. So what you would put is your query right here, and your document that you returned


here, and your document that you retrieved. And then you would you would simply output.


Now, here is an assumption. And the assumption is often baked into these data sets, you assume


that if you have the correct document, the span, the answer is somewhere in in the z document right


here. So y is somewhere in here. And what you would do is you would classify the start and the end of


the span of y, right, these correspond to these. So that's your training signal right there. As I


said, this is not always the case. But very often, especially in these data sets, it's the case that


it is a single contiguous span as the answer. Okay, so that's basically the architecture. As


I said, the architecture is using inner product to retrieve retrieving top whatever k documents,


in this case, I think it's about five, they retrieve five documents. For each document,


they run it through this bird in this joint way, like on the bottom. And then they classify the


output. And you can you can do it with a top one document, but you can marginalize over the top


documents for both pre training and for actually answering a question. There's lots of stuff you


can do. The important thing right here is that this thing is what the paper proposes a new it's


basically saying how do we do masked language modeling, pre training with a system like this?


Right? Okay. And the rest of the paper basically goes into more detail, like how do you how do you


join? How do you exactly what's the input right here? And we've already seen you just concatenate,


whatever you have, you concatenate your query and your documents and so on. So the important thing


is, it's two distinct like there are three models right here, right? Model one, model one is used to


take a document from the corpus and map it into a vector in these in this vector space right here.


Okay, that's model one. That is the model that you want to build this index for right every now and


then you take that model and build an index for your whole corpus. Then model two is the model


that takes a query, okay, a question to answer or a masked string, and also generates a vector


in this vector space right here. Okay, that is a different model than the model that embeds the


document and you don't build indices for that you continuously train it. And you just because you


only need to embed every query once and if you were to not build an index for model one, then you


would need to re embed the whole corpus for every training step. And then model three is something


yet completely different. Model three takes whatever documents you retrieved right here as z


along with the query as text. So not the vectors, but it takes the text of these documents and it


takes the text of the query and it produces an answer y which is either the mask token or the


answer span in the document. Okay, but this is again, this is a text model. This is nothing to


do with the vectors from before. Alright, so that was the architecture and the pre training. Now they


go into a few details. Namely, first detail is how do you actually what? How do you even see that


this does something sensible? Okay. And thereby, they analyze the gradient of this thing. So if


you look at the gradient, here's the gradient of the of py of x py is this is as we said the answer.


And this is the question. And this probability distribution has everything in it we've discussed


before, like retrieving the documents and then marginalizing over the retrieved documents,


and so on. Okay. So here, you can see that the gradient is first of all, it goes into the


direction of this inner product. Okay, this f here, that's, that's the inner product between


the embeddings of x and the relevant documents z are relevant according to their relevance.


Okay, so the gradient of the entire model is goes into the direction of the gradient of the inner


product. So that's already a good thing, right? Now, we can mask ourselves, we can ask ourselves,


when do we want the gradient of the entire model to be strongly correlated with the gradient of


this inner product? When not, that of course, depends on the document itself. And this is


quite this quantity are specifies how much that is. So if this turns out, like we want it, then we


can say, okay, the training of this model does something sensible. So what's this quantity are


the quantity are notably has this ratio right here, this ratio minus one. Now, what does it say


if if the top of the fraction is larger than the bottom of the fraction, then this is a positive


number, right? And if the bottom is larger than this is a negative number. Okay, so let's look


at the two elements, the top mean the top the ratio basically means that the difference here is


this z. So the ratio is larger than one, if the probability of the answer rises when you


have z in there, versus when you do not have z right here, there is no z. So what it basically


means is that the document helps if the document helps for class for answering the question x,


then that probability is larger than the bottom probability. If the document is irrelevant, then


that's one right and the entire thing becomes zero and therefore, no gradient. And if the document is


counterproductive, and that's often the case, actually, because this documents, they can


introduce noise, like noise is often counterproductive for the systems because you


have more input and then the the distribution of y will become more noisy and therefore flatter.


And this fraction would be lower than one. So this is going to be negative. So this quantity is


quantity is positive, the more relevant, the easier it is to answer the question with y, given the


document. And that's exactly what we want out of a system like this. So if you look at the gradient


of the system, it shows you that what we want to happen, namely, that the system is trained in such


a way that the relevant documents will help it is actually happening. Okay. So that's the left hand


side. And there's a little bit to be said about this thing right here, the probability, this is


proportional always to the probability that your retriever outputs this document. Okay, so the this


quantity r is going to be even larger if your retriever outputs that document frequently. So if


it is a helpful document, and the retriever outputs it very frequently for the given question, then


this quantity r is super large. And that's exactly what we want, right? Okay, so the next thing they


do is they they have to they have to sort of take care of the initialization here, because the problem


we've spoken of before is that if your retriever is bad, right, it will not retrieve the good documents.


And so it won't retrieve this z here very often. And then it really doesn't matter what this


quantity is right here, because this is going to be very low, like even if it hits upon a correct


document, and probably it doesn't because there's like 13 million documents and you retrieve five


or so. So very probably, you're not by chance going to hit the correct document. So you never


have a chance to get the document that would actually help you answering the question, then


you get a bad gradient, then you screw everything up even more, and so on. So the problem is that


if you just train this from scratch, you have a pretty bad learning signal. So what they do is


they have to take care of initialization. So they have to initialize things such that they are


already working fairly well before anything else happens. And this is sort of if I had to, you know,


criticize these systems a bit, it's that there are many hacks to getting them to work, right,


you have to really take care of initialization and so on, because they sort of build in a loop,


right, the better the retriever, the better the model that can answer the question and the better


the model that can answer the question, the better gradient you get for the retriever. But the


retriever only samples, so it doesn't even see all the documents. So how can it ever learn that a


given document is going to be relevant if never sees it and so on. So there's quite an interdependence


and you you only can do that with good initialization, as you know, is the case for a


lot of these language tasks. But here, even the pre training, so that's the point, even the masked


language model pre training, where they already have this, you know, retrieval step in there,


even that needs to be itself initialized at a good point. Otherwise, it doesn't help otherwise,


because you want to train the retriever such that the masked language model becomes easier.


And you have to take care of a bunch of stuff. So here they say at the beginning of training,


if the retriever does not have good embeddings, the retrieved documents will likely be unrelated


to x. This causes the knowledge augmented encoder to learn to ignore the retrieved documents. Okay,


so it basically just falls back to a model that does not have these other documents because none


of the retrieved documents are relevant. Once this occurs, the knowledge retriever does not


receive a meaningful gradient and cannot improve creating a vicious cycle to avoid this cold start


problem we warm start the embedding of the input and the doc. So these are these are models one


and two, right? I think this is what I called model one, this is what I called model two.


Using a simple training objective known as the inverse closed task, where given a sentence,


the model is trained to retrieve the document where that sentence came from, we refer to this


paper. So this paper, I believe is the the the orca paper. And just quickly for the knowledge


augmented encoder, we warm started with Bert returning. So this here, I think this is this


is model three. So this is model one. This here is model two, that's model three. So this paper


here, I believe that's the orca paper. The orca paper is very, very close to this paper, it also


has this retrieval step and so on. But it it took it said that it introduced this inverse closed


task as pre training for its own model. So you can see this paper right here as sort of an evolution


where they go from from orca and basically use that as an initialization for their own model.


Now, it's not exactly the same and so on. But this inverse closed task in that orca paper was


quite a central point. So what you want to do is you simply take a document from your corpus,


any document, and then you select a span, like this span right here. And then you make two things out


of that. First of all, the span is going to become your x, okay, and then the document right here,


the document but without the span, obviously, so the span you just leave empty, that's going to


become the thing to retrieve. And you simply now train a model your models. So this case,


this is model one, and this is model two, you train them such that the inner product between


between the two, so your embedding of x times your embedding of z is going to be large,


I guess they have a weight matrices in front of that, but it doesn't matter. So you can see that


you train the model to retrieve the document where a piece of text came from. Okay, and you


train these model in conjunction with each other, you simply make the inner products large. And you


can do negative sampling for this in order to contrast this with other documents where the text


isn't from. If you don't know what negative sampling is, I've done a bunch of papers,


most notably the word to vec paper where that was sort of introduced. So that's your pre pre


training task. And I'm going to just take a wild guess here. And I'm going to guess that in or in


this ICT pre training test, this here is started from the public BERT checkpoint or something like


this. So technically, this, you have the masked language model of models one and two would be the


pre pre pre training, and then this ICT would be the pre pre training. And then the masked language


modeling with the retriever based on I see built on ICT is going to be the pre training. And then


the question answering using that retriever is going to be the actual training. Okay.


So there's a lot of build up here. One thing to say is that, yeah, as you see here, so here is this


pre training on the left unsupervised, where you simply, again, the the thing the way you have to


think about it is what document do I have to retrieve to make the job of filling in the blank


here easier? Okay. And the hope is that that correlates well with the job of what document do


I have to retrieve to answering the question easier? Okay. What document do I have to retrieve


to make to make the job for the model that answers the question easier? I guess that's the way of


formulating it. Alright, so the next few things you have to do to get it to work is prohibiting


trivial retrievals. They say if the pre training corpus and the knowledge corpus are the same,


which I guess they sometimes are because, you know, it pays off to do the pre training on the


same corpus as your knowledge corpus if it is large enough, a trivial retrieval candidate z that is too


informative, right? There exists a trivial retrieval candidate. If the mask sentence comes from


document z, the knowledge augmented encoder can trivially predict y by looking at the unmasked


version of it. Yes, of course. Like if you do this masked language modeling and you take your


sentence from that corpus, then the retriever can simply go look for that document. And then it


becomes very, very easy to fill in the blank, right? Because you just do this pattern matching.


And that's of no use, because what you want to teach the model essentially is to kind of look at


the semantics of a document. So you simply prohibit that particular thing. So this is during


pre training, this is for your masked language modeling, pre training, what we call here realm


pre training. During that, you simply prohibit. For this reason, we exclude this trivial candidate


during pre training. So that's one thing you have to do. And I feel here is, you know, where the


specifics of your task and your data set come in. Because, you know, on the internet, many things


are copied, and sort of copied and translated and so on. So if you were to do this, not in Wikipedia,


but in a more unstructured way, this would be one of the pain points, I guess, because


imagine, you know, there is just a website that translates all the other websites to French,


and then your model can simply learn to translate from French and always retrieve the French


document and fill in the blank using that it will learn nothing about the word like it will not


require acquire any retrieval of along semantics of world knowledge, it will simply learn to


translate to French and so on. So I think that this is rather more crucial than this simple


one paragraph appears to, to have it. Then they also introduced this null document,


along with the things they retrieve. So if they retrieve, maybe not five, but eight, I think they


retrieve eight in the experiments if they retrieve. So they retrieve seven documents, the seven


closest ones in inner product space plus a null document, such that the model has the opportunity


to ignore all the documents, right. So it can basically just go to the null document, assign a


large weight to that and just answer the question outright. So if the answer is already contained in


the question itself, it can just, you know, point to that it doesn't need the an additional


document to answer the question. So they leave room for this possibility right here. Now,


this would also be a good metric to assess how much the model makes use of the other documents.


And I think they have this further down. And then the last thing here is the salient span masking.


So when you do masked language model pre training, what you'll do is simply you'll drop out, not even


words, but word pieces, right. So, so here, let's say this, you have this span of text, what you do


is you just drop out like at random words, or, as I said, even worse, if this is BERT or something,


you have word pieces. So you maybe just drop out this CUS right here, and the low. Now, people have


observed that this is now pretty easy for the model. And most notably, it doesn't require a lot


of world knowledge, it doesn't require even a lot of attention to the other parts of the sentence,


which is what you would like to induce with this pre training, all you basically need to do is


you need to say, oh, there is something and then Cal. And maybe you look at the words around it,


and you can pretty easily deduce that it's locale. Also, to fo on, you can pretty easily


deduce that to focus. So this kind of pre training doesn't really mean the model learns some long


range dependencies or understands language pretty well. So people have been upping the kind of


smartness with which they drop out things. So the most obvious thing is to drop out entire words,


even though you know, BERT works in word pieces, you can simply always enforce that entire words


are dropped out now it's a bit harder. Then what people do is like salient span drop outs. And


that's what they do right here. So what you want to do is you want to drop out things that are sort


of kind of little snippets that are belong together. So for example, if I drop here, local


context, if I drop this out, right, then I need, you know, some masculine spans Rick only require


what? Right? And that requires much more world knowledge to answer that question requires much


more long range dependency resolution in my language model and so on. In order to see that


there is world knowledge. And this is exactly what you want to induce here, right? You want to induce


your model to learn, learn more global knowledge, more world knowledge, more semantics of the


language. And you can relate this to sort of pre training or data augmentation, I'd say, in image


in image in vision. For example, there you have the random cropping. So you only crop out part of


the picture and then you crop out another part, maybe here. And then you ask the model, does this


come from the same or from different images, these two parts, and the more you crop sort of the more


the model has to cannot rely on just single pixels somewhere, but actually has to understand


image scenes and so on what direction is up and whatnot. So we see qualitative difference between


pre training methods, and augmentation methods for images, it only makes sense that we see a


qualitative difference and different in differently induced inductive priors in text if we do this.


So what they do is they say, since we want to induce this kind of thing, we will not only drop


out entire words, we actually drop out entire salient spans, such as right, United Kingdom or


July 1969, we use a bird based tagger to identify named entities and a regular expression to


identify dates. We select and mask one of these salient spans within a sentence for the mask


language modeling task. We show that this significantly outperforms other masking strategies


in section 4.5. Now, while I agree with the notion of salient span masking, I have big troubles with


the way they do it here. And I think this is where you kind of start to overfit on the particular


data set. So I guess they looked at the data set and, you know, you kind of as a developer, you


kind of look at your, you know, just kind of questions are there and they saw often it's,


you know, questions about entities, question about dates, and so on. So, you know, we can just


pre train with those things in mind. And that yeah, that's where it gets a bit wonky and really


specific to your task, really specific to your data sets, and so on, to do that. So this is already


baking in a bit of knowledge, or a lot of knowledge, I would argue about the task itself. And


we're going to see that this is actually fairly important in the results, the salient span masking.


And yeah, this, it's sort of, I get it, you get better numbers with it. But also it's kind of


dirty, and very, very specified to the task, I won't actually see. And I don't know if people


have actually have done this. But the way I would do it in a kind of more principled way is if you


have a piece of text, what you do is you start by masking one word, okay? Like, I'm asked spans here.


And then I would ask my own model, right, my own half trained model, which, which, if I want to


predict this one, right, if I want to do mask language model with this one, I can use one of


these saliency methods to ask which other words are most relevant to predicting this one, okay? And


it will probably be say, okay, salient is really important, right? Because if I know that there is


salient in front of it, I can predict that there spans really easily. And then I can say, well,


okay, so I'll mask salient as well. Now I have masked these two. And I do that up to some point


I have masked these two. And I do that up to some threshold, right? So the saliency in my mind


should come directly from the model you're training. By that you're basically saying that,


you know, model, you've sort of learned your local dependencies. Now I want you to go beyond that.


You're, you're basically really mean to the model, you, you forbid it from using everything it has


learned so far, to make the task more challenging and more challenging over time. I think this is


kind of a built in curriculum learning. And that's how what I would see if you, if this is already


done, maybe someone's already done it, just let me know in the comments. This already exists kind of


expanding the masks by assessing the models own saliency. Alright, so let's jump into the results


and the results, as you've already maybe seen in the abstract are pretty, pretty good. So on these


open domain questioning data sets, they outperform all the previous state of the art not only by a


little but by significant margins, as you can see here. And they do it in when both the pre training


corpus is the same as the knowledge corpus and when the pre training corpus is actually a different


one. And that tends to work actually even better in in two of the three tasks. So fairly cool. Also,


not more parameters than you know, previous models, especially not this, this t five. So this t five


here is an example of just you know, where everything is baked into the language model.


Whereas I believe these models right here, they have a retrievers along with it. Yeah, you can see


here they all have retrievers along with it. But their pre training objective and their architecture


sometimes is different. I believe you can also see the fact here that orca has the same amount


of parameters. It's very close to the model right here. It's just that the pre training here is


different. And you also see right here, they do some ablations, where they say, okay, how important


are the different parts right here. So you can see on the development set, you get what your 38.2


exact match score. If you only train the retriever, but you reset the encoder before. So that's the


thing that actually answers the question. If you reset that before fine tuning, you drop a little


bit. If you reset the retriever, you drop actually you drop more. But still, it's I would say it's


fairly competitive, as you can see, now this is probably the test set. But still, it's fairly,


fairly competitive right here with the with the sorry, the previous state of the art. Oh, yeah,


here, here is the baseline, it's 31.3. Now, interestingly, as you can see right here,


if you have uniform masks or random span masks, which is the two types that I have masking that


I discussed where either you drop out just word pieces or, you know, entire words or entire spans,


so you just you just take that idea further, you say, oh, I'm asking an entire span, but


nothing with their saliency. So no, no reg X's for dates, no entity taggers, and so on, you,


you drop quite a bit, especially with the uniform masks, you see here, you drop quite a bit. Now with


the random random span masks, you also you drop, you drop for the random spans, and then you drop


again for the uniform masks. So this seems to be pretty, pretty important. So never forget,


when you see things like this, that there are these engineering choices that can make


as big a difference as the the actual idea in the paper itself. Okay, so you can see this


is pretty, the improvement is it's like three points from uniform masks to random span masks,


and then three points again, from the random span masks to their realm pre training. And the actual


improvement with the uniform masks over the baseline right here is not as high. Now the


baseline, you know, uses a different thing, it uses this ICT as pre training, but still, I haven't


seen the saliency masking. Maybe I've seen it, maybe it's somewhere else, but I haven't seen it.


Okay, they also have an interesting thing right here. Oh, they also have an interesting plot in


the appendix, where they show the number the performance of the different masking styles,


with respect to this retrieval utility and the retrieval utility compares this, these two things


that we've looked at. So it compares how good is document z and answering the question why versus


this null document. So the null document is basically just answer the why, right? So if


let's let's play devil's advocate and say that all of this retrieval stuff, it's just bollocks,


right? You know, the knowledge is still baked into the language model, they were critical


that this helps, and so on, then this would also always be zero. You can pretty easily or you can


pretty easily see that this would be zero, right, there will be no improvement having the document


versus not having the document having the null document. So if this is high, that means these


retrieved documents are actually relevant. So you can see that if you do random uniform masking,


then it's it's okay, it gets above zero. All right, if you do random span masking,


it gets even higher. And if you do salient span masking, it gets very high. So again,


you see here the difference between the salient masking and the others is, you know, I would say


higher than the difference between not having the document at all and doing the random uniform


masking in pre training. So again, you know, something to think about. At last, they have


one example right here where they can show actually helps this is just a concrete example.


So the question here is an equilateral triangle is easily constructed using a straight edge and a


compass because three is a and then blank prime. So this is the masked word right here. If they


just ask the model what they should feel what it should fill in the probability and Fermat is the


correct answer is super duper low. Okay, then if they give it the correct document, they just


they just search out the correct document, which is here the conditional probability with this


document to 57 is a Fermat prime does a regular polygon with 257 sides is constructible with


compass. So you can see that it has it has some overlap like the constructible with compass,


okay, the constructible with compass, it's not an exact overlap. So it's debatable whether a


classic search engine would find this probably but not and then the a something prime a something


prime, they are here. So given this document, you can see how a model could easily classify


Fermat as the correct answer. And in fact, the probability is, I guess it's not 1.0. But it's


around that 1.0. So if you give the model the you know, model three, your if you give it the relevant


document, it immediately knows what the answer is. And if you give the if you do this whole


retrieval step in between, so this is marginal probability marginalizing over the top eight


retrieved documents, so now they don't tell it what the correct answer is, but they actually


let it do its whole retrieval thing and marginalize over the top documents, then it still assigns a


very high probability. And I'm going to guess that's the top probability for all of the words,


but you see, there is a considerable decline. So it's not like it's not like it's always super


sure. And I think there is quite a bit of improvement still to be to be done right here,


because as a human, if I go look for an answer for this question, and I find even if I consider


the top eight documents, I don't think they would confuse me to the point where I'd say that Fermat


is only 12% likely, even though it might be more likely than any other word, I would assign it


probably a much higher probability. So I think there's there's a bit of improvement still to be


made right here. And I'm looking forward to what people can come up with. Alright, I hope you


enjoyed this video. I know it's been a bit of a long rant, but I wanted to make sure the individual


parts are clear. Let me know what you think of it of the model itself. And I wish you a good one. Bye


Gradient Surgery for Multi-Task Learning

Hi there, today we're looking at gradient surgery for a multitask learning by Tianhe


Yu, Saurabh Kumar, Abhishek Gupta, Sergei Levine, Carol Hausmann and Chelsea Finn.


So in this paper the concern is a thing called multitask learning. Now what is


multitask learning? So this is has some very subtle distinctions from other


things that's I think why it's important to look at it a bit. So let's say you


have multiple tasks, a learning problem in multiple tasks, this seems easy enough


right? So what we mean is that we have the same input but then we want to


perform two different tasks, so task one and task two, sorry two. So it could be


something like task one, if the input is a food, right, a food object. The task


one could it be, is it a fruit? Right, task two could be how many calories,


calories does it have? Right, the input is this food item here and you want


to know both things, is it a fruit and how many calories does it have? And ideally,


so what you could do is you could train two separate machine learning


classifiers, right? Classifier one simply does the is it a fruit thing, task two


simply does how many calories does it have? Let's say this is, let's actually


say this is a food picture, right, since Instagram is full of food pictures, we


have lots of training data, right, at least unsupervised, people usually label


it and we could train two different things but it would be nice since


they're both kind of dealing with the same input, so they, not kind of, they


actually deal with the same input distribution, it would be nice if we


could kind of share a representation, right, so maybe we have some neural


network here with many layers and then we have at the end we take this hidden


representation here and we just have maybe one or two fully connected layers


for each individual task but our goal would be that the hidden representation


here is shared, so shared representation. And why could that help? Because we might


have, maybe we have lots of training data for the how


many calories does it have, right, but we don't have that much training data for


is it a fruit, so lots of training data here, big database, but only like a


handful of data points for the second task, or we we might just not have much


training data at all for both tasks and we just might benefit from training this


shared representation, you might have already seen this with something like


BERT, so in BERT's case the input is text, right, and then you do something


different, that's why BERT is different than multitask learning, what you do in


BERT is you do first you do this masked language model pre-training, so that's


step one, and then in step two you take this and then you you fine-tune it on a


number of tasks, right, so here question answering, sentiment detection, entailment


and so on, this is different, this is called pre-training and fine-tuning,


in multitask learning we actually want to train on different tasks at the same


time, maybe they have different data, right, and we simply want to create this


shared representation and we hope that by combining these tasks we might learn


them better than if we were to learn each task individually, alright, so this


paper says there are, there's a big problem with things like this and they


illustrate this in this example right here, so let's say you have a


multitask objective and the learning landscape looks like this, so the


objective for task one is the following, so this you have to have to imagine this


is maybe a neural network with just two weights, right, here is weight one and


here is weight two, and this is what the optimization landscape looks like


for task one, if you're not used to this kind of depiction the light parts up


here and here are high values for the loss function and the darker parts are


low values for the loss function, so you want to get to the darker parts, now


usually we discuss things like this in terms of optimization, so for example we


would talk about SGD and we would talk about the fact that oh if we have two


large of a, so if you're here where does the gradient point, the gradient points


towards the direction of steepest increase, so here, so the negative


gradient would point down, if we have a SGD maybe would go here and then we take


another gradient step we would go here, right, now we've gone too far, right, so


the gradient now points this direction, so we go here and then we just continue


this, right, so this is a problem with SGD and what we can do is we can


decrease the step size for example and then we converge in this, or we can use


something like Adam that adjusts the gradient to the variance of the


gradient landscape, things like this, right, so these are these are problems in


optimization, but what happens when you have a multitask objective is that for


just task one the optimization landscape would look like this, right, if you were


just to train your neural network, if you were to just train this part and we just


look here is like theta one and here is theta two, these are the two weights we


care about right now, everything else let's say is fixed, task one looks like


this, but for task two, because it's a different task, right, we need to set the


weights differently to get our desired output, it looks different, so our loss


function is going to be a combination, so our loss function is going to be the loss


of task, loss function for a given sample, it's going to be the loss on task one of


that sample plus the loss of task two on that sample, so that's going to be the


combination, the combination you see on the right, so this plus this equals this


right here, so you can see in task one it almost, let's say it didn't matter whether


we were here or here, both had a relatively low loss value, right, but you


can see in task two this point here is not an optimum, well this point or


maybe these are slight, these are somewhat close together, so if you add


them you can see that now this thing here still has a low value but not as


low as this is much darker, right, so the landscape for both tasks together looks


differently from the landscape of either task alone, so your goal is to find this


optimal point, an optimal point here that works for both tasks, now the paper


identifies many, sorry, sorry, the paper identifies problems with this


multitask learning and they say the problem is that you can have what are


called conflicting gradients, so if you look, if you look at, if you look at where


the gradients point in the different, in the different tasks, so if we go by task


two, sorry, let me put that in again, and we care about the point right here that


they care about, right, and they use Adam in this case and their starting point


is right here and they've come this way so far, so we're going to draw this in


here and draw this in right here and we'll stop a little bit before that


valley, right, so let's analyze the gradient, the gradient in task one


actually points in this direction, you see, down the valley, right, and it's


pretty big because it's pretty steep, right, you can see the curves here


getting closer and closer together, that means the gradient is pretty steep and


it points in that direction, whereas for task two, if you're here, right, the


gradient actually points in this direction, but not as steep, right, because


here the lines are pretty far apart still, so that means it's relatively


flat, this is what the paper calls conflicting gradients and they're drawn


in here, I'm going to draw them just a little bit larger, so these two gradients,


first of all, they have different magnitude, you see that the magnitude of


this is much larger than the magnitude of this and also their angle between


them is large, that means conflicting, that they're more than 90


degrees apart from each other and this results, if you calculate the resulting


gradient, of course, this results in a gradient like this, right, so our


algorithm wouldn't actually go down this valley, it would go up the hill again


because you have differently sized gradients from the different tasks


that go in different directions. Now the important point, I was wondering for a


long time, what's the difference between this and simply saying, look, any


data set, right, your loss on any data set D is just the sum of the loss of your


individual data points Xi, because it is the same case that you can have


different data points and the gradients that you get, right, so that would


result, if you've never done optimization, I'm sorry I'm going a bit fast, that


would result in the gradient with respect to your weights of your loss


over the entire data set is of course approximated by the 1 over n in your


mini batch, so by the gradients in your mini batch, right, so let's call this the


loss of Xi, this is completely illegible, but what I'm saying is that your total


gradient is the average of your individual data points and these might


be conflicting as well, right, you could have that one points in this


direction and the other one points in that direction and we've done this just


and things like Adam and SGD actually are able to handle that


just fine because we do this average operation. I think what is different here


is in multitask learning is that the multitask, the task distribution is not


like stochastically IID let's say, so in this case you can always count on


that the expectation will average out this noise, so this noise if you go


in expectation, right, if you do mini batches and aggregate over the whole


data set then that will kind of even out because for the different data points


okay one gradient might be larger, one might be smaller, but there is no


systematic error or there's no systematic bias that comes from the


different data points. Here you have, as we said, one task might be much harder


than the other task, right, or you might have much more data or the loss function


is just larger like magnitude-wise, so you can have any number of systematic


biases that the different tasks have with each other and therefore the


conflicting gradients seem to be a problem. So this paper does a good job of


analyzing the situation of conflicting gradients and what I find particularly


interesting is that they, first of all, they propose an algorithm to deal with


these conflicting gradients, so they say whenever two gradients are conflicting,


right, what we would do is we would project them on the normal plane of each


other, right, so for example here in the in step B we take the gradient of task I


and we project it onto the normal plane of gradient from the task J, right, and if


we do this and they have a whole algorithm where it's a general, so if we


do this for multiple tasks, so basically we get a mini batch of tasks, right, so


they generalize this to that you have a bunch of tasks, we get the different


gradients and these can be stochastic because we can we can do this with


stochastic datasets, we go through the batch and if the gradients are


conflicting we simply project the gradients onto each other and


that will result now in a set of non-conflicting gradients. You might be a


bit appalled by this, I was at first when I saw this, but they actually do, as I


said, a good job of analyzing this, so they have two theorems here which I find


interesting, so theorem one is assume these are convex and differentiable, so


somewhat standard assumptions in optimization, they say then the PCGrad


update rule with a step size smaller than 1 over L, L is the Lipschitz constant,


will converge either to a location where the cosine is exactly negative 1 between


two gradients, you can, that never happens except if you construct it, or the


optimal value, right, so this is basically a consistency theorem saying that this


algorithm will still converge to the optimum value, this here is, this is the


loss, so this loss is the sum of loss 1 and loss 2, right, of these two tasks, so


for two tasks they prove that the algorithm will still go to the correct


point if you run it long enough, doesn't say anything about the speed though, this


is where theorem two comes in, theorem two says suppose L is differentiable and


the gradient of L is Lipschitz continuous, this again same assumptions


except no longer need convexity, let theta MT, which is the multitask gradient,


and theta, sorry not the gradient, the parameters, theta PCGrad be the


parameters after applying one update to theta with G and PCGrad modified G, so


this MT is the, that would be kind of the original algorithm without their method


and this here would be with their method,


moreover assume a bunch of things which we'll go into soon, then the loss


function of the PCGrad theta is smaller or equal than the loss function of the


MT of the original, so what does it mean? It means that if you're in your


optimization landscape and you're somewhere here, right, and your optimum is


somewhere here, and your loss function is kind of how far away are you from this


from this optimum, it means that as long as these conditions are given, if you do


your update without their method, which would be, so here would be theta MT,


or with their method theta PCGrad, then the loss function that you get from


their method will be smaller than the loss function that you get without their


method, right, so this is a theorem, they prove it, and for this to be the case


they need these three things, so let's go from the back, the third one is a


condition on the loss function, sorry on the step size, and you can say


okay the step size needs to be large enough, you can set the step size, this


here, what is this? This here needs to be a, this is a condition on the, on this


epsilon, so what's this thing? It is a curvature bounding measure, and that is


compared to little l, and little l here is this thing, it is a constant that must


be smaller than h, and h is up here, is the curvature, so it depends on the


curvature, right, it depends on the curvature fulfilling some condition, they


state down here the curvature of the multitask gradient should be large, yeah,


and the first condition we've already seen is that the cosine of the angles


needs to be smaller than negative something that depends on the gradients,


and this here turns out actually to be the magnitudes of the gradients, so this


this first, this here, we can, we can neglect, that's a step size condition,


this here means the gradients should be conflicting, and this here means that


there should be sufficient curvature in the loss function, this is exactly what


we saw at the beginning in this, in this thing here, so there was a sufficient


curvature, because in one direction the gradient was very steep, and in the other


direction it wasn't, which basically means there is a change of steepness,


right, there is a change of steepness in one direction versus the other direction,


and also the two gradients were conflicting, which we saw right here, if


this is the case, then this algorithm will bring you faster to the optimum


than the, the normal algorithm, but only if this is given, and notably this can


change step to step, they actually call this the, I think the holy trifecta, evil


trifecta, something, they have a name for it, but I'm going to read you the


conditions that, how they describe it, the conditions are first, the angle


between the task gradients is not too small, i.e. the two tasks need to conflict


sufficiently, second, the difference in magnitude needs to be sufficiently large,


third, the curvature of the multitask gradient should be large, and fourth, the


learning rate should be big enough such that large curvature would lead to


overestimation of performance improvement on the dominating task and


underestimation of performance degradation on the dominated task. So


here you see a little subtle theme. I said before that this condition here was


negligible because you can set the task size. In actuality, this, you can, so I'm


not meaning to rag on this, but what does it mean, the learning rate should be big


enough such that blah blah blah, and what comes here seems to be negative, right,


such that the large curvature would lead to overestimation, which basically means


this method, this thing here, counts if the step size is large. So that means if


I were to play devil's advocate, if I have a problem like this, I could either,


right, I could either use their method PCGrad or I could just decrease my


learning rate and use the classic algorithm, because if I just decrease my


learning rate relative to the curvature, then this theorem would no longer hold


and it would no longer be the case that their algorithm gives me a faster


convergence. So there's two ways of looking at these things. It's like,


yes, in these conditions this algorithm is better, but it is better because


someone has set the learning rate too high and this algorithm kind of fixes


that. Now the upside to this is of course that usually you don't want to


kind of set your learning rate in accordance with the curvature of the,


with the curvature of the problem and so on. You don't know the curvature most of


the time, so you just set some learning rate and their algorithm appears to be


working also when this learning rate is smaller. It's just not guaranteed to


outperform the classic algorithm. But I just found, find this interesting in


terms of how you read a paper, right. If you read a paper you come across


something like this. These conditions you can always see them as here is what


needs to happen for us to succeed or here is what needs to happen for the


others to fail and therefore we're the only ones that succeed in this regime.


Though, yeah, as I said it's a cool algorithm but I found that to be funny.


Alright, so they test this on multitask which these MT10 and MT50 benchmarks are


these robotic manipulations. So multitask doesn't only mean like supervised


learning. In this case actually multitask reinforcement learning. So here you have


everything, you have mini batches, you have episodes and you have, you have


multiple tasks. So this is everything together. Very cool. And you, in their


actual implementation they say what they do is they have these multiple tasks so


they have the agent and they first select a task. So for example here pull


this, right. Then they generate an episode by interacting with the


environment forth and back, forth and back. Then they put that episode into a


replay buffer. Then they maybe select another task and so on. So until they


have a bunch of data in the replay buffer from different tasks. Then they


sample episodes from different tasks, right. From task one, task two and so on.


And that will become a mini batch in the learning procedure. So pretty intricate


thing but of course you, the hope is that you can learn kind of a shared


representation that you can then perform all of these tasks faster than if you


were to learn them each independently. So the MT10 and MT50 come from this and I


think they also have goal condition pushing where the task is simply to push


something to a what they call goal conditioned. And the cool thing about


this is it's not only 50 tasks but you can produce an infinity of tasks because


you can always specify a new location where you should push something to, right.


So that's fairly cool and oh yeah the curves. So you see that if you do


something like soft actor critic or multi head soft actor critic. So this


multi head soft actor critic is probably the closest to what I defined in, to what


I defined at the beginning where you have this shared representation and then


and then the individual heads and you can see that severely underperforms


against the SAC plus PCGrad plus their method that seems to outperform fairly


consistently even against learning the tasks independently. So it learns much


faster than if you were to learn these tasks just independently from each other.


Which is pretty cool, right. So I think that's pretty cool. All right so they do


actually interesting investigations. First of all they research okay in during


these learning runs how, what is the curvature here and the curvature of the


loss function they measure like this. So basically all this is is a consequence


of a Taylor approximation. So if you have like f of x you can you can write this as


f of some x0 plus the gradient of f that plus the gradient of f times x here


sorry at x0 times x in this direction. And then if you subtract so this is a


first-order approximation to this to the function on the right then if you bring


this over here you or if you sorry if you subtract the two sides from each


other then you can see there's the difference between the actual function


and the first-order approximation of the function that must be or that is most


likely the curvature. Now it is not it is like every higher order term but the


assumption is that the dominant higher order term will be the curvature. Right


so this is this would be this except they don't they they do it not doing the


x and x0 they do it at theta t and theta t plus 1. So you can see this is the


first-order approximation and this is the actual function value after they do


a step and the resulting thing will be the curvature or dominated by the


curvature. So they analyze this over the course of learning and they see that it


actually increases as you as you go on and just I'm not a big fan of like just


large numbers but the numbers seem to be large right just compared to what you


can handle with a computer the numbers seem to be large and they seem to be


getting larger in order of magnitude steps across training iterations. So I'm


going to believe them that this curvature is given I would have liked to


have it seen compared to just a single task instead of a multitask it's


instead of you know comparing these things which is useless because they


reach different losses right so it's pretty useless to to compare their


curvature across the number of iterations. What I would have liked to


see is a comparison multitask versus single task and to show me that in


single task learning this curvature doesn't happen. Here you have the


percentage of update steps where conditions A and B are held. You remember


condition A was the condition on the conflicting angle, condition B was the


condition that the curvature is large enough and you can see that as you go on


with learning these dotted and dashed lines the conditions hold almost


entirely at the beginning of learning but then still hold by in a big time of


the steps so here is like about half the steps still at the end of training


these conditions hold. So it is fairly fairly good evidence that often the


problems that they say are real are really there and then therefore their


algorithm helps right so here is the average per task average return and


interestingly they say in the text look this task here seems to be easier right


and the task 2 which is the dotted line seems to be harder so SAC the baseline


algorithm never really manages to learn task 2 whereas this PC grad manages


after a while to learn it and at that point something happens over here which


I'm not super sure yeah that's what they say in the text but I have to squint a


lot to see that exactly at that position something happens. Suffice to say that the


PC grad is able to learn the task that SAC isn't able to learn because probably


task 1 is completely dominating the gradient at that point right. Alright so


[Drama] Who invented Contrast Sets?

I love me some good Twitter drama, look at this, this is awesome. So after this


[Drama] Who invented Contrast Sets?

contrast set paper appeared and I've done a video on that, the author of it


[Drama] Who invented Contrast Sets?

tweeted it out with one of the these long Twitter threads with screenshots


[Drama] Who invented Contrast Sets?

and all this seems to be the new marketing tool of academics and as you


[Drama] Who invented Contrast Sets?

know I'm not a fan of this paper I think that the number that comes out of such a


[Drama] Who invented Contrast Sets?

contrast set is very either useless or counterproductive and you can see my


[Drama] Who invented Contrast Sets?

video on that. In any case there was another researcher, Zachary Lipton, who


[Drama] Who invented Contrast Sets?

felt like he needed to jump in here saying before the media blitz and


[Drama] Who invented Contrast Sets?

retweet party gets out of control this idea exists, has been published, it has a


[Drama] Who invented Contrast Sets?

name and a clear justification is called counterfactually augmented data.


[Drama] Who invented Contrast Sets?

This is amazing, look at that, and here's the published paper of course and if we


[Drama] Who invented Contrast Sets?

look at the published paper this is it right here of course Zach Lipton is an


[Drama] Who invented Contrast Sets?

author on that paper and so let's just just read the abstract I haven't read


[Drama] Who invented Contrast Sets?

the paper but let's just read the abstract. So I have it classically I


[Drama] Who invented Contrast Sets?

have it here my nifty thing here so we can analyze it. So this paper if you read


[Drama] Who invented Contrast Sets?

the abstract it does sound similar right despite alarm over the reliance of


[Drama] Who invented Contrast Sets?

the learning system blah blah blah blah spurious correlation so it talks about


[Drama] Who invented Contrast Sets?

the same problems. Now what do they say? Given the documents and their initial


[Drama] Who invented Contrast Sets?

labels we task humans with revising each document so that it accords with a


[Drama] Who invented Contrast Sets?

counterfactual target label, retains internal coherence, and avoids


[Drama] Who invented Contrast Sets?

unnecessary changes. Right so this sounds very similar to what these contrast sets


[Drama] Who invented Contrast Sets?

do so the counterfactual target label would be the necessary of the


[Drama] Who invented Contrast Sets?

contrast set to change the label, retains internal coherence which is the in the


[Drama] Who invented Contrast Sets?

contrast set this simply given by it supposed to conform to the intent of the


[Drama] Who invented Contrast Sets?

data set makers which the intent probably includes internal coherence and


[Drama] Who invented Contrast Sets?

it avoids unnecessary changes that conforms to the contrast set only


[Drama] Who invented Contrast Sets?

searching in the local environment of a test set sample so you see that the


[Drama] Who invented Contrast Sets?

definition of these is pretty similar then we go on and say they say classifiers


[Drama] Who invented Contrast Sets?

trained on original data fail on their counterfactual revised counterparts and


[Drama] Who invented Contrast Sets?

vice versa this experiment was also done by the contrast that paper and then they


[Drama] Who invented Contrast Sets?

say classifiers trained on combined data sets perform remarkably well just shy of


[Drama] Who invented Contrast Sets?

those specialized in either domain so immediately we see some differences as


[Drama] Who invented Contrast Sets?

well right the main difference I see is they say we task humans and then they


[Drama] Who invented Contrast Sets?

train on the they train on the counterfactual revised counterparts


[Drama] Who invented Contrast Sets?

which probably means they use some mechanical Turks here when they say


[Drama] Who invented Contrast Sets?

humans because if you want to create a training data set you need lots of data


[Drama] Who invented Contrast Sets?

so they probably take a data set and run its training data set again through


[Drama] Who invented Contrast Sets?

something like mechanical Turk to get annotations this is exactly what the


[Drama] Who invented Contrast Sets?

people of the of the contrast sets claim is wrong with the current pipeline and


[Drama] Who invented Contrast Sets?

they so here we have this this thing counterfactually augmented stuff so the


[Drama] Who invented Contrast Sets?

contrast sets what they say is we actually need the experts to do this that


[Drama] Who invented Contrast Sets?

this the these humans are exactly the wrong people to make the data sets so it


[Drama] Who invented Contrast Sets?

has the CFA has some elements correctly the same namely how they construct these


[Drama] Who invented Contrast Sets?

labels but who construct the labels and for what reason so here it's experts and


[Drama] Who invented Contrast Sets?

for what reason it's for testing it's they say the experts that make the data


[Drama] Who invented Contrast Sets?

set should provide an additional contrast test set so this is I mean if


[Drama] Who invented Contrast Sets?

if this is just my opinion if this is the same idea of course it's very


[Drama] Who invented Contrast Sets?

similar but if this counts as the same idea then 95% of all research counts as


[Drama] Who invented Contrast Sets?

the same idea as something that you're in Schmidhuber has done in the 1990s


[Drama] Who invented Contrast Sets?

which of course you can Schmidhuber will eloquently argue exactly he invented