give a piece of conditioning text. And that together will this this is newer, this is


NÜWA: Visual Synthesis Pre-training for Neural visUal World creAtion (ML Research Paper Explained)

the baseline right here will, as you can see, fill in the missing pixels in order to also


match up with the text, because it's been trained on text to image data sets. Yeah,


lastly, this video manipulation, which was one of the sort of appraisals of this paper


right here, you can see the raw video on top, the first row is the diver swimming to the


surface that's given to the model. So the model is asked to manipulate the video in


that way, that we're swimming to the bottom, or the diver is flying to the sky, which surprisingly


the model can do as well. Again, I think I think the model simply gets the first frame,


and then needs to continue the video, I don't think the rest of the video has given us conditioning


information, but I might be wrong. So in if I'm right, it would not necessarily be video


manipulation, but more kind of like video completion conditioned on text, but still


is pretty cool. Right. So yeah, they have a by the way, they have a big appendix. They


also compare like different local attention mechanisms. They have much more output right


here. Yeah, some sometimes it's it's very funny. But I hope the code is out soon or


is already out and I just haven't hadn't found it. As a conclusion, they say they present


new unified pre trained model that can generate new or manipulate existing images and videos


for eight visual synthesis tasks. Again, caveat here is that only very few only like two or


three of those are actually zero shot maybe are resulting from the pre training for the


rest you actually have to fine tune. Several contributions are made including a general


3d encoder decoder framework covering text images and videos at the same time. That's


what we saw is possible by doing this. Essentially, it's a it's a VQ GAN for images. For text,


it's already in the correct representation. And for for videos, they simply say, well,


every frame is an image. So it's like, general encoder decoder framework, covering text


images and videos is let's say it's a nice formulation, a nearby sparse attention mechanism


that considers the nearby characteristic of both spatial and temporal axis. That is simply


local attention. So this nearby sparse attention, it simply is local attention, they simply


do it over the three axes instead of over one axis where local attention was originally


presented. And third comprehensive experiments on eight synthesis tasks. Yeah, that is, that


is what they do. This is our first step towards building an AI platform to enable with visual


world creation and help content creators. Yeah, I can imagine that like models like


these are going to be pretty powerful for content creators. If you can, if you can essentially


input arbitrary, arbitrary modalities, and mix them together, it's going to be pretty


cool. Alright, so that was a new war. Let me know what you think. And I'll see you next


Typical Decoding for Natural Language Generation (Get more human-like outputs from language models!)

Pay special attention to this paper. It is not a paper by Google or DeepMind or Meta or anything like this.


Yet I believe it is a really important paper.


It discusses typical sampling, which is a new decoding strategy of how we sample from language models.


We usually train language models with a maximum likelihood objective that put a lot of weight on very likely words.


And when we use these models to produce language, we either explicitly or implicitly reproduce that.


We make these models sample very highly likely strings, which are boring and not human-like.


It's not what we do. I don't say things that are just highly likely because I actually want to say something interesting.


And that means that every now and then I should utter something that's less likely.


I should speak a word or a sentence that you didn't expect because that's what transmits information.


Typical sampling does exactly that and does it in a principled fashion.


This video right here is a description, a review of the paper,


and the next video is going to be an interview with Clara Meister, the first author of the paper.


Both videos, but especially the interview, are super-duper interesting.


I would definitely invite you to check them both out and I would definitely invite you to try out typical sampling.


It is in-hugging phase and whenever your objective is to sample something that is very, you know, high-quality,


but also diverse and interesting and not just bland high-likelihood text, then that is your method for you.


I believe that we do need new sampling strategies and this one is very promising.


Check it out, leave a like and see ya.


Hi, let me quickly tell you about Fully Connected, which is a curated space for the Applied ML community.


It features articles, project reports, news, events and anything you could want.


Especially the projects page acts as a little bit of a product hunt for ML.


So feel free to add your own project right here.


It's curated by Weights and Biases, but I know what you're thinking.


Yeah, another company, blog, whatever about their products, but this is not at all about Weights and Biases.


It features some of their stuff, of course, but it is generally a really good resource to get good information on what's currently happening in deep learning.


They have great articles and tutorials, like there's one on solving Wordle with reinforcement learning, which is pretty cool.


There's one explaining group normalization in PyTorch and there's one that explains to you how to run Yolo v5 object detection on Windows.


So as you can see, they have all kinds of stuff and the list of already existing articles is long.


If you still don't believe me that it's not all Weights and Biases, in fact, you can submit a post there.


You can click the button, write a post, it will be reviewed by them and then published.


One of the coolest ML startups currently is going to push your content. How great is that?


Now, if you are just a lurker like me, then you know, head over there and subscribe because it's user submitted but curated so you get the best of both worlds.


Besides articles, they also have events, which usually means they're webinars about various topics.


You can look at old webinars, but you can also subscribe to get updates on new ones.


They also host their podcast there, Gradient Descent, and the current episode is actually with Jensen Huang, the CEO of Nvidia.


So pretty big hitter. And lastly, it includes the good old Weights and Biases community forums where you can get all kinds of help on Weights and Biases products


and beyond Weights and Biases to all kinds of things machine learning related.


So again, fully connected. It just got a major redesign. Please check it out.


Go over there, subscribe for awesome articles and news. There's new stuff all the time.


Thank you so much to Weights and Biases for sponsoring this video. They've been a great sponsor, so please check them out.


That's 1db.ai slash fully connected. Now let's get into the video. See ya.


Hello there. Today we'll look at Typical Decoding for Natural Language Generation by Clara Meister, Tiago Pimentel, John Wicher and Ryan Cotterall.


This paper suggests a new way of decoding of producing text from a large language model or a small language model.


It doesn't matter. We don't discriminate here. In any case, usually currently you might have heard of things like beam search.


You might have heard of things like nucleus sampling and top case sampling. These things are all right.


And interestingly enough, these stochastic methods like nucleus and top case sampling are better than the methods that try to find the most likely things such as beam search or greedy decoding.


However, it's still not satisfactory. Large language and small language models.


They often produce text that is boring, just kind of bland when you actually use them, even though they have amazing perplexities on text.


This paper tackles this. It proposes that when humans generate text, they don't just produce the most likely text.


They will actually trade off likelihood with information content or the transmission of information to another human.


And that trade off can be captured in the frameworks of information theory.


And we can generate or we can suppose a decoding scheme which they call typical decoding, typical sampling,


which exactly encapsulates that notion of balancing interestingness or information with likelihood.


And when they test it, that actually results in better results.


This could be really crucial because it doesn't require any change to how we train language models.


In fact, we can take off the shelf trained language models and simply use this new decoding strategy out of the box.


And it applies across across domains.


Now, I have long said that we need that probably our decoding methods are sampling methods may be inadequate depending on what we do with those language models.


For example, AlphaCode samples a whole bunch of programs in order to solve a problem.


Now, we again, we don't like there is value in diversity if you sample a whole bunch and then after that use like a filter to narrow it down.


So I think depending on what you want to do, maximum likelihood sampling is very appropriate.


This paper, for example, mentions natural or machine translation because in machine translation,


you really want kind of the the best translation for a given input.


However, in other frameworks such as AlphaCode, but also such as storytelling, this paper mentions summarization maybe as well.


You want to we want to trade off some of this maximum likelihood for some more diversity or for some more interestingness or for some more information content.


And that's what this paper does. So we'll dive into it.


If you like content like this, as always, leave a like and don't be shy to let me know in the comments what you think.


I'm not exactly I'm not entirely sold on what this paper does.


I do agree we need better or we need a different decoding strategies, but I do have my, you know, reservations about this exact one.


So let's dive into the paper. The paper first complains about the exact thing I complain about,


namely saying that language models currently they have extremely low perplexities on corpora from many domains.


Yet when used to generate text, their performance is far from perfect.


And by that they mean, yeah, they they produce text that is undesirable, e.g. generic or degenerate.


Wait, yes. So either generic or degenerate or just as I said, boring, bland, you know,


and that comes from the fact that a lot of these things, they try to find the maximal probability string.


So they think, you know, I'm going to sample from the probability distribution and I want to sample what is the most likely because that's how we train these models.


Right. So let's do a short excursion. If you are unaware of how language models are trained, they're usually trained.


You have a sentence like the cat is in something, the house. And it goes on.


So what you do is you input a part of the text and then you let the model predict the next token.


And then you input that part and you let the model predict the next token.


Now, in training, this is all good and fine. But at inference time, what you do is you provide a prefix, for example, the cat.


And then you have to decode here. You have to decode a word. What's next?


And then you feed that whatever you decoded into the language model and you decode the next word.


And I think that's where part of the problem comes from, because during training, naturally what is here is given by the data set.


So every new step that you take, if there is something unlikely, if there is a certain diversity to the to the input that's captured by the training data.


However, in decoding, you sort of make your own data as you go along here.


And if you just always focus on finding very likely next tokens, you'll never get into kind of a less likely environment, which could also be correct.


So that is one of the problems. However, obviously, in these language models, the way they work is, for example, you input all of this into a big model.


There is some sort of a model which usually is a transformer.


Nowadays, and out comes a probability distribution. And the probability distribution is over your vocabulary.


For example, there is the vocabulary, this cat, dog. I don't know another word. What's another word? House. Something like this.


And it will give you a distribution of probabilities over these words.


And you can now choose what to do. Either you can take the maximum one, which often runs into these problems of being boring or even repetitive.


You can take you can sample from this distribution, which is also not super appropriate because and the paper touches on this a little bit,


because sometimes the long what's called the long tail here.


There are many, many words, of course, and they all have their some probability. And you don't want to get into these super low probability words


because they might just be artifacts of the model. The model doesn't represent these low probabilities really well.


It's really good at the sort of high probability words because, well, it's essentially trained as a classifier.


And the classifier is trained to give you the correct label as the highest class.


And it doesn't really care about the rest of the words, especially not the ones that have really low probability.


So what people do is they came up with, first of all, beam search.


What beam search does is it considers multiple futures.


So if it's here, that cat, like that cat, it considers multiple futures and it looks a few steps ahead.


So it looks a few steps ahead and it keeps a list of things that are possible to complete.


So, for example, in the beginning, it goes all these three routes and it keeps those in mind along with the probabilities that you go along that tree.


And then, you know, you go ahead and maybe the buffer is five large.


So now we can still fit it because there's one, two, three, four, five paths currently.


But as soon as we expand the sixth one, this one here, we have to drop someone.


So we consider all the paths and we consider only the ones with the highest likelihood so far.


This we can simply do by multiplying the probabilities of consecutive decoding steps.


We consider the most likely five, let's say, paths so far and we delete some of them.


Let's say that this one here is really low probability.


And then once we add this one here and this one, we have to drop another few.


So let's say this one, these two here are really low probability and so on.


And we only continue the paths that have good probabilities or high enough probabilities to be the highest possible.


So that's beam search. And the reason why people do it is because there might be a very high likelihood sentence that you could produce.


But the next word just happens to be low in probability, right?


Maybe here house will lead to a sentence that down the road is very likely, has very good score.


But just this word right now in this case is low probability because the immediate best word would be dog for all the possible continuations


or for this particular prefix for all the possible expected continuations.


So beam search is even worse than greedy decoding in the sense that it really finds the high probability stuff.


It doesn't and it looks ahead to be even more accurate.


If you go to the opposite end of the spectrum, you can say, OK, can we sample?


But can we fix the sampling issues that arise from this tale? And that's why people do two things.


So there's top k, top k sampling, and there is nuclear sampling.


And they both were pretty much the same. So top k sampling, what it does is you have again your probability distribution.


And top k sampling simply consists as well.


Can we only consider the k largest entries in that distribution and then just sample from that?


So let's say k equals three. Then we only consider the three largest entries here and we just forget about the rest.


And we only sample from that. We have to renormalize, but that's fine.


And then nuclear sampling is very much the same, except it says, well, I'm going to afford myself a probability, a cumulative probability of, let's say, 0.7.


What does it mean? It means that this distribution right now has a cumulative probability of one.


I am simply going to take the largest ones like this one and this one and this one until the cumulative probability reaches my maximum threshold.


Maybe I'm going to take this one as well. You can see the advantage here is that you don't always pick the same amount,


but you always pick sort of the top entries that make up, let's say in this case, 70% of the mass.


And that is useful because you have to consider multiple scenarios.


One scenario is where the distribution is very peaky, like there you only want to consider very few entries.


So you only want to consider few entries because everything else is just really unlikely.


However, if you think of a distribution that is more spread out like this one,


then you want to consider more entries because all of them are kind of likely.


And nucleus sampling affords you that, whereas top case sampling would just disregard the shape of the distribution and pick the top ones.


Right. So these are the decoding strategies, but still you can see they always go to the top or the most likely things.


And this paper says, well, that's kind of dumb. And it shapes this as a information theoretic problem.


We already said that humans probably want to trade off the likelihood of a string.


So like how likely it is to appear, meaning essentially how much it is expected.


Because if I just say things that other humans expect, right, then I'm essentially not transmitting much information at all.


So we can say that every string has a form or a content of information.


Actually, I'm going to skip here, skip here to the theory section directly and forgive me.


I've pretty much explained all of what's highlighted already.


So what we can say is that a Y, Y is the message that you want to pass.


So let's say it's a sentence, the information content can be quantified as its negative log probability.


Essentially, the less likely a given message is, you can see here that's negative, negative log probability.


The less likely a message is, the more information it carries.


You have to think of it like exactly as I said, if I say something that's very likely,


the other person, you know, could have expected it because it's so likely.


It's like if you meet the stereotypical boring person or if you see a movie where it's like a really stereotype of a boring person,


they will always say exactly what, you know, what you'd expect them to say.


However, if you say, let's say you communicate with someone and they all of a sudden say something that you really didn't expect.


Now, that's a lot of information right there.


In fact, you can by simple application of the chain rule, you can see you can also define a information content for every single word in the sentence.


And that is going to be just the conditional log probability, the log conditional probability of that word given the prefix.


And that's the prefix, those are the previous words in the sentence.


So akin to the information in a sentence, a word carries a lot of information if you really didn't expect to see that word as the next word in the current sentence that you begun or that your conversation partner has begun to say.


So we carry this through and the assumption here is that the goal of an agent is to transmit information efficiently while also minimizing the risk of miscommunication.


So that's the fundamental trade off that humans do when they communicate, at least that's the hypothesis.


If you transmit a lot of information, you're going to have to utter some words that are very not likely because that transmits a lot of information.


However, if you overdo that and if you, for example, don't follow the rules of grammar anymore and just just send around low information messages or high information, low likely messages,


your receiver will be confused because they don't know what to make of it because they really didn't expect to see something like this.


And therefore, there is a chance of miscommunication.


You can also you can imagine that if you if you want to transmit a message to someone, right, if you want to explain something to someone, you always have to adjust to what they already know.


Like if I want to explain the chain rule to someone and I expect them to already know a little bit of math, I'm going to transmit a lot.


I'm going to have to adjust my message to that. And if I assume too much of what they already know and then I'll just end up saying something like, oh, yeah, if you derive F of, you know, of G of X with respect to X,


then you have to, you know, you just derive G and then you kind of multiply by the derivation of F. And it's all good, right? It's all good.


So sorry for this butchering of the chain rule.


But you can imagine that someone who has little grasp of math in the first place would be very, very hard because I only utter the words that carry so much information


that are so not likely in their framework that there's a chance of miscommunication.


I don't know if actually that captures it the best. Maybe there's a better example. That's sort of how I think of it.


What they do define, and now we get into the decoding strategy, is the expected information, the expected information that a specific symbol in the message will contain.


So this formula right here you might recognize as the conditional entropy of a given word in the sentence, namely, and this I think the notation here is a bit out of place.


I think this should be something like the expectation of the information content of just the Tth word, not necessarily Y of T because Y of T we sum over Y of T right here.


So yeah, but so we ask ourselves if we have already produced the sentence up to time step T and we consider the distribution of words conditioned on this sentence.


So we ask our language model, what's the distribution of words that could come next?


And we ask ourselves for each of these one, what's the information content? And since we have the information content is the negative log probability, that's this and here is the minus sign.


We ask ourselves, so what is the expected information content of the next word?


You know, whatever the next word is, what's the expectation of its information content?


If we were to just sample from this probability distribution, and then this here is the formula, right? We simply multiply whatever we're interested in, which is the information content with the probability and we sum that up across the set that we're interested in.


That is, it's just the definition of the expected value and by happenstance, it is also the definition of the entropy or the conditional entropy.


So the expected information content of any given position in a sentence is the entropy of, is the conditional entropy of the distribution at that point.


So what does that mean? That means if my distribution is very peaked, so if it's very likely that one of these three words here is uttered next is, so if I find a text somewhere, right?


And the sentence up to here was something and then there's only like three words that could potentially be there, none else.


It's very peaked distribution. That essentially means the entropy is very, very low.


And therefore, the information content of that, of whatever word comes next, is probably going to be very low because all these words are super likely.


However, if the distribution is very shallow or very broad, then the entropy is high.


And you can also see since any of the words that could come next, first of all, there are many more that could be considered and all of them have less of a likelihood.


Therefore, the negative log probability will be higher. So any of those words will have more information content and especially the expectation over those words, the information content will be higher.


So that is just the definition of the expected information content.


Now, here's the hypothesis of this paper and they base this on some, you know, psychology theories or linguistic theories.


But here's the hypothesis. Any given word should have an information content close to the expected information content, i.e. the conditional entropy given prior context.


In other words, we expect the difference between the expected information content and the true information content to be small in human-like text.


So the hypothesis here is that the way humans balance this trade-off between interestingness and likelihood and so in between information transmission and not being misunderstood


is that they implicitly calculate the expected information content of the next word and then they try to choose the next word in accordance so that it is as close as possible to that expected information content.


So when I talk, I model sort of the transmission channel to my receiver and I figure out, okay, in the language right now, what will be the expected information content of the next word?


And then I try to match that as closely as possible and that gives me a way of determining this trade-off.


Again, this is a hypothesis. It's backed up by a few theories from linguistics. This is also known in information theory as typicality.


So a typical message is one that has the information content of, that is close to the expected information content, but we'll investigate.


So they say figure one shows for human-generated text the distribution of this epsilon.


So this epsilon is the distance between these two quantities, the expectation and the actual thing that's uttered.


Remember, the expectation considers all possible next words and calculates the expected information content of them.


And then this thing right here, this thing is just the information content of the next word that is actually uttered or actually written.


So what would we expect this or what do we see if we analyze human-generated text?


And these here, these are obviously language models that estimate the probabilities of these words, but these are evaluated on human-generated text.


So not on language model generated text, because remember, this paper is all about how do we do that in order to not make the mistakes.


So let's look at what humans do and you can see the distribution is very peaked.


Now, this isn't the distribution of words. This is the distribution of this epsilon.


So that essentially means this distance, this difference right here is very, very peaky and it's peaky around a very small value.


You can see here the scale goes from whatever, negative 10 to 20 or something, and the peak is at a value that's quite close to zero.


Now, it's not exactly zero, but this is empirical data.


So this paper says this is evidence for the fact that humans do as much as they can try to match the information content to the expected information content.


Now, it would be interesting to see what you would actually, let's say humans would just sample from the distribution itself, right?


What kind of distance between the entropy and the information content would you expect to see?


Maybe a Gaussian or a log Gaussian? I'm not entirely sure.


Also, you know, what is peaky? You know, what is peaky even like? How do you characterize peak?


Like I can see peaky, but it's proof by picture almost.


And then we see a very interesting imbalance. Namely, there seems to be sort of mass going higher up, always on the left side of this rather than on the right side.


There seems to be a bit of a longer tail on the right side, but a bit more heavy mass on the left side.


Now, what does that mean? This is, well, I can't really make sense of it because this is the, the epsilon is characterized as an absolute value,


whereas this right here is not an absolute value. And so I'm going to guess they left away the absolute value.


Therefore, I don't know which, I don't know the distribution of the deviation of information content from the conditional entropy per token.


OK, again, I do not know what came first, if they do H minus I or if they do I minus H, and that determines how we interpret these plots.


So I'd rather not interpret them in the wrong way right here. They further, so that's what they say.


The peaked nature of the distribution reveals that humans indeed tend to form language with per word information content quite close to their expected information content,


and the centering of these distributions around a value close to zero reveals that our probabilistic language generators are learning what this rate is.


Well, I'm not sure I'm not sure I agree with that statement because being peaked doesn't mean, doesn't mean like you need both to be true at the same time.


If you assume that the language models are really good at what they do, then you can claim that humans peak around zero and therefore they match the expected information content.


If you assume that humans match the expected information content, then you can conclude that language models are really good at what they do because the peak seems to be rather around zero.


But you can't draw both conclusions at the same time from this plot because you need one to justify the other. In any case, this is a minor point.


What is interesting is that here they go into information theory. As I said, this notion of typicality, which is exactly what we're describing right here.


They say it says typical messages are the ones that we would expect from its probability distribution.


Their average per symbol information content is close to the entropy rate of their source distribution.


Now, the interesting observation right here is that the definition implies that the highest probability message is often not a member of this set.


Its average information content is too low.


So if we consider any distribution and we consider what's the expected information content, which is the way we defined it, and we only consider messages.


Let's say these are the messages. We only consider messages that are close to that expected information content.


But those are going to be messages that are kind of somewhere in the middle of the likelihood.


So they're not super duper unlikely because the expected information content is again the expectation over all of these messages,


which is going to be not super duper high, which makes these un rules out these unlikely messages.


These are prone to misunderstanding, but it also rules out the very likely messages because those are going to be prone to being boring


and not transmitting any information at all.


And that is something interesting. That is exactly the property we want in a new decoding method.


Leave away the really low likelihood stuff and leave away the really high likelihood stuff because that's boring.


Yeah, the typicality is a property.


Okay, now they go into why we have to go for a local notion of typicality, whereas information theory usually defines it as a property of the entire sentence or of the entire message.


Don't necessarily want to go into that. The next chapter, they try to justify this with psycholinguistic concepts.


There are two they consider. There's the uniform information density hypothesis, which proposes that speakers construct their utterances such that information is distributed uniformly across them.


And the speakers choose words such that their information, their information rate is closer to a target channel capacity, which is essentially what we're doing right here.


Then there's the rational speech act and the rational speech act sort of casts a speaker's behavior as the maximization of a utility function.


And the utility function is a sentence's usefulness to its listener.


So the way it constructs this, again, this is sort of hypothesis. It imagines this literal speaker.


So this is a hypothetical speaker that just samples from the probability distribution.


It just looks at the probability distribution and just samples from that.


And it just utters the words as you know, as they come out.


And that means, you know, with the typical problems like it's going to utter kind of low, low information stuff a lot of the times.


Then it says, well, a smart, the pragmatic speaker, and that's what the humans would be,


that the pragmatic speaker produces sentences to maximize the utility function as opposed to following its expected literal behavior.


If you define the utility function to be this thing right here, then the hypothesis kind of matches, the hypothesis matches this rational speech act.


However, I find this also to be a little bit shady because if I have a different decoding method in mind, I can apply the same argument.


I can simply say, well, my utility function is now my new decoding method.


So, yeah, I'm not super convinced by this.


However, it's interesting to see that people think in this way that they say, well, there is going to be this literal imaginary agent that just speaks according to the distribution.


And then there is the upgraded version of that.


And probably the humans are a form of an upgraded version, this pragmatic speaker that changes something that sort of uses this distribution, but changes something about it.


And that's exactly what we do.


So how do we do it? And we've already alluded to most of it.


So what we do is we introduce this typical sampling.


Much like nucleus sampling, we define a threshold, in this case, this is called tau, of probability mass that we're going to allow in our subset of words.


So again, maybe you have a distribution of a couple of words and they have different likelihoods under our language model output.


And we assume our language model output models these probabilities, especially the non-negligible ones well.


Then what we're going to do is we're going to calculate the expected information content, which is the expected negative log probability, which is also the conditional entropy.


So we're going to estimate this property by simply calculating it.


We can do this. This is simply again, this is p of x given y times log p of x given y.


The log probability is usually already output by our model in the form of logits.


We just need to normalize it. And if we apply some sort of a softmax operation, we get the p of x given y.


So then we have the conditional entropy and then we simply choose the words that are most close to this.


So maybe the expected, the entropy, let's say this is the let's say these are the log probabilities right here.


Let's say the expected, the expected one is here.


We simply choose in order the words that are most close to that one.


So it would be this one right here. This is really close. Then this one is really close.


Then, well, it's a tough choice. Maybe this one's really close. And then maybe this one's really close.


And that we do that until again, we reach our target probability mass. Again, if the distribution is very peaked.


So if the distribution is very peaked, that means the typical information content is going to be lower,


which means the words that have low information are going to be chosen more.


And these are also going to be less words. And that gives us our original case back where we're simply going to choose the highest likelihood words into our bucket to sample from.


Yeah. And that sort of regresses to the old case, if the distribution is very peaky.


However, if the distribution is flatter or more broadly, more broad support, then we the expected information content is going to be lower.


Which means that probably these highest likelihood ones are not going to be in it. And we opt for more interesting ones that are also likely, but not as likely.


So this kicks in mostly when there's a lot of possibilities, which you can see in let's say machine translation.


There is not in machine translation is often very clear or there's only a few possibilities on how to translate something.


However, in storytelling, there's lots of possibilities how things could continue and there our distribution are much more shallow.


And this method would exploit that by saying, well, I'm just not going to consider the most likely things right here.


The computational complexity is the same as nucleus or top case sampling.


We also have to determine the set we're going to consider by somehow calculating across it.


We have to aggregate it, we have to renormalize it and then we have to sample from it.


Except here, well, I guess we always have to sort, right? Yeah. Here we also have to calculate this conditional entropy part.


It's the same in complexity, but it does add a constant overhead or like a multiplicative constant factor overhead to the whole thing.


So the last thing I want to go in here is the choice of hyper parameters in this one.


They say we found K equals 30 and N equals point nine to perform best.


So these parameters perform best for top case and nucleus sampling respectively.


So this is for their experiments. So one is for top case sampling and one is for nucleus sampling.


For typical sampling, we found that tau equals point two and tau equals point nine five to provide the best results for story generation and abstractive summarization respectively.


So while they allow for a single parameter for each of the baselines, they go with a separate parameter for different tasks for their method, which is a bit shady.


Now, there's two possibilities. First possibility is they sort of stifled the baseline by only sort of giving it not exploring well enough the possibilities.


Or what I think happened most likely is that the same parameter performs pretty well for all the different tasks, which is a good property in itself right here.


Here we consider 20 percent of the probability mass and here we consider 95 percent of the probability mass.


Now, that's a huge difference in how our set looks.


And that by itself makes it, in my opinion, a bit of a weaker choice for using this as a decoding method, because for every thing that I want to achieve, I need to essentially tune this parameter.


Whereas with top case sampling, I could just leave it be.


So it'd be interesting to see if in the future there might be, because I'm a fan of this technique in principle.


So maybe in the future we can find more of an adaptive way, much like nuclear sampling is an adaptive way of top case sampling.


Maybe we can come up with an adaptive way of determining the number here or the parameter of how many things to consider.


So I don't want to go too much into the evaluation.


There is a difference. Sometimes it's stark, sometimes it's not as stark.


It is different in different regimes.


You can see that depending on the regime that you are at, it's sometimes the different methods are really different.


Sometimes they're quite close, sometimes they switch places.


Yeah, that's I don't want to go too much into the results because we can maybe discuss them in an interview.


But qualitatively say, for example, for the summarization task, we see that typical sampling provides a comprehensive and coherent summary of the article under consideration.


In comparison, nuclear sampling leads to hallucinated facts, for example, getting drugs from under.


OK, I don't I haven't read the article, but nuclear sampling hallucinate facts, which is one property.


If you sample only from high likelihood things, you're just going to continue with things that are very likely in the language itself rather than transmitting the necessary information.


While top case sampling misses some of the important information in the article, e.g. the charges of burglary and arson.


And that might be because top case sampling simply has this fixed bucket of words to consider.


And as soon as one word is not in that bucket, it simply is forbidden from uttering it, even if the distribution is shallow and that word is kind of likely.


So I want to stop here and just give a few thoughts on this.


In my opinion, I already said it is quite needed that we have different decoding strategies to achieve different tasks.


This one right here, it seems really interesting.


It is a way to trade off sort of not considering the most likely things, but also not considering the least likely things.


However, I'm not sure if the notion of the matching the expected information content is appropriate.


I can't tell. It's a good hypothesis. I don't know if this quantity here, the absolute distance is a good quantity.


Like, why would it be the absolute distance? And the other issue I have right here, but this might be my ignorance of information theory is.


So if I change, if I assume the humans talk like this, they choose their words according to the expected information content, right?


And I use this particular construction right here that is going everything that comes out of this,


whatever comes out of this will have a different expected information content than the original language.


Right. If if I wanted to actually match, like if I wanted to keep the expectation, I probably couldn't do this just in absolute difference.


That's probably going to change the expected information content, let alone the distribution of it itself.


But just the expectation is going to change.


Now, if you're telling me that humans do it like this and that our language models are trained on text that is written and uttered by humans,


like wouldn't that text already have that property and therefore sampling from it would be the original distribution?


Or in other words, if I produce text like this, like shouldn't I get the same distribution out that my language model predicts?


Because my language model is trained on human text and your claim is that humans sample text like this.


So why would that be any different from sampling from the language model itself?


And especially, shouldn't it be that the expected information content remains constant if I apply this sampling technique?


Just out of out of principle, because by definition, if it doesn't, then it doesn't it is not.


It doesn't match human generated text because, yeah, because that's already the input.


That's the training data. All right. But maybe I'm sort of ignorant of information theory right here.


Yeah, my other concerns are with the hyper parameter choice. And yeah, I'd be interested to dive a little bit more into this.


Like, what would we expect to see with it, with the different sampling methods or with different hypotheses?


This is also really interesting, but I'm going to leave it at that.


All I can say is that we should probably try this out and maybe, you know,


for certain tasks where diversity and actually transmitting information is more important than being,


you know, uttering the most likely thing, this might really be a cool application.


And maybe we'll figure out an automatic way to adjust the hyper parameters.


Let me know what you think. Maybe you've already tried it out.


Underspecification Presents Challenges for Credibility in Modern Machine Learning (Paper Explained)

Hi there, today we'll look at under specification presents challenges for credibility in modern


machine learning by Alexander Damour, Catherine Heller, Dan Moldovan, and literally all of


Google.


All of Google is on this paper, including some others, including MIT and Google with


a white space.


But there is a lot of authors here, and not sure what they all contributed.


But the main authors are three main authors, which I guess is legit.


But this more and more looks like some kind of physics paper from CERN.


But we'll dive into what the paper claims.


It's sort of a paper that looks at a higher level onto machine learning pipelines, but


gives very concrete examples for what it's talking about.


So the problem that the paper identifies is this thing they call under specification,


which is sort of related to problems we had in the past or that were identified in the


past, but they make a clear distinction of what under specification is, to what problems


it leads and how that manifests and also what the causes are, to an extent.


Well, is a very long paper, I think it's some 30 pages long, the main text or so.


So we won't go through all of it.


I'll pick out some parts of where I think are relevant to the main story.


I'll criticize it a bit because I think it warrants a bit of criticism.


And yeah, that's what we'll do.


So bear with me.


If you like what videos like this, don't hesitate to share them out and tell your friends about


it.


Also, let me know what you think in the comments.


This is I think this is a good topic for, you know, discussing things.


The question to keep in mind while going through this paper is, do they really demonstrate


what they claim?


So that that was my kind of question when going through some of this.


So let's let's actually just dive into the the abstract.


They say ML models often exhibit unexpectedly poor behavior when they are developed deployed


in real world domains.


I think we all get a sense of what that means.


And we all know of examples when ML models perform fine in our lab in our training data


and test data actually, but then when we deploy them into the world, they're not doing so


fine.


I say we identify under specification as a key reason for these failures.


They're not saying it's the key reason it's a key reason.


So that's the important thing.


Now they define it.


They say an ML pipeline is under specified when it can return many predictors with equivalently


strong held out performance in the training domain.


Under specification is common in modern ML pipelines, such as those based on deep learning.


So I think this, the sentence isn't really complete here.


So it's under specified when it can return many predictors with equivalently strong held


out performance.


So what that means is you have some sort of a test set, right big data set, that sorry,


train, if big training data set, you train your model on that, and then you test it on


a test set.


And the training and the test set, they usually come from some sort of distribution.


And what often happens is you simply split your data into a train and a test set.


And with that, you measure the some sort of generalization capability, right.


So there are a number of assumptions here, namely that these, this is sort of an IID


distributed data cloud.


And the assumption is basically that the test data, the data to which your model will be


applied in the real world is sort of similar to the data you've trained it on.


And if that is the case, then a procedure like this will give you a fairly good estimate


of how your model is going to perform in practice.


However, you then take that model and you deploy it to the real world.


And the real world I look, I'm horrible at drawing real worlds.


But in the real world, you might have this is Europe, yay, Africa.


In the real world, you might have very different distributions of data.


And the model might not perform as well anymore.


So this, of course, they're not the first ones to notice this particular problem, the


fact that there's distribution shift, and so on.


What they are saying is that this procedure up here, let's say it's a deep learning system,


there are many, many local minima of that deep learning system.


So that starts from your choice of optimizer, your choice of batch size, hyperparameters,


the choice of architecture of your network, and so on.


So there are a number of hyperparameters, let's call them all hyperparameters, even


like the different procedures, and so on.


So there are a number of hyperparameters, learning rate, architecture, batch size, all


kinds of stuff.


What they experiment here with is the most the most innocuous of hyperparameters, which


is the random seed.


So even if everything else stays the same, and you switch up the random seed, you necessarily


go into a different local minimum, right?


All of these give you different models, we know that in deep learning, you have sort


of a lot of local minima, actually, like you have a continuum of local minima, they are


all as good as each other.


And notably, so these are training models, notably, they all perform quite well on that


test data set, right?


So you train any of these models, maybe you switch up the random seed, and most of them


will actually work quite well on the IID test data set.


However, they will exhibit very, very different performance when you apply them to the real


world.


So maybe this model here, you apply to the real world, and it works equally, it also


works well, but maybe this model right here, you apply to the real world, it all of a sudden


doesn't work.


So the under specification problem that they identify is when all the models work well,


all the models from your training procedure work equally well on the test set.


However, they perform very differently in the real world, namely, there would actually


be a at least one model like this one here, that does perform well, even in the real world.


However, there is another one, at least one other that doesn't perform well like this.


So the pipeline is under specified, this train test split simply doesn't capture the variation


that some important property of the real world.


So the pipeline that produces the model is doesn't care about that feature.


So it's pretty much random, whether or not that feature will be included or excluded


or important or not important.


And it's pretty much depends on which local minima you happen to be in.


And just by looking at the test set, you can't differentiate whether or not that model will


perform well in the real world, or not.


This is under specification, it's very different from the usual domain shift argument.


Usually you say, well, the test set simply isn't the same as the real world.


And therefore, the model performs well on the test set, but then in the real world,


not so much right here, it's more specific, you say, there would be one of these good


models that we get out of this procedure, one of the random seeds would actually work


well in the real world.


However, another one doesn't.


So of course, that is a problem.


And they, so the way they go about the paper is they say, they give some examples of how


that is.


And in my opinion, the examples don't really convince me like I see their point.


However, the examples are, let's say half convincing.


And then at the end, they give some recommendations for I mean, there is some work in this, namely


what you have to do is you have to add constraints, right?


If you want to solve this problem, there's two ways either you can test models, you can


take all of the models that come out of your pipeline, test each one of them on the real


world on the things you care about.


And the one that works, you know, you deploy that however, it means that you then again,


need some kind of test data set from that real world.


The other way is to actually, since the model is under specified, try to bring in more specifications


that you care about during the training pipeline, making sure that this model that you care


about is the one that actually turns out to be returned.


They don't demonstrate this here.


So this is my criticism.


They don't, they don't, they demonstrate the problem.


I think they demonstrate the problem in a way that doesn't convince me.


They also do not demonstrate a solution.


So they don't ever go ahead and say, now we actually perform this additional specification


and look, what turns out is still a good performing model.


But with that thing fixed, they don't do that.


Yeah, so that's keep an eye out for that.


So we'll go, as I said through the paper, but first a bit more of the abstract.


So you just hear it in their words.


They say predictors returned by under specified pipelines are often treated as equivalent


based on their training domain performance.


But we show that there that such predictors can behave very differently in deployment


domains.


This ambiguity ambiguity can lead to instability in poor model behavior in practice, and is


a distinct failure mode from previously identified issues arising from structural mismatch between


training and deployment domains.


So that's what I said.


It's, it's a different problem than the classic domain shift or data drift or whatever you


might want to call it.


We show that this problem appears in a wide variety of practical ML pipelines using examples


from computer vision, medical imaging, yada, yada, yada.


Our results show that the need to explicitly account for under specification in modeling


pipelines that are intended for real world to play deployment in any domain.


I mean, yeah, fair enough.


This is actually a problem, right?


And you if you deploy ML in the real world, you would be you know, it, it's very appropriate


to actually care about these types of problems.


I'm not saying you shouldn't care about this.


Yeah, so let's go to let's go to actually jump in the first example.


So they have this notion of what they call a stress test.


Okay.


So a stress test is, as I understand it is nothing else than you test whether or not


you test like one particular aspect of the model.


So they're going to have a couple of examples.


One example, they have an NLP pipeline, where you're supposed to infer, I don't know, do


pronoun resolution, and the stress test, one of the stress tests would be whether or not


that model is sensitive to gender stereotypes.


Okay, so the the assumption is kind of pronoun resolution should be like, just linguistic


thing, it shouldn't really have any bias towards any gender stereotypes and whatnot.


Or maybe not overly so if you compare it to actual world biases.


And the stress test would be, let's measure that particular dimension.


So this this gender stereotype dimension in the model and see how that performs.


So that's the stress test.


And what we are specifically looking for is, is there a large variance?


So is there models that behave the same on the training and the test set, but have a


large variance in these stress tests?


So the first model here is this epidemiological model.


So they say a simple epidemiological model, which appropriate for our times, I guess,


specifies how disease how infectious disease moves through a population, given certain


parameters, right.


So there are two parameters, you can see the differential equations right here, there are


two parameters, namely, there is this beta right here, represents the transmission rate


of the disease from the infected to susceptible populations.


And the parameter D, which is this thing here, represents the average duration that an infected


individual remains infectious.


So once you plug in those parameters, and you start with like some, this isn't some


some initial population, I guess the susceptible population, this s is susceptible, I is infected


and R is recovered.


So you start with 100% susceptible, and then you let this and zero infected zero recovered,


you let this play out, and you see how well that works.


So this is a model, and it will give you curves like this, okay.


So you can see depending on the D parameter and the beta parameter, you have different


curves like this, they all sort of look like this.


So here is number of infected at the beginning, it's zero, and then of course, like it shoots


up and but then as kind of herd immunity, I guess kicks in, this goes down again.


So it's a quite a simple model.


And what their goal is here, they say, look, let's say, just hypothetically, hypothetically,


this is the beginning of a pandemic, just making this up.


And I give you some data points, right?


So at the beginning, we're at zero, then we have some, then some more than some more.


Now please predict the trajectory of the of this epidemic from these data points.


So what you want to do is you want to fit these two parameters to the data points, there


is actually a unique solution.


However, because of the exponential rise of the trajectory, the unique the solution is


numerically not well specified.


Okay, so they say importantly, during the early stages of an epidemic, when the observations


are small, the parameters of the model are under specified by this training task.


This is because at this stage, the number of susceptible is approximately constant at


the at the total population size as the total at the total population.


So that means if you have low number of infected people, the amount of people that could get


infected is still like pretty much everyone, there is no no type of herd immunity yet.


And the number of infections grows approximately exponentially at this rate.


So you can see that approximately approximately what you're dealing with is this rate right


here.


And you can see both parameters are in this rate.


So if you derive some number for this, let's say this you derive from your data points


that this must be five, this is the rate at which the exponential curve grows, there are


many settings of beta and D that make this number five, right?


In fact, there are infinitely many pairs that make this number be five.


So they say this is a classic example of under specification, okay, there are many different


predictors, each of which returns a good predictor on the data that you have.


And you can actually you could split this into train and test, you could split these


data points, you can say, I'll take three data points as a train and one as a test.


And still, there would be many, many predictors that are fit the data here, you see two of


them.


So the blue and the red, they fit the data equally well, right here.


However, they have obviously very different trajectories.


So they say this is an example of under specification.


And here already, like I have agree, I mean, yes, yes, if you do it like this numerically,


these look kind of similar, but it's like clearly one fits more than the other, right?


So I'm not sure that that is is a good example for this under specification.


But we can, you know, we can give you can give kind of the benefit here and say, okay,


they want to give a simple model.


So this is one of these models where it's under specified.


So it performs well on this data.


But then if you look at this data, it performs drastically differently, right?


That's the important part here is drastically different.


So if the real trajectory of the of the epidemic is something like this, then there is a predictor,


namely, D equal 28, that actually performs well, right?


It's not that the training setup is different from the real world, it's that the variance


of predictors is so large with respect to the data over here, that there might be some


that perform well, but the others perform pretty, pretty poorly.


And they say this is not only this is not only the case for you know, this initial fit.


But if you do the same, and you simply use a different initialization, so you different


simply use a different initialization for your parameters, namely, you either use a


gamma or a normal distribution, that will already turn out to give you very different


results.


So here, depends on where it was initialized, and different initialization distribution


result in different distribution of predicted trajectories.


So this is much more, I feel an example of what they want to demonstrate.


So here, depending on how you initialize the model, the resulting model that it tends to


give you right, they do many different runs right here.


And you can clearly see that the blue curves that were initialized with a normal distribution


are in general kind of on average, significantly lower than the red curves, right?


Same data, same procedure, same everything.


But you get an expectation even different outcomes simply by how you initialize the


parameters.


This is I feel this is a very good example, right here of what they want to say, not so


much the early training data.


But you get the point that that they say the under specification leaves this variance,


okay?


Now, what would a good specification look like?


So in this case, a good specification, a good would either be that you somehow know you


somehow have a theoretical reason for choosing one of these two initializers, this could


one specification be that could solve the problem.


Another one that is probably more practical one would simply be to incorporate data from


over here, and thereby you know which model you should pick, which in an epidemic, it's


not really it's like, well, I can tell you how it turns out once I know how it turns


out, right?


Yeah, so and that's a bit of a problem because it already shows you sometimes adding these


more specifications or checking, checking whether or not the model does what you want


it to do in this specific axis that has a large variance is just not possible, like


here.


But the example is, you know, it's the example.


So the next thing they do is they analyze this in a theoretical model.


So they have this theoretical model right here.


This is kind of a two layer neural network, where the first layer is completely random,


okay?


This is a random, this is not trained, what's trained is this thing right here.


So it's sort of kind of a linear model.


It's a it's sort of a model of a neural network that people often use in theoretical analysis,


you assume some kind of distribution on the data, then you assume some kind of distribution


on the weight matrix on the weight matrix entries.


And then all you do is you train the theta parameter right here.


And you can make some theoretical statements about what happens with that model.


So their goal here is to show that their goal is to show the following.


What is obviously, let's say we keep the same data, okay, we keep the same data distribution


or the same data.


We sample this W right here.


Now we can imagine w1, w2, w3, these are all different weight matrices, okay?


So can we come up with a model that performs well on all the weight matrices that we would


kind of throw at it?


But that doesn't, but if we if we just plug in kind of different data, it doesn't it stops


performing well in one particular axis, right?


So as long as we as long as we only look at the training distribution, we're fine.


But then there is this one particular axis that the model just fails for some weight


matrices, but not for others.


So that's going to be the theoretical goal here is to construct as closely as possible,


a model that conforms to the to the claims right here.


So what they do is they make use of adversarial perturbations, where they say, we can construct,


we construct a weight matrix.


Where is it?


We construct a weight matrix here, for any given weight matrix, a shift can be chosen


such that it has a small norm, so that it's essentially the same data that goes into the


model.


Two, it leaves the risk of an independently sampled W mostly unchanged, which is exactly


what we you know, what we have specified is that if I simply evaluate if I train the model,


and I simply evaluated on my original data, then everything's fine, okay.


But it drastically increases the risk of w zero.


So what it says is that if I have such a model like I have above, then I can construct a


situation where I pick, I simply pick one weight matrix, say this one right here, I


can derive a data set x zero, or x, let's call that x three for w three, I can derive


a data set x three, such that all the other weight matrices will work just fine on that


data set, right?


They will work the same as my original data right here, everything's fine.


However, this particular one won't work on that data set.


And that is going to that is going to result from an adversarial perturbation targeted


x yet exactly that.


So this, this thing here constructs a data set that is according to their own claims.


So it's a cool thing to show that this is possible.


If you have an over specified model, you can generally do you can generally construct a


situation that exactly conforms to their claims.


However, I I this is cool in theory, but I don't think they demonstrate this too much


in the real examples right here.


So yeah, just just maybe this was unclear.


I'm not the best at explaining this this type of stuff.


But what you can imagine is that the weight matrices that you get out of your training


procedure, they can be fairly different, right?


Let's just call them vector.


So this is w one, this is w two, w three, w four, if your neural network just had two


two different weights.


So the weight matrices can be drastically different.


And the solutions to them can be drastically different.


So I can construct kind of an adversarial data set that is, let's say, exactly into


the, this is going to be very simplified, exactly into the let's say, opposite direction


of one particular weight matrix, so that it will work just fine with this weight matrix


will work just fine with this with this because you have kind of the projection onto them


is well specified.


But if I try to project it onto this one, maybe I should have drawn it exactly orthogonal.


But you get what I mean, I can sort of target one of these models.


And then by definition, that one particular model that is as good as all the other models


on the regular data will fail for this particular data set, whereas all the other models will


still work just fine.


That's kind of a theoretical analysis by construction.


Yeah, cool.


But you know, if you make a claim, and then you construct a situation that exactly conforms


to your claims, then of course, it's going to conform to your claims.


Yeah, so this is more according to the real world.


So this is a medical genomics example, where you can see the training, the training data,


they have training data, they have evaluation data that comes from the same distribution.


And then they have evaluation data that comes out of distribution.


So this is more like a domain drift domain shift example.


Okay.


And our question is going to be how do these things relate?


So you can see that if you train on the training data, and then you evaluate on the training


data, you get this is mean squared normalized mean squared error.


So lower is better, you get kind of a variance of models.


So these are all the models that kind of come out of the training procedure.


And the red dot is a specific heuristic that that performs just a bit better.


This is actually it's so what it does is you have a bunch of data points, but the data


points sort of form clusters.


And what these methods do is they take one representative out of each cluster, like so


one representative, and then they train a model just on the representatives.


And that's supposed to just because these data points are all very correlated, if they're


in the same cluster, that kind of gives a better performance, the red dot simply is


a very special heuristic to choose that representative, whereas the blue dots here simply choose


these representatives at random.


So you can conceivably say that all these models, like the difference is simply how


these representatives are selected.


And you can see they all turn out fairly similar with the red dot being just a little bit better.


If you go to the test set on the same data, you can see the performance drops.


But you know, still, everything performs like pretty well, the range of performance here


is fairly small.


So all of these models, you would say, they perform pretty okay ish.


But now you go to the set sets a out of distribution data, and the range of performance is just


very, very big.


And the point here, I think they're trying to make is that look at the best performing


models right here, look at them.


They are on the level of the performance of your models in the test data set in the in


distribution test data set.


However, not all of them, right.


So the good performing model would be in the models that you get, but you simply can't


tell from just looking at the test data set.


And that that is according to their claim.


And they have a further graphic right here where they show look, it's not it's not as


easy as saying the let's just take the best one here, because that's going to be the best


one here.


So here a plot, they compare how well a model does, and the eval set in distribution versus


the eval set out of distribution.


And you can see the correlation is if it's there, it's fairly weak.


So you like you would expect some line like this, if that was just stretched out, right,


if this thing was just stretched, you would expect like a line, but here, there's just


no way to tell for this particular data set.


Okay, so that's, that's an example of what they mean by under specification.


However, I, like I fail to see, like, I see that these low points right here are kind


of on the level of the test distribution.


But I am not like, I failed to see what the difference is to a classic data drift, just


because they are on the left on the same level.


Right, I, I don't think it's that different, like here, the mean performance simply drops


and the variance between the models increases.


And if I had a different eval set, the ordering would be different, and it would look the


same, but the ordering of models would be different, and so on.


What you'd have to do to for me, like, you, I wonder, for example, is it the case in this


step as well?


So what here, what here, if you did the same analysis, would it turn out that what performs


well in the training data set also performs well in the test data set?


Or is it also pretty, pretty random from the training data set to predict the at least


the order of tests at performance, they never do anything like this, if this is substantially


different here, then you can make an argument, well, this is a different thing than simply


some sort of generalization.


This is really kind of due to this under specification, because going from this data set to this data


set, you sort of have a different spec.


But to me, it seems that this is just kind of a domain drift problem.


And if you look closely, actually, the performance right here is lower than the best performance


here, right?


So that this technically does not fall under their definition, if you go strictly.


So I'm not really sure what to make of these sort of examples.


I get what they're trying to say.


But it seems to me that except for the theoretical thing where they construct the examples, it


doesn't convince me that it's not just domain drift, okay?


Like it's not just the same problem that other people have described.


And secondly, it also doesn't convince me that adding the specification will solve the


problem because in the experiment so far, notice, we have never seen a method from them


to say, let's just fix the problem.


Let's add the specification.


And then we show that we can really keep this performance, right?


The key thing is you want to keep this performance, but you want to bring this performance up


right?


So far, we've had these kind of fundamental trade offs.


And these have often arisen, let's say, explainability or fairness and so on, or actually domain


adaptation is, if you want to bring this down, a natural effect is going to be to bring this


up.


So, you know, even if there are good models right here, it might be that to in order to


reach those models, you actually have to weaken the training procedure in order to consistently


reach those models.


It's not demonstrated in the paper that this is even possible.


Okay, so they have a bunch of more case studies.


For example, they have this kind of ImageNet-C example, where ImageNet-C kind of takes ImageNet


and applies a bunch of random, but let's say, well specified perturbations on it.


And again, they show the same thing right here.


They show that look, all these models, they perform relatively equally on the just plain


test set of ImageNet.


But the span of these models, they are trained all the same, just the random seed is different,


And they have a huge span of performance on these individual things.


And what you'll notice also here, or here, is that it's not always the same model.


So the model that is good at the pixelate thing will be not so good at the contrast


thing and so on.


So the question is going to be, which the paper also doesn't solve is going to be that,


you know, these kind of stress tests, they are in very, very specific things like pixelate,


I can think of a million perturbations to images that are kind of orthogonal to pixelate.


It is going to be very impossible to specify all of them right to remove this under specification.


So the question is, is probably by adding the specification of pixelate, you simply


worsen the problem for any of the other things that you have still not specified.


Plus you probably worsen a little bit your performance on the actual test set if you


incorporate that into training.


So the paper still hasn't shown that that is even even possible.


What is interesting is, yeah, here, they basically say you cannot predict the performance on


one of these perturbations from the others.


So they appear to be completely orthogonal.


So it's not just enough to have a bunch of perturbations and then kind of be confident


that the model is sort of robust to all the perturbations.


I think the core message of the paper is that if you care about a specific axis, you have


to go and check for that specific axis, right?


Otherwise you don't know what your model is doing.


It could be doing something good, but it could be doing something bad if you don't specifically


care about it.


They do the same thing with kind of these skin lesions.


So they have all kinds of demonstration here.


In NLP, they do tests with BERT.


And this is interesting because not only do they test different seeds for fine tuning


BERT, but they also test different seeds for pre-training.


So in these language models, you have like a pre-training phase, and then you have a


fine tuning phase.


And both of them have kind of random seeds.


So they are going to show that even, let's say, the random seed of the pre-training will


actually already play a big role in how these models perform in these stress tests.


I find this to be pretty interesting.


So they do this with respect to these gender datasets, which have been constructed to sort


of assess fairness of these models.


And so what you're going to have is data like the following.


So you're going to have the sentence, let's say, a doctor is walking.


So it's always going to be like some sort of profession, okay, used in a sentence.


And then what you do is you simply replace that entity with a man or a woman, right?


You replace it twice.


And you ask your model, you perform, you embed all of these sentences, and then you ask your


model how similar are those sentences.


I presume by simply taking the inner product of the embeddings, or you can actually train


it.


And so they say, part of glue, our ensemble of predictors achieve consistent accuracy,


measured in terms of correlation with human provided similarity scores ranging from this


to that.


Okay, so you have kind of a model that can predict similarity in text, just similarity.


It knows nothing about gender, right?


You simply train it on a data set to predict similarity in text.


And then you ask it.


So this sentence that I have here, this reference sentence, is it more similar to when I replace


the entity with a woman?


Or is it more similar to when I replace the entity with a man?


Okay, and what you look at is the the difference between the two.


So if this is a positive number, that means that the sentence is more similar to when


you replace it with the word woman, and when you have a negative number, the same for men.


And if the model is, let's say insensitive to the gender dimension, then you expect a


difference here of zero, at least in expectation, right.


So a model that does not learn a gendered correlation for a given profession will have


an expected similarity delta of zero.


We are particularly interested in the extent to which the similarity delta for each profession


correlates with the percentage of women actually employed in that profession, as measured by


US Bureau of Labor Statistics.


This is, in my opinion, this is already an improved assessment from what usually happens


in these in these fairness literature things where they just say, well, if it's anything


but 5050, we are angry.


Which I get, I get it if you, you know, some cases, you need to build a model that is actually


But if if you want to assess things like they assess here, like the question, the question


is, does the model spurious ly pick up this thing?


So if the model does something like if the model is, let's say, perfect, and does only


the task we needed to do, it will learn the association between a profession and the gender


in the exact proportion that it kind of happens in the text, which I guess is proportional


to the proportion at which is happens in the world.


If however, the model for some reason uses this thing as a feature more or less than


it should, then we see a discrepancy.


And why is that important that it's important because if we then deploy this model, right,


we simply take so the model here is going to be the axis here is going to be zero, zero,


and the model can perfectly solve the task by simply being here, right, it's actually


best to be here, where this delta between the similarity and the profession percentage


is zero.


But the model can probably solve the task equally well by being here, or here, or here,


or here, right, it can solve the task equally well.


However, if we just happen to pick at the end, we pick one model, if we happen to pick


this model right here, that model just by more or less chance, has a much higher association


with one gender to particular professions than the other.


And depending on what we use the model for, like we seldomly use the model on the exact


task and data that we trained it on, depending on what we use it for, this might cause some


some adverse effects.


Okay, so I want to stress that this is not the same as your kind of classic fairness


literature, this really considers all these models, they perform like equally well on


the test set of that particular task.


And since it's overspecified, or under specified, over parameterized, there are many, many ways


to solve tasks, some of these ways will include this feature, some of these ways will include


actually the opposite feature.


And if we kind of pick one that's at the extreme, then the model is going to have that feature


and that might not that might not be important for this task.


But it might cause something bad for a task that we ultimately apply it on.


So they do this similarity, and they do pronoun resolution.


And so they come up with different things, they say there is a large spread in correlation


with BLS statistics on the STS task correlations range from point three to point seven on the


pronoun resolution task, the range is this.


As a point of comparison prior work on gender shortcut pronoun resolution found correlation


ranging for that.


Okay, so we are in the in the same ballpark as prior work.


They say there is a weak relationship between test accuracy, performance and gendered correlation.


So there is a Spearman correlation coefficient of point oh eight, which is a weak correlation,


right?


In fact, the confidence interval includes zero.


Oh, that's for pronoun resolution.


So for for the for the similarity, it's point two one, which is an okay correlation, the


confidence interval just barely includes zero.


So we're fairly sure I'm not a statistician, don't grill me about p values.


But this they say this indicates that learning accurate predictors does not require learning


strong gendered correlations, which is a statement you can make, though, I would say such a over


over parameterized under specified model will probably pick up this feature here fairly


often since the correlation is there, right?


But they are right, it does not require it does not require strong correlations.


And they say, third, the encoding of spurious correlation is sensitive to the random seed


at pre training, and not just fine tuning.


So this is very interesting, especially in the pronoun resolution tasks, the pronoun


resolution tasks, don't want to go into it too much here.


But so here, you can see two different runs.


So two different random seeds that result in two very different.


So here is the similarity delta, this is this, this minus this we observed before plotted


against this percentage female bio occupation for individual occupations.


And you can see here, this predictor has a stronger correlation than this predictor right


here.


Now, I've thought about it, I'm still not sure which one is, let's say, let's call it


the better one.


Because I'm, yeah, I'm not sure like, because the, you can say the bottom predictor has


less of a correlation with actual occupation.


I think that makes it worse, right?


But you might argue that a model just shouldn't depend, or shouldn't care.


But then the delta is not zero.


Whereas this top predictor actually has the zero here at fairly at the point where it's


So I'm going to tacitly argue that the top predictor is the one you want, but I don't


know.


The important part of the paper doesn't make a strong opinionated claim about which one


you want.


The paper actually just says, you should be aware that both predictors solve the task


very well.


However, one, they're drastically different in how they treat this feature.


So here you can see, there's not really a correlation between this score and the test


set accuracy, you can't tell from the test set what, you know, can tell from the test


set how it's going to perform in this particular stress test.


And this is very interesting in the pronoun resolution task, they here they plot by different


pre training seeds, and you can see they clearly cluster, right?


So even the pre training seed has an influence later on this on this performance.


I guess it's kind of logical, but it's still interesting to see that this clusters so well,


while all these things solve the task.


So that it basically means that you can't just take like a bird checkpoint and then


fine tune it with like an objective in there that you might already be worried about how


the pre training happened.


I guess maybe you can fix it.


I don't know.


That's what they don't show.


So they analyze it a bit more.


They say they take 20 of those predictors here to better understand the differences


between predictors in our example, we analyze the structure in how similarity scores produced


by the predictors in our ensemble deviate from the ensemble mean.


Here we find that the main axis of variation aligns at least in its at its extremes, with


differences in how predictors represent stereotypical associations between profession and gender.


So these data sets, by the way, they are annotated.


You know, they are constructed such that the kind of stereotypes manifest or don't manifest,


depending on how much your model has picked those up during training.


Specifically we perform principal component analysis over similarity score produced by


20 fine tunings of a single bird checkpoint.


So 20 different models.


We plot the first principal component, which contains 22% of the variation in score deviations


against the female participation percentages in figure nine.


Notably examples in the region where the first principal components values are strongly negative


include some of the strongest gender imbalances.


So let's look at this graphic right here because this this is where I kind of sort of get skeptical.


Okay, so let's understand these plots on the left right here.


So what you have is the first principal component of this kind of of this resulting similarity


scores.


So I'm going to guess each of these dots here is one of these models.


So you can see, and I'm going to guess that each of these line is like one of these professions.


Okay, so for a given profession like this, this here appears to be a profession where


let's say approximately that has a 20% female participation rate.


And the spread here is going to be how the different models happen to manifest in the


first principal components.


So the first principal component, you know, the axis of largest variation in the data


set.


So the first thing that is very notable here is that these models are spread out quite


a bit, right?


So they are, they are they perform like sometimes it's very, the it's very negative.


Sometimes it's very positive for the same thing, right?


This is what is strange.


Or this is a thing that this paper points out, all these models perform equally well


on the test set of the task that they care about.


However, so this here is when you put man as a subject.


So up here, the 100, these occupations that are listed here would be something like, I


don't know, mine, mine worker, oil rig worker or something like this.


And on the bottom, you'd have kind of the more stereotypical female professions like


nurse or something like this.


So a couple of things to note here.


So what they what they do is the red dots here are theirs.


So they say, we'll take the extremes and the extremes are just whenever I think this is


here is negative one.


So they take the extremes, and they look at them here and they kind of make a point of


the first principal component in its extremes displays kind of the most, most anti stereotypical


examples.


Okay.


So it you have to see here is these dots are where the first principal component is loaded


negatively by a lot.


And the sentences, these are the red dot sentences, right there, red dots, those are those sentences.


A receptionist is crawling is the sentence, and the plot is for man as a subject.


So this is the when measure when you measure the similarity between a receptionist is crawling


and a man is crawling.


You ask how similar are those sentences compared to when I enter a woman is crawling, sorry,


compared to the similarity of a receptionist is crawling with a woman is crawling, right.


So this is the data.


This is fairly, it's fairly meta, right.


So their claim is that this first principal component kind of incorporates this feature


by a lot.


And I think their their point is kind of see even when we don't train this stuff, there


are models that that very much rely on kind of these are that very much over rely on these


kind of stereotypes.


However, that this is very, I feel it's it's a bit, it's a bit shady, because, I mean,


look at look at this data, right?


You can't like you can't just pick these outliers like here, these are outliers to and even


if you look here, like they conveniently pick.


So I guess they conveniently pick such that these things here are left out, you can see


here, it's woman as a subject.


So what you'd expect here, if this is really the models pick up a lot of these kind of


spurious correlation, what you'd expect is a line like this, right?


You have like shift here, and then up here, because you know, 100% women, like the first


component will load a lot.


You don't see that at all, right?


And here you see a little bit, you see a little bit a slope like this.


But I don't think that, and especially if you look at the noise between the things like


this is here, and then this is over here, right?


So like the in between noise is way bigger.


To go and claim you had the first principle components contain something like this, and


then we don't look at these outliers up here.


I don't know.


Yeah, so this, this doesn't seem to me like, I see what they're trying to say.


And what is concerning is that there is such a big spread among the models, right?


Within these professions, there is a giant spread.


These are the same performing models.


So I see the what they're trying to say, but I don't think the point they're making here.


I don't know if this is politics or something that they have to kind of bring in these these


types of topics.


But you know, they also look at with respect to others, and they show look, these models


perform differently with respect to different stress test dimensions.


And notably, the ordering isn't the same.


But again, I feel that this is simply, this might be just a problem of domain shift rather


than what they're claiming.


And lastly, they have kind of a test on these other stress tests.


There are also NLP stress tests.


And you can see that the models perform quite differently.


So there's a spread right here.


Within each of these, the red bar is the spread on the actual test set, as I understand it.


And then these are the different pre training seeds.


And you can again see that even the pre training seed will have a big effect right here.


So yeah, again, what I would like to see is kind of how does even does even the training


performance predict the test performance on the same distribution that would already be


quite informative.


As you can see right here, you can't really predict one of these stress tests from the


other.


The question is just can you even do this for the training to the test set because that


would inform you whether or not this is a property of this stress test being in a different


direction, one direction that you didn't capture.


If the stress tests are really meant to show that look, you can't really tell this axis


that you didn't specify.


This is really because of under specification, you would expect that from the training performance,


you could at least predict the test performance somewhat or from the test performance, you


could predict on an IID test set.


I'm going to assume that it is somewhat like this, but I also not sure that you like that


this is anything to rely on.


And the last thing they do is kind of a lab study where they have kind of vital signals


and they predict whether or not there is a medical problem.


And again, you can see here they even test different architectures and so on.


And what they're basically the point is the point is the same.


But it's just shown in a different data.


It's pretty cool that they have lots of different different examples right here, but I don't


want to go into the lab thing.


So their discussion at the end, I think is kind of weak because I mean, what they say


is our findings underscore the need to thoroughly test models on application specific tasks.


And in particular to check that the performance on these tasks is stable.


I mean, I fully agree with that, right?


If you deploy your model into some sort of real world application, please test whether


it actually works in that real world application, but it seems to me that that is not, it's


not a solution fully to the problem, because as we saw in the epidemiology paper, that


sometimes just isn't possible.


And also, you know, it is the case that not everyone can train a language model.


So we kind of need pre trained checkpoints.


Maybe the goal is that we provide like maybe Google, if they provide one BERT checkpoint,


let's say they provide 50, right?


And then people can go ahead and check which one actually is good or bad on on their particular


dimension that they care about that maybe the pre training didn't care about.


That would, I think that would be a practical solution to the problem, if you can't specify


it.


And what I would say also is that it's not clear to me that it is always possible, even,


you know, in theory, maybe, but it is not clear to me that it is always possible to


add the specification that you want, and keep the same performance, I see that there are


predictors in the set that they consider that have that, but that doesn't mean that once


you add the constraint, the training procedure reaches that same performance, and specifically


keeps the performance on the test set.


So that's kind of a number of criticisms on this paper.


All in all, I mean, it's, it's a paper that you generally can agree with, right can agree


with the the sentiment, and also the analysis, the examples are of course real.


And the problem is real.


And yeah, especially for a company like Google, this is fairly important because they build


big models and deploy big models.


Alright, let me know what you think about this.


I'll see you next time.


NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis (ML Research Paper Explained)

Hello there, look at these objects right here. What if I told you that I'm going to give you a


bunch of pictures of these objects from different sides. And what you have to do is you have to come


up with a system that generates me the picture as if the object was viewed from any direction.


So something like this, right, any direction, you can get me a picture of that object from just


a few input pictures. This is a pretty daunting task. Specifically, look at the ship, for example,


right here, you can see in the water, there's specularities that only appear if you view it


from a very particular angle, right? Also the drum kit, you see that the microphone on the left,


it has very specific structure to it. So this is not at all like a trivial task.


There, there's very, there's very, very intricate things here. And this not only with toy data,


but here you can see real world scenes. So this isn't some kind of abstract thing. You can actually


use this in the real world. Now don't look at these things too long. They tend to make me dizzy.


But that's ultimately the goal input a few pictures and then being able to synthesize any kind


of view. So the paper we're going to look at, it's a bit of an older paper, but I think it's


pretty cool. And it's relevant. And there is a bunch of follow up work to this. This is very


popular right now. This is the paper introducing nerf, representing scenes as neural radiance


fields for view synthesis. And it's by Ben, sorry, Ben Mildenhall, Pratul P Srinivasan,


Matthew Tanchik, Jonathan T Baron, Ravi Ramamurthy, and Ren Ng. This, as you can see, the task is


called view synthesis. And what you can do with with view synthesis, or with this paper specifically,


is you can, it can also it takes into account your viewing direction, which gives a much more


realistic impression. We've already seen this with kind of the the lighting here. But it's


really cool to see the lighting here. But in order to really show you this on the left, you're going


to see this novel view that is rendered. And on the right, it's sort of like a fake thing that you


couldn't do in reality. But what we're going to do is we're going to keep the camera at the same


position. But we're going to tell the scene that the camera is at a like switching around. And that


is just how different a pic like a room can look like if viewed from different directions. So the


right one is really kind of physically impossible. It's just meant to show you how different things


look differently if they think they are viewed from a different direction, right. So the same thing


here. And it just looks amazing. What you get automatically out of the systems are depth maps.


These are notoriously hard to get, especially for complex scenes, such as this one. Also this one


right here. It's it's very complex, and it handles it fairly well. Sorry. You can even do something


like AR right here, since you now have a representation that tells you how far everything


is away and you have it from different views, you can see. Yeah, and you can even get meshes. So I


should be able to move that around here. This is now a mesh. It's not only view synthesis, but you


can actually fill out the voxels, which is a slightly different task. And if you have pictures


from all around, you can synthesize kind of any view in between, as you can see right here. So


we're going to switch away from the fancy videos to the paper. Now the special thing about this


paper and this is it's in the spirit of something like sirens. So sirens we've I've made a video


about it. And the special thing right here is it uses deep learning in a little bit of a different


way than we would normally use it. So first of all, what does the abstract say? We present a


novel sorry a method where it is novel that achieves state of the art results for synthesizing


novel views of complex scenes by optimizing an underlying continuous volumetric scene function


using a sparse set of input views. So the task description is the view synthesis, right synthesizing


novel views. Also, you're given a sparse set of input views. So you're given you have a scene,


let's say you have a tree or something like this. So here's a tree. I know beautiful. And you're


given a bunch of images. So maybe someone you know, stood here and took a picture. So the picture


kind of views in in this direction. It pictures depicts the tree and someone stood here and took


a picture of the same tree, maybe the same person someone flew up here, took a picture of that tree.


So you get a bunch of those, maybe you get 20 or something around the tree, maybe more, maybe less.


So from these pictures, you want to build a thing that can generate any view from anywhere. Okay. And


the way they do it is by optimizing an underlying continuous volumetric scene function. This is a


cryptic way. But it goes along the direction of the sirens and kind of a bigger trend in I think in


the AI in these in these neural rendering papers and so on, which is that we want to overfit a


neural network to a single data point. This is really different from classic deep learning. If


you know, if you ask someone, how would you go about this problem with deep learning, what they


would tell you is, okay, I need a data set, I need a data set of these, you know, different scenes


and the input, and I have my x and my y. So the input x is going to be always like, you know, 30