a bolt on and we're not like super clearly looking at, you know, or distinguishing when


Author Interview - Improving Intrinsic Exploration with Language Abstractions

is it okay to have language, you know, just be a complete drop in replacement versus just


some additional information. So yeah, I think I think we're showing that, you know, in general,


like if you're if you're trying to add language into these environments, you're seeing a gain.


But how precisely that that gain manifests is, you know, still still requires some more


exploration for sure. So I guess more generally to your comment on using extra data. Yeah,


I mean, I think we have some intuition that that this data should help, right? It's a


fairly clean linguistic signal. But how to use this data concretely is an open question,


right? And so that's kind of where I view the contribution of this paper as even though


we have some intuition that adding extra data will help, we actually need the equations


written down, right? And here are two concrete ways in which we can operationalize this data


for the purposes of actually getting better performance in your environment. And there


are a lot of examples of this in machine learning, right? So like you have some large language


model, for example, and then you want to fine tune it for some domain or you want to fine


tune it on human preferences. I mean, that's fundamentally, you know, you're adding extra


data for the purposes of getting something that works well on a task that you care about,


right? And how to use that data is the open question. The other point that I would say


is that, you know, we have some deep seated intuition that this language should help.


As you say, it's really high quality. It comes from an Oracle, it comes from the game engine.


But we actually still need to get that kind of empirical verification that it works, right?


And there's actually a lot of reasons why maybe these experiments might not have worked


out. For example, the language is Oracle generated, as I mentioned, but it is also very noisy.


So as I described in kind of the method section of the paper, most of the messages that you


see in the environments are actually not necessary to complete the extrinsic task. You know,


and I kind of exhaustively like show like which of the messages do matter. And so it


could be the case that, well, you know, the language signal, at least in these environments,


is too noisy. The state abstraction captures all of the factors of variation that you might


care about in an environment. And so you don't ultimately need language. Right. And that's


that's an empirical question that we have to measure. And so I view this paper as providing


that empirical verification, which in hindsight, I think is a fairly straightforward intuition.


You know, it's something that I definitely thought would happen. But yeah, it's nice


to see those results kind of in writing.


Yes, it's easy. I think you're right. It's easy to look back and say, of course, like,


well, all you do is you know, you do this, but exploration has been since since, you


know, people have thought about reinforcement learning, they've obviously thought about


exploration methods and intrinsic rewards are like as old as Schmidhuber himself. And


we you know, the fact is that, you know, new things are developed. And this is at least


one of the first things into into really the direction of incorporating. There have been


incorporation of languages before but a systematic adding it to the state of the art methods.


And yeah, it seems like I am I am convinced the method at least the L-amigo method is


quite well outlined, I think in these diagrams, the contrast of the left being the original


amigo and the right side being the language amigo. The question I had right here is that


on the left side, you have this teacher network and it simply outputs a coordinate to reach


and it has to pay attention to the fact that the coordinate is not too hard and not too


easy, right? Therefore, it has to learn that too easy coordinate. Yes, one that is, you


know, close, but also it has to learn maybe unreachable coordinates or coordinates that


are inside the walls, right? They can't be reached or something like this. However, on


the right side in the language, I mean, you seem to split these two tasks out into one


network that that determines which goals can even be reached and one that then orders them


essentially, why? Why are you doing this? Like, what's the, is there a particular reason


behind why one network can do both at the same time? Yeah. So the reason why we split


the L-amigo network up into two parts and as you say, we don't have to do this and there


are ablation studies in the appendix that shows what happens if you get rid of the grounding


and you just have a single network predicting both goal achievability and, you know, actual


goal that's seen by the students. So it kind of a goal difficulty network. It does fine


in some environments, especially in mini-hack, but it doesn't do as well in other environments


such as mini-grid. And part of the reason, as you've described, is that at least in these


environments, the coordinate space stays consistent across episodes. And so you're right that


there are some coordinates that are perhaps unreachable in certain environments and not


in others, but there's much less variation than the set of language goals that are achievable


in an environment because the environment will have different colored doors, for example.


And so the goal, go to the red door only makes sense in, let's say, half of your environments.


So it's possible for the teacher to, the L-amigo teacher to hopefully learn this distinction


kind of just through, you know, the policy gradient method. So basically just like Amigo,


but this is relatively sample inefficient because the problem is that when you propose


a goal that's simply impossible in the environment and you get negative reward, that negative


reward only comes after the student has tried to complete the goal for, let's say, a few


hundred steps. And so it's a relatively sample inefficient way of telling the teacher, hey,


the student did not achieve this goal in the environment. And moreover, that negative reward,


you know, there's two possible sources of that reward. So if the student never completed


the goal, is it the case that it was just too difficult for the student, but it is achievable


in practice? Or is it that the goal is simply never achievable in the first place in the


environment? Right. And those kind of two failure cases are a little bit hard to distinguish.


Whereas we have kind of this more frequent source of supervision, which is simply, you


know, as the student is randomly exploring in the environment, it's encountering a lot


of goals, a lot of messages because we have a language annotator and we're kind of, you


know, if we kind of ignore that signal, that seems like something that we should be using.


And so we have kind of this dual thing where we have a grounding number, which is updated


more frequently in the environment, which is updated from the messages that are seen


by the students. And then finally, the policy network, which is actually trained to satisfy


the kind of difficulty objective and actually get the student to complete goals in the environment.


Can you go a little bit more into because that was, I think, the only part that confused


me a little bit, which is the how exactly you train this grounding network. There is


a, there is this, this notion of whatever the first language description encountered


along a trajectory being sort of the positive sample, and then the rest being the negative


samples. And that kind of confused me because it means the negative samples would also include


goals that were encountered just not as the first message. Could you maybe clarify, maybe


I didn't understand something right? Or maybe I don't, you know, see the reasoning behind


this exact choice?


Yeah, so no, I think your intuition is correct. I think you've described it correctly. It


is kind of a weird thing to do, which is that we are treating negative samples as basically


all of the goals besides the first one that was achieved. And of course, that is incorrectly


treating negative samples of goals that were achieved later. So yeah, these negative samples


are noisily generated, as I say. In the limit, this noise should even out though. So you


can compare, you know, like we're just kind of noisily generating negative samples here.


We can compare that to maybe a setting where we had a more oracle sense of when a goal


is truly infeasible in an environment. And so what happens is, you know, just in general,


a goal is going to appear in this negative sample term more and more often as we train


the network. But because we're kind of down weighting all possible goals in the space,


the idea is that hopefully, you know, this noise of incorrectly classifying a goal as


unachievable in an environment kind of evens out over time. And so yeah, it's a little


bit tricky because we don't have the oracle saying, oh, you can't achieve this goal in


an environment, right? We only know that, well, you know, that the student just didn't


happen to achieve the goal in this environment. So I could imagine other ways in which we


try to come up with some heuristic that better captures this idea of kind of unachievability.


But this is what we came up with, which seems to work reasonably well in practice. An alternative


way that you can interpret this is we're not really measuring true achievability. Like,


you know, is this at all possible in an environment? What we're really trying to have the grounding


network capture here is what are the goals that the student tends to reach? So like are


feasible at the current state of training, right? The current policy, what goals can


it reach? And that's really what we need, right? Is we need to propose goals that at


least for now are eventually reachable by a student. And that doesn't mean that it's,


you know, unachievable in all possible students under all possible environments, but at least


just for current, you know, in the current stage of the training process, it's a reasonable


target.


I can imagine that this gets very, that this may require an adjustment or that this breaks


down in environments that are more causally structured. For example, if I always have


to go through the green door before I reach the red door, right, then the goal would always


be in any trajectory that I do, the green door would always be the first goal. And therefore


my grounding network would never recognize the red door as a reachable goal, because


that's always going to be at least the second goal, right? So I guess depending on the environment,


it's not hard to make a change to this, obviously, in that case, but I guess that's one thing


that might have to adjust a little bit to the environment at hand.


Yeah, that's a that's a great point is that we do not. There are settings where you might


just, you know, want to run it without the grounding network. And obviously, that's actually


a simpler version, so it should be fairly easy to experiment with that. And also in


the setting that you describe, what will happen is, like you say, you know, the the the green,


the go to the green door goal will get a lot of weight, but hopefully can be counteracted


to some degree by the policy network, which will, you know, learn to not put any weight


on that once it realizes that it's getting absolutely zero reward for that setting. But


I agree that this kind of introduces some weird training dynamics that we don't really


want and might be cleaner just to remove the grounding network entirely.


If if you as as you say, you've looked at my paper review a little bit. I didn't go


too much into the experimental results as such. Is there also I didn't go into the appendix


at all because honestly, I haven't read the appendix because I sometimes I don't I think


I should probably. But is there anything that you want to highlight specifically about the


experimental results or or maybe something that you did in the appendix, which is also


has a lot of experiments in it? You know, things that you think people should take away


from the paper from the experiment section?


Yeah, so. Broad takeaways are and I think that you mentioned this in the review is,


you know, we're in these kind of DRL environments and and the individual training runs are just


incredibly noisy, you know, and that can be sometimes like rather difficult to get a sense


of, oh, is my method actually working better than others? Right. But there has been some


great recent work from, I think, a team at Mila, which won an outstanding paper award


at NeurIPS last year, which was called Deep Reinforcement Learning on the Edge of the


Statistical Precipice. And the basic idea is, you know, we're compute constrained. We


have these environments. They're very high variance. But even despite all of this, you


know, what are the kind of statistical best principles that we can follow to really see


whether or not our methods are actually making a measurable and replicable difference in


the environments that we're testing? And so they have a lot of good recommendations, which


we try to subscribe to as close as possible in this setting. Right. So these training


curves here give you kind of a qualitative sense about not only kind of the ultimate


performance attained by any of the models, but also of the differences in sample efficiency


that we see. Right. So it could be the case that, well, ultimately, both Amigo and El


Amigo reach the same asymptotic performance, but Amigo just gets there faster or more reliably.


And that's something that you can sorry, El Amigo gets there faster, more reliably. And


that's something that you can look at in these graphs. But I think the more kind of statistically


rigorous way of verifying that language is giving a gain in the environments is in the


subsequent figure, which is figure four, which should be right below this one, I think. And


this is really, you know, us trying to statistically verify, you know, is there an effect happening


here? And so these here are bootstrap confidence intervals, five runs in each experimental


condition. And we're plotting the 95 percent confidence intervals for the interquartile


mean of models across tasks. So this is kind of like the mean performance, assuming that


you drop some of the outliers, because, again, these runs are very high variance. Right.


And so this is kind of a statistical recommendation from the authors of that deep RL paper. And


we show that, yes, the individual runs here have really high variance naturally. But as


you begin to look at the runs in aggregate across both the mini grid and mini hack environment


suites, we begin to see a trend that it's clear that overall we're seeing a good effect


of language in these environments.


And so this is obviously these are aggregate metrics, overall metrics and so on. When we


look at the plots themselves, there is quite considerable variance even in the in the ranks


of the method. Do you have an intuition of between the language methods, which works


better in what kind of environments and in what kind of environments does language even


maybe hurt and why? Do you have an idea?


Yeah. So the trend that I try to highlight in the paper is that in larger environments,


language exploration does better. And the reason why you might expect this is that in


larger environments, Amigo and Novelty kind of suffer from this problem of increased noise.


Right. There's a lot more coordinates, for example, that you can propose, which essentially


describe kind of the same semantic action. Right. You have like you want to get the agent


into one room of this maze and because the environment is larger, now there are four


or five different coordinates that all kind of mean the same thing. Whereas as you increase


the size of the environment, the language set, the set of language goals is relatively


more consistent. Right. It's kind of one of those complexity analyses. Right. It's like


kind of space complexity almost of the goal space. And so you can see this trend happen


a bit, for example, in the Wand of Death task. So WOD, this is in the top right corner here.


We have WOD medium and WOD hard, where in WOD medium, Amigo actually outperforms El


Amigo. It actually gets you to higher performance quicker. Whereas in WOD Wand of Death hard,


Amigo is actually not able to learn at all. And the only difference between these environments,


it's fundamentally the same task. But the only difference is that in WOD hard, the room


is a lot bigger. So instead of a narrow corridor, you actually have to search for the Wand of


Death, that's the task, in some room beforehand. And you can see that just simply increasing


the size of the possible coordinate spaces results in both traditional novelty and traditional


Amigo doing much worse in this environment. And I think that kind of shows that these


kind of state based exploration methods are very brittle to the size of your state base.


Right. So you can kind of increase your state space infinitely and it'll make these methods


perform worse, even if the underlying semantics of your environment haven't changed yet.


Do you have a feeling maybe, if this is a property of the world in general, like let's


say I as a human, right, I'm put into a small, whatever environment or a big environment,


would my descriptions of language also not grow very much? Or is it a property of just


game developers? You know, I add a few extra rooms, I can reuse these language, you know,


I just kind of tile, you know, all the big games, I mean, the biggest games are procedurally


generated like Minecraft, there it's really, it's just the same thing over and over. But


even in like the like, these big open world games like Grand Theft Auto or so, the same


textures are reused and the same cars and the same NPC characters, right? Is this a


property of the world or of the video game developers?


Yeah, so this is a really deep and almost philosophical question. Yeah, it's something


that I think about a lot is you can certainly, and this is a totally valid statement, right?


You can say, well, there are a lot of language actions that you can describe in our world,


and even in the video game world, which just describe these like kind of infinitely complex


and nested sequences of actions, which have absolutely nothing to do with the extrinsic


task, right? I could tell you to, you know, run at the wall six times, do a 360 and then,


you know, continue hitting the wall eight times, right? And that's like an incredibly


difficult goal, which you can imagine a very structured curriculum to get to that point,


right? I'm just like infinitely kind of bumping your head against the wall, which satisfies,


you know, maybe the difficulty threshold of LME, but it's absolutely orthogonal to the


tasks that we care about. And I can imagine that there are settings where the language


is kind of useless and doesn't end up, you know, giving you any gains in this setting.


And so there's kind of this open question that we haven't really touched on sufficiently


in this paper, which is how good does the language have to be in order to get this to


work? So as I say, you know, the language is Oracle, it's game developers, but it also


is noisy. There's a lot of actions like running into walls or trying to throw stones at a


Minotaur that are ultimately useless in the environment. The argument we're making here


is that hopefully, you know, the noisiness of language scales a little bit less than


the noisiness of your state environment, right? But there's still a lot of kind of edge cases


and kind of unexplored territory here. I think more philosophically, if you think about our


world and our environment, right, there are a lot of ways that we can describe actions


that are not particularly useful in the world that you and I inhabit, right? I mean, I can


again tell you to do handstands and hit a wall and, you know, walk around and write


endless, you know, trivial things in the dust. But at the same time, there's a lot of our


action space in the real world that we simply don't have language descriptions for, right?


So like every single precise movement on my hand and my arm, you know, I could presumably


come up with some language to describe, oh, I'm actuating this joint, you know, by 0.03


degrees. And there's like, you know, how many joints in my hand, right? I mean, there's


like endless complexity in terms of the possible action space just by moving a hand that in


language we have absolutely no words for, right? And so it's really, it's a really tough


question, right? Like we have a lot of kind of ways of describing useless actions in the


world. But at the same time, it's very clear that the language that we do use to describe


the world is operating at a higher level abstraction than perhaps the kinds of actions that RL


agents have access to, right? And for example, actuating some sort of limb or something.


You make a good point that in the paper that language is a strong prior over what is, you


know, essentially important to humans, right? If I can describe something with a short piece


of language, like, of course, I can say do three backflips, and then you know, do eight


of that and so on. But it's a fairly complex sentence in itself. If I can describe something


with a short piece of language, usually that is something that matters to some human somewhere,


right? Otherwise, that wouldn't be mapped to a short string. But that brings me a bit


to a different question. And that is the question of isn't isn't the I think in these environments,


there's always a goal, right? There is one reward at the end that you need to reach.


I can imagine though, that novelty or not novelty in general, or how how important a


state is, is really dependent on your goal. Now, whether I, I circumvent the minotaur


at the you know, below or above, that might not be important if I want to reach whatever


the goal behind it. But it is really important maybe for a different task, you know, is likewise


I as a human, whether I move from here to there by walking forward or backward doesn't


matter if I want to get to the fridge, but it matters really if I'm if I'm dancing, right?


So is that something that like, how does that interplay here with these with these language


things? What do you do when a language? It almost like needs to incorporate a piece of


the goal that you want to reach in order to be useful or not?


Yeah. So I think thinking about or trying to filter the language descriptions that you


have to language that is relevant for your task is going to be important if we scale


this up to environments where it's clear that using unfiltered language is not helping,


right? And again, as I mentioned, the robustness of these kinds of exploration methods to the


noisiness or relevance of your language signal is still an open question. If we do have task


descriptions, so we have extrinsic task descriptions, like your job is to, you know, defeat the minotaur,


and it's really intuitive that we should be able to use that as a signal for kind of waiting


how relevant a subgoal or language description that we encounter, waiting how useful that


is for the extrinsic task, right? So if the extrinsic goal is combat, then we should be


prioritizing combat related messages. If the extrinsic goal is buying something, then we


should promote acquiring money and things like that. And so that's something that I


think is a kind of natural extension of this is you extend this to a multitask setting where


you have task descriptions and the task descriptions ought to kind of heavily filter what subgoals


should be relevant for the task. I think when you include task descriptions, there are some


more comparisons to related work. So there's been some related work which you mentioned


in the paper where let's imagine you're doing basically hierarchical reinforcement learning.


So you have some extrinsic goal and then you want to explicitly decompose the extrinsic


goal into subgoals that you want to complete in order, right? And those are certainly kind


of relevant methods to look at when you start thinking about multitask or goal condition


settings. But this is kind of a slightly different focus where we're not trying to identify subgoals


that need to be completed on the way to some extrinsic goal. There's still kind of this


exploration component, which is a bit of a different use of language than this kind of


hierarchical stuff. But certainly I would say that there are people who have looked


at kind of language conditioned RL and hierarchical RL that think a lot and very deeply about


this problem of proposing subgoals that are relevant for the extrinsic goal, assuming


you have some structured description of what the extrinsic goal is.


Although I can imagine you run into sort of the, let's say the more abstract problem of


the exploration problem is that, you know, without an outside signal, I don't really


know what to do. And there is no clear, let's say, gradient towards the goal, right? Otherwise,


the exploration problem in RL would be relatively easy. Now, when we say, well, we'll just filter


out all the messages that don't have anything to do with our combat goal, right? It is like


we could run into the exact same thing again, where, you know, maybe in order to acquire


a weapon, I first need money, right? That doesn't, that's not directly related to my


combat goal. So there is like another exploration problem again, on top of the of the thing


we introduced. I guess maybe we can, we can hope that if we introduce enough levels, the


highest level of abstraction will have a small number of states so that that, you know, random


exploration works. But it's kind of funny that the problems repeat or replicate.


Yeah, yeah, it's really tricky. And that's essentially just kind of a deeper or more


nested failure case of not knowing what's novel and not knowing what's relevant for


your goal, right? So if you're prioritizing words that have combat in them because your


extrinsic goal is combat, but you first need to buy something, then your semantics, you


know, your measure of novelty or relevance is just not good enough. Right. And so that's


going to just be a fundamental problem in exploration is how do we know whether it's


states or language, you know, how do we know when a state is relevant for the ultimate


task?


Yeah, and I guess humans aren't very much different, right? I mean, science is a really


hard process. It's not, you know, that exploration takes millions of humans and hundreds of years.


So we can't fault our RL agency for not not doing that great of a job. Here, I found these


plots to be really cool, like the analysis, sort of the evolution of what the teachers


propose. And of course, these being language, it's quite insightful and understandable what's


happening in the algorithm. My surprise was a little bit, aren't these things kind of


subject to like catastrophic forgetting or things like this, I can imagine, right? If


I train these things online, and they're at some difficulty level, all of a sudden, they


forget that reaching the red door is kind of really easy, or so. Is that have you ever


thought, is that a problem? Or was that ever a problem? Did you encounter that? Or why


don't we encounter that?


Yeah. So I expect that that is a problem that happens in these agents, I don't think we


really precisely tried to measure whether or not catastrophic forgetting is a problem.


I think the fact is that we evaluate in environments where we are not testing the agents kind of


continuously for mastery of all of the skills that it has learned in its curriculum proposed


by the teacher. And so this problem of, oh, you know, you forgot how to specifically open


a specific color door is not an issue as long as the student is still quite good at completing


whatever goals it needs to complete to try to achieve the extrinsic goal that is currently


being set for the teacher, right? And so if you forget things that are at the very beginning


of training, that's not a big deal. So long as whatever path that the teacher is leading


you on is something that will eventually get you to the extrinsic goal that we care about.


And I think that happens to be the case in these environments because there was only


one extrinsic goal and because we're not testing it to, you know, master every single skill


from kind of low level to high level abstractions. But if we were in a setting where being able


to complete those lower level goals kind of, you know, on a dime and kind of switch, kind


of do context switching like that, if that were more important, then we would have to


deal with this problem of catastrophic forgetting. An important point here is that we really


don't care about how well the student is able to follow instructions proposed by the teacher.


That's I mean, we hope the goal is that that property emerges such that we can complete


the extrinsic goal, right? But we're never actually trying to learn a student that can


follow instructions. We never really evaluated exclusively in an instruction following setting.


Let's, if we think ahead a little bit, and I'm gonna, I'm gonna just scroll down to the


environments, just because. Yeah, maybe this this will inspire us a little bit. If we think


ahead a little bit beyond this work, here you have this very, this oracle language descriptor.


And you say also in the outlook of future work that that is something obviously that


we're trying to get rid of, because not every environment, like the fewest of environments


actually have such a built in language description or easily accessible one. So we might have


to regress to something else. So I want to I want to think about three different external


models that we could bring in. And I wonder what you think of each of them, like how these


could fit in the first would be something like GPT-3, like just a pure language model.


How could that help us? Maybe in combination with with these things, because we need some


starting point, right? But how could a pre trained language model that knows something


about the world help us then something like clip, maybe something that can, you know,


take an image and language and say whether they're, they're good together or not. And


then maybe even something like or maybe a captioning model, right? And maybe something


like Dali, like like something that takes language and generates is there in this cloud


of models? What possibilities do we have to bring in sort of to replace this oracle thing


with, with learn systems? It doesn't even need to be learned online, right? It can be


pre trained. I'm probably much more excited about that.


Yeah. Yeah, these are, I think, going to be the most fun questions to look at in kind


of language conditioned RL going forward is taking the boom in pre trained models in large


language models, and resulting, you know, bringing these into concrete and actionable


gains in reinforcement learning. It's funny that you mentioned this kind of what I just


described as almost a gradation from ungrounded language models like GPT three, right, which


are trained on text only corpora, and whether those can actually help in these environments,


which I would call are fundamentally grounded, right there, they're grounded in some, some


visual or perceptual world. Can ungrounded language models still result in gains in these


settings? And my intuition is, yeah, they probably still can. Because, you know, even


if you don't exactly know what it means to acquire a wand or kill a minotaur in some


environment because you don't know what a minotaur looks like or what a one looks like.


GPT as I mentioned, you know, this idea of priors, right, GPT has strong priors on sensible


sequences of actions, right? So in so far as these environments are testing kind of


sequences of actions that humans kind of have an intuition for, you know, it's some fantasy


world, but we have some intuition, oh, in order to defeat the minotaur, we need to get


a weapon first, we probably look around for a weapon. Maybe there's a shop, maybe we can


buy a weapon from the shop, right? Video games are testing knowledge that we have very like


deep seated common sense knowledge that we have that hopefully generalizes to these fantasy


worlds. And GPT certainly contains a lot of that information, right? So you might imagine


we should reward or filter the kinds of descriptions that we see to those that seem sensible narratives


that GPT-3 would generate, right? So a sensible sequence of actions along the way to defeating


the minotaur is collecting a wand and buying it and things like that. And I think you actually


already see some examples of this happening in more goal conditioned or instruction following


RL. So there's been some recent work from, I know teams at Berkeley, maybe Google as


well, that are looking at using pre-trained language models, which are not necessarily


even grounded. They're just, you know, GPT-3 using them to construct sensible plans, action


plans or sub goals for completing certain actions. So in some home environment, for example,


maybe my action is get a cup of coffee. And then the goal of GPT is even though I don't


really know what my environment looks like, I don't know what kitchen you're in. I know


that sensibly this should include finding a mug and then heating up the kettle and things


like that. And so we already see some promising use of kind of ungrounded models for improving


grounded decision making settings. Yeah. Did you want to comment on that? Or I can also


No, no, that's cool. I think, yeah, I've, I think I've even had one, at least one of


these works here on the channel in this home environment that does exactly. I was also


really cool to see. Obviously, these models know a lot about the world. Right. And I think


people overestimate how or underestimate maybe, well, whatever. The thing if we humans look


at a board like this, like at a mini hack board, we see a map, right? We see paths to


walk on and stuff like this, even if we've never played a video game. But this is this


is such strong priors built into us. And we we sometimes think like, why can't that dumb


computer just like, walk around the wall, right? What's up? But and and I think these


large models are a way we can really get that knowledge from the human world into into this


world. So yeah, I think that's, it's a great it's a great outlook also with the models


that combine images and text. I feel that could be that could be a really, like adding


a lot of value to the RL world, at least the RL environments that are like human environments.


Of course, there's reinforcement learning for computer chip design, and things like


this, I don't think those are necessarily going to be profiting that much from it. But


yeah, yeah, really cool. Is so you're you're at Stanford? Or did you do the work at Stanford?


Or were you at some internship?


Yeah, I did it while I had an internship last fall. So this is fall 2021. Okay, continue


to work a little bit while at Stanford. But it was mostly in collaboration with some people


at fair or meta, I guess now, in London, reinforcement learning is notoriously also kind of hardware


intensive. Although this work right here seems like maybe not that much because you describe


a little bit sort of what what it takes to investigate a project like this. Yeah, unfortunately,


I think even for these environments, it's fairly hardware intensive, certainly still


feasible, I think, on, let's say, a more academically sized compute budget. But for being able to


run the experimentation needed to iterate quickly, you know, you do really definitely


benefit from kind of industry level scale, which is one of the unfortunate things about


this kind of research is that it is a little bit less accessible to people in smaller compute


settings. So maybe the typical kind of RL environments you think of are compute heavy


are the ones that are in 3D simulation, you know, very, you know, need physics, need soft


joint contact and all of these things to model. And those are really expensive. I think compared


to that, these are kind of more symbolic grid worlds. You know, the whole point as to why


mini hack or net hack was chosen as a reinforcement learning test bed was because the code base


is written entirely in C and is very optimized. And so you can run simulations very quickly


on modern hardware. But that being said, it's still relatively compute expensive. Again,


the just amount of experience needed by state of the art deep RL methods, even with extrinsic


or intrinsic exploration bonuses, is still very expensive. Right. So, for example, one


of these runs, we would typically have, let's say, 40 CPU actors collecting experience at


the same time in parallel and then kind of one or two GPU learner threads in the background


kind of updating from this experience. So even just a single computational experiment


here needs non-trivial hardware for sure.


Yeah. And ideally you want to do that in parallel, right? Because you want to try out a bunch


of things or repeat it a bunch of times because one experiment really tells you almost nothing,


right? Unless it succeeds, right? If it succeeds, it's good. But if it fails, you never know


if you repeat it a bunch of times. Yeah. But I mean, it's still, it's not the most extreme


thing, right? Like two GPUs or so and a bunch of CPUs. As you say, that's still academically


doable, which I find cool. Could you maybe tell us a bit about the process of researching


this? Like did everything work out as planned from the beginning or where was your starting


point? And what changed about your plan during the research? Like maybe something didn't


work out or so? Yeah. Yeah, I feel, I don't know, I feel it's always good for people to


hear that other people encounter problems and how they get around problems.


Yeah. So, yeah, it's a great question. The intuition that I think me and my collaborators


started with was, you know, fairly sensible. It's language is clearly going to help in


these environments. You know, it has some nice parallels to human exploration. And so


let's just see whether or not language will work in these environments. What's funny though,


is that we actually started out the project less about the more abstract question of like,


does language help exploration and more a very concrete question of how do we improve


upon Amigo? So how do we improve upon an existing state of the art algorithm for exploration?


Let's propose something that we argue is better than everything. It's like, we're going to


propose a state of the art exploration method called El Amigo, which will get 100% accuracy


in all of these environments. And none of the existing methods will work. Right. That's,


that's kind of the narrative that you set up for yourself when you're starting research


is I'm going to build something that's new and that's the best. Right. However, I think


this focus of this paper and the story has shifted considerably. I think it's shifted


for the better actually. And part of this shift happened because we implemented El Amigo


and it was working fine and it worked better than Amigo. So we were quite excited, but


at the same time, the field is moving so fast. And at NeurIPS last year, some researchers


came out with this method called Novelty. And we ran Novelty and Novelty also did really


well. And you know, in some environments it totally like blew Amigo out of the water,


right. And El Amigo. And so part of our thinking was, well, okay, now we can't really say,


oh, we have El Amigo and it's the best model, it's the best environment and you should only


use this. And at first I thought, you know, this is derailing our narrative, right? We're


not proposing anything new. We're not proposing anything state of the art. So what's the point?


But I think after some kind of juggling and shuffling, we realized that what we're really


interested in is the scientific question of does language help exploration? So take existing


method X and then do X plus language, right? And that question can be answered kind of


agnostic to the specific method that we actually use, right? And so it was that juncture where


we actually decided, okay, let's actually, you know, look at Novelty closely and let's


imagine adding language and Novelty as well. And do we see the same kind of results? Right.


And so I think this is kind of an outcome of the paper that was kind of, you know, on


the fly changed, but I'm very happy with, which is that we're not trying to claim that


we have a method that is state of the art or that is best or that, you know, anyone


should be using our method. We are very agnostic to the particular choice of method, right?


We're trying to answer kind of a more abstract question, which is when does language help


exploration? And I think this is a little bit more egalitarian. You know, we're not


saying that our method is better than anyone else's. And we also don't have to exhaustively,


you know, compare to like a lot of existing work. We're just saying that if you take whatever


method that we have and you add language, you do better. And here are two examples where


that happens.


Cool. And it is a good way to preempt some reviewers from saying that you didn't train


on ImageNet and that's bad. Yeah. Is there anything else that you want to get out to


viewers? Maybe a way they can get started if that's possible or anything that you'd


like them to know?


Yeah, I think that we've discussed a lot about these kind of higher level ideas of one Holy


Grail is that we have clip generating descriptions or open, you know, GPT-3, and then we're evaluating


in these really high dimensional spaces with actual motor joints. And we're going to, you


know, show how language helps in these like MuJoCo style, like really deep RL, you know,


realistic environments. And maybe you can transfer to the real world. I think that's


the broad vision, but I think it is still very far away. I think, you know, we even


in this paper abstracted away a lot of difficulty of the problem, right? We're assuming that


we have Oracle language annotations. We're only looking at these kind of symbolic grid


worlds. And although it's tempting to dive in and say, okay, now let's kind of, you know,


straightforwardly, let's extend this to a real world environment where I have to actually,


you know, move my coffee mug to make coffee and tea. I think we're still quite far away


from that, you know, broad vision of kind of household enabled robots in RL and is probably


not the most, I think, like beginner friendly way of starting, right? It was just, there's


just so many deep problems that need to be solved jointly from perception to action,


to planning. And before we even consider how we better incorporate language into the mix.


And so I think the way to build upon this work is just these kind of very small progressive


relaxations of the assumptions that I, and many of the other people who have worked in


this space have, right? So again, let's imagine, let's just imagine we get rid of the Oracle


language annotator and we train a model to emit states for these simple environments.


You know, we didn't really explore that, but that's a very sensible way to extend this


kind of work while keeping the environment and the models fixed, right? So this goes


back to the very beginning when you mentioned, Yannick, the kind of way in which we approached


this paper was to keep everything fixed and then just look at this kind of very small


change and see how that results in, you know, different performance in our environment.


I think that's really just kind of the way to go. It's very slow. It's very incremental


work, but hopefully it's getting us more towards that kind of guiding star of eventually having


these models that operate in these realistic environments and use, you know, pre-trained


model language to help exploration.


Cool. Jesse, thank you very much for being here. This was awesome.


The Dimpled Manifold Model of Adversarial Examples in Machine Learning (Research Paper Explained)

Hello there, today we're going to look at the dimpled manifold model of adversarial examples


in machine learning by Adi Shamir, Odelia Melamed and Oriol Ben Shmuel. This paper on a high level


proposes a new way of looking at the phenomenon of adversarial examples in machine learning,


specifically in deep learning. And they proposed this model called the dimpled manifold model,


essentially arguing that classifiers put their decision boundaries right next to the manifold of


data, while only slightly sort of curving it around the data like this. Now the data manifold being


low dimensional, this results in a situation where you can cross the decision boundary really easily,


if you simply go perpendicular to the data manifold, which also is perpendicular to the


decision boundary. And if because it's just such a small dimple there, the decision boundary is


pretty close. And that's how you end up with adversarial examples that are super easy to get.


So it's not a new attack, a new defense, anything like this, it's simply a mental framework of


explaining why adversarial examples exist on a high level, they have some conceptual thought


experiments, they have some explanations and some real world experiments. Now, I personally


don't think that this is entirely, it's not necessarily incorrect, but I don't think that


this is really useful to think in this way. And I'm going to explain why. In general,


my opinion of this is it doesn't really add anything. And I think it explains less than


the models we already had. Yeah, so that's, that's my opinion, I'm going to get to it


specifically also the experiments they propose, I think that there is a big Occam's razor failure


right there. But as I said, we're going to get to all of this, I'm going to go through the paper,


and I want you to make up your own mind, even though I'm going to try to bias you. So yeah,


this is this is not a neutral channel in case you haven't noticed. All right. So if you, you know,


like content, or if you dislike it, tell me in the comments, tell me what you think of the paper,


whether it makes sense, whether it doesn't make sense, and so on. I'd be very interested to see


what you have to say. Yeah, I read the comments. So please. They say the extreme fragility of deep


neural networks when presented with tiny perturbations. Yeah, but okay, this starts


out how every single adversarial examples paper always starts out saying, okay, deep neural


networks are extremely fragile. There's this phenomenon of adversarial examples. Now, if you


don't know what adversarial examples are really briefly, essentially, what this is, it's a


phenomenon where you take an image like the thing here on the left, the neural network thinks it's


a plane with a very high probability, and you change it to this thing right here, which you as


a human can't even tell it's different. However, the neural network will think that this is now a


bird with very high probability. And the this is the change that you made. It's magnified for you


to see it kind of looks like random noise. But it's a very particular noise that makes the neural


network think it's something different. And this is just it's tiny in the in its norm, right, so


you don't see a difference. Now, bird here is kind of close to plane, but you can change this into


anything, literally anything you want, you can change this into banana, or, I don't know, dog,


or any class you want, using these techniques. So it's not about being close, it's really kind of a


separate phenomenon. So that's adversarial examples. And many frameworks have been proposed in order


to explain these adversarial examples. And they make a they make a nice overview right here.


Many have been proposed over the last eight years that DNNs are too nonlinear, that they're too


linear, that they were trained with insufficient number of training examples that are just rare


cases where they err, that images contain robust and non robust features, etc. They say, however,


none of these vague qualitative ideas seem to provide a simple intuitive explanations for the


existence and bizarre properties of adversarial examples. So that is pretty harsh criticism,


specifically, the first ones are kind of Yeah, but specifically, this last one that images


contain robust and non robust features, which is sort of the leading hypothesis right now,


of why adversarial examples exist and what they are. And then you're saying none of these can


none of these vague qualitative ideas seem to provide a simple intuitive explanation for the


existence. Like, let's see whether or not they're gonna do better. Okay. So also, in the abstract,


they go on and they say, okay, they introduced this new conceptual framework, which they call


the dimpled manifold model, which provides a simple explanation for why adversarial examples exist,


why their perturbations have such tiny norms, why these perturbations look like random noise,


and why a network which was adversarially trained with incorrectly labeled images can still


correctly classify test images. Now, this last part, if you're not familiar with the literature,


it might come to you a bit random, this Y network, which was adversarially trained with incorrectly


labeled images can still correctly classify test images. This is a famous experiment from the group


of Alexander Modri, where also this this hypothesis, this one, the robust and non robust feature comes


from. And any, any attempt at explaining adversarial examples after this paper has to explain why that


experiment makes sense, because it's kind of a non intuitive experiment. And we're going to get


to that as well. But just so you know, that's why they write it in the abstract. Now, I personally


think they don't have a good like this model here doesn't have a good explanation for why that works.


They're sort of hand wavy trying in any case. So they say in in the last part of the paper,


we described the results of numerous experiments, which strongly support this new model. And in


particular, our assertion that adversarial perturbations are roughly perpendicular to the


low dimensional manifold, which contains all the training examples. Okay, also remember this


experiment, they strongly support what, in particular, the assertion that adversarial perturbations are


roughly perpendicular to the low dimensional manifold, which contains all the training examples.


Now, remember this, that the experiments are supposed to support this particular claim,


because also that is going to be important down the road. Okay, so let's get into the dimpled


manifold model. What is it? What do these authors propose? And I'm going to try as best as I can to


say what the authors are saying in the paper. So they claim that there is an old mental image


of adversarial examples. And the old mental image is is here, it's they say we think the old mental


image is based on the highly misleading 2d image on the left side of figure one. And that's this


thing right here. So the old mental image is that there's a there is a data space, right, this here,


if you think of pic of images as data points, this would be the pixel space, right? So this is


images with two pixels right now, in this conceptual framework. But you have to sort of


think yourself into higher dimension. So they claim the old mental image is the following you


have sort of the data distributed somehow in this space, the data being the all the set of natural


images or images you consider, which is kind of these the subspace, these subgroups right here.


There are a bunch of images right there and there, and also there and there. So these are


images of two different classes, the red class and the blue class. Now, they're distributed like this,


and what is a classifier supposed to do? A classifier is supposed to put a decision boundary


between them. And that's what they draw in here. So this would be sort of a reasonable decision


boundary between the two classes, right? So now what do you do if you want to create an adversarial


examples? Well, necessarily, you have to start at an image of a class, this one maybe, and you have


to cross the decision boundary, right, you want to fool the classifier, ergo, necessarily, by


definition, you have to cross the decision boundary. So what do you do? The the easiest way to do this


is to sort of go straight towards the decision boundary, which is approximately in this direction


right here. And then once you cross the decision boundary, you are done, you're on the other side,


you have created an adversarial example, provided, of course, that the image still kind of looks like


the original image. So they say, this has this has many, many problems. Here, they say the in this


mental, this mental image adversarial examples are created by moving the given images along the green


arrows towards some kind of centroid of the nearest training images with the opposite label,


in which they mean this, this thing right here. So we would move the images towards the other class


towards images of the other class. And they say, as stated, for example, by Ian Goodfellow,


in his lecture, at this time, I'm going to cut this in right here, I've said that the same


perturbation can fool many different models, or the same perturbation can be applied to many


different clean examples. I've also said that the subspace of adversarial perturbations is only about


50 dimensional, even if the input dimension is 3000 dimensional. So how is it that these subspaces


intersect? The reason is that the choice of the subspace directions is not completely random.


It's generally going to be something like pointing from one class centroid to another class centroid.


And if you look at that vector and visualize it as an image, it might not be meaningful to a human,


just because humans aren't very good at imagining what class centroids look like.


And we're really bad at imagining differences between centroids. But there is more or less


this systematic effect that causes different models to learn similar linear functions,


just because they're trying to solve the same task.


Okay, so it really appears like Goodfellow says this thing right here. However, they claim now


they claim this doesn't make sense. So they claim that you should think about adversarial examples


in a different way. And this is their dimpled manifold hypothesis. So what is their dimpled


manifold hypothesis, they say, what you have to do is you have to think about the data manifold


in the higher dimensional space that the handy higher dimensional input space. So in this case,


they consider instead of here, this 2d landscape, they consider the 3d landscape. So this would be


the pixel space, right now we consider three pixel images. And the data is embedded in a low dimensional


manifold in this higher space. So because if you think about all combinations of pixels that are


possible. So not all of them are natural images. In fact, only very few of the possible combinations


of pixels are natural images or images that make sense to you as a human or are images that you


could potentially generate by going out with a camera. So the data you're considering lives on a


very low dimensional manifold in this big space, and you have to explicitly think about that. Now


the data is the data manifold here is represented in this in this sheet in the middle. And on this


manifold, you're going to have your different classes of data here, the blue or one class and


the red are the other class. What this paper claims is that what classifiers do what neural


networks do when they classify the training data here is they go and they lay their decision


boundary instead of so in the old model, you would have thought maybe something like this happened,


where you put your decision boundary sort of in the middle between the two classes, right,


crossing the manifold right here. So you sort of put it in the middle between the two classes.


And then when you have to create an adversarial example, again, what you would do is you would


maybe start here, what you would have to do is you would go straight towards the decision boundary


right here, okay, crossing the decision boundary, and then on the other side, you'd have an


adversarial example. In this new model, what they claim is the decision boundary actually doesn't


look like this right here, okay, the decision boundary actually is very much aligned with the


manifold of data, as you can see right here. So this mesh that they show is the decision boundary


now. And their claim is that that usually just aligns with the manifold of data. However,


around the actual data around the training samples, what the classifier will do is it will


create these what these dimples, okay, and these dimples are just tiny, well, dimples, tiny


perturbations in the decision manifold, such that the data is on the correct side of the decision


manifold, sorry, of the decision boundary, right, so the blue points here are under or one side of


the decision boundary and the red points are on the other side of the decision boundary. And for


the rest, the decision boundary just aligns with the data, the data manifold. Now, if you want to


make an adversarial example, now, what you have to do again, you start from an image. And again,


you walk straight towards the decision boundary. However, now, you don't have to go like this,


you see what you can do is you can go simply perpendicular to the data manifold. And you will


cross the decision boundary very quickly, because the dimple you're in is kind of shallow. And they


give a reason why the dimples are shallow, because they claim this is results from training these


models. And that explains something. So the difference is the difference is we started out


from this, to make an adversarial example, we have to go towards the decision boundary, okay,


if we sort of transfer this image into higher dimensions, it looks like this in the middle.


Again, in order to make an adversarial example, we have to go towards the decision boundary. Now,


in the old mental image, going perpendicular to the decision boundary means walking on the data


manifold because we walk from this group of data towards this group of data. Okay, you can see


right here that we're walking on the data manifold, when we walk perpendicular to the decision


boundary, whereas in the new model, walking perpendicular to the decision boundary coincides


with also walking perpendicular to the data manifold. So this is the difference right here,


that they that they claim. So this they say there's, we call this conceptual framework,


the dimpled manifold model. And note that it makes three testable claims about the kinds of decision


boundaries created by trained deep neural networks. First, natural images are located in a k


dimensional manifold where k is much smaller than n. Second, deep neural network decision


boundaries pass very close to this image manifold. And third, the gradient of the classifications


confidence level has a large norm and points roughly perpendicular to the image manifold.


Alright, so these are these are the claims that they're going to make to be tested and to be


supported by experiments, I guess. So I hope I've represented enough what the authors claim right


here. I hope they would agree that I've represented this is accurately. So now where is the problem


with this, in my opinion, the problem isn't necessarily with what they claim right here.


It's, it's, you know, I don't necessarily disagree with this mental image, I don't necessarily


disagree with these claims. In fact, that the data is on low dimensional manifold, this we've,


this is kind of commonly agreed upon assumption, right? As I said, not all the possible pixels


combinations make good natural images. And that the fact that it is then a manifold is a commonly


held assumption. decision boundaries pass very close to the image manifold. Well, the fact that


we can generate adversarial examples, right, already means that decision boundaries pass very


close to the image manifold. So this also is not news, this this has been like, in everybody's


conceptual framework for the last five years, at least. And then third, the gradient of the


classifications confidence level has a large norm, and points roughly perpendicular to the image


manifold. And this claim right here, I'm pretty pretty sure there. So this is not a trivial claim,


which Yes, okay, this is not something that was like, set around much. However, I'm going to


claim that their model is not the only model by far that makes this happen, or any something like


this. Specifically, when we go look at the experiments, I'm going to show you that this


doesn't necessarily support their claims. It doesn't disprove them, right, but it also doesn't


necessarily support them just because they show that. Okay, so the other problem I have with this


is that this in this thing they build up as whoo, this is this is the old mental image. This is how


people thought about adversarial examples. Until now, I look I disagree. Like this. It's a bit of


a it's a bit of a straw man, almost I feel like this, no one, no one thought no one that is sort


of in the literature of adversarial examples, thought or thinks that this is an appropriate


model for what is happening. Like we know that these distances here are very small, right, the


distance until you cross the decision boundary. And we know also, like if this were true, you should


just be able to go to decision boundary, and then go the same distance, right. And then at some


point, you would actually arrive at a sample of a different class. So you could you could actually


transform images into the other class, by simply going into the adversarial direction, which is


precisely what we don't see, right, we see the image still largely looks the same, what gets


added looks like a bit of noise, okay, so no, no one was having this mental image, because clearly,


this mental image is it is not appropriate for adversarial examples, as well as saying, look,


if you think of this in sort of higher dimensions, and I realize I've drawn this decision boundary,


but this is what they describe in the text, then I don't I don't see that this is the correct way


of like there are many different kinds of decision boundaries that are compatible with with the


decision boundary right here. By the way, this decision boundary I drew doesn't even separate


the classes all the classes correctly. What I'm saying is that also, if you consider the decision


boundary that for example, looks like out of colors, looks like this, that also crosses here.


However, it's sort of kind of flat like this, but it's still a linear decision boundary, right?


Like this, okay, so this is above, and the other part is below. If you think of this, if you project


this down, it looks the same in 2d. And in 3d, it also explains that decision boundaries are very


close to the data samples. It's a bit different though, than this dimpled manifold hypothesis,


right? If you, I think the, at least in my estimation, what's happening is much more that


you have just a bunch of these kind of linear decision boundaries flying around right here,


partitioning up the space and so on. And this might result in a similar situation as here,


but it has quite different predictions in form of what it does, then what it does right here,


here, it's sort of a flat manifold, the dimpling around the data, whereas here, it's kind of the


classifier separating the space into many regions, always trying to sort of distinguish one class


from the other. And yeah, so might end up a bit the same, but I don't think they give a fair shot


at what we know so far. Like, we that this model is not a model that people hold in general,


especially the one on the left, I can make an attempt at making a mental model that people hold


so far, maybe it's just me. But I have a feeling this is a bit more. So the model that I call,


let's call it something because they call it there something right, I call mine the squishy,


the stretchy feature model. Okay, let's contrast this with the stretchy feature model. So what I


want to do is I have two features, and this is a coordinate system in feature space. Okay, so there


are two features this in feature space, I mean, sort of the last representation before the


classification layer in feature space, the two classes look like this. So there is the red class,


and there is the blue class. And you can see right here, there are two features. And for some reason,


the network can classify along these two features, maybe because there are other classes, other data


points. So we can't put a decision boundary like this, between the two, we can classify along the


two features. Okay, so you can see there are two features right here, feature one, and feature two,


and both features are actually pretty good features for keeping these two data points apart. Okay,


now there are empty spaces, as you can see right here, which we're going to get to in a second.


But you can you can use both features. And ideally, a classifier would actually use both features,


it would say, you know, if feature one is high, it's the probably a red class, if feature two is


low, it's probably the red class. And the combination makes even more of the red class.


Okay. However, since we're in a deep neural network, which is has transformations, it transforms the


data along the way, if you look at the same situation in input space, so in the actual pixel


space, it looks different. And this is due to not necessarily the non linearity of things, but


actually, it is due to the linear transformation is actually the problem of adversarial examples,


at least in my estimation appears to happen in the linear layers. If you think of, for example,


like eigenvectors of matrices, and the largest eigenvalues determine how far you can go in a


particular direction by having a sort of a standard input, delta. And the same happens here,


by the way, this is why spectral norm regularization tends to work at least a little


bit against adversarial examples. So what I mean is, if you look at the scale of these features,


right, they are like 12345 of this features 12345. If you look in the input space, some of the


features are going to have roughly the same scale right here. And these features are going to be


features that you have to change the input a lot in order to change the feature a lot. What do I


mean by this, this is something like the shape of an of an image, okay, if you think of a cat,


the general shape of a cat, you know, it has it has two ears pointy, it has a head, it has a


head pointy, it has a head, and so on. That's the general shape of a cat. Sorry, that is actually


the left right feature, right? This is the the left right feature is the shape. And I have to


change the input a lot in order to affect the feature, right? So they're roughly on the same


scale of what I have to change to change the feature. However, the other the other feature


in the input space has a much different scale than it has on in the feature space. And this


might be something like the first structure of the cat. So the first structure of a cat, like is I


can change the pixels a tiny bit. And I'm going to change the first structure by a lot, I can change


the first structure of a cat to the first structure of a dog, by just changing the by just changing


the pixels a little, however, it will be different. And now it will be the first structure of a dog.


So how does this change now in input space in input space is going to look something like this,


where one feature dimension is going to look rather the same. And the other feature direction


is going to be very, very stretched. Okay. Now remember, both of these features are good features.


They both can be used to read to classify the images. So you can see changing the shape requires


a lot of pixels changing the first structure however requires just a little pixel. Now if I


take some image, and I draw an l two ball around it, which was what we usually do when we create


an adversarial example, we say, only, we only allow small perturbations, you can see that in in this


in in this direction, it's a very, you know, you don't get very far in feature space. But if you


go the same distance in the in the input space into this direction, in the feature space, you're


going to walk a lot, you're going to walk like way far. And this is just by definition, there are


going to be many features that you can use to classify images, and they're going to be good


features, they're not going to be errors or aberrations, like the first structure is a good


feature to classify a cat, there are going to be many features in there. And some of them are going


to be of large magnitude, and some of them are going to be a small magnitude. And this is just


what happens, okay. So I call this the the stretchy feature model. And this is sort of a direct result


of this paper that they cite by Alexander Modri's group, which we're going to get to in a second.


Right, but keep those two in mind. And we're going to see how which one explains the phenomena


better, and which one doesn't. Okay, so they say, why deep neural networks are likely to create


dimpled manifolds as decision boundaries. And the the idea here is that, okay, we have to now explain


why this even happens. So if you consider the data manifold in green right here, and here we have


just one dimensional data, and you can see it's not linearly separable, right, so we have to have


sort of a curved decision boundary around this. And why would this result in a dimpled manifold?


So they say, look, if you start off your your deep neural network training, you're maybe your


decision boundary is going to be somewhere like here, okay, not very effective. What's going to


happen is, let's say what you want, what you want is you want to have the blue data, you have the


blue data above, and the red data below the decision boundary. So right now the red data is,


is, oh, that's the other way around. The red above, and the blue below. So right now the blue


are fine, like the blue don't complain, you do get a gradient out of the red examples pushing the


entire decision boundary down, there's no resistance, right, the blue ones, they, they're


they're fine. So you're going to push down, this is your next decision boundary. Okay, same


situation, you're going to push the entire decision boundary down. Now you're here. Now you're too far.


So you're going to push the entire decision boundary up, because now the red ones are fine,


the blue ones complain. And this results you being sort of right on top of the data for once,


okay, and then both gradients kick in. So now the red data are gonna push such the decision boundary


down the blue data are going to push the decision boundary up, which is going to result in this


sort of dimples around the data. Otherwise, the decision boundary coinciding with the data, okay,


this is their explanation for why the why this works. I hope this makes a little bit of sense.


Now. Yeah, so they claim that that this is happening. contrast this with the mental model of


having a bunch of linear half spaces, which would result in something like you know, a decision


boundary being through here, a decision boundary being through here, a decision boundary being


through here, and through here, through here, which would also explain what we see. But this is


their claim why this decision boundary looks the way it is. To me, it's, it's a bit it's a bit weird,


right? Like here, why should the decision boundary align with the data manifold? Maybe it doesn't,


maybe they don't, they don't claim that I should not complain about this. But, for example, in


between the data, why does it do that they give some examples right here, the decision boundary,


it should be rather simple, right? It doesn't like to curve a lot. They say, the new model can help


to understand why the training phase of a given network typically converges to the same global


optimal placement of the decision boundary, regardless of its random initialization,


they're going to make a claim right here, why this happens. To demonstrate this point,


consider the old model in which you sprinkle at random locations in the two dimensional square


a lot, as the large number of classes depicted in figure three, sorry, I'm, I was confused for a


second, I am no longer. So they're talking about this figure right here, they say, look, in the


old model, you have, if you want to pass sort of simple decision boundaries through this, you have


to sort of pass them, like some of the gray ones we see right here. And they are not going to be


so good. Okay, so our goal is to pass a decision boundary of bounded complexity. And this bounded


complexity comes up again and again, they claim, of course, their decision boundary is very smooth


and very simple, which will best separate the red and blue clusters. They say there is a large


number of way to do ways to do this, like the green lines, and most of them will be about equally


bad. In particular, any decision to pass one side or the other of some cluster can make it harder


to accommodate other clusters elsewhere along the line. Consequently, there will likely be many


local minima of roughly the same quality. In the dimpled manifold model, however, there is likely


to be a single globally best decision boundary shape, since there is no conflict between our


ability to go above one cluster and below a different cluster when they do not intersect.


So their idea here is that rather putting the decision boundaries like this, what they want to


do is you look at this in three dimensions, and then they kind of put a sheet over top of it and


above the blue ones, and they're below the red ones in all of the three dimensions, right. So


you go above the blue ones and below the red ones, rather than this these gray things like here,


which are not very optimal. Now, this one, I'm not really sure what to make of this because


first of all, they say it typically converges to the same global optimal placement of the decision


boundary regardless of random initialization. We know that this is not true, right. I've


specifically made videos on research by Stanislav Ford, who shows that if you randomly initialize


a network differently, what it will happen is you will reach the same accuracy, but it will,


it will make mistakes on different samples of the test set, right. And there's actually a


structure to how these decision boundaries are going to be different, depending on your random


depending on your random initialization, which actually would support what they claim is the


old view right here. Second of all, I have no trouble making a decision boundary here that


separates red and blue, right, I can go something like this, like this, come here, okay, you get


here, right, I have no trouble separating red and blue, I guess this should go here. So there,


this this kind of, this kind of bounded complexity does a lot of work here, them saying,


well, the decision boundary should be simple, and so on. And that's why they insist that


these decision boundaries should be somehow straight. But then a lot, but I disagree that


their decision boundaries are so simple. If you have to curve around every data sample,


and otherwise follow the image manifold, that seems to be like a rather complex decision boundary,


honestly. Because it's, it's, it's kind of a generative model of the data, right? If you


follow the data manifold. So I disagree that there's is so much simpler, right, just because


it doesn't bend that much. And here it like bends a lot. That's also something they say, like you,


you don't want to bend decision boundary so much that hardens training. And third of all, why do


they give their model the benefit of the third dimension? Right? So they claim like, oh, look,


the old model doesn't work. Because if you have to place decision boundary between the data points,


you're going to end up with a bad decision boundary. However, in order for their model to


work, they need the third dimension, they need to pass like, under and over the data in the third


dimension. Whereas, if you actually go into the third dimension, you know, every single lecture


you have on kernelized SVMs and whatnot, they show you like, if you go in higher dimensions,


these things are actually separable, like you would make, if you have like RBF kernels, these


would become a cluster, these would become a cluster, and so on. This is sort of the first


lecture on going into higher dimensions in order to linearly classify stuff. So it's not like their


method can explain anything more than any other method, if you give it this third dimension. And


the fact that they don't give the old model the third dimension, but they give themselves the


third dimension in order to explain it is a little bit, I'm not I don't know, it's like,


yeah, so I don't think this is any argument for their model, it just simply shows that if you have


a lower dimensional manifold of data, and you classify it in a higher dimension, there are ways


to do that, right. And if you like, if you have relu networks, and linear classifiers,


it's going to look like more chunky, it's going to kind of divide the space into these kind of relu


cells where you classify the data. All of this is compatible with what they're saying, not just their


dimpled manifold hypothesis. Alright, so this is, yeah, I don't, I don't see the big explanation


here. So they claim, what can they explain with their model explaining the mysteries of adversarial


examples, okay, there are five things they claim they can explain with this. First of all, the


mixture mystery, right, how can it be that a tiny distance away from any cat image, there is also an


image of a guacamole and vice versa. Okay, if these and if these classes are intertwined in such a


fractal way, how can a neural network correctly distinguish between them? Our answer is that all


the real cat and guacamole images reside in on the tiny image manifold, but below the real cat images,


there's a whole half space of pseudo guacamole images, which are not natural images of guacamole,


and above the guacamole images, there's a whole half space of pseudo cat images. So their idea


here is that, okay, you have this one dimensional data manifold. Here are the cats, here the


guacamoles. If you have your dimpled manifold curving sort of around the data right here,


you know, all of this is technically guacamole. So if you go from the cat to here, you reach a


non natural guacamole image just by the fact. So the explanation here is that the explanation is


that this, this, the decision boundary lines up with the data manifold, except around the data


where it creates a small dimple. And therefore, you can cross the dimple into the other region.


Okay, you this is very, it's the same effect as this model right here, you know, I can draw this


dimpled manifold, I can draw it right here, right, if I classify the image, I can draw this dimpled


manifold, I get the same effect. However, this model here explains much more, it actually


explain like here, there is no reason if you think about a multi class setting, right? If you think


of this in two classes, fine. But if you think of this in a multi class setting, there is no reason


why this region right here should be guacamole, it can be any other class, right? If the if the


idea is the decision boundary follows the data manifold, and then just dimples around the data


to make the data correct, classically classify, the only constraint here is, is that these are


cats. It says nothing about sorry, it says nothing about why on the other side, there is guacamole


instead of anything else. And that does not coincide with what we know about adversarial


examples. Like this region here is a consistent region. What? So first of all, first of all, my


bigger problem is why does this even generalize? Why does the dimpled manifold hypothesis even


generalize? Right? Like if it follows the, if it follows the data manifold, largely except around


the training data? Why does it exactly generalize well to test data, you have to like argue that the


test data is here quite close, because otherwise, it would be it would get very confused on test


data, which would be somewhere else on the manifold, right? But we know that generally


neural networks classify data that's on the manifold of natural images quite well, they


generalize quite well. However, this model is sort of an anti generalization model. But okay,


maybe you can claim that their test images are close enough to the training images such that this


works. But for example, we know that if that this, this is a consistent region, what do I mean by


this? We know for example, we can make universal adversarial perturbations, which means that we can


find directions that no matter from which image or from which class we start from, they will always


result in guacamole. This is not explained by the dimpled manifold, there is no reason why these


regions on the other side should be of a consistent label in a multi class setting. We also know that


adversarial perturbations are transferable, which means that we can make an adversarial perturbation


in one classifier. And then in a different classifier, even if it's trained with a different


data set, actually, we can, we can apply the same adversarial perturbation, and it will most likely


still be of the same like the adversarial perturbation going towards the same class,


there is no reason in the dimpled manifold hypothesis that explains these phenomena. If you


think of this of the stretchy feature model, this is really easy, right? If I create an adversarial


example, I go across the decision boundary right here, what do I do, I change the fur


without changing the shape. Now I change the fur by so much that, you know, now there is a conflict


right in feature space, I go up here. Now there is a conflict, it has the fur of a dog, but the shape


of a cat still. Now I, there is a conflict, but neural networks in the final layer are linear,


which means they just weigh the different features. Now I just pumped that fur to be so doggish,


right, that it overpowers the shape feature of the cat neural networks are biased towards sort of


structure anyway, over shape already. So I just I just hammer that fur. And now the neural network


thinks it's, it's a dog, and a different neural network trained on the same data will also think


it's a dog because it will also have learned to classify images by shape and for therefore,


therefore, it will it will be vulnerable to the same attack, right? This is super easy to explain


in this model, there is no reason why this should happen in the dimpled manifold model,


unless you amend it by some more hand wavy things. They say, the direction mystery.


When we use an adversarial attack to modify a cat into walk model, why doesn't the perturbation


look green and mushy? Okay, so they say, well, in the old model, you would have to walk along


the image manifold from here towards the guacamole images. And that should mean that your image should


sort of change to look like a guacamole. In our in the dimpled manifold model, you go off the


manifold perpendicular. And that explains why the adversarial perturbation looks like a little bit


like just random noise. Again, no one thought this in the old model. In fact, we have a pretty good


explanation why it still looks the same. And that's because humans are much more receptive to this


thing right here to the shape, whereas neural networks also or much more consider this thing


right here, the fur. So they consider fur and shape in different proportions than the humans do. And


so that's we already sort of knew this. And it's, in fact, a better explanation.


The uniformity mystery, you know why the decision boundary is ever present. So they claim because


the there's this dimple right here, even you know, the most far away cat image here has a close


crossing to the decision boundary. So there is no cat images that are kind of closer to the decision


boundary. But this is I think this just a property of a high dimensional classifier, I think that


here our 2d view of the world betrays us. And yeah, especially if we can go really far in feature


space with a tiny perturbation and input space. This is not not a mystery, not even a mystery,


the vanishing gap mystery. Okay, which is about adversarially training, I think, which we're


going to skip here. And then there is the accuracy robustness trade off mystery. So this is if you do


if you train a model adversarially, which means that here, look here, I have my cat, okay, I train


I have a data set of cats and dogs, I train my neural network on it, it's vulnerable, what can I


do? What I can do is I can create adversarial images, this is a cat, right, I can create


adversarial images by making this into a dog. Okay, so this is a dog because I changed the first


structure a little bit. This is an adversarial example. Now I add this. So this is comes from


the data set. Now I add this to the data set. But I tell it this is a cat to write, this is a cat,


and this is a cat. If I do this with my neural network, the neural network will become robust


to adversarial examples to a degree, not fully, but to a degree, this is the best method we have


so far of defending against adversarial examples called adversarial training. Now, what you do,


when you do this is you train the network to to sort of classify the advert to did Yeah, classify


to incorporate the adversarial ness into its decision making process. And this results usually


in a degradation of the generalization performance of the network. So as it becomes more robust,


it becomes less accurate on real data, right, you gain accuracy on adversarial data, you decrease


the accuracy and real data, which makes sense intuitively, but it is a strong effect, which is


not the same as you know, I simply teach my model to do yet another class. It is quite it is actually


a trade off. Now, they try to explain this right here. When we train a network, we keep the images


stationary and move to decision boundary. By creating dimples, when we create adversarial


examples, we keep the decision boundary stationary and move the images to the other side.


By allowing a large perpendicular derivative, we make the training easier since we do not have to


sharply bend decision boundary against around the training examples. So this is when you train


normally. When you train without adversarial examples, they say there is a large perpendicular


derivative, which in the like the what they mean is that the data samples sort of push these dimples


out that that's the large perpendicular derivative, the perpendicularity is to the image manifold.


And that makes it easy because you don't have to bend the decision boundary a lot. So you can kind


of remain here, and you have to kind of create these dimples. Again, their argument is you don't


want to bend this boundary a lot, which makes training easy. However, such a large derivative


also creates very close adversarial examples. Yeah, this is their claim that now the decision


boundary is pretty close because you don't bend the decision boundary by too much around the data


because you do dimples. Any attempts to robustify a network by limiting all its directional


derivatives will make the network harder to train and thus less accurate. I'm not super sure


how to interpret this. So I might be doing this wrong right here. But if you create adversarial


example, what you do is you essentially have this data point and you create an adversarial example,


this data point is here. Well, these are of the same class. So now the now the D, the decision


boundary has to sort of bend harder, okay, which makes it more hard to train. And at some point,


it so it's harder to train. And that's why you have less accuracy. And at some point, it says,


Well, actually, I don't want to bend that much, I'd rather make a mistake here, and just bend


around both of these data points. And now you have a wrong classification. So that's sort of


their explanation of why this happens, which I find a bit handwavy, you have to argue like,


ooh, ease of training, bending the decision boundary, and so on. In this model right here,


super easy, okay? What happens if I create cats that have cat fur and dog fur, and I tell the


network, these both are cats? Well, essentially, I tell them, I tell the network, look, there are


two features right here, the fur and the cat. And you know, the fur just, just disregard it, just


don't do that. Don't regard the fur as a feature, because it's useless now, because I now have cats


with cat fur and cat with dog fur. So the network can't use that to classify anymore. And that


explains why it gets less accurate, because I take away one useful feature, okay? So, you know,


now the network has less useful features. And that's why it gets worse. This, it's a pretty


simple explanation in the stretchy feature model. It has, there's a lot of work to make this happen


in the dimpled manifold model. So lastly, they try to explain and they became an interesting mystery


in this this paper that I have cited throughout. And what that is, is that it's kind of the same


experiment as here, where we create adversarial examples, and we add them to the training set,


except for two things. First of all, we don't have the original, so our new data set is not going to


contain the original images, it's only going to contain the adversarial examples. Second,


it is going to contain the adversarial example image. But the label isn't going to be the correct


label, quote unquote, correct, from where we created, but the label is actually going to be


the adversarial label, the wrong label. Okay, so we're going to tell the network, this is a dog,


please learn that this is a dog, right? It's a cat with dog fur. And the old training images are


nowhere in the data set, we just do a data set with these wrongly labeled images. Now, when we go


and we apply this, we train we use this, we train a network right to classify cats and dogs. And now


we once we've trained this network, we go, we take one of these samples of the original data set,


we classify it, it's going to give us a correct classification, right? So we're going to take


classification, right? So it will recognize that this here is a cat, even though we told it that


this here is a dog. Now, how does it do this? It does this by looking at the fur, you know, we've


we've doubled down on the fur here, right? So this is like, we really made that fur feature super


strong in these adversarial examples. So it's going to look at the cat fur. And even though


none of the cats had the shape like this, we sort of, we sort of supercharged that fur feature. Again,


in this model, not a problem. Essentially, what we've done is we've created two data classes,


you know, one up here, one down here, that have the first supercharged. And now it's just going to


mainly look at that first structure. And that is a useful feature, right? So this, this what's called


what's called the features not bugs paper, adversarial examples are features not bugs,


or other way around not bugs, they are features has demonstrated with this experiment, this notion


that there are adversarial examples result from useful generalizing features in the data set


that are simply of, by definition, the features that are not large enough for humans to see


what they call non robust features. How do they explain this? They say the original people try


to explain this highly surprising growth by distinguishing between robust and non robust


features in any given image, where some of them are preserved by the adversarial change, and some


are not. However, it is not clear what makes some of the features more robust than others.


Definition, just definition, like, like, if you have features and you order them by their size,


like by their how much you have to change the pixels that some features are going to be larger


than other features. And then some features going to be below that cutoff where you define adversarial


examples budget, this is definition makes them such that some of more robust it's not it's not


clear. Our new model provides very simple alternative explanation, which does not necessarily


contradict the original one, okay, at least this, which is summarized in figure four. To simplify


the description, we will use 2d vertical cut through the input space and consider only the


decision boundary that separates between cats and anything else. Okay, so they have this example


right here. They say, look, we have a decision boundary that distinguishes cats, see from non


cats. And the green one here is the image manifold, and the gray is the decision boundary. Okay. So


now what we do is we create adversarial examples in frame two right here, you can see that we make


the cats into non cats and we make the be the bats into bats aren't very popular lately, the


badgers into into cats. So we make the badgers into cats, and we make the cats into these,


whatever DS ducks. Okay, and now we relabel those. And that gives us a new data manifold. So the new


data manifold is this data manifold right here. And we have also new labels. And now they claim


the resulting decision boundary in figure four, as you can see right here, this is the resulting


decision boundary, the gray one, it is, it is very similar to the decision boundary in the first


frame. And therefore, we shouldn't be surprised that this new decision boundary that results from


this perturbed data results in the same decision boundary as the original one. Okay, however,


like why, like why. So their whole, they have two notions. Notion one is that the decision boundary


follows the data manifold closely, except it sort of bends around the data a little. And you can see


this right here, like this decision boundary kind of follows the data, yet it just happens to be on


the correct side of the data points at any given moment, which, okay, we can see that the decision


moment, which, okay, okay. However, they also make the claim in different parts of their paper that


bending the decision boundary and so on is not good, you'd rather want to have a simple decision


boundary. So to me, there's no reason why the decision boundary couldn't just look like this,


it would correctly classify this new data set, right? However, it would not correctly classify,


it would not correctly classify the, let's say, the C that was right, where was it right here,


or right here, these data points, it would not correctly classify. So you see that this,


until now, they've always had this data manifold to be sort of super duper straight and smooth. And


that's how they can also say, well, following the data manifold and not bending too much and


so on. Those are not in conflict with each other. But now that they are in conflict with each other,


you have to give it going to give up one or the other. And only in one of them do actually does


this experiment here still make sense in the other one, it doesn't. And but if you give up the,


ooh, bending too much is bad, then, you know, you lose a bunch of explanations that you have up here.


So, yeah, like it's one in my mind, it's one or the other. And there's, there's still no reason,


I think no good reason why this like the decision boundary should align super closely with the data


points. Like if there, if there is nothing here, right? If this is perpendicular, really, to the


data manifold, like why would the decision boundary align so closely with this data point?


Why would the decision boundary align so closely with the data manifold in that point? I don't know.


Okay, so they ask, why are DNN so sensitive and humans so insensitive to adversarial perturbations?


Essentially, their argument here is that humans project the input data onto the image manifold,


which is not a widely accepted claim, right? I don't, I don't think that is a,


I think that is not not a widely accepted. I mean, it's, it's certainly possible. But also,


I'm not sure, I'm not sure that humans do project, they have like an internal manifold of natural


analysis and image. And also, also, the Yeah, how do you project, right? Like, like both of these


features are useful. Okay, so both of the features are useful. If you project an adversarial example,


like why do you project it onto the shape dimension and not onto the third dimension,


right? Why? There's no explanation right here. We know that sort of humans are more receptive


to shapes and so on. But just projecting won't get you there. So now they're going to into


experiments. And I want to highlight one particular experiment right here. They have


synthetic experiments, they have their experiments, I want to highlight this experiment right here.


Remember, they said their experiments are going to give, you know, strong support that, and this


experiment right here, what they want to claim is that okay, you have the data manifold here,


if you are if you have a data point, and you make an adversarial example, the question is,


do adversarial examples go along the image manifold? Or do adversarial examples go sort of


perpendicular to the image manifold? They, they, their claim, again, is that the this here would


give support to the old view of adversarial examples. And this here would support the dimpled