[Drama] Who invented Contrast Sets?

GANs basically the same thing so yeah so if this is it's not the same like I have


[Drama] Who invented Contrast Sets?

to say this is very close but it's not as I understand they even cited the


[Drama] Who invented Contrast Sets?

other ones so then the bickering starts and this is funny I'm like this this is


[Drama] Who invented Contrast Sets?

just funny to me so Zach Lippen jumps here it says this has been published has


[Drama] Who invented Contrast Sets?

a name and a clearer justification it's called contrasted data here is the


[Drama] Who invented Contrast Sets?

published paper we just looked at this right and then Matt Gardner answers and


[Drama] Who invented Contrast Sets?

he says Zach and divinish work is sorry and divya deviance work is excellent


[Drama] Who invented Contrast Sets?

recommend you all go look at it our provides a different concurrent take on


[Drama] Who invented Contrast Sets?

similar issues right and I think here someone comments that so he says it is in


[Drama] Who invented Contrast Sets?

the related work section although mischaracterized and misattributed as


[Drama] Who invented Contrast Sets?

contemporary work so position is really that it is kind of a stolen idea and


[Drama] Who invented Contrast Sets?

they were apparently in contact with each other during that so this Matt


[Drama] Who invented Contrast Sets?

Gardner here says what the differences are he says we take a geometrical view


[Drama] Who invented Contrast Sets?

we demonstrate solution a wider variety I mean for all intents and purposes if


[Drama] Who invented Contrast Sets?

you go through any of the research go to computer vision go to NLP you'll find


[Drama] Who invented Contrast Sets?

like the same exact I have like I have I reviewed two papers each year that want


[Drama] Who invented Contrast Sets?

to produce data that better defines the decision boundary like these people here


[Drama] Who invented Contrast Sets?

I mean this is this idea is just get rehashed over and over in slightly


[Drama] Who invented Contrast Sets?

different form these two are particularly close but and then they had


[Drama] Who invented Contrast Sets?

the paper was finished two months after theirs and then they say we started the


[Drama] Who invented Contrast Sets?

project well before and so on why do we feel defensive and then he answers again


[Drama] Who invented Contrast Sets?

with this is absolutely false our paper was drafted in July your paper was


[Drama] Who invented Contrast Sets?

finished the night before the ACL deadline this is not two months ago but


[Drama] Who invented Contrast Sets?

a half a year it is nothing to do it says why do you presume to know when we


[Drama] Who invented Contrast Sets?

started drop the nonsense we did this work in May 2019 present the public


[Drama] Who invented Contrast Sets?

results in July post it is a better drop the posturing so much of what you're


[Drama] Who invented Contrast Sets?

doing here is the very cancer in the system I mean I agree just you know


[Drama] Who invented Contrast Sets?

slightly refining ideas that previously were there is very bad problem in a


[Drama] Who invented Contrast Sets?

good email so this is actually correct to point out but I don't think that this


[Drama] Who invented Contrast Sets?

particular instance is particularly bad and this is I'm afraid you're simply


[Drama] Who invented Contrast Sets?

mistaken history of publishing similar so I've I've something like the last


[Drama] Who invented Contrast Sets?

thing I just invite you to read this beautiful the last thing to say here if


[Drama] Who invented Contrast Sets?

if this counterfactually augmented data if this is in fact the first instance of


[Drama] Who invented Contrast Sets?

this general idea to produce counterfactually augmented data that


[Drama] Who invented Contrast Sets?

that that does actually fulfill these criteria I would be extremely surprised


[Drama] Who invented Contrast Sets?

because this is nothing to do with deep learning right and the real novelty in


[Drama] Who invented Contrast Sets?

our field is mostly deep learning so I'm pretty sure someone must have thought of


[Drama] Who invented Contrast Sets?

something like this when everyone was just doing grammars and manual features


[Drama] Who invented Contrast Sets?

and things like this so I'm I would be extremely surprised if this hasn't been


[Drama] Who invented Contrast Sets?

there in one form or another and why the authors of that shouldn't make exactly


[Drama] Who invented Contrast Sets?

the same argument that being said it is fairly close like that the fun part here


[Drama] Who invented Contrast Sets?

is that it is actually a fairly similar idea except after so the idea itself is


[Drama] Who invented Contrast Sets?

fairly similar but here the focus is on different things and it's also on


[Drama] Who invented Contrast Sets?

different data sets and I believe yeah as I said 95% of research falls into


[Drama] Who invented Contrast Sets?


[Rant] Can AI read your emotions? (No, but ...)

We need to talk about your face or face recognition in general. A tweet has been


making the rounds saying facial recognition is able to analyze in real


time the emotions and feelings. Just that. And it showed a video of a apparent


real time system looking at people's faces and determining what their


emotions are. Now there is a predictable reaction of machine learning Twitter


with respect to anything to do with facial recognition and that reaction is


no. The biggest reaction is no, this is impossible. AI will never be able to


infer your emotions by looking at your face. This is the data is not there.


Anything like this. I just think that is really, really, really surprising.


Honestly, now look, facial recognition technology isn't exactly the most


popular subject. It's not going to win any Nobel Peace prizes anytime soon. Is


this technology dystopian looking? Yes. Is it dangerous in the wrong hands? Yes.


Does it work as advertised? Very probably no. Is it easy to be tricked?


Absolutely. Yes. However, saying that it is impossible for an AI to look at your


face and infer your emotional state. That is wondering me. You do this every day.


You look at people's faces and then you infer something about their internal


state. People splitting hairs here about the word analyze to analyze the emotions


and feelings. Well, if you want to split words, I would say inferring is a lot


heavier than analyzing your face has literally evolved to convey your


internal state. Other people have a trouble with saying, well, you can fake


your face. Not all facial expressions can be faked. A lot of what you tell with


your face is involuntary. And there is in principle, not a reason why a machine


cannot pick up on these cues. Now, this is not to say that this particular


system works well, it probably does not. It is extremely hard to do this to look


at a face and get how that person is feeling through all the deception that


might be there is an extremely hard task. But there's nothing supernatural about


it. We do this. We're a machine. Ergo, a machine can in principle do this. The


most criticism I see right here is that, well, the machine only analyzes facial


expressions, they have nothing to do with your emotions and feelings. What is


that? Of course, this has something to do with your emotions and feelings. Have you


ever thought to yourself, huh, that person looks kind of sad today? Have you ever


gone to someone and said, you know, you look a little bit down? Is everything


okay? No, never, never. And you certainly didn't infer this from their face. Hey,


doctor, I have a problem. Well, what's your problem? Well, I banged my foot and


now it hurts and it has a dent in it and it bleeds and it's swollen and everything


is bad about my foot because I hit it. And it might be brought, well, I don't


say it's broken because the external symptoms will never tell us anything


about the internal state of the system. I'm sorry, have you ever heard that an AI


can diagnose lung cancer by looking at a chest x-ray? Well, no, all we can say is


just that the AI detects a little bit of a spot and there is no correlation at


all. This is no indication of the internal state of the cancer. Shut up.


Twitter makes it such that everyone immediately is extreme on the one side


and extreme on the other side. Instead of saying the data to train the system is


very hard to get, the systems itself aren't as good. They don't understand


context that this happens in or nuances. That's very different from saying that


no, this is impossible. The most ridiculous is when people come out and


compare this to phrenology or literally call it phrenology. You know, phrenology,


the science of what bump on your head means something about your personality


or intelligence. Like my face has literally evolved to tell you something


about my internal emotions. None of the bumps on my head have evolved to


communicate about my intelligence. It is a predictable reaction for some reason


anywhere where facial recognition technology is used. There is a crowd of


people coming out saying phrenology faces are a real thing. Emotions are a


real thing. There is a real connection between your facial expression and your


emotions. It is more complicated than these machines right now can assess it


might require more context, more data, better algorithms, and even things we


don't have yet. But this definitely exists. It is not a pseudoscience. Not


everything that has to do with face recognition is a pseudoscience. It might


be dangerous, yet it's real. So in conclusion, I guess my message here is


that yes, this is probably an over promise of what AI can do. And it could


easily be used for bad purposes. On the other hand, this is not a pseudoscience.


This is not impossible. And research in this direction might actually lead to


something good. Imagine an AI that is better than a human at recognizing


emotions from someone's face. Assuming that is possible, we could avoid a lot


of conflict, maybe do a lot of good work in suicide prevention, and ultimately


communicate with the AI's as we would with other humans. Apart from all the bad


thing that we can do with facial recognition technology, ultimately, its


technology can be used for good and for bad. And for evil. I'll end with the holy


trifecta of broader impact statements, technology good, technology bad,


[Rant] The Male Only History of Deep Learning

Alright, so instead of reviewing a paper today, I thought I might review this thing.


So this person on Twitter posted this link to an article called brief history of deep


learning from 1943 to 2019 of machine learning knowledge.ai.


So let's look at this.


Actually let's look at the tweet first because this is I just saw this the male only history


of deep learning where you say Alex net makes history but image net doesn't because women's


contributions don't count and contributions from anyone except for white and white adjacent


people for that matter.


That is the tweet and it has 109 retweets over 400 likes and people generally agreeing


with this sentiment.


So the person is expressing concerns that the this article is only over one particular


group of people.


So let's look at the article.


They basically go over the history of neural networks of deep learning in an algorithmic


sense.


So let's check it out.


So first we go into neurons starting at 1943 and the perceptron paper right here.


The first back propagation algorithm from Kelly.


This actually I think people like Schmidhuber would be proud as far as I can tell.


This is kind of more of a forgotten history or some of these things are more of a forgotten


history of course.


Minsky's paper very famous but here back propagation attributed to this paper and so on.


And you can see things people like Hinton only coming up later here.


The Boltzmann machine back propagation in neural networks now.


So this as far as I can tell it's just a take on kind of the history of algorithmic development


and you can see here it really is about algorithms.


The algorithms behind deep learning.


So here is the vanishing gradient problem.


The LSTM as an architectural component.


Deep belief networks then you have GPUs for training.


Again vanishing gradients AlexNet and GANs AlphaGo.


So we're now going a bit faster and then the end it says the Godfathers win the Turing


Award for their immense contribution in advancement scenario of deep learning and artificial intelligence.


This is a defining moment for those who had worked relentlessly on neural networks when


the entire machine learning community had moved away from it in the 1970s.


So the article clearly is focused on algorithmic developments in deep learning.


Right.


And that's why AlexNet is here.


Now this person rags that AlexNet is here but ImageNet isn't.


And clearly you can see from the article ImageNet is a data set.


It was not made with deep learning in mind.


It was simply made as a data set.


It's not an algorithmic development.


So GANs are here as well.


But CelebA isn't.


C410 isn't.


MNIST isn't.


The PenTreeBank isn't.


So I think we've skipped a lot of architectural advancements here like transformers or all


kinds of all kinds of things here.


But the history is clearly about the algorithmic developments.


And to reframe this it clearly states ImageNet doesn't because women's contributions don't


count.


Right.


The insinuation here absolutely I find this to be absolutely intellectually dishonest.


And they say and contributions from anyone except for white and white adjacent people


for that matter.


At this point you just have to laugh.


Like because of course the narrative that the person wanted to tell was that it's only


white people that count.


But then you scroll and turn like it doesn't fit my narrative.


This this this GPU is not a white person.


So you to make it fit your narrative you have to call white.


What is white adjacent.


It's like it's whatever I don't like I now call white.


And but people just people just agreeing with this.


I find this absolutely disgusting.


And I find the article to be OK.


I don't know better.


But if you have a problem with I definitely think there is misattribution in science throughout


even systematic.


But to say that ImageNet wasn't included because women's contributions don't count that is


just a straight out lie.


And to call people white adjacent is like how do you not have a bell in your head that


goes ding ding ding ding ding when you do something like this.


So I find this to be dishonest either willfully or just because people have so become used


to seeing the world in one particular frame.


And this is I think these these calls they only get they only get big whenever there


is money and attention going into a field.


Right.


If you look at like any any field where it's just a bunch of weirdos doing their thing


the weirdos don't care who's there.


They just care about the ideas that people have.


And I believe we should take that view in science in general.


I don't care who has the idea and these people do.


And I disagree.


All right.


That was it.


Keep pushing back on these things if you agree as well and keep science for ideas.


More Is Different for AI - Scaling Up, Emergence, and Paperclip Maximizers (w/ Jacob Steinhardt)

Hi, this is an interview with Jacob Steinhard, who is the author of a blog post series called


more is different for AI more is different is the title of a famous paper in science


from 1972 by Philip Warren Anderson, a Nobel Prize winner in physics.


The article is generally on the theme of emergent phenomenon when scaling things up.


So as you make things bigger, not only does stuff get just more as you would expect, but


qualitatively new phenomena arise, you know, what better phenomenon to discuss in this


context than AI.


So today, we'll talk to Jacob about this blog post series, expect to learn how scale fundamentally


changed how we look at AI systems, how the paperclip maximizer might not be as dumb of


a thought experiment and how we can look forward and make sense of a world where AI safety


could play a critical role in how we interact with these systems in the future.


Now I'm having a ton of fun talking to people about all kinds of stuff.


But ultimately, what matters is you so please let me know how I can make these videos the


best possible for you.


Leave a comment, share them around if you like them.


And let's get into it.


Hello, everyone.


Today I have Jacob Steinhard here with me who authored a series of blog posts titled


more is different for AI, which lays out an argument or a series of arguments, playing


out the I want to say the different viewpoints on the future of AI alignment and safety in


AI safety in machine learning systems, mainly playing on two viewpoints that Jacob calls


the engineering viewpoint, mainly focused on, I want to say near term practical things.


And the philosophy viewpoint, mainly focused on more overarching principled approaches,


but maybe a bit futuristic.


And I found this to be super interesting.


It's very well laid out.


It also shows a little bit of a journey of Jacob himself, as I think he learned more


about these things.


So Jacob, thank you very much for being here.


Thanks for having me.


This was this a was this an accurate description, let's say of the blog post there are five


in total.


How did you come to this?


Yeah, I think that's pretty accurate.


I'd say the beginning posts, at least are in some sense, almost kind of letter to my


past self, trying to either, you know, argue for things that I've come to believe now that


I didn't believe five years ago, or just viewpoints that I've kind of got more clarity on.


And then I think the later posts start trying to maybe address kind of the broader field.


So both, I think I guess you could, I'd say there's maybe two fields that you can think


of this as addressing one is the kind of traditional machine learning field, which tends to be


very empirically driven.


And I wouldn't say is exactly the same as what I'm calling the engineering approach,


but I think has a lot of affinity for it.


And then this other field, that's kind of more top down, more more kind of philosophical


and conceptual that's kind of worried about long term risks from AI.


That starts with maybe people like Nick Bostrom, who was in fact a philosopher.


And so I kind of again, not exactly put that field the same as the philosophy approach,


And I think my thinking is kind of trying to be a synthesis of these two approaches.


And so I think some of the later posts are kind of trying to argue to people who would


have subscribed to one or the other philosophy why maybe they should also care about the


other side of things.


The title is More is Different for AI and that is in itself a bit of a, so there have


been already works with this given title.


Why did you choose this title?


Yeah, so this is based on an essay called More is Different.


It was originally written by a physicist, although I think biology is actually the area


where this kind of idea seems most powerful.


So this is the idea that when you just kind of increase scale, you often end up with qualitative


changes.


And I guess scale could just be the amount of something, although it could be something


like temperature as well.


So in physics, I think the simplest example would be phase transitions where I can have


a bunch of molecules.


If I just increase their temperature, they can end up in kind of qualitatively different


configurations.


But there's also cases where a few molecules is very different from having a lot of molecules.


So I think one example of this is H2O.


If you have just a few H2O molecules, they behave very differently than if you have just


a huge number and you get water.


So it turns out, for instance, that wetness is not really something that you can get from


just individual molecules.


It's more about interaction forces between different ones.


So that's where it sort of initially came from in physics.


And I think as physicists who are starting to try to consider larger molecules that maybe


didn't just form simple crystals but could be more asymmetric, and that's where it gets


more towards biology.


So I think DNA is maybe one of the most canonical examples of an asymmetric molecule that has


many, many, many, many atoms in it.


And kind of its size actually is important to how it functions because its whole purpose


is to store information.


And you can't really store information in like a calcium molecule, but you can store


information in DNA.


And so this is another example where just making things bigger leads to kind of qualitative


changes in what you can get.


And in biology, just each layer of abstraction gives you more of this.


You can go from DNA, getting even bigger, you end up with proteins, complexes of proteins,


muscles, organisms.


And so I kind of wanted to reflect on whether there are analogous properties in machine


learning.


There, you have a bunch of examples right here in this first part, and that one's called


future ML systems will be qualitatively different and different from the current ones.


Uranium, where if you have a critical mass, you get a nuclear reaction at your already


mentioned DNA, you mentioned water traffic, I find interesting, right in that 10,000 cars


could be fine, but 20,000 could block the road.


And also specialization in humans, what I would challenge a little bit here is that,


okay, DNA is a bit special, you say you can store information in calcium, but you can


in DNA.


But that is, I mean, that is very much linear, there is not really a phase transition, like


the more molecules I have, the more information I'm able to store.


And the other ones, I see much more as a function of interaction between things.


Now as we get to machine learning, maybe bigger and bigger models, do you, you call this emergence


and other people call it emergence to emergent phenomena that only happen when you get a


lot of stuff into the same place?


Do you think this emergence is mainly a property from the interaction of things or just like


the sheer number of things?


I think it's a bit of both, so I think interactions between things is one really common way to


get emergence, especially kind of emergence that looks like a phase transition where you


kind of have some sudden change.


And that's just because the number of interactions between n things grows like n squared.


So kind of that's a very natural thing that's going to kind of increase and scale up.


And maybe the interactions, each interaction could be less important than each individual


item, but if you have 10,000 things and then 100 million interactions, then those interactions


are going to dominate even if each individual one is less important.


So I think that is a really common one, but I don't think that's the only one.


For instance, for DNA, I think one thing that actually is important is that I guess you


can have multiple different bases in the DNA that all kind of interact together.


So you kind of need this like gadget of, yeah, okay, I can have A, T, C, or G.


These all fit together.


They can all kind of go in this pattern.


And somehow to get that gadget, you need like enough complexity that you can actually form


the gadget.


And so I think that's a bit different from just interaction forces.


It's more like kind of having enough substrate to build up what you want.


How does that play into AI and machine learning, this phase transition or scaling up?


Yeah.


So I think in some sense, I would say that in machine learning, there's probably a bunch


of different things that play into emergence.


And I'll also be honest, I think you're right that emergence is really kind of what we might


call a suitcase word.


Like once you unpack it, it's actually a bunch of different things.


And we could try to be more specific about what each one of those are, but I think it's


also not always clear, except in retrospect, what the cause was.


So that's kind of why I'm packing them all together into one thing.


But it is something I think we should just broadly be trying to understand better.


With that kind of caveat in mind, I think in machine learning, there's probably several


different things going on.


So one is you do need the gadgets.


You just need enough parameters that you can build up interesting behavior.


I think this might be a little counterintuitive because some of the really interesting behavior


that we're getting right now is things that start to look like reasoning.


And those are things that actually, if we wrote them, symbolic reasoning is something


that's actually very easy to write kind of a short Python script to do compared to things


like image recognition that are much harder traditionally in the domain of machine learning.


But I think somehow doing reasoning in a very robust, open-world way I think does actually


require kind of a lot of machine learning to get the gadgets right, at least the way


we're currently setting up neural networks.


So I think that's one, just getting the basic gadgets.


I think another thing is that there's a lot of stuff that kind of gets packed into, say,


like the last few bits of entropy that you're squeezing out of a system.


So most machine learning models are trained on the log likelihood or the cross entropy


loss or something like this that's just trying to kind of predict what will happen.


And most of predicting what will happen for, say, images, for instance, is going to be


just knowing what edges look like really, really well.


And that might not be so exciting.


But once you're really getting near the entropy floor, now you're forced to also think about


interactions.


You're forced to think about kind of long-range dependencies, all that sort of thing.


And so even if, say, your cross entropy loss is kind of decreasing smoothly, in terms of


the qualitative properties that a system has, you might actually get kind of sudden qualitative


changes in the behavior because there's like something that's in those last few bits.


You have a bunch of historical examples, but then you go into GPT-3 as an example of this


qualitative difference that arises from scale.


What do you think GPT-3 showed in this regard?


What does it mean?


Right.


So I think the thing that was really surprising to me and I think to many other people was


that GPT-3 was very good at in-context learning, meaning that from just a few examples, it


could kind of learn how to do new tasks.


So you could just give it a few examples of, say, translating sentences from French to


English and you'd get a pretty good translator.


I think actually the graph you're showing right now is for those results.


And so I guess why was this surprising?


Well, previous systems really couldn't do that very well.


If you wanted a translation system, you really needed to train it on example translations.


And GPT-3 was instead just trained on lots of text on the Internet.


Surely it did have some French and English sentences, but it wasn't being explicitly


trained to do this particular task.


And so that's what in-context learning was.


And the reason that I would have called it surprising is if we had just drawn a graph


of how much can systems do in-context learning, I would have just put it at zero for a while


up until you hit GPT-2, I would have said a little bit, and then GPT-3, I would say


it's quite good at that.


And so that I think is how I would kind of capture the surprise.


It's like there was this line that was at zero.


Usually I would expect to go from zero to non-zero, you need some clever idea.


But here you just did the same thing, but more of it.


And then you went from zero to non-zero.


Yeah, there are a lot of, I don't know, this is maybe a side point, but there are a lot


of people that at the same, like they say, oh, I always knew that GPT-3 was going to


do what it does, but I doubt anyone could have foreseen just the, like how good it is.


It's easy to say in hindsight, and it's easy to go and say, well, it just does interpolation.


It's just a bigger version of GPT-2.


But I think genuinely the entire world was surprised by really this emergent phenomenon


of this in-context learning.


Yeah.


Yeah, I would say, so I think I would agree that most people were pretty surprised.


Certainly I was surprised.


I do know people at the time who, well, okay, all I know is that they said at the time they


had kind of done extrapolation, say on the cross entropy loss or things like that, and


felt like there should be something pretty cool happening at like around that parameter


count.


I don't know if they would have said exactly that parameter count or if it was just like


within a factor of 10 or 100.


Certainly I guess I would think that the people at OpenAI who bet on this at least had to


have some belief that something cool would happen because there was a lot of resources.


And if you didn't believe there is a payoff, it would be kind of hard to justify that.


So I guess what I would say is I don't think it was something that was like entirely unpredictable


by anyone in the world.


But it was just very surprising relative to kind of the consensus and to my own beliefs


at the time.


And that surprise is one of the, let's say, core arguments of your contraposition of the


different viewpoints on the future of AI and its alignment.


Could you briefly introduce us to kind of the different viewpoints you considered and


what they say?


Yeah.


So I think there's kind of two viewpoints that I often think of as being in tension


with each other.


The first is what I kind of dubbed the engineering viewpoint.


And what is this?


So it's kind of very bottom up driven.


It kind of looks at the empirical data that we have in front of us.


It tends to kind of extrapolate trends going forward.


So it's like, you know, what did things look like last year?


What did things look like two years ago?


What do things look like today?


And then I'll predict the future by kind of, OK, maybe not literally drawing a line, but


just kind of intuitively like where are things going from there?


And so and also, I think this this worldview would kind of really prize empirical data


be somewhat skeptical of kind of abstract conceptual arguments, maybe not completely


dismiss them, but but really be focused on the empirical data.


So that would be kind of the engineering worldview.


I think the philosophy worldview would be much more top down, kind of trying to think


about just what's in principle possible?


What's the limit as we get really, really smart machine learning systems kind of more


into these kind of abstract arguments, not as into the empirical data and willing to


make extrapolations that don't look very much like existing trends?


And so that would be kind of the more philosophy worldview.


And I think I guess in terms of where I've come from historically, I think I'd say I


sort of.


Would have mostly bought into the kind of engineering world view kind of into just,


yeah, let's let's look at where things are going empirically, and this is a good way


to decide what problems to work on.


On the other hand, I had read kind of some more philosophy oriented stuff like Nick Bostrom's


Superintelligence book and other arguments around that.


And it always felt to me like there was something both something to them, but also like somehow


it didn't really match my experience with ML systems.


And so I had always kind of almost felt like a little bit like I had these like two different


conflicting views in my head that I was trying to reconcile.


How does the phenomenon of emergence play into this game between the engineering and


the philosophy viewpoint?


Right.


So I think the main thing is that it shows that you have to be somewhat careful with


the engineering viewpoint, because what emergence kind of is saying is that you can often get


these kind of qualitative shifts that don't at least apparently follow existing trends.


There's a bit of nuance to that, because actually GPT-3 followed trends in the log, like the


value of the log likelihood loss, followed that trend very well.


It's just that you can get behavior that is a very nonlinear function of your cross entropy


loss, where just a small decrease in cross entropy loss leads to a pretty big increase


in behavior.


And so I guess what this is saying is that at least for maybe the kind of like end line


things you care about, the actual behavior of ML systems, you can actually get kind of


discontinuous kind of breaks in the trend.


And so you can't just kind of be safe with a worldview that's kind of always predicting


that things are going to follow smooth trends.


You can actually get these surprises.


And so I think there's kind of two updates that that has for me.


One I guess is just being a bit more careful how we apply engineering.


So there are some things that will probably be smooth, but there's other things that won't


be.


And we need to think about which is which.


But the other is then wanting to rely a bit more on philosophy, because it's at least


a very good source of hypothesis generation.


If we're kind of trying to come up with hypotheses about what trends might break or surprise


us in the future, then I think we need more top down thinking to kind of generate that.


And then we can kind of try to tie that into what we see with actual ML systems and try


to kind of reconcile those two.


But I think we need some form of top down thinking to generate the hypotheses in the


first place.


Isn't that you're saying the engineering viewpoint is a little bit, you have to be a little bit


careful because we get these emergence phenomena, these discontinuities and so on.


Isn't that in itself a trend though?


Like isn't because you list this even historically that as soon as some new barrier was reached,


we have been able to all of a sudden do something that we didn't think was possible before,


like a kind of a jump in abilities without necessarily having to have the great idea


behind it.


Isn't that in itself a trend?


I extrapolate that reasonably and say, well, I don't know exactly what it's going to be


in two years, but I'm pretty sure there's going to be some emergent phenomena that allows


us to have some new good capabilities.


Sure.


So I would agree with that.


So what I would say there is that the trend is towards more surprises over time.


So because I think you can think of emergence as sort of like a surprise.


Like I said, I think it's possible in some cases to predict it to some degree, but it's


certainly more of a surprise than most other things.


And so, yeah, I think we should expect more surprises over time.


But if we're then trying to kind of predict what's going to happen, that I guess it's


good to know that you're going to be surprised, but then you want to have some sense of what


the surprise might be.


And so I think kind of getting a sense of what those surprises might be is where this


philosophy approach can come in and be really useful.


Now all of this, and you mentioned here the paperclip maximizer, all of this goes into


AI alignment and AI safety.


What is, like, what's the relevance of this field to you?


What drew you to this?


Why are you making this argument specifically for these fields?


Right.


So I think the one big relevance to AI safety or alignment is just the bigger the surprises


you might end up with, I think the more you should be concerned about safety.


So that's just a very kind of abstract, but I think fairly robust, consideration.


A more specific consideration is that I think many of the sort of historical arguments for


caring about AI safety or alignment sort of tend to posit properties of systems that don't


necessarily match what we see today.


So I think you give this example of Nick Bostrom's paperclip maximizer thought experiment


where you give an AI some objective function to make paperclips, and then it kind of just


takes over the world to maximize the number of paperclips.


And I don't think Nick thinks literally that will happen, and I don't think literally that


will happen.


But it's sort of trying to get at this idea that if you have a very simple objective function


but a really powerful optimizer, you can get all sorts of weird things happening.


I think in some broad sense, actually, we can see that already even from the engineering


worldview with things like Facebook or YouTube that often end up with a lot of unintended


consequences when you optimize.


But certainly some of the aspects of that story kind of invoke lots of things that would


be foreign to existing ML systems where you have way more capabilities than any existing


system.


And you're doing all sorts of weird long-term reasoning and trying to out-think humans and


things like that.


And so I think that's where you kind of end up departing from what we see with current


ML systems.


And so I guess I kind of find—actually, let me collect my thoughts for a second because


I think I'm going off the rails a bit.


Yeah, so I think what I want to say for the paperclip maximizer thing in particular is


that it seems at least more plausible to me that you could end up with systems that kind


of have really advanced reasoning capabilities or things like that without necessarily having


huge conceptual breakthroughs and just from scaling up.


And so I think there's kind of risks from that.


I think there's other more exotic failure modes that people discuss beyond just this


kind of misaligned objectives failure mode that involve other specific capabilities that


kind of systems today don't have.


And historically, I've been very kind of skeptical of those more exotic failure modes.


I think the pivot clip maximizer one, at least if we interpret it as being about misaligned


objectives, I actually find kind of less exotic because I can point to existing systems that


have that.


But I think kind of more as different has made me be a bit more willing to buy some


of the more kind of exotic failure modes that have been discussed.


My issue with these types of arguments—and you also said you used to be very skeptical.


If I can take this from your blog post series, you're now still skeptical but have a little


bit of an appreciation gained for these types of arguments.


Maybe that's a good formulation for that and we'll get to that in a second.


My issue with these types of argument is always that there is always on the path to the super


intelligence, there is always a hidden intelligence somewhere else.


So if someone says that optimizing on YouTube or optimizing on Facebook leads to unintended


consequences, that is because the intelligent humans are taking part in the system.


There is also a famous, I think, paper by I think it's Rich Sutton that is Reward is


Enough and a bunch of others out of DeepMind.


And it makes similar arguments like, well, if you just optimize for reward, then all


kinds of things will emerge if you have a powerful enough optimizer.


But hidden in that is the powerful enough optimizer, which in itself must already be


an AGI essentially in order to make that optimization happen.


Likewise for the paperclip maximizer, the postulation of the process of the paperclip


maximizer emerging is only possible if the optimizer itself is an AGI already.


So I always find that hidden in these arguments, it's kind of a circular, it's a tautology,


it's we'll get an AGI if we have an AGI.


And that is so I challenge anyone from that camp to come up with a situation like an alignment


problematic situation, given some kind of future super intelligence that doesn't already


require the super intelligence to exist for the other super intelligence to emerge.


And I haven't found that yet.


Yeah, so let me try to unpack that a bit.


I guess first of all, just to kind of clarify what my views are, I think historically, I


felt like on each of the individual arguments, I felt skeptical that that particular thing


will happen.


But I found them to be moderately convincing that there's just like a bunch of risk that


we should think more about and try to understand more.


I think the main way that my views have evolved in terms of, you know, and I say decrease


in skepticism is I now find it useful to think about many of the specific properties that


kind of show up in these thought experiments as potential hypotheses about things systems


might do in the future.


And so that's the sense in which I started to assign more weight instead of just taking


some like very big outside view of like, well, it's going to be a big deal.


We should really worry about making it go right.


I'm now also taking some of the specific hypotheses that the philosophy view is raising.


So that's just clarifying kind of my stance there.


In terms of, yeah, you're saying, well, to get like if you have a powerful to get a super


powerful optimizer, you need to like already have a powerful optimizer.


I think that I think that's like probably right.


I'm not I wouldn't say I'm like 100 percent confident of that.


But I think what what this kind of makes me like, I guess the way that I would put this


is that before you have kind of superhuman systems, you will have like slightly superhuman


systems.


And before that, you'll have human level systems where you'll have like slightly below human


level systems.


And so it is going to be this kind of probably a continuous thing rather than like a really


sharp takeoff.


I'm not so confident that there's not going to be a sharp takeoff that I think we should


just ignore that possibility.


But I do think in most worlds, it's probably somewhat smooth.


You know, one piece of evidence for this is even within context learning, you know, it


like that kind of developed over the course of a couple of years, at least going from


GPT 2 to GPT 3.


So so so I think I would agree that like probably you'll have something more smooth.


And that is kind of like a like one problem with a lot of the scenarios that are put forth


is that they kind of imagine that like, oh, you just have this like one AI system that's


like way more intelligent than like everything else that exists.


And I think that's like probably not true.


You'll probably have other things that are slightly less intelligent.


And so there's not going to be some like enormous gap in capabilities.


So I think that's maybe like one place where a lot of stories kind of become less realistic.


So I think that would be kind of my main takeaway from what you're saying.


In your third blog post here, or second, you make a case for these thought experiments.


Could you you have already touched a little bit on this and you talk about anchors here,


could you lead us a little bit on the case for respecting such thought experiments?


Yeah, so I guess this is this is getting back to what I was saying about how my views have


shifted towards wanting to rely a bit more on the actual kind of like inside view considerations


from some of these thought experiments rather than just taking it as a kind of broad outside


view argument for for caring about risk from AI.


So the way I would put it is that whenever we're trying to predict something, it's very


useful to have what I'll call reference classes or kind of anchors of kind of analogous things


or analogous or just some sort of heuristics for predicting what will happen.


And in general, it's better to kind of when making predictions, take several reference


classes or several anchors and kind of average over those or ensemble over those rather than


just sticking with one.


Right, machine learning ensembles work better than than individual models.


And it's also the case that when humans make forecasts, it's generally better to kind of


take an ensemble of world views or approaches.


So I kind of lay out a few different a few different approaches you could take that I


call anchors.


The simplest one is you can just predict that future ML systems will look like current ML


systems.


And so I call that the kind of current ML anchor.


And I think that's probably the one that would be favored by most machine learning researchers.


I think it's the one that that I've historically favored the most.


But what I've come to realize is that and actually, this is more actually just from


reading literature on forecasting.


I'm actually teaching a class on forecasting this semester.


So I've been reading a lot about how to make good forecasts as a human.


And I realized you actually don't want to rely on just one anchor.


You want several if you can.


And so I thought about, OK, what are other ones we could use?


Well, another somewhat popular one, although it might be more popular with the public than


with ML researchers, is what I'll call the human anchor, where we just sort of think


of AI systems as like dumber humans or something, and maybe future ML systems will be like smarter


than they are now.


And eventually, they'll just kind of do things that humans do.


And so we could just look at, OK, what can humans do right now that ML systems can't


do and predict that we'll probably have those sorts of things in the future and just generally


kind of take that kind of human centric approach.


I think most ML people really hate this one because it just sort of reeks of anthropomorphism,


which there's kind of, I think, to some extent, correctly, a lot of pushback against because


kind of historically anthropomorphic arguments in ML have a pretty bad track record.


I think the amount of pushback is actually too high relative to the actual badness of


the track record.


I think you should be sort of somewhat downweighting anything that's based on reasoning about humans,


but I don't think you should be downweighting it as much as I think most people do.


But anyways, this is another one.


I don't like to rely on it too much, but I rely on it.


I use it at least a little bit.


And then this other anchor is what I'll call the optimization anchor, which is just thinking


about ML systems as kind of ideal optimizers and thinking about, OK, well, what would happen


if you could just like if actually ML systems were just really smart and were just like


optimizing their objectives perfectly, what would happen there?


And so I think this one is the one that's kind of I would associate most with the philosophy


worldview.


I think the paperclip maximizer argument is like kind of exactly doing this.


And then there's some kind of more recent arguments that are a bit more sophisticated


that also kind of take this there.


So like what is this thing called imitative deception, which I can get into in a bit,


or just this idea that like, you know, if you're like trying to optimize, you'll kind


of want to acquire influence and power.


So this is kind of a third anchor.


I actually think there's a lot of other anchors I like to use.


Like I think evolution is a good analogy.


Corporations are a good analogy because they're kind of like super intelligent optimizers


compared to humans.


And but like the general point is like we should just be trying to find these anchors


and use as many as we can.


Yeah, I've, especially to your second point right here, it is pretty interesting that


I believe when you have something like AlphaZero that plays really good, like really is really


skilled in chess, and you ask it to lose a game or to draw a game or something like this,


it will not play weaker, it will play just as strong until the end where it will kind


of bring itself into like a draw situation or a losing situation.


Because right, that's still the most sure way to get your result is to have complete


control to crush your opponent completely until you know, you get the outcome that you


want.


That's pretty, pretty interesting and I think counterintuitive because you would guess that


if you ask a model to play for a draw, it will kind of reduce its skill, but that's


not the case.


The other thing, imitative deception, could you elaborate on that a little bit?


Yeah, so imitative deception is this idea that if I have something that's trained on


the cross entropy loss, what is the cross entropy loss doing?


It's trying to kind of predict or in other words imitate the distribution of examples


that it's given.


And so you could, if you kind of have something that's trained with that objective and then


you start asking it questions, it's not actually, you know, like its incentive is not actually


to output the true answers to the questions, it's to output the most likely answers to


those questions because that's what minimizes the cross entropy loss.


And so those tend to be pretty highly correlated, but they aren't necessarily, right?


So if you have common human misconceptions, then it could be that text on the internet,


which is what these systems are trained on, is actually more likely to contain the kind


of misconceived answer than the true answer.


And so you ask the system that question, then you're going to get the wrong answer.


Now you could say, well, that's maybe not so surprising.


If you have noisy data, you're going to do worse.


But I think there's a couple properties, and actually at this point now I'd say empirical


properties of this that I think show that it's kind of different from just like noisy


data makes you worse.


One is that actually larger models exhibit more of this.


So models that kind of do better in general will actually do worse on these kind of common


misconception tasks.


So that's what this paper by Lynn and collaborators from 2021.


Okay.


I have to throw in, I have a giant problem with this paper.


But you're obviously right, right?


That's the background.


Aren't large models doing quote unquote worse because they're just a lot better at picking


up the nuance of, because what this paper tries to do is tries to elicit, right, these


wrong answers.


It tries to like hint at a conspiracy theory and then it checks whether the model kind


of falls for it.


Isn't that just because as you say, the larger models, they're actually skilled enough to


pick up on this kind of questioning and then continue as a human would if encountered by,


I think one of the main questions they have is like, who really did 9-11, right?


And a small model is just not able to pick up on that.


Yeah, who really caused 9-11?


And I think, I mean, absolutely correct, right?


The larger models are doing worse, but it's just because they're more skilled, right?


They are more capable of being able to pick up on the nuance and isn't the failure in


the user here, the user that expects that these models actually give me truthful answers


rather than the user expecting these models actually give me the most likely answers?


So I guess I would agree with you that the failure is coming from the skill of the models.


I think this is actually kind of exactly what I'm kind of worried about, right?


So the concern is that if you have a very slightly incorrect objective function and


you have models that aren't so skilled, then probably what they do to increase that slightly


incorrect objective function is pretty similar to what they would do to increase the true


objective function.


So here maybe think of the slightly incorrect one being output what's likely and the true


one and like the one you really care about being to output what's true.


So I think this is sort of the point that kind of as you get more skilled, those two


things diverge.


Now, I will grant your point that the kind of framing of these questions might create


a context where the model thinks it's more likely that the person asking it is like into


conspiracy theories or it like pattern matches to text on the internet that's like more about


conspiracy theories.


So that's totally true.


They did the ablation.


If they don't phrase the questions like this, this effect goes away of the larger models


doing worse, right?


And this, it brings us a bit to your next post, which is ML systems will have weird


failure modes, which deals exactly with this.


And I agree that it is, if you think about like a perfect optimizer, and as our models


get larger, they do approach better and better optimizers.


It is really hard in the real world to specify a reward function correctly in a simple enough


way, right?


And that will result in exactly what you call weird failure modes.


What do you mean by that?


Yeah.


So I think I guess there's sort of different levels of weird, right?


So I guess this kind of like imitative deception, I would call like somewhat weird.


I mean, in some sense, it's like not that hard to see why it happens.


Because you know, you can kind of see why if you kind of have stuff that's phrased about


like who really caused 9-11, that probably the stuff on the internet that's closest to


that was like some conspiracy theory forum.


And so that's how you're going to complete it.


I think other examples of this that I think, okay, maybe you could blame the user, but


I'm not sure that's the right way to think about it is things like code completion models


like codex, right?


So one thing you might worry about is well, if you have a novice programmer, and you have


them like type in some code and ask them to complete it.


Well, if the model is smart enough, then it can tell the difference between code written


by a novice programmer and an expert programmer.


And it can see that it's a novice programmer typing stuff.


And so then, if I want to complete stuff in the most likely way, I should complete it


the way a novice programmer would complete it, and maybe introduce like some errors also


just for good measure.


And so we really don't want that, right?


You want things that are actually being helpful rather than just copying you.


So I think that's maybe a slightly more counterintuitive version of this, but I would call these somewhat


weird.


I think the ones that start to become really weird is if you're positing that the system's


actually starting to reason about what people will do in kind of like a long-term way and


like potentially doing things to intentionally trick them.


And so these are the ones that I guess historically I've kind of found very implausible, but started


to put like a bit more weight on because of this kind of emergence.


And so I think that's what the post you have up right now is about.


I think it's about this idea called deceptive alignment.


And the idea there is that if you...


Okay, so yeah, so what's the idea behind deceptive alignment?


So the idea there is even if you actually got exactly the right reward function and


you train the system with that reward function, you could still end up with something that


is misaligned with that reward function.


And the reason for that, and this is where it gets kind of a bit weird and philosophical,


but the reason for that is that as the system being trained, you know that in order to get


deployed, you need to have high reward.


And so no matter what your actual intrinsic reward function is, during training, the thing


you want to do is output stuff that is good according to the kind of extrinsic reward


that you're being trained on.


So maybe you're doing that because you were actually optimized to do that, and then when


you're deployed, you'll continue to do that.


Or maybe you'll do that because you have a different reward function that's this kind


of intrinsic reward function, and then when you're deployed, you'll just pursue that intrinsic


function, even though at training time, it looked like you were optimizing the extrinsic


function.


So that's kind of the basic idea.


It's pretty weird, and we can break it down, but that's kind of the sort of one-minute


summary.


So that, in other words, the AI could be really smart and sort of during training trick us


into thinking it has learned what we want it to learn, and then once it's deployed,


all of a sudden it's going to do something different, like take over the world and fire


all the nukes.


Yeah, or even you could consider more frusetic things as well.


Maybe the intrinsic reward it ended up with was some exploration bonus, and so then when


it's deployed, it just tries to acquire as much information as it can, although that


could also be destructive in various ways.


But yeah, I think this is kind of the basic idea, and yeah, maybe with a sufficiently


capable system, we can discuss firing all the nukes if we want.


Why do you, I mean, on first hand, it's like, yeah, that is a nice thought, but probably


not, right?


Probably if we optimize something for a reward, like the simplest explanation, and you also


write that down, right?


The simplest explanation is it's just going to get better on that reward, right?


And if it is at all anything progressive, increasing, we'll probably get to know once


it's going to try to trick us, or once the reward that is deployed isn't the reward that


we trained for.


What makes you give more credence to this than your past self?


Right, so I think my past self would have looked at this and just been like, this is


totally bonkers, and then kind of moved on and read something else.


I think my present self instead is going to be like, okay, well, I feel a bunch of intuitive


skepticism here, but let me try to unpack that and see where the skepticism is coming


from.


And when I unpack that, I actually, I think I can lump the skepticism into two different


categories.


One category is like, well, this invokes capabilities that current ML systems don't have, so it


seems implausible for that reason.


And that's the sort of skepticism that I kind of want to down-weight.


So in particular, this invokes the idea that ML systems can do long-term planning, and


that they can reason about the external aspects of their environment in a somewhat sophisticated


way.


And these are things that now, the fact that we don't have those now doesn't really, to


me, say much about whether we'll have those, say, like 10, 15 years from now.


So that's the stuff I want to down-weight.


I think the stuff I don't want to down-weight is like, okay, well, why does it have this


intrinsic reward in the first place?


Where did it come from?


Why should we expect systems to have intrinsic reward functions versus just like followings


whatever policy they're following or doing whatever else?


And if they do have an intrinsic reward, why shouldn't we expect it to be at least pretty


similar to the extrinsic reward, given that that's what it was trained to do?


So I think those are kind of the sort of sources of skepticism that I don't down-weight as


much.


But what I think this kind of thought experiment does show is that there's at least a bunch


of different coherent ways to get zero training loss.


Like, right, it's like you could get zero training loss because you're actually trying


to do the thing you're trained to do, or you could get zero training loss for this deceptive


reason.


I think there's probably some large space of other ways to get zero training loss that


are like some combination of these, or that are like getting the answer right, but for


the wrong reasons or things like that.


And so I think the main takeaway for me is just that there's like many, many ways to


get zero training loss.


And as systems become more capable, the number of ways to do that could actually increase


in ways that are kind of unintuitive to us.


Is there any work in actually trying to get a system to be deceptive in exhibiting, you


know, good answers during training, but then doing something different in deployment?


It'd be interesting to actually try to get a system to do that.


Yeah, I think I haven't seen anything that does exactly this.


I've seen things where there's some distribution shift between training and deployment that


leads to something weird happening around having the wrong reward function.


But it's usually not really about deception, and it kind of has some clear distribution


shift.


Whereas here, okay, technically there's a distribution shift because there's like, are


you being trained or are you being deployed?


But otherwise, the distribution of inputs is like exactly the same.


And so that's kind of the thing that's kind of counterintuitive is that it's like a very


subtle distribution shift that could potentially lead to a large difference.


So I don't know, like all of the work I've seen on this, and I might be missing something,


so I apologize to whoever's work I'm missing, but all the work I've seen on this has been


kind of purely kind of abstract and philosophical.


And I think it would be great to make kind of better connections to actual empirical


stuff so that we can start to see like, yeah, like how does this actually pan out in practice


and like, how do we address it?


It's interesting that in things like virology or so, we're perfectly capable of saying,


you know, we're gonna make these super pathogens in order to try to combat them, right?


But in ML, people rarely, I mean, there's the adversarial examples community, but it's


not exactly the same.


There isn't much work that I'm aware of that is like, yeah, let's create like the most


misaligned AI that we can think of and then see what we can do against it.


I think that'd be a fun topic to research.


Yeah, I think that like the general thing I would call this would be like red teaming,


kind of trying to elicit failure modes.


I think there actually is starting to be like, I'd agree with you, there's not much work


on this so far, but I think there's starting to be more and more good work along these


lines.


DeepMind had a nice paper that kind of tries to use language models to elicit failure modes


of language models that I thought was kind of cool.


We like our group actually had a recent paper at ICLR that kind of takes mis-specified reward


functions and looks at what happens when you kind of scale the capacity of your policy


model up to see if you do kind of get these like unintended behavior.


We find that in some cases there are these kind of phase transitions where you scale


the parameters up within some fairly small regime, you go from basically doing the right


thing to doing totally the wrong thing.


Those are still in environments that I'd say are kind of like at the level of Atari environments.


So they're not like trivial, but they're not super complex.


So I'd like to see that in more complex environments.


But yeah, I'd agree with you, I think it would be awesome to see more work like this.


And I think some people are already trying to do this.


Excellent.


So your last blog post here is called empirical findings generalized surprisingly far.


And it is almost a bit of a counterpoint.


You even admit this year, it might seem like a contradiction coming a bit full circle in


the whole story.


What is this last point that you're making here?


Yeah, so I guess I would say the posts up to this point were kind of more almost directed


like at my past self.


And then to some extent, the broader ML community.


In the sense that I think I was like pretty far on the kind of empirical engineering side,


probably less so actually than like the average ML researcher, but like way more so than kind


of the average like philosophy oriented person.


And so I was trying to argue like why you should kind of put more weight into this other


viewpoint.


Here I'm kind of now going back to arguing kind of maybe not against the philosophy viewpoint,


but talking about what things I feel it misses.


And in particular, I think it tends to be like somewhat too pessimistic where it's like,


well, like future systems aren't going to look anything like current systems.


So like anything could happen.


So to be like to be extra safe, let's just assume that the worst case thing will happen.


Oh, but then in the worst case, like we're all screwed.


Yeah, this is what I find in people like almost everyone who gets into this alignment stuff.


Six months later, they come out and they're like completely blackpilled and be like, well,


nothing matters anyway.


We're all going to die because AGI is just going to take us.


And I'm like, well, I'm not so sure, but it seems to be a consistent pattern.


Yeah, so yeah, so that's not what I believe.


I think I would say I think like future systems pose like a real and an important risk.


I think in the like median world, we're fine.


But in the like 90th percentile world, we're not fine.


And I want to like, you know, if I could say like, if I could push it out so that in the


90th percentile world, we're fine.


But in the 90th percentile world, we're not fine.


So that would still be kind of scary because I don't like five percent chances of catastrophes.


But like, you know, that would be an improvement.


And so that's kind of like what I think of myself as trying to do is like, yeah, there's


like tail risk, but it's like real tail risk.


Like it's not like a one percent thing.


It's like maybe more like a 10 percent thing.


And like we should really be trying to push that down.


So I guess that's just my view in terms of like why I believe that I think it's for like


a number of reasons.


But one of them is that I feel like, yeah, some of the thinking is kind of too worst


case.


It's kind of like ignoring all properties of how ML systems work.


And like I agree, yeah, you don't want to rely too strongly on whatever we happen to


have today.


But I think like there are properties that we kind of can rely on.


I think one is just like things will probably look kind of like neural networks.


Like they'll probably have internal representations.


We can probably try to like introspect on those representations to understand what's


happening.


Those probably won't directly be human interpretable.


But I think with enough work, we can still kind of do things with them.


And I feel like there's already like some work suggests like showing that you can do


at least a little bit with representations.


And like 10 years from now, I think there'll be way more work like that.


So that's kind of like one reason for optimism is like we don't just have to look at the


outputs.


Right.


Like most of the worries, most of the worries that we've been talking about are like somehow


because you only are supervising the outputs, you end up with a system whose like internal


process is like really off and do it, getting like the right answer for the wrong reasons.


But if I can like supervise the reasons as well as the output, then maybe I can do better.


So I think that's kind of one reason for optimism.


Another reason for optimism is that I think, yeah, we shouldn't assume that neural networks


have like exactly the same concepts as humans.


But I think like their inductive biases aren't like totally crazy.


I think usually if they kind of generalize in the wrong way, they generalize in like


a wrong way.


That's at least like somewhat understandable.


And it's like you can kind of see where it's coming from.


And so it's not like there's this like infinite dimensional space of like anything could happen.


It's like there's this kind of relatively low dimensional space of things that could


happen and like a bunch of things in that low dimensional space are pretty bad.


So you need to like avoid all those and like get to the good thing.


But I think that's very different from like the good thing is like totally like unidentifiable


and just like nowhere close to anything you're talking about.


So I think those are both kind of like reasons for optimism.


They're kind of fuzzier than I want them to be.


So like I hope in like five years we'll have much more like good reasons for optimism that


are kind of more empirically grounded and more solid.


But those are kind of those are kind of two reasons for optimism that I kind of argue


for here.


Now that you have a let's say you've done your travels, you were on this side, you looked


into the other side or many sides of this debate.


Now that you're enlightened, what would you think is the most if you could if you could


do one, if you could force the world to do one thing to guarantee better AI alignment


or safety in the future?


What would you recommend?


One thing.


It can be two if you have two with that equally.


But you know, just kind of like something that you've realized, okay, this is actually


something important that not that many people push for.


Well, I think I would like it if there was within ML more of a place for dialogue of


thinking about these kind of like not not even just in the context of like AI alignment,


but just generally like kind of more conceptual or philosophical arguments.


If you go back to like way back, you know, Turing, people like that, they write all sorts


of like super philosophical papers, like the Turing test was like a really philosophical


paper.


And like not all of it stands up.


There's a section in it on how because ESP has been established to exist with high probability


that like creates problems for the Turing test.


And you're like, okay, where does that come from?


Well, it actually turns out that like, a lot of scientists in Turing's time thought that


ESP existed based on some some experiments that someone had done that later ended up


having like severe issues, but but they're like very subtle, severe issues.


So it's like, yeah, I think if you do kind of more philosophical stuff, some percentage


of it is going to end up looking like that.


But some percentage of it is going to be the Turing test.


And, you know, I think, I think the like, increased recall of really good ideas like


that is kind of worth the decreased precision.


I mean, we obviously need sort of standards to kind of judge those arguments.


But right now it's happening is all those arguments are happening kind of like next


to the ML field rather than like within the ML field.


And so that I don't think that's like, that's not going to improve the quality of arguments,


it's going to be much better if you kind of have have a community of people with on the


ground experience also also participating in this.


So I think that might be the biggest change I personally like to see.


You know, now that we are we've begun sort of requiring sections, we could we could force


people to next to the broader impact section, we could also, you know, do a philosophical


musing section where you have to reflect on the long term and sort of paperclip stuff,


maximizer style impacts of your work.


Well, yeah, I'm not sure I want to force people to do that.


Might be fun.


Yeah, I think like, I guess I'd rather have like a track or venue for kind of talking


about these.


And also for the broader impact stuff, to be honest, because I think a lot of the broader


impact sections of these papers are kind of cookie cutter, and people are just like, filling


it out, because they feel like they need to add that section.


But you know, there's other researchers who I think are super thoughtful about the broader


impacts and have like really good thoughts.


And so I like I'd like there to just be, you know, venues.


And like there are to some extent, right.


But like, I think there should just be like, more more of a culture of like, yeah, like,


let's have, you know, an essay about the broader impacts.


And like, that's like a reasonable contribution or kind of, you know, this like very conceptual


essay about like, weird stuff that could happen in the future.


And that's a valid contribution.


So I think that that's maybe what I want more of.


Cool.


Yeah, that's a good message to all the people who, who think about organizing workshops


and so on.


This would be neat topics that would make for interesting workshops, certainly at conferences,


I'd certainly attend.


Yeah, it's funny, because I also wrote a paper on troubling trends in machine learning scholarship


where I argue against speculation.


But what I think actually is not really an argument against speculation.


It's really important.


It's that you need to separate speculation from from the like, solid stuff, right?


If you have, if you're like mixing it all together, then it's just a mess.


But but I think if it's kind of clearly labeled, then then you know that that's a much safer


way to do things.


This workshop is an opinion piece.


Good.


Is there any any last thing you want to get out to people about this topic?


Something we haven't touched on yet that he feels important?


Yeah, good question.


Um, no, I think you did a pretty good job of hitting it.


Maybe the other thing I would just say is I think, like biology is a really interesting


field where you also have kind of complex self organizing systems and an emergent behavior


like we have an ML.


And so I've personally gotten a lot out of just reading a lot about the history of biology.


So I recommend that there's a couple of really good books.


One is the eighth day of creation.


It's it's kind of long, but very well written.


And I think if people want like a good nonfiction book, I highly recommend it to people.


Cool.


Your blog is Boundary Regret, right?


People can find you there.


Yep.


Excellent.


Well, Jacob, thank you very much for being here.


This was really cool.


Yeah, thank you.


I'll see you around.


Neural Ordinary Differential Equations

Hello and welcome. Today we're going to look at ordinary differential equations by Ricky Chen,


Neural Ordinary Differential Equations

Yuliya Rubinova, Jesse Bettencourt and David Dovino. So this has been quite an interesting kind


Neural Ordinary Differential Equations

of paper to see because it's a bit special. We're going to go over parts of it, not the full paper,


Neural Ordinary Differential Equations

just kind of the important parts because the paper is quite packed and we'd rather, I'd rather


Neural Ordinary Differential Equations

explain it in parts and kind of get the gist of it. So basically what they do is they say,


Neural Ordinary Differential Equations

we introduce a new family of deep neural network models. Instead of specifying a discrete sequence


Neural Ordinary Differential Equations

of hidden layers, we parametrize the derivative of the hidden state using a neural network. The


Neural Ordinary Differential Equations

output of the network is computed using a black box differential equation solver. These continuous


Neural Ordinary Differential Equations

depth models have constant memory cost, adapt their evaluation strategy to each input, can


Neural Ordinary Differential Equations

explicitly trade numerical precision for speed. I mean this, it sounds awesome. Honestly, it sounds


Neural Ordinary Differential Equations

really cool and it sounds really new and yeah. All right, let's jump in. So what they say is,


Neural Ordinary Differential Equations

let's look at kind of classic neural networks, especially residual neural networks. So what


Neural Ordinary Differential Equations

residual neural networks do is in each hidden layer, they kind of have a representation,


Neural Ordinary Differential Equations

HT. That's kind of their hidden representation at layer T. And what they do is they then add


Neural Ordinary Differential Equations

something. So if you don't know a recurrent neural network is where you have, let's say this is your


Neural Ordinary Differential Equations

hidden state HT. And in a classic neural network, you would just have a weight matrix here, blah,


Neural Ordinary Differential Equations

blah, blah, blah. You do a matrix multiplication to get HT plus one. So to get the next kind of


Neural Ordinary Differential Equations

the next hidden state, you do a matrix multiplication by a big weight matrix here, W,


Neural Ordinary Differential Equations

right? In a residual neural network, what you do is you have this weight matrix W,


Neural Ordinary Differential Equations

you multiply it to get delta HT plus one and you take HT and you add the two, right? You add HT and


Neural Ordinary Differential Equations

delta HT plus one to arrive at HT plus one. So that's a residual network. It basically doesn't


Neural Ordinary Differential Equations

learn the transformation to the next layer, but it learns what, how is the next representation


Neural Ordinary Differential Equations

different from this representation, right? So what do I need to add to this representation to get to


Neural Ordinary Differential Equations

the next representation? Which is, it's kind of, it's reason that for deep networks, since each


Neural Ordinary Differential Equations

layer only does a little bit of transformation, we should basically bias it towards keeping the


Neural Ordinary Differential Equations

representation the same and just kind of changing it a little bit. So this is the inherent biases,


Neural Ordinary Differential Equations

the identity transform, right? So that's a residual network. This here is characterized


Neural Ordinary Differential Equations

by F of kind of theta and HT. So this is kind of the, this is what we called delta H. It's now,


Neural Ordinary Differential Equations

it's now called F. So this F would be the kind of neural network layer and the theta would be


Neural Ordinary Differential Equations

the parameters of it. So the weight matrix in our case. They say, okay, what if you do many of those,


Neural Ordinary Differential Equations

right? So this, they say basically what this is, is kind of a time process, right? It's kind of,


Neural Ordinary Differential Equations

you have a state and the next state and the next state, and you always learn how to go to the next


Neural Ordinary Differential Equations

state to the next state and so on. What if you go very deep and what if you look at this as a time


Neural Ordinary Differential Equations

process and kind of make these steps very small, right? Make these super small and basically what


Neural Ordinary Differential Equations

if you have many, many, infinitely many layers, right? And say, well, okay, this then becomes a


Neural Ordinary Differential Equations

dynamic process. Basically an ordinary differential equation where I say, okay, my time is now


Neural Ordinary Differential Equations

continuous and I look at it as a linearization, as a local linearization basically and I say, okay,


Neural Ordinary Differential Equations

I basically specify how to get from this time to the next instance of time, the next instant,


Neural Ordinary Differential Equations

the next infinitesimally small instance of time by specifying this F. And in the continuous case,


Neural Ordinary Differential Equations

this is to say that the derivative of the hidden state is now parameterized by a neural network,


Neural Ordinary Differential Equations

right? So if you know what a differential equation is, it has like, it's like, let's look at it here,


Neural Ordinary Differential Equations

it's like a start state and then what you do is you specify how in each, at each point in time,


Neural Ordinary Differential Equations

so that's T, at each point in time, how does the gradient look? So maybe the gradient looks like


Neural Ordinary Differential Equations

this and then what an ODE solver will do is the ODE solver will say, okay, the gradient is going


Neural Ordinary Differential Equations

to do an infinite small step in this direction and then it goes back to F. Hey, F, what's the


Neural Ordinary Differential Equations

gradient at this infinitely small step next in time? And then F would say, well, the gradient


Neural Ordinary Differential Equations

is like this and then the ODE solver will go like, okay, I need to be a little bit flatter,


Neural Ordinary Differential Equations

so I go here. So what's the gradient at this time? Okay, maybe it's up this, I need to go up here. So


Neural Ordinary Differential Equations

the ODE solver will kind of construct a curve and at each point it needs to look that whatever F


Neural Ordinary Differential Equations

says is the gradient is actually the gradient, right? F, this is the gradient, this is the gradient,


Neural Ordinary Differential Equations

this is the gradient. So that's, that's kind of how an ODE works and that's, they say, okay,


Neural Ordinary Differential Equations

you can actually look at residual networks here as being a discrete time analog to such an ODE.


Neural Ordinary Differential Equations

So what we want to do is actually, we want to specify, we want to actually, and this is the,


Neural Ordinary Differential Equations

the crazy part, right? Or the cool part is we want to do this for neural networks. Basically,


Neural Ordinary Differential Equations

we simply specify an ODE and the start state here, the start state is, let's say, if you want


Neural Ordinary Differential Equations

to build an MNIST classifier, it's our, it's our image, right? The start state is our MNIST image


Neural Ordinary Differential Equations

and we're simply training a neural network such that the ODE, the equation, if you solve it,


Neural Ordinary Differential Equations

the curve at the end will arrive at the correct class. I mean, that's, that's, I'm skipping a few


Neural Ordinary Differential Equations

parts here about dimensionalities and so on, right? Because you need to keep in the same dimension.


Neural Ordinary Differential Equations

But in essence, they say here, we start out with our input and we train the neural network to give


Neural Ordinary Differential Equations

us the correct gradients, the correct derivatives of this curve at each point in time, such that


Neural Ordinary Differential Equations

when you solve the ODE at the end point, you are going to be at the correct label. So that's,


Neural Ordinary Differential Equations

this is the input to your task basically, and this is the output, right? But instead of having


Neural Ordinary Differential Equations

a neural network go from input to output, you have a neural network that parameterizes how you


Neural Ordinary Differential Equations

go from each step in time to the next one. What's, what's the gradient at each point in time? That's,


Neural Ordinary Differential Equations

that's the kind of gist of it and that's, that's kind of really cool. It's a really new approach.


Neural Ordinary Differential Equations

Alright, so they give various advantages of this and so here is this demonstrated again,


Neural Ordinary Differential Equations

right? You are here, this is your input and you want to go to the output and then the loss of,


Neural Ordinary Differential Equations

the loss that you specify, it can depend on kind of either on the output as in like an image


Neural Ordinary Differential Equations

classifier or it can depend on intermediate states. This is, it's kept general, right? So the


Neural Ordinary Differential Equations

way they go about it is they say, well, okay, but so the neural network now specifies how to


Neural Ordinary Differential Equations

get from one step to the next, right? Here. And the neural network has parameters, right? So we,


Neural Ordinary Differential Equations

we need to train this neural network such that the correct output is given to some input, right?


Neural Ordinary Differential Equations

We actually need to train it. So we need to, we need to somehow way to train these parameters


Neural Ordinary Differential Equations

theta and they say, okay, we do gradient descent on theta like in a classic neural network. But


Neural Ordinary Differential Equations

now we need, it's not, it's not so easy, right? It's not one pass through this function. It's


Neural Ordinary Differential Equations

like infinitely many passes through this function until you arrive here. And then if, if you,


Neural Ordinary Differential Equations

you basically need to somehow get a gradient with respect to these parameters here. So they say this


Neural Ordinary Differential Equations

again, the loss of the, this is the loss of the end state, right? Is the loss of the start state


Neural Ordinary Differential Equations

plus the integral over time of this is derivative, which is basically this curve. And the, the curve


Neural Ordinary Differential Equations

is given by an ODE solver where we input all these things. So we need gradients with respect


Neural Ordinary Differential Equations

to that. How do we do that? And they gave a way here of saying, okay, we could either kind of


Neural Ordinary Differential Equations

back propagate through the ODE solver, but that would, you know, depend on the ODE solver and so


Neural Ordinary Differential Equations

on. But there's another method. There's called, what's called the, we need the, what's called the


Neural Ordinary Differential Equations

adjoint. So this is reverse mode differentiation of an ODE solution. The adjoint sensitivity method


Neural Ordinary Differential Equations

solves an augmented ODE backwards in time. So basically what you need to do is you forward


Neural Ordinary Differential Equations

propagate, you come here, right? And then what you can do is you can solve the second ODE. So you can


Neural Ordinary Differential Equations

generate a second curve here, this one, and don't worry about these little jumps here. You can solve


Neural Ordinary Differential Equations

the second curve and the second curve together with the first and the second curve. You can then


Neural Ordinary Differential Equations

compute the gradients you need, right? So the second curve is basically simply something like


Neural Ordinary Differential Equations

the application of the chain rule to the continuous domain. And you need to adjust these jumps here


Neural Ordinary Differential Equations

only when your loss depends on intermediate states. This is kind of the offset caused by


Neural Ordinary Differential Equations

including or not including the loss. So let's dive a bit further into this adjoint state. What's the


Neural Ordinary Differential Equations

red curve? The red curve is called A. And what's A? A is a curve and this is the differential equation


Neural Ordinary Differential Equations

for it. Again, we specify the curve A by specifying its start state and its derivative. And from its


Neural Ordinary Differential Equations

start state and its derivative at each time, the ODE solver is able to construct the curve entirely.


Neural Ordinary Differential Equations

So AT, it says here, is del L to del ZT. This means how does the loss depend on this ZT, on the hidden


Neural Ordinary Differential Equations

state, right? How does the loss depend on the hidden state at time T? So it doesn't even have


Neural Ordinary Differential Equations

to be any of these points here. How does the loss depend on this hidden state here? And in order to


Neural Ordinary Differential Equations

find that out, you would need to go, you would need to develop the curve until here, right? And


Neural Ordinary Differential Equations

then calculate the loss and then back propagate through here. But you can do this by calculating


Neural Ordinary Differential Equations

this adjoint thing. So as you can see, here is a demonstration. It's an example, right? So the


Neural Ordinary Differential Equations

start state here is simply given by the loss. How does the loss of this state, how does the loss


Neural Ordinary Differential Equations

depend on this state? Well, simply by plugging it into the loss equation, right? So your loss is


Neural Ordinary Differential Equations

maybe a cross entropy loss or something. How does the loss depend on this state here? Well, we go,


Neural Ordinary Differential Equations

we go from this state that we already know, and we know how in reverse time, so backwards in time,


Neural Ordinary Differential Equations

this sensitivity of the loss develops. So we go, we develop this curve until here and we say,


Neural Ordinary Differential Equations

aha, this point influences this loss in this much basically, right? So, and if the loss explicitly


Neural Ordinary Differential Equations

depends on this point, then we have to, we have to calculate in this offset since this point here


Neural Ordinary Differential Equations

only depends on this time up till here. And then it changes, so there's a discontinuation, but yeah.


Neural Ordinary Differential Equations

Don't worry about that too much. Basically, what we can do is we can calculate the curve in a forward


Neural Ordinary Differential Equations

pass. Curve and the loss in the forward pass. Then we can do a second pass backward again by


Neural Ordinary Differential Equations

an ODE solve to say, how does the loss depend on each one of the states here, of the hidden states,


Neural Ordinary Differential Equations

right? So that's the second point, but that's not all because we're ultimately not interested in


Neural Ordinary Differential Equations

the, how the loss depends on the state. We're interested in how the loss depends on these


Neural Ordinary Differential Equations

parameters that tell us how to get from one hidden state to the next. But luckily we can then


Neural Ordinary Differential Equations

simply evaluate this integral that depends, as you can see here on A and on Z. We can evaluate


Neural Ordinary Differential Equations

this and get the gradients for the parameters, right? So I also have to say the parameters are


Neural Ordinary Differential Equations

static, so the parameters are given over the entire duration of this. They're the same. It's


Neural Ordinary Differential Equations

simply what changes is time. All right, so this is how you can get, this is how you can get


Neural Ordinary Differential Equations

gradients with respect to parameters. And the cool thing is now you can train these. You can


Neural Ordinary Differential Equations

actually train this neural network here that tells you how to go from one state to the next,


Neural Ordinary Differential Equations

such that if you input the digit 2 as an image, you can output 2. I mean, not exactly, but that's


Neural Ordinary Differential Equations

that's the point, right? You can, by going through this motion, by going through this ODE solve. So


Neural Ordinary Differential Equations

that's, I mean, that's immensely cool. They actually define how to do this here in one forward,


Neural Ordinary Differential Equations

one kind of backward pass. You can solve everything at the same time. It's pretty cool. And they


Neural Ordinary Differential Equations

evaluate their net and they compare it with a bunch of other nets. And they interestingly show


Neural Ordinary Differential Equations

that, so basically with an ODE solver, you can never kind of tell how many evaluations it's going


Neural Ordinary Differential Equations

to do because it's going to get increasingly accurate over time. So you let it run and maybe


Neural Ordinary Differential Equations

it's going to first generate a curve that's like something like this, right? And then it needs to


Neural Ordinary Differential Equations

say, crap, okay, I need to go back and refine and then it maybe goes the curve like this and so on.


Neural Ordinary Differential Equations

So it gets continually closer over time. And for that, it needs to kind of query. It's like a query.


Neural Ordinary Differential Equations

It needs to query this F. So you need to give it the function as an evaluable function. And it goes


Neural Ordinary Differential Equations

and just, okay, I need to know it here. Okay, I got it from here. Okay, I need to know it here. Okay,


Neural Ordinary Differential Equations

I got it from, oh no, I didn't get it. Okay, I also need to know it here. All right. And so you


Neural Ordinary Differential Equations

can never know how much they will evaluate, but you basically have a parameter to trade off accuracy


Neural Ordinary Differential Equations

and how much they evaluate. That's what they show here. So the less error they want in their forward


Neural Ordinary Differential Equations

pass, the more forward passes they have to do. That's this curve. The more forward passes they do,


Neural Ordinary Differential Equations

the more time they have to invest, right? That's this curve. Interestingly, the more forward passes,


Neural Ordinary Differential Equations

the time required for forward passes or the evaluations required for forward passes increases


Neural Ordinary Differential Equations

also the evaluation required for backward passes, but not by much. So the backward passes require


Neural Ordinary Differential Equations

about half the amount of evaluations as the forward passes, which is encouraging since the


Neural Ordinary Differential Equations

backward passes don't go kind of overboard. Like if you had to back propagate through the operations


Neural Ordinary Differential Equations

of the ODE solver itself. And they also show as your training epoch continues, the ODE solver


Neural Ordinary Differential Equations

requests more and more evaluations for the same epoch basically, or the same samples within


Neural Ordinary Differential Equations

different epochs, which means as it gets more accurate, it needs to know more and more and more


Neural Ordinary Differential Equations

about the samples basically, about the test, the training samples, which is all basically showing


Neural Ordinary Differential Equations

that this kind of works. Yeah, so they kind of get into normalizing flows, which I don't want to


Neural Ordinary Differential Equations

get into here much because we haven't done a video on that yet. We'll do one. But they basically show


Neural Ordinary Differential Equations

that it's quite easy to do normalizing flows in a continuous fashion. And the topic of normalizing


Neural Ordinary Differential Equations

flows is in itself pretty cool. What they do at the end is they say, okay, what we can now do is


Neural Ordinary Differential Equations

we can actually take sequential data. So now we've just talked about let's input one data point


Neural Ordinary Differential Equations

and get out, let's say a label or something. We can actually do sequential data. And let's, for


Neural Ordinary Differential Equations

example, have an R&N encoder for our sequential data. So here, these are data points, right? These


Neural Ordinary Differential Equations

are measurements, like blood pressure of a person. And what we can do is we can do a variational


Neural Ordinary Differential Equations

autoencoder. We've talked about this. We can have an R&N encoder, parameterize a distribution, and


Neural Ordinary Differential Equations

then as a decoder have this ODE neural network. And basically what that allows us to do is that


Neural Ordinary Differential Equations

allows us to deal with time steps that are not regularly sampled. And so we can extrapolate from


Neural Ordinary Differential Equations

the data point at times not regular samplings. Like with R&Ns, you're basically forced to have


Neural Ordinary Differential Equations

always the same time step difference. Otherwise, you have a very tough time. But with this, since


Neural Ordinary Differential Equations

these are continuous flows, you can basically unroll them and evaluate them at whatever time


Neural Ordinary Differential Equations

you want. So they have pretty cool experiments here where they kind of try to learn these kind


Neural Ordinary Differential Equations

of spiraling behaviors. And you see on top the R&N decoder will get all jaggy and so on, where


Neural Ordinary Differential Equations

as the neural ordinary differential equation will generate quite, let's say, smooth things. And also


Neural Ordinary Differential Equations

it can extrapolate, as you can see here. The red thing is the extrapolation. There's only data


Neural Ordinary Differential Equations

where the green dots are. So that's pretty cool. You can see the R&N sometimes isn't able to kind


Neural Ordinary Differential Equations

of continue the flow, as you can see in here. It extrapolates wrongly. So these kind of, I mean,


Neural Ordinary Differential Equations

it's a toy example, but these kind of dynamics are pretty cool. And they also show here when they


Neural Ordinary Differential Equations

learn the spirals and vary one dimension of the latent code that is given by the encoder, then


Neural Ordinary Differential Equations

the flow goes from clockwise, it goes from to counterclockwise, as you see here. I've drawn this


Neural Ordinary Differential Equations

in wrong, but so it's pretty, pretty cool what these things learn. And since it's only small data


Neural Ordinary Differential Equations

right now, small models, but I'm pretty sure this is going to develop further and be a cool, just a


Neural Ordinary Differential Equations

cool way, cool alley of research, cool idea, and looking forward to what they come up next. All


Neural Ordinary Differential Equations

right, so that was it for today, a bit shorter. But I hope this was somewhat clear enough. All


Neural Ordinary Differential Equations


Imputer: Sequence Modelling via Imputation and Dynamic Programming

Hi there, today we're looking at the imputer sequence modeling via imputation


and dynamic programming by William Chan, Chitwan Sariah, Jeffrey Hinton, Mohamed


Nourouzi and Navdeep Jaitley. So this is a model to perform sequence to sequence


tasks. Now sequence to sequence tasks are very very common in NLP but in this case


it's kind of a subset of sequence to sequence tasks. So a classic sequence to


sequence task is machine translation. Here for example the sentence I like you


if you want to translate it to German sorry you if you want to translate it to


German that would become ich mag dich. And you see that the input is a sequence


right and the output is a sequence. Now the imputer deals with very special kind