Deep reinforcement learning, symbolic learning and the road to AGI
By: Jeremie Harris
Tags: Machine Learning, Ai Alignment And Safety, Symbolic Ai, Tds Podcast, Reinforcement Learning

Please find below the transcript for Season 2 Episode 2:

Jeremie (00:00):

Hello again, my name is Jeremie and I’m the host of the Towards State of Science podcast, and part of the team over at the Sharpest Minds Data Science Mentorship Program. And today, we’re continuing our series on emerging problems in data science and machine learning, by welcoming Tim Rochdechel to the podcast. Tim is a research scientist at Facebook AI research, where he’s working on a number of projects related to advanced AI systems, including sample efficient machine learning, which is focused on reducing the amount of data needed to train machine learning models, as well as things like symbolic AI.

Jeremie (00:31):

Now Tim is deeply knowledgeable about the current state of the art machine learning, and we’ll be talking about just that, as well as the potential future direction of artificial general intelligence research. I hope you enjoy the conversation.

Jeremie (00:43):

All right, I am here with Tim Rochdechel. So he’s a research assistant, sorry, research scientist, at Facebook AI research and lecturer at the Center for Artificial Intelligence at the University College London. He’s going to fill you in about all this stuff, but his work is really interesting, focused on what he’s calling sample efficient and interpretable machine learning. We’ll get into what that means, I’m sure. There’s a whole discussion we’ll have about that, as well as some of the recent developments in the space, reinforcement learning as well, Opening Eye’s GPT-3 model, I’m sure will come up a little bit. We’ve got a lot of ground to cover, but for starts, I just want to ask you, Tim, can you introduce the field in your own words that you’re working on? Some of the interesting problems that you’re working on day to day.

Tim (01:43):

Sure, yeah. First of all, thanks so much for the invitation. It’s really, really great to be here. So, I guess I’ve been working at the intersection of reinforcement learning and natural language processing in the past, and also currently. So in reinforcement learning, I think I would imagine that many of the listeners are familiar with the setup of basically, you have an environment, you have an agent. Your agent isn’t acting with the environment, it’s obtaining [inaudible 00:02:16] from the environment and aims to maximize expected future rewards over time, some of the future rewards.

Tim (02:23):

So that’s a very general approach, right? Something that’s very generic that you could apply potentially to lots of different situations, and then at the same time, I guess the progress of reinforced learning in terms of being applicable to real life problems has been relatively limited over [inaudible 00:02:42] we can talk about that later.

Tim (02:44):

And then at the same time, as I mentioned, I’m occasionally working on natural language processing problems, so here the question is, how can we develop machines that can process, and to some extent, make use of natural language texts? So, for instance, a common problem in natural language processing is question answering. Can I develop models that give it a question, that give you a free form answer, right?

Jeremie (03:12):

Mm-hmm (affirmative).

Tim (03:12):

Or, not so much of a actual real life problem, but a very popular class of models, so-called language models, so models that, given some text, are really good at predicting what’s the next word in the text and can then generate text. So we’ve seen a lot of progress in that over the last few years.

Jeremie (03:34):

Interesting. And is there a reason for the marriage between reinforcement learning and natural language processing? Because historically at least, this is my understanding from a 30,000 foot view or whatever, historically, reinforcement learning and, say, computer vision might’ve enjoyed a tighter interaction, focus there being on self driving cars, maybe on robotics as well. Is there a reason that you’re seeing NLP as one of the interesting ingredients here, or is that, it’s just two independent lines of research that happen to be happening concomitantly?

Tim (04:08):

No. I definitely see a lot of synergies between the two, so it’s a really excellent question. So first of all, people have been traditionally applying all kinds of techniques from reinforcement learning in natural language processing tasks. So, people have been using reinforced to optimize directly for machine translation performance instead of using negative [inaudible 00:04:30] training for [inaudible 00:04:31] models. So that has been done quite a bit. And then, conversely, people have been also looking into instruction following, so given a certain natural language description of a goal that an agent needs to achieve, how can we design agents that can condition on such natural language information to then achieve a goal in the environment?

Tim (04:53):

The reason why I strongly believe in that marriage between natural language processing and reinforcement learning is, to a large extent, due to the fact that so far, we’ve been seeing a lot of success of reinforcement learning in environments where you basically can train tabula rasa. So that means your expiration policy allows the agent, over time, to learn or to explore the environment sufficiently in order to come up with really, really good policy. And that, for instance, works in environments where your environment dynamics are somewhat limited. So for instance, a good example, I think, is the game of Go, which has been approached a few years ago, right? By Deep Mind.

Tim (05:46):

And it’s a really impressive accomplishment, in that they were able to train an AI that beats the human experts in the game of Go. And it’s, however, important to note that the dynamics in Go are relatively simple, right? The rules are very clear, so you have a stated time step tee, there’s a specific action, it’s absolutely clear what will happen at the next time step. So that allows you to, basically, do a Monte Carlo tree search. That means you can actually try to plan ahead what will happen with certain sequences of actions.

Tim (06:23):

But then there are so many other environments where people are bringing so much domain knowledge and common sense knowledge and work knowledge, and only that is the thing that allows them to do well in these environments, right? So I think a good example, for instance, is Minecraft. So if you play Minecraft, there’s a lot of things that are quite similar to how things would behave in the real world, right? So if you have a torch and you start burning a tree, it just burns down. And I think that’s really exciting, right? So how do we make sure that our agents can tap into the wealth of knowledge that humans gathered over time, that they write down in Wikis, that they write down in tables and whatnot? And they [inaudible 00:07:15]

Jeremie (07:17):

And part of this, too, or at least as I’ve always understood it, is, part of our knowledge, of course, is learned over time as we’ve, as you say, read Wikis or read textbooks or just been taught things by the world, but then part of it is also, I guess, innate in the sense of having been taught to us by a much slower process, which is evolution. And so we’re born, not as a blank slate, but we’re born with specific biases towards faces that look a certain way and so on and so forth. I guess, in a way, this seems to cover both of those aspects. If you actually can summarize a body of human knowledge, I don’t know, the Wikipedia corpus or all the language people have written, I guess the hope is you’re covering both of those bases? Is that fair to say?

Tim (08:03):

Well, I think it’s a really great point. I do believe you have to separate them to some extent, right? You have to say, “Okay, there’s knowledge that people write down. Things that they acquire over the course of their lifetime and they want to share the knowledge with other people, and potentially, at some point, right, with agents, artificial agents.” And that’s something I’m focusing on most of my time. Then the other aspect that you mentioned, evolution and the innateness of certain behaviors that allow you to, for instance, either directly behave really well in the environment or allow you to learn really quickly what are good policies in the environment. I think that’s a really exciting area as well. I’m not too familiar with what is going on in that area of research, other than, I believe, I strongly believe that we need to have agents that are couriers to explore.

Tim (09:01):

So it’s not enough, for many environments that are realistic, it’s really not enough to try to maximize reward based on what the environment gives you as a reward, right?

Jeremie (09:10):

Yeah.

Tim (09:11):

In many environments, I think it’s absolutely crucial that you set yourself goals, that you reward yourself for somehow encountering unexpected things, for increasing your knowledge about the environment. I think that’s a really crucial point. That’s why people like to play games like Minecraft, that they just learn about how certain things work. And I’m talking a lot about Minecraft here, but actually, I believe there’s other games that are maybe more suitable for reinforcement learning.

Jeremie (09:41):

Like what kinds of games?

Tim (09:42):

So we released, a few weeks or months ago, we released a NetHack learning environment. So this is a reinforcement learning environment centered around a very ancient game called NetHack. And NetHack was originally released in 1987. It’s played in the terminal, so this is really a time where there wasn’t even any colored screens.

Jeremie (10:06):

Yeah.

Tim (10:06):

You really played this in a Linux terminal. So all the observations are [inaudible 00:10:12] characters. It looks really arcane, it’s really exciting. So basically, you’re thrown into this dungeon. Your player is this “at” symbol. You can move around in that dungeon. Monsters are other askey characters, like D could be a fox or could be a dog or something like that. You walk around, you find items, weapons, armor and so on. And one of the aspects that’s really exciting about this game, similar to Minecraft actually, is the fact that it’s procedurally generated. So, maybe it’s more fun to talk a bit more about that.

Tim (10:48):

When deep reinforcement learning became really popular, at was at a time when researchers were using the arcane learning environments. So Atari games, in order to test whether agents can learn to play games like Pong or Ms. Pacman or Montezuma’s Revenge. And that really drove reinforcement and research for a long time. But interestingly, by now, I think almost all these games have been eaten by artificial intelligence.

Jeremie (11:19):

Mm-hmm (affirmative).

Tim (11:20):

And as often, this tell us a bit more about the actual environments and problems that outstand, it tells us about how intelligent our AIs are. And the reason this is important is because in Atari, the environment is, in some sense, static. So every time you play this game, it’s going to be the same.

Jeremie (11:40):

Yeah.

Tim (11:41):

Right? So every time you play Pong or Breakout, there’s not really much variation in terms of the observations that you’ll see. Similarly, if you think about Super Mario, right? Every time you play Super Mario, the level’s going to be the same.

Jeremie (11:53):

Yeah.

Tim (11:54):

So over time, what you will learn to do is almost memorize what kind of sequence of actions you have to do in order to do well.

Jeremie (12:00):

Yeah, [crosstalk 00:12:01] and you over fit to your environment.

Tim (12:03):

Exactly, right? And specifically, the environment is deterministic. That means if you find yourself in the same situation later on, you will do the same action, you will see the same result. So it’s deterministic and what I would call somewhat static in terms of the variables of the environment observations. And recently successful approaches haven’t explored that. So there’s really amazing work that came out by a Uber AI, 2018, I think, called Go Explore, where they solved this really, extremely hard Atari game called Montezuma’s Revenge. And they do this by basically making sure that they reset the agent, or the environment actually, they reset that to a previously visited state of the agent, and then explore from there. Right? So, really over the time, the agents are memorizing certain sequences of actions through that game.

Tim (12:56):

And that led multiple researchers now to consider what’s called procedure generated environments. That means there, there’s a generative process of, every time you start an episode, the observation is basically generated in front of you. That means every time you start an episode, you will see things that you’ve never seen before, like in Minecraft, right? The world is generated every time you play a new game of Minecraft.

Tim (13:20):

And NetHack is like that. So every time you’re thrown into a dungeon, you don’t really know what to expect in terms of the topology of the dungeon or the arrangement of specific rooms and secret doors and so on. And that really forces our agents to, in order to do well, to generalize to these novel situations.

Jeremie (13:36):

Interesting. So, this is, and I guess this is a qualitatively different problem from Montezuma’s Revenge. If I remember, that one had a similar issue where rewards were often hidden or a little bit further. You had to do some counterintuitive stuff to get to those rewards. I guess that’s a different class of problem from just, now you’re faced with a completely new environment, and what you’re actually learning to do is not just, well, in a way, it feels like an exploration/exploitation thing. You don’t want to over invest in exploiting this one deterministic environment. You need to get good at, as well, exploring and orienting yourself in this new environment and the skill of figuring out where you even are is something that, now, people are starting to work on? Is that the idea of this research?

Tim (14:21):

Yeah, I think that’s a relatively good assessment. So, in Montezuma’s Revenge, you’re right, this was a sparse reward environment. That means you’re not really bumping into many extrinsic rewards that come from the environment, so you really have to explore for a long time before you get a positive signal from the environment that you’re doing something right. But interestingly, Montezuma’s Revenge already exploits a lot of human prior knowledge and biases. So, in Montezuma’s Revenge, you have to collect keys and you have to open doors, and if you run around in Montezuma’s Revenge as a human player and you see a key and pick it up, then you know, okay, there must be a locked door somewhere, and then when you see a locked door, you open it.

Tim (15:01):

But for reinforcement learning approaches, at least the ones that have been around for longer, they have very dump exploration strategies, right? They occasionally try out a random action and see if that really helps. So what we really need are approaches that do structured and goal oriented explanation that motivate themselves, right, to explore, in a very target way, certain aspects of the environment. That learn, over time, about the environment dynamics, about the things that you can interact with and how to interact with those.

Tim (15:36):

And just to draw, maybe the connection, again, to NetHack, in Montezuma’s Revenge, you have keys and doors and so on. In NetHack, you have hundreds of objects. All kinds of things, right? All kinds of weapons, all kinds of tools like tin openers and land and keys and whatnot, right? And there’s really such a great wealth of objects to learn about and also such a great wealth of enemies to defeat, that human players, when they learn to play NetHack, almost all of them have to consult external knowledge sources themselves. So there’s a really large Wikipedia called the NetHack Wiki, where human players collect their shared wisdom about how this game works and what you can do and what you should do and not do. And people, when they play this game, they really learn to, well, do things in the environment while also looking up things on this NetHack Wiki.

Jeremie (16:30):

So so much of this seems to depend on your ability to create, or to have a coherent model of the world to supplement your exploration. I think this ties in, actually, to GPT-3, which Opening Eye, of course, released fairly recently, the last couple of months. The reason it brings this to mind, GPT-3 is one of those natural language oriented ways of at least attempting to construct a model of the world that’s fairly complete, fairly coherent. I could imagine, just based on some of the things that GPT-3 seems to be capable of, some of the connections it seems to be able to draw, that it would not be a bad knowledge base to start from, in terms of taking on problems like this. Do you think that’s a fair assessment? Could you use that for transfer learning and combine it with RL?

Tim (17:18):

Yeah, I think there are at least two different ways one could utilize such pre-trained language models. So, one approach is to, let’s say you have an external textual knowledge source like the NetHack Wiki and you want to condition your reinforcement agent based on that textual information. Then usually, it’s a bad idea to try to learn about language and try to learn about how to behave in the environment from scratch at the same time, right? So reinforcement learning is hard, natural language processing from scratch is hard. If you do both together at the same time, it’s not going to make things easier. So then what you want to do is, you want to have a good starting point, right?

Tim (17:59):

You want to have a model that can already build for you, relatively good representations of the sentences that are written in this Wikipedia so you can then run a pre-trained language model over the entire Wikipedia or the NetHack Wiki. You build up all the sentence representations or word and context representations or whatever you want to call that, and then you can try to query these neural representations while acting in the environment. So that gives you one way to operate with this Wiki.

Tim (18:32):

And while that hasn’t been done as far as I know, that hasn’t really been accomplished yet for complex reinforcement learning tasks, like the NetHack learning environment, there’s reason to believe this is a sensible approach because there are question answering systems that are able to condition on such large Wikis. The entire Wikipedia, for instance. Right?

Tim (18:54):

And the second approach would be to, or the second direction would be… And I think that’s probably what you meant. Is to maybe use pre-trained language models such as GPT-3 as a good prior of where, for instance, affordances of objects, right? You can query GPT-3, I have a door, what can I do with it? And it would probably tell you, well, you can open it, open it with a key, kick it in, whatever, right? It will give you, probably, quite a number of sensible things you might want to try out, and that, obviously, could be used for a better exploration strategy that your reinforcement agent could apply [inaudible 00:19:35]

Jeremie (19:35):

Yeah. It always struck me that language modeling, especially more complete language models like GPT-3 and GPT-4 and whatever else is going to come down the line, would be particularly promising for this sort of thing, that tabular rasa idea that you brought up, the blank slate. How do you teach an agent to orient itself in the world from scratch, or how do you avoid having to do that? Yeah, giving it that leg up with these pre-trained models, it seems like a promising strategy anyway. So, I think one cool theme, as well, that we could talk about in the context of your research is, it seems a large part of it is involved, finding ways to convey human level information to neural networks, to machine learning models, I guess in a very abstract sense.

Jeremie (20:21):

We’ve talked about your focus on NLP, the interaction with RL a little bit. You also mentioned before we started talking that you’d been doing this work on deep learning systems and trying to get them to basically work with logical rules, so allowing humans to convey logical rules to constrain the behavior of deep learning systems. Would you mind exploring that a little bit and explaining what went into that work?

Tim (20:43):

Yeah, absolutely. So this was work that I did mostly during my PhD. I think the idea there was that deep learning systems are really great. Deep learning systems like GPT-3 or like deep reinforcement agents, they’re really great at learning from a lot of data. Right? But then, the problem is that, for many domains, we don’t have a lot of training data, or we might want to make sure that we have certain guarantees that, after we’ve been training the system, it will make some predictions. It will always predict that man are mortal, to just give you one example. And it’s quite interesting because there’s a very long tradition in symbolic artificial intelligence where people have been thinking about how to work with logical rules, how to infer new things based on existing knowledge, so existing facts and existing logical [inaudible 00:21:46]

Tim (21:46):

And there, it’s really funny because you don’t need any training data, right? So you can write down the facts about a specific domain, you can write down certain rules about that domain, and then you can just make inferences. And also, you have then really strong guarantees. You know that, if you have a rule that tells you every man is mortal, every time you have a particular man, you will predict that that man is mortal. If you look into deep learning systems, there are so many examples of adversarial attacks, right? Where you have, let’s say, a system that classified traffic lights and you change on specific pixel in the image and the prediction will be different, and that obviously has all kinds of catastrophic consequences.

Tim (22:31):

Now, the problem with logical rules in symbolic systems is that they can’t generalize beyond what you explicitly write down. Right? You have to write down all the things you need to write down, basically, right? So, to give you a concrete example, let’s say you have The Simpsons and you have Bart Simpson, Homer Simpson, and Grandpa Simpson. Abe, right? Because of Grandpa Simpson. And you have facts that Homer Simpson is a parent of Bart Simpson, and Abe Simpson is the parent of Homer Simpson, and you have a rule that states, every father of a parent is a grandfather. Right? So now you can make that predication that actually, Abe Simpson is the grandfather of Bart Simpson.

Tim (23:15):

But that’s just for that specific symbol, namely the grandfather of relationship. But what about grandpa? Right? Grandpa and grandfather, they are two different words, they’re two different symbols, so how do you make sure that we can also make softer inferences? And there, I looked a lot into, I guess, combining the merits of neural representations of deep learning as well as symbolic systems, so rule based systems. And we’ve… Yeah.

Jeremie (23:44):

That makes sense because you need that dexterity of… In a way, you need an embedding for grandpa, and then you need the words that are close in that embedded space to say, “Oh, yeah, these are probably synonyms.” Is that part of the idea here?

Tim (24:00):

Yeah, so we followed two directions back then, and one was, you have a neuron that works, that try to predict, given a specific fact, like Abe, grandfather of Bart. Trying to predict whether or not that fact was true, and there are models out there that try to predict that based solely on neural representations of the two entities. In this case, Abe and Bart, as well as a representation of the relation, grandfather of. And there, we use logical rules to constrain or directly regularize these learned representation such that in the future, whenever we have a grandfather of relationship in a neural representation [inaudible 00:24:42] it’s very likely that we predict it between somebody who’s the father of a parent of someone.

Tim (24:47):

So that was one line of work, and the other line of work was closer to what you were mentioning. So it’s a really good idea. I mean it’s a really good comment that you made. So that was a NeurIPS paper from 2017 where we actually took an existing data lock proof system, and we basically turned the entire so-called, a backward chaining algorithm that’s used in prologue and data log, old school AI systems, turned that into a neural network, basically. So that was at a time where people looked a lot into using existing algorithms and data structures, and do what, yeah, was called neuralizing them, basically turning them into neural networks, making them [inaudible 00:25:31] such that they can do these kind of soft comparisons and can train [inaudible 00:25:35]

Jeremie (25:34):

That’s really interesting. So, one of the reasons I find this so compelling is a big focus… Actually, a big focus for this podcast series and a big focus for me, personally, as well has been this issue of AI alignment. Just the idea that, as our intelligent systems get more and more powerful, more and more effective, we need to be able to convey human values to them, we need to be able to convey the constraints that, we don’t want an RL system that is physically embodied or whatever. We don’t want to have a system like that running around, doing harm to people because it’s operating, let’s say, in an out of distribution context or it hasn’t considered some weird, adversarial possibility.

Jeremie (26:13):

And it just seems like this idea of allowing us to communicate at a symbolic level, using logic, directly to neural networks, at least opens up a certain avenue for making sure that, for example, you could actually convey the idea that, don’t kill people, for instance. Or something like that. Not that it’s that simple. Obviously, there’s tons of downstream complexity, as you mentioned. How is the word “people” going to be interpreted? How is the word “kill” going to be interpreted? And so on. But it does seem to make it a little bit easier for us to communicate with machines, with these algorithms in a way. I’m using the word communication loosely, but do you see that as playing a potential role in this outer alignment problem of having humans at least convey to machines what they want in a more effective way?

Tim (26:59):

I think it’s a fantastic question. First off, I should honestly say that I’m not an expert in AI alignment or AI safety, so anything I’m saying in that regard should be taken with a huge grain of salt. I do believe that we, nowadays, have new ways to constrain artificial [inaudible 00:27:20] using domain expert knowledge, for instance in the form of logical rules. However, that assumes that you can write down all the things you want to teach to your AI system in form of logical rules, right? That’s a big if. I could imagine tons of scenarios where that simple rule that, don’t kill someone, right, is super problematic. I’m pretty sure that, actually, I know that people have been thinking about exactly this kind of exemptions a lot. So that’s one issue.

Tim (27:48):

The other issue is that I don’t think we have already cracked to what extent do we completely enforce, let’s say, logical rules, to what extent do we want to give models enough freedom, right, to learn about the messy world we’re living in, right? If you take natural language, for instance, there’s so many messy things in there, right? There’s so many ambiguities and problems that we, as humans, even face, and our AI systems or pre-term language models and natural language understanding systems still struggle to work with. So, I do believe we have, still, an enormously long way to go.

Tim (28:32):

But at the same time, I think there’s, in a way, somewhat a number of synergies, in that it’s not just about making sure our AIs are aligned with our values, it’s also about how we even make sure that our AI systems are sample efficient, so it means, can they learn quickly to adapt to novel situations? Can they explore environments efficiently? For that, they also have to learn to understand, I think, natural language and our intentions and our priors and biases.

Jeremie (29:05):

Actually, I think one assumption that’s been in the back of my mind, and unspoken, that your last statement just brought to mind is, why is it that natural language is such a good candidate for constructing these models of the world? Maybe, I don’t know, do you want to elaborate on that thought a little bit? Why would a language model be better than, say, I don’t know, a computer vision model trained on ImageNet or something like that? What is it about language that makes it really promising as a source for models of the world that then can be applied to reinforcement learning agents, to give them a leg up so they don’t have to learn these things from scratch?

Tim (29:48):

Well, I can give you one example. Again, this centers around NetHack, this game that I mentioned earlier, in that you can maybe use… If there was a, let’s say, 3D rendering of NetHack, which doesn’t… Actually, it does exist, but let’s say there’s a 3D rendering of NetHack. Obviously, you can use pre-trained computer vision models to give your model a sense of what are certain objects, right? There’s maybe a chair somewhere, something, like armor somewhere else. But what’s important in games like NetHack or in many, I think, actual environments that we care about in the real world, is that we have knowledge written down and text that is procedural that tells us about how to do certain things, right?

Tim (30:33):

So NetHack, there’s a lot of strategic guidance on the Wiki that tells you what you should be and should not be doing in certain situations, that kind of knowledge, we, as humans, at least convey using natural language. That doesn’t mean that natural language is the best way for building world models. I could imagine you could have AIs, if they’re really smart, they might come up with all kinds of internal representation to encode how the world works, and that might not reflect or connect to our human natural language. But at least right now, it’s the only source, for instance, I have in order to condition agents to learn about NetHack, right? Because people have been using natural language to write about what to do and not to do in [inaudible 00:31:19]

Jeremie (31:19):

Yeah, and I guess, to the extent that humans can convey, pretty well, arbitrary knowledge to one another through language, to the extent that, I could teach you or you could teach me, anything that it’s possible for a human being to know by writing it out. That implies that language is this knowledge graph, is this implicit model of the world that is as complete as humans are almost able to store. Would you disagree with that idea?

Tim (31:47):

Well, I think there’s still a lot of knowledge that we can’t convey by natural language, right? I think there’s certain situations where I would have to say, “Look, I can demonstrate to you how I do certain things, but you have to really experience that for yourself.” And I can try to explain to you, as much as I want, how that smells for me or how that feels for me in the right [inaudible 00:32:13] when I’m doing certain, whatever karate move or so. But you really have to train it, you have to experience it yourself.

Tim (32:21):

So, I think, yeah, it’s fair to say there is still a lot of knowledge that we can’t express or convey in natural language. But I do believe, for everything that’s, I guess, on this level of reasoning, which is obviously very [inaudible 00:32:35] term, right? But this, it’s reasoning that I would expect in AI to be capable of when thrown in games or in simulated environments. I think a lot of that is, indeed, conveyed in natural language, and for many games, we have these textual Wiki resources that we should be utilizing when we train reinforcement agents. And I think also, that is something that then, long term, will allow us to train reinforcement agents that are capable of solving more real world tasks and not just these simulated ones.

Jeremie (33:07):

Maybe this is a good segue into the longer term picture for you. For you, what’s the goal with the AI research that’s being done right now? Is it about approaching, for example, some AGI finish line? Is it about getting to the point where we have AI take off, or is it more, do you see that as being further away, and right now, we’re just working on a series of narrow tasks instead?

Tim (33:33):

I think that’s a brilliant question. I honestly don’t think I’m somebody who would say, is striving for artificial general intelligence. I think there’s an open debate to what extent even humans are capable of general intelligence. What we see in reality is that people are specializing heavily in certain domains. So for instance, I might be an okay research scientists, but I’m definitely not able of piloting an airplane right now. I could probably learn it, but then that comes with all kinds of additional costs, obviously, and there’s a limit to what I can learn about in the real world. And this idea that we’ll have AIs that are generally capable of all kinds of task, I think is an open question if that’s even possible. And to be honest, I’ve been… I’ve been trying to set quite ambitious research goals, right? Having agents that can learn in these procedure generated environments that have to condition on external knowledge sources.

Tim (34:41):

I think that’s something we believe is not going to happen in a year or two. I think this is a longer term effort, and I’ve been very happy about recent progress in pre-training language models, but I don’t believe that will take us all the way there. I think there’s a lot of things that will be necessary before we have agents even able to solve NetHack. As funny as that sounds, right? Because that’s a game [inaudible 00:35:07] silly.

Jeremie (35:06):

Yeah. So I’d love to pick your brain on that because I think this is such a hot button topic for so many people in this space. And obviously, spurred, in large part, by what GPT-3 has shown. Actually, maybe it’s worth just mentioning briefly, what GPT-3 is, because we’ve touched on it a couple times. I think I’ve been a big haphazard, my use of the term. Would you mind providing a quick summary of what GPT-3 is, and then maybe we can tie it into this bigger picture?

Tim (35:32):

Sure. So, GPT-3 is a language model. Language models have a long tradition in natural language processing. They have been, for instance, used as part of machine translation systems. A language model is basically just a model that allows you to score a sequence of words, and it allows you to score how likely that sequence of words is. For instance, you might want to train a language model for the English language, and now it can give you a particular sentence and the language model will tell you that sentence has a certain probability. And you can imagine how that’s useful because if I have a machine translation system that needs to translate from a specific source language to a [inaudible 00:36:13] language, if I can enumerate multiple possible translations, and then I can use a language model to score which one is the best one, right? That allows you to, for instance, have a really good decoder for translating this, from a specific language to another language.

Tim (36:30):

Previously, quite a few years ago, people have been using all kinds of statistical methods for language models, so-called MREM models, have count in the large textual corpus, how frequent are certain unigrams, bigrams, and trigrams, and so on, of words, and then that is something you can use to score sequences of words in terms of how likely they are. And then, again, I guess a couple of years ago, people started using deep artificial neural networks for language modeling. And most recently, so-called transformer architectures.

Tim (37:09):

So these are models that, first of all, have a representation for each word in your vocabulary, and the look, basically, through large sequences of text, so maybe 500 or so words. And at every time step, they try to predict the next word, given the history of previous words. And it turns out that if you make these deep artificial neural networks really big and you collect a huge textual corpus, so we’re talking about gigabytes of textual data… We talk about over 100 billion of parameters in these artificial neural networks. You bring this together, so a really large model and a really large, high quality textual corpus, then you get something like GPT-2 and then GPT-3. So, language models that are really good at producing given history of tokens, [inaudible 00:38:04] history of words produced the next [inaudible 00:38:06]

Tim (38:07):

And what’s really, I think, surprising or really astonishing is the fact that you, in the case of GPT-3, you can prime these models, so you can start, for instance, giving certain examples to GPT-3 like, can you code me in Java, how to reverse a list? And then you write down the Java program that [inaudible 00:38:31] list. And give it a few more of these examples, and then it’s primed and you can start to actually somewhat code in natural language. You then just give, I guess, queries in natural language like, can you now please count the number of elements in the list? And it would give you executable, let’s say Java code that does that for you. So that’s mind blowing, but at the same time, not extremely surprising given, I think, the whole scale of the model and scale of the high quality textual data. That’s used [inaudible 00:39:00]

Jeremie (39:00):

And this was a controversial question, as I recall, about two or three years ago. Outside of the AI alignment community or people who were worried about scaling these models, the idea that you could just, this so-called scaling hypothesis, that you could just make the model bigger, throw more compute at it, throw more data at it, and that you would get not just higher quality results, but more generalization ability out of it, too. That you would go, for example, from a glorified auto complete algorithm to something that now, as you said, with relatively minimal priming, can code in Javascript or something. I think it’s fair to say, that aspect of it is a new thing in the last two or three years, is that fair to say?

Tim (39:42):

Yeah, I think that’s definitely fair to say. I think there were always voices, and I’ve been one of these as well, who would say, “Look, I don’t really believe that we just scale models more and more and more, and we just throw more data at it and more compute, we’ll get more impressive results.” But so far, this has still been the case, and I’m pretty sure we’ll have GPT-4 at some point, which will be more impressive. So, we just haven’t, I guess, hit the button there yet, but I think there is something fundamentally problematic about just training language models.

Tim (40:17):

So, people have been already starting, obviously, to take this apart and looked into the systematic generalization abilities or the reasoning abilities of these models, right? To start, for instance, giving it a sequence of prime numbers and then you want it to continue generating more prime numbers, and it just doesn’t work and doesn’t get it, right? Whereas a human, at some point, they would realize, okay, ah, this is [inaudible 00:40:38] generates more prime numbers for you.

Tim (40:41):

So, there’s still a lot to be done, and whoever thinks that, I think GPD will be the solution to all of our problems and will lead us directly to AGI, I don’t think I would agree to that statement.

Jeremie (40:56):

Yeah, no, that’s an interesting view, and I have heard some people say it, albeit not necessarily hopefully. I think there is an undercurrent of concern as well, because AI safety research and AI alignment research just isn’t yet ready to really accommodate something that would be generalizable or general level human intelligence. To your point about a lot of the communication difficulties with neural networks, we don’t know really how to embed what we want into these neural networks, how to set up our loss functions so that they’re morally coherent, among other things. But I guess, one of the things that I found remarkable, too, about it was that GPT-3, or sorry, GPT-2, basically could not do addition or really struggled with simple arithmetic addition, even of single and multi-digit numbers.

Jeremie (41:45):

And GPT-3, at least the larger version of it, you can see actually, as they scale it up, you can literally see the efficacy at these larger and larger additions with more and more digits. And I guess there’s a bit of an open question there, to what degree that qualifies as reasoning. But my sense has been that we’ve been moving the goalposts to some degree, and that potentially, what we’re learning here is maybe more, not that GPT-3 is more powerful than we thought, but maybe just that the human brain is less powerful than we thought. Maybe that, because we’re optimized, not just for parameter efficiency, but also for energy efficiency. We have all kinds of extra evolutionary constraints. An airplane isn’t nearly as energy efficient as a bird, but it can go bloody fast compared to a bird. I guess one of my, excuse me, one of my concerns around this issue is, maybe we’ll be taken by surprise again and again, and GPT-4 might present us with a level of capability that we’re not quite ready philosophically to accommodate. Is that something that you see as possible or not really at this stage?

Tim (42:55):

I think we’re very far away from that. I think, first of all, you’re right that we’ve been moving the goalposts all the time. We’ve been doing this in the past all the time, right? We’ve been doing this with chess. I have been in the meeting of this episode. I’ve been doing this with goal, right? I’ve been saying, “Look, it’s super impressive and the result is super impressive,” but there’s literally zero that you have to transfer from the real world in order to learn about goal, right? Is something like Minecraft, NetHack is different in that regard.

Tim (43:24):

So we’re constantly moving the goalposts and we’ll be doing this in the future as well. I think GPT-4, definitely, it will be impressive, but even with GPT-3, I think if you [inaudible 00:43:34] long enough, if you present it with numbers it just has never seen before, that are extremely large, right? If I tell you at these two numbers, one is a trillion something, something, the other one is a trillion something, something, you would be able to do it, right? Because you can write down an algorithm or you can write down some symbolic representation of the task that I gave you, and you’ll be able to solve it reliably every time. Whereas with something GPT-3 and, I think, also GPT-4 when that comes out, you’ll still struggle as long as you only look into training a language model. It will be impressive, it will also have all kinds of downstream applications, right?

Tim (44:17):

This is fantastic, right? We can potentially use those, as I mentioned, for conditioning on textual data, for reinforcement learning. There are all kinds of interesting applications that come out right now that build upon GPT-3, but somehow, I am convinced that we’ll hit a wall at some point. I can’t really pin it down, right? If I could, I would probably write a paper about this right now. But my intuition tells me that this will, it won’t get to this same level of reliable reasoning that we are capable of, if we’re training to do it and if we are concentrated.

Jeremie (44:57):

Interesting. I guess, as well, the thresholds of reasoning matter, too, here because to some degree, for those people who are worried more about the downstream, even existential risks associated with the tech, it’s like, how much reasoning do you really need to get an AI system to the point where it has some thoughts about how to iteratively self improve? And it’s unclear how much of that would be novel reasoning, because perhaps in its corpus, there’s enough information for it to just pattern match to the point where it figures that out, or something else. But I think it’s a really interesting possibility space. It’s so counterintuitive, too.

Jeremie (45:34):

But to give you a last really, and unfairly tough question here, just because you’re so specialized in this area, if you had to guess what technology you would have to couple to something like a GPT-3, to break this mold of, you’ve got this model that, it doesn’t seem to have the ability to, as you say, reason. It certainly lacks any kind of agency or intrinsic motivation. I’m sure there are a whole bunch of different possibilities, but just to ask you the unfair question and put you on the spot, do you have any thoughts about what would be that next step?

Tim (46:07):

So to me, what’s absolutely unclear, and I think there are probably lots of other researches who have been exploring that area much more from the, let’s say a neuroscience perspective, is this emergence of concepts and emergence of symbols that we humans are capable of, right? We came up with mathematics, we are able to externalize our thought process. If I put that on paper, I’d be able to write down an algorithm or I’d be able to write down equations to solve a specific task. Unless we have something in these deep learning systems, artificial neural networks, that encourages that kind of forming of crisp concepts that then they can reason about on a symbolic level, I think we’ll hit a wall. So that combination of neural processing and symbolic processing, I think we will have to make a lot of progress in that.

Jeremie (47:02):

So, would it be fair to say that that would be your AGI fire alarm, then? If you were to see GPT-4 exhibit what seems like symbolic levels of logical reasoning and inference, would that, for you, be like, okay, now I’ve changed my mind, now I think this kind of strategy might work? Or…

Tim (47:23):

Yes, but, and it’s a big but, we are still capable of, while we do all this logical reasoning, we’re still capable of being thrown into completely novel situations and adapt quickly, or at least be somewhat sensibly. And I think if you combine both, right? If you see both happening, then, I think, yeah, I’ll be quite impressed.

Jeremie (47:48):

That’s interesting. Yeah, actually, what this is making me realize is that there’s a difference, too, between… So first off, we have this very fuzzy definition of AGI. I think the G is really what trips us up. What does it mean to be generally intelligent? People talk about the Turing test as being this thing that we’re going to use, and now the Turing test, mostly, I think it’s fair to say people are, well, we’ve made systems that can pass it plausibly, some of them by pretending that they’re an incompetent or inarticulate… I think one of them was pretending it was an inarticulate Russian teenager.

Jeremie (48:20):

And so, it’s like, can an AI system act such that it seems like an inarticulate Russian teenage? Well, okay, yes, you’ve technically passed the Turing test, but what does that really show? But I guess what this is revealing, at least for me, is there’s a difference between AGI and potentially risky level, high capabilities, AI. That may be technically narrow, but nonetheless, might pose a risk. Would that align with your views?

Tim (48:47):

Yeah, absolutely. And we already have a lot of, I guess, open, ethical issues with the current AIs that we are capable of developing, right? There are so many examples of face detection or gender detection or whatever detection based on images of people. We are capable right now of synthesizing on very high quality human faces, right? There were examples of that as well. And you can imagine all kinds of misuses of that. You can also imagine all kinds of misuses of language models. Yeah, coming back to your comments on the Turing test, I think one of the things I would be curious about… Basically, when people ask me, “Okay, what should we use to test systems for being generally intelligent?”

Tim (49:35):

I would like to just take a philosopher and put them in the room with such a system for a day, and if the philosopher afterward says, “Well, I had an interesting conversation for the entire day,” that would be saying, okay, well, apparently, the system did a lot right. And just to be clear, [inaudible 00:49:52] language models right now, they’re not capable of that at all. The text looks, on a superficial level, [inaudible 00:49:58] if you look into the details, you’ll find it very easy to spot. What’s the [inaudible 00:50:04] what’s the [inaudible 00:50:04]

Jeremie (50:05):

It seems like symbolic reasoning, then, really is the holy grail as far as you’re concerned? If it can do symbolic logic, then we’ve got something here that’s really significant?

Tim (50:13):

Yeah. As I mentioned before, I guess, one has to be careful in that. Symbolic reasoning is working really well with symbolic solvers and symbolic AI systems, right? So that’s not sufficient. It has to, at the same time, also demonstrate this generalization and sample efficient learning and distribution generalization

Tim (50:35):

So both together. You have to design, basically test if that’s what you want to do, right? You have to design a test that tests for both of these capabilities. If you test the symbolic AI system to what extent it can generalize to, let’s say detecting a specific traffic sign in the middle of a hurricane, it will fail completely, right? But some might also [inaudible 00:50:54] artificial, or artificial neural networks. You have to basically test for both of these capabilities, I would say.

Jeremie (51:02):

And if both capabilities existed, what then would you say are our moral responsibilities with respect to an agent that is capable of both of those things? Do you actually see that as behavior that would be indicative of a morally recognizable agent? Just for you, personally. I’m sure this isn’t a thing that you’ve done research on, but…

Tim (51:22):

Yeah. The closest experience I have with that kind of question, I would say, is for instance, in the movie Her, which I’m pretty sure you know about.

Jeremie (51:34):

I actually haven’t seen that movie, but sorry, please, go ahead. Yeah.

Tim (51:38):

Sure. At some point, if a artificial intelligence behaves according to its own set intentions and shows empathy and all of these things, right, this boundary between what’s artificial and what’s not becomes extremely blurry and there’s lots of science fiction movies that have that as their core theme. At that point, for me, personally, right, I would find it very difficult to shut off such a system, right? At this point, right, you think more about killing something that has a personality, but to me, at least right now and for the really foreseeable future, that is science fiction from what I can say, right? Looking at what breakthroughs we have in AI recently and trying to extrapolate, at least I find it difficult to see that becoming a reality during my lifetime.

Jeremie (52:35):

Oh, interesting. During your lifetime even. So you have a really long time horizon on this stuff.

Tim (52:41):

Yeah. Yeah, I’d say so. I think we will potentially see another AI winter, we’ll go beyond that as well, but I don’t think we’ll have something that’s really indistinguishable from human intelligence, that has all these aspects of empathy, zero shot generalization and symbolic reasoning all combined in the very near future. Yeah, I think we’re talking really about more than decades.

Jeremie (53:13):

Fascinating. Okay, very cool. Well, thanks so much. I really appreciate you indulging, as well, some of the more philosophical side of the conversation at the end there. So, Tim, is there anywhere where people can follow you as well on social media if they want to follow your work?

Tim (53:26):

Oh, yes, I’m on Twitter. Just look for Tim Rochdechel. And then obviously, you can look at my personal webpage for publications.

Jeremie (53:38):

Sweet. Yeah, we’ll be sure to link, actually, to all that below. Tim’s done a ton of work on all the topics we’ve talked about and more, so please do check that out, especially if you’re interested in RL, deep learning, any of the things we’ve talked about here today. Tim, thanks so much for joining us for the podcast.

Tim (53:51):

Yeah, thanks so much for the conversation.