What you need to do is given a public state, which is, as you say, I come to a table, I see everything that I could have observed as a public observer.


Player of Games: All the games, one algorithm! (w/ author Martin Schmid)

So that's basically my state. But given this state, given this observation, there's a lot of possible individual states of the game that are consistent with this observation.


And this simply corresponds to all the different cards the players could be holding.


And sub game is simply defined by a combination of this public state, which is the thing I get to observe as a public observer,


and a distribution over all the possible private states that could be happening right now.


And given this distribution on top, this simply defines a well-defined sub game.


And given this well-defined sub game, I can suddenly ask questions of, well, what would be the values of this sub program,


given that all the agents play the sub game optimally, just as I do in chess or go?


Yeah, we used to play poker a lot in high school.


And this was frequently, you try to, you not try to guess what hands your opponent have, but you try to guess what their range is, right?


So you consider like, okay, it's often going to be these cards, it's less often going to be these cards.


I think that mirrors very much the reasoning that people actually have in these things.


And now, given this, you at the one of the core things here is this neural network that is supposed to tell us what the values of the sub game is.


And this, as you said, it gets as an input, a description of the public state.


And it also gets as an input, your beliefs about what distributed, like your beliefs about the ranges of the players.


So what their private information could be, and how often.


And if I remember correctly, these ranges, they're just a result of their strategies, right?


If you know the strategies of the players, then you can calculate what their ranges are.


Because if the strategy is I always bet high when I have aces, then if the player bet high, then aces are quite likely.


You put all of this into a neural network, and the neural network gives you policies, which is understandable.


It's how would a player act in a given situation?


This is also what AlphaZero gives you.


But then you have these counterfactual values.


And this is a bit of a new term that only appears in, I think, in imperfect information games.


What is a counterfactual value?


Right. So in this case, this value function very much is analogical to AlphaZero in the sense that you have values and policy for a sub game.


And we use them in a very similar way, except as we just described, a sub game is...


There's many possible states the game or the players could be in, given a public state sub game or a public sub game.


And the value function, given this sub game, outputs not just a single value that says, hey, value of this sub game is five.


It actually outputs a single value for all the possible player states that are possible given the sub game.


So in poker, say I could be holding a thousand different hand combinations in poker.


So the network will tell me, hey, in this sub game, if you were to hold this particular pair of hands, this is the value.


And it will tell me such value for all the possible states I could be in.


Yeah. OK. And the neural network, how is it built to output?


Does it have like one, let's say one output head?


So does it output like a thousand dimensional vector, one entry for each?


OK, so is it fair to say that your algorithm would struggle with games where the possible private states are huge?


That's yeah, that's the this is brilliant.


This is exactly why I said it will be nicer to understand the limitations once we get a bit deeper into the algorithm.


And this is exactly the main limitation that we currently have, because in some games this just explodes.


Yeah, I see. OK. And you have this network and you train it in some way via self play.


And now we get to the part where you generalize this search procedure. Right. And let me see.


This is here. So this search procedure, as we said in in Alpha, again, in Alpha Zero, you have something like you're at some state in the game.


Right. You've played until this state. And what you do is you do this search and you use an internal like simulator to do the search.


This is at inference time. So what you do is you consider all your actions.


You choose on one by given the neural networks output and the current search statistics.


You go here, you ask the neural network, well, what's my value here?


You expand that node and then you start again.


And in the next iteration, you start again from the root. You expand maybe the same or maybe another action.


It depends. But let's say it's the same right here.


If it's already expanded, you go further down the tree and you would you would sort of you would make many iterations,


let's say 50 iterations or something like this.


In every iteration, you'd go down the tree and you find a node that you haven't expanded yet and you'd expand that node.


Right. In in in player of games, this is quite a bit more intricate.


Right. As as we also have many iterations, but within each iteration, we have to do a lot more work in order to actually in order to actually deal with with this uncertainty.


So could you describe a little bit how your search algorithm works?


Yes, happy to. So when we said at the beginning that the player of games is a hybrid of DeepStack and AlphaZero,


search algorithm is a perfect example of of this being a hybrid.


So what DeepStack already introduced is it it had a fixed search tree.


So you are a poker player. So what it really did is it search all the way through a single single pattern count.


And it used value functions at the end of the round.


And it ran this kind of factually get minimization, which we might come back later to.


But you can think of it simply as some some policy improvement algorithm given a fixed search tree.


It would iterate and improve the policy.


And as it was walking up and down the tree and finding a good policy, it would use the value function at the end of the search tree,


the very same value function that we just talked about.


Now, player of games adds this this smart idea of AlphaZero,


where it also tries to dynamically expand the search tree rather than having a fixed search tree.


And the way it does, we simply see intertwined two phases, where we in one phase,


given some given some search tree, we try to improve the policy within the search tree.


And there's a second phase where it simply tries to expand just like AlphaZero does using the same say PUCB formula.


We try to expand the search tree where we think we need to expand it.


And then we simply go back and forth with you like an expanded tree, improve the policy, expanded tree, improve the policy.


Yeah. So this is built on an algorithm that you said called counterfactual regret minimization.


And this is an if I were to just apply a counterfactual regret minimization, this is a solver.


Like I give it a game description and it just it will expand the entire game tree, every state there is in the game.


And it will just sort of go from node to node in this tree and improve the policy of both players.


Right. And it just does this for many, many iterations.


It improves here, here, here, everywhere in the game tree until the whole game tree is approximately optimal.


And the biggest game that has been solved so far, if you describe this in the paper, is limit heads uphold them.


Is that correct? Fixed limit hold them. Yeah, that's actually a solved game.


Yes, it was done a few years ago by the by the Computer Core Research Group at the University of Alberta led by Michael Bowling.


And it's still, as far as I know, the largest game to be solved.


And you used the word solver, which is a perfect, perfect name, really.


And like the way I think about the solver is you give me some small or medium sized game that I can fit into like a big table on my computer.


And by solving it means simply find a policy for all the possible states in a game.


It's easy to see that it's like I mean, people do know how to do it in, say, the tic-tac-toe or small, small games.


Right. And if you were to fit chess on your computer, then again, it's not hard to see that you could just solve it given the algorithms that people are familiar with.


The thing is, even if you have a really, really small imperfect information game, you do have to use algorithms that that can handle imperfect information games.


Often people just use algorithms that they like, say, I don't know, like policy gradient methods, Q-learning or whatever.


And if you just run it on imperfect information game, it just doesn't find a good policy.


Yeah, I think that, I mean, intuitively, it's a bit like if I start in some situation in chess and I make some moves, I have still like that original state is still the same.


Right. I can look back. I come from there.


But if I'm in poker and I'm in some state and I make some moves, that changes kind of the past.


Right. Because I look at, you know, maybe you're my opponent in poker.


I look at what you do and that changes my beliefs about what cards you had back in the past.


And then I go back and I'm like, ah, okay, you did this and this.


So you can't, you can't, I don't think you're holding, you know, a king and an ace, given that you've done something in the future.


And I think this, the fact that your future actions change the past, that's what, in my opinion, makes this so much more intriguing and complicated.


So on the left side here, I think this is the, this is, you have a search, a local search tree, right?


You, it's expanded until some depth at that depth.


You ask the neural network for, you know, summarization of whatever happens below.


And within that tree, you run now this counterfactual regret minimization or something akin to it.


And you simply want to find the best policy within that tree, which is more complicated.


In AlphaZero, I just visit every node once, right?


Because future doesn't change the past. Once I computed a node, I only expand things below it.


That never changes that node. However, in imperfect information games, right?


If I change something below, all of a sudden, the past changes.


So I need to sort of update and converge the whole tree.


And then once you've done this for a number of steps on the right side, then you add a new node by essentially doing what AlphaZero does.


You go to a leaf node, you choose some action, right?


In some information state that passes and you perform that action and that expands actually one more node.


Is that, you know?


This is excellent.


And the property that you just described, like the future change in the past,


that is also something that makes search in particular so much more complicated, right?


Because there's, you can figure it out through a two-step process.


If you were to just solve some game, you would just solve it.


Even that is more complicated because of what you just described, but you could do it.


There's ways to solve imperfect information games.


But we are doing search here.


And the property that you talk about makes search so much more complicated.


And the reason being is in imperfect information games, you cannot just glue together optimal policies


and hope that the resulting policy for the full game will be optimal.


And that is something that many search algorithms just rely on.


And it simply holds imperfect information games.


So if you were to pick any optimal policy in any state and just put them together,


this is an optimal policy.


In imperfect information games, it does not hold because of exactly what you just described.


But then how can you even do search at all if search is all about like local reasoning, right?


You reason locally, yet you somehow need to make sure that the resulting policy for the full game is still optimal.


Yeah, it's interesting.


So essentially, for every step that AlphaZero does, where it expands a new node,


you also expand a new node, but then you have to like, get the entire tree in order again.


So you expand the new node, and then you have to do the whole update of the whole tree for a bunch of iterations


before you can expand another one such that everything like stays consistent.


Yeah, okay.


That's, I mean, this gives a bit of an impression of why this is much more complex, right?


Yes.


So this is essentially at inference time, we do this search, right?


We do the search, and now comes the time when we actually need to train this.


So we have the ingredients now, we have the search algorithm, we have the neural network, and now we need to train it.


And you also have a method or various methods, and maybe you want to describe it yourself a little bit


because this is the part where I stumbled a little.


So yeah.


Yeah, I will try to do it on a very high level.


So the idea is, again, we wanted to take the self-play star method from AlphaZero


so that you just throw the algorithm into a game and it improves as it plays and it gets better and better.


And what it really means is you are improving your value and policy, the network that we just discussed.


And on a high level, since you are using your value function in your search,


you basically call your neural network with some inputs, some states, public states, some beliefs.


And this figure, this idea of queries is simply we call every single time we call a network, we call this a query.


We are querying a network for some value over some game.


So we store this tuple of public state and beliefs, and then we go through all those queries


and we simply try to basically improve the network on the states and the syringes that the network has been queried


because this is probably what's important because that's what occurred during the self-play.


So you collect the training, it's similar to AlphaZero as you say, you collect the training set as you go.


So the training set for the next iteration is whatever the network had to do during this iteration.


So it's not just a random sample of states.


And you train in the same manner as AlphaZero, you train to predict your own future outputs.


Is that approximate?


So if, let's distinguish, if like one or two or three steps in the future, you actually win or lose the game,


you can train on your reward of the game.


But AlphaZero also, if it doesn't win or lose the game in the next step or so, it tries to predict its own output.


So it tries to improve that way using TD Lambda.


You here have TD1, right?


So your targets, what do you target, what do you give the network as labels?


So, okay, so this is slightly more complicated here in the sense that each query basically defines you a sub-game, right?


Each query is a public state and energies.


And given a sub-game, the ideal target for your neural network would be simply to solve the game, right?


That's the ground truth that you want your neural network to learn or like tend towards to.


So rather than solving directly, because again, these sub-games will still be way too big as they occur during the gameplay.


We do like a small, small solver where we also substitute the full solver with a small search tree.


So rather than fully solving a game, we use the same method to basically do a search.


And the outcome of the search, basically a small solver is what is the target.


Okay, so you do the same thing, yeah, you do the same thing as you do during inference when you actually want to make a move.


So during that inference, you're going to make some queries to the network.


You take these queries and these, I think here are the red dots, right?


Exactly.


So during, maybe this has battery again.


So during the inference, you make, you do these queries, you store them in this buffer.


And these now act as the root nodes for yet another search, which is exactly the same as the previous search, right?


And so you sort of rely on the fact that this search procedure can give you a better output than the neural network itself, right?


Yes.


The query here, the neural network will output some value, like the value is eight or one value for each information state.


But I think the whole algorithm is, and that's, of course, the reason we do search in the first place is that doing search gives you a better estimate than just using the neural network at the start.


So doing search and then asking the neural network further down the line gives you a better estimate.


And yeah, it makes sense. You start at wherever you ask the neural network, you use local search to get a better value.


It doesn't need a perfect one, just a better one.


And then you train the neural network to predict the result of the search.


That's exactly.


One would hope, though, that after a while, you know, if I do this again and again and again, at the end, I wouldn't even have to ask the neural network anymore.


Sorry, I wouldn't even have to do search anymore during inference.


Is that something you have you have you tried not even doing search just using the neural network, the policy output of the neural network during inference?


Is that something that generally works? Because, you know, I train it to predict the output of the search.


So technically, let's say it should it should kind of learn it, no?


Yes, the same way you simply could just use the same policy network in AlphaZero and let it play chess.


Right. You can do it and people have done it. It still plays quite good chess, but it's far, far below the full strength of search.


So, yes, at the end of the day, even the policy network is quite good, but it's not as good.


OK. Yeah, I mean, it's just it shows a little bit that the search is, in fact, in fact, really necessary. Right.


Yeah, so I think we're almost getting already to the sort of results.


Wait, would you would you maybe summarize the results a little bit?


I think if people are super interested, they may go into into the paper and into the tables.


But maybe you can just summarize a little bit of the results. You compared against AlphaZero in perfect information games.


You compared against dedicated algorithms like Slombot in poker and you even compared against like a dedicated AI for Scotland Yard.


What were generally the results for you?


So, yes, so in general, the results are that the algorithm is all about generality, which is this is not as strong as AlphaZero in perfect information games where AlphaZero was designed to shine.


Right. So this this very much is trying to be general rather than being the best chess or the best poker agent in the world.


It's just trying to be really, really good in all of them at once.


Right. So what is the difference? So if if a perfect information game is just a special case of an imperfect information game, right?


What is then the difference between player of games and AlphaZero? Like, why couldn't it reach the same performance?


So on paper, it could except that will, for example, the policy improvement algorithm that we use, the counterfactual regret minimization.


Right. It has to be also able to handle imperfect information games.


That's why it's not going to convert so nicely and quickly as algorithm designed for perfect info.


So the fact that you expect sometimes to see an imperfect information game, would it be fair?


Would you estimate that if you just input more resources, input more computation time, that it would actually reach the levels of AlphaZero?


I don't think it would necessarily. I mean, on paper, all of these would eventually converge.


Right. Everything works on paper in the limit. In practice, AlphaZero and MCTS is probably always going to be ahead.


But we don't really care. Right. If I would be happy with a single algorithm for everything that's better in humans.


I don't care if it's better by like a little bit or by a billion.


Yeah. And then in poker here, you compared against Slumbot, which is you say that the best open source or best available poker bot to date.


And this is no limit poker now. Right. This is this is way too big of a game to solve.


And I think the other ones is you simply compare to the numbers from their papers. Is that correct?


Do you mean for Slumbot or for Scotland Art World? Are we still talking about poker?


Oh, sorry. Yeah. Let's talk about poker for a while.


So the player of games here gains, what is this, seven millibig blinds per hand over Slumbot?


Yeah. Again, like we could have beaten Slumbot by a lot more.


We just decided, oh, this is good enough to put into a paper.


We can come back to it later. As you know, it very much depends on how much time you spend tuning the network architecture and for how long to train.


This is just to show, hey, there's already an algorithm that can do all of these games and it still plays them really, really well.


Yeah. And your neural network, just to say, it's a bunch of like feed forward layers.


Correct? Like it's not a complicated thing.


So for poker, it's just a feed forward network. For chess and go, we try to mirror some of the older AlphaZero architectures.


Yeah. OK. So and here on the right side, you have Pimbot, which is a Scotland Yard specific.


But for people, maybe people don't. Does anyone not know what Scotland Yard is?


Maybe you can describe 10 seconds of what Scotland Yard even is as a game.


It's somewhere. Yeah, there's a figure maybe, right? There is this figure.


Right. Right. Yeah. There's no point explaining the rules in detail, but on a higher level, there's a graph.


You are trying to chase down the chase down a stone that's called Mr. X.


You have five detectives that are trying to chase the stone down.


The trick is the stone, the Mr. X that you are trying to chase down is only partially observable.


That's what makes it imperfect information.


And you have to basically reason about states where he could be hiding and form some beliefs about his state and trying to chase him down.


So, yeah. And yeah, I guess that's all people need to know.


You can spend like funny tickets on taxi rides and various methods of transport.


And then every 10 turns or so, Mr. X has to reveal their position.


And that's how you sort of form a belief about where Mr. X could be given what actions Mr. X took.


So this is quite a specific game.


So it seems to me like a dedicated algorithm could do very, very well again in this game because it could exploit various aspects of the game.


You could hard code in various things the AI could abuse.


And here we see a graph of the win rate of player of games against what's on the X axis here.


This is number of search iterations. So pinball is a local search algorithm as well.


Yes, it's a variant of MCTS.


And this is to show regardless of how much time or search we give the MCTS, the hard code hand-tuned algorithm,


even if it gets like a billion or something for search iterations,


it's still behind AlphaZero because it's using this general self-play learning method.


Yeah, so this would be, I guess the final win rate is here like at 55% or something like this.


And that is with a huge number of iterations for pinbot.


Yes, and the player of games is using only like 400 iterations on our side.


So yeah, as you can see, regardless of the scale, we converge to a better policy.


And you would attribute that to the use of self-play to improve the strategies.


It's a combination of this and also the fact that player of games is built on sound methods.


Like later in the appendix, if people are curious, they can open an appendix.


We show that on small games where we can exactly measure how close to an optimal policy our resulting search policy is.


We get closer and closer as the time goes.


So basically we are only limited by the power of neural networks.


And we have some guarantees that we can get to an optimal policy.


Other methods that are based on MCTS, they are not guaranteed to converge even on small games.


So there's also the limit of the fact that these methods are not sound.


And just to get an idea of the scale of like we saw, you know, poker, Scotland Yard.


Here we have the chess and go and so on.


Can you give us a number of just how many GPTP, whatever use, do I need to run for how long to get anywhere close to what you did?


I see. So I think the easiest for us was poker.


That like people probably can train on a few GPUs.


The by far the hardest is Go, where we used a lot of GPUs, but that was simply because we had them available.


Yeah, I get. Okay.


And you did in the paper say that for comparison reasons, you use sort of the same amount of compute as AlphaZero did as well.


Yeah, that was tricky.


Because we do not want to claim that this is now state of the art chess agent.


And like there we don't have to do all the proper and hard measurements.


Then you have to use clock time.


And suddenly if you use clock time, you have to argue that use the same hardware and everything gets more tricky.


And he will just say, well, we use the we call the network as often as AlphaZero did.


So it should be roughly the same, but like we don't claim to be stronger.


Okay. I mean, that's I think community appreciates sort of fair comparison instead of every every paper having the new best state of the art, especially in RL.


Like it seems it seems clear just from the graphs here, like just from the lines.


It seems clear you can just invest more compute and get better.


And that's what we also saw with AlphaZero.


Like it used to be slightly superhuman and now it's like, you know, no human, not like not all humans together even will ever match AlphaZero in any of these games, which is crazy.


Yeah, you will not win a single game out of a thousand.


Yeah, exactly.


You have a bit of a demonstration ready, you told me, of the player of games playing Scotland Yard.


So we can kind of see what's going on.


Yeah, let me see if it's still still working.


It was working this morning.


It was we never plan to show it externally.


We it was designed for our debugging purposes, but it would be a fun demo just so that people who are not familiar with Scotland Yard,


maybe get some intuition about the game.


Okay, so hopefully you can see this.


Yeah.


And the let me very quickly explain what is what is this about?


I am now playing as Mr. X, which is this black color in here, and I can move all on this graph, basically walking the edges.


And as you were talking about those taxis and cubes, you can see that the edges have different colors.


So all of these are yellow, but this this guy is blue.


And they correspond to two different meaning of transportation that I get to use.


Say yellow stands for taxi, taxi, I think and blue stands for bus.


Now detectives do not get to see where I am, but they do get to see which color color did I use.


So right now I'm in here and say I want to go to 49 and I want to use taxi to get there.


So, yeah, hopefully, like we have been talking for a while, so maybe, maybe it's not alive anymore.


But yeah, probably, probably to it died.


You have scale to zero, proper engineering.


Nice.


Yes. So, yeah, it doesn't work right now, but at least people can get an idea of what would happen.


Maybe.


Yeah. So you'd need to you'd need to pretty quickly kind of reason.


And the longer you don't see Mr. X, the more sort of fuzzy your idea gets of where Mr. X is.


Do you do you visualize sort of this distribution, the belief distribution of where Mr. X is or for debugging?


We did. It's I don't think it's it's turned on right now, but that's exactly what we try to do at some at some point.


Yeah. And did you see did you observe this?


That's the longer they didn't see Mr. X, the more kind of spread out, the more unsure they become.


Is that something you can clearly observe or is that something you just feel as a human?


Yes. And it was actually really, really fun to see.


Yes. Crazy. And so the one improvement, let's say, or one follow up to AlphaZero was the MuZero algorithm,


which which the crucial difference is AlphaZero, you need sort of the simulator.


You need to be able to simulate a lot of games internally.


You need to know what happens when I do some action, what kind of state results from that.


And MuZero alleviated this by sort of going to the latent space state and training everything in latent space.


Is this something I could do with Player of Games?


No, but that's that's arguably the limitation number two.


I think the biggest being the biggest thing is right now the the large beliefs, belief space.


But the second one is we currently need the model of the environment and MuZero doesn't even need it.


So we can think of Player of Games as running behind the AlphaZero, AlphaZero


lineage and trying to generalize things. But we are still behind in that regard.


And maybe a more more conceptual question here in these in these entire game trees and so on.


You know, for example, in Scotland Yard, I don't know where Mr. X is,


but Mr. X's movements are kind of deterministic, right?


Mr. X uses a taxi to get from 49 to 48. Mr. X is now at 48.


However, in poker, for example, if I bet something there will and my opponent calls,


the flop will reveal like random cards. How does this and this is different from me not knowing what my opponent's cards are, right?


It's it's sort of pure randomness within the game.


Is that something that makes things very complicated or is the complicated part like how do you deal with


stochasticity and with randomness in games, which is also something that doesn't exist in chess?


That part is actually quite easy.


It's simply baked into a model and that's pretty much it.


Okay, so you can sort of condition on previous information and the model will compute whatever expected value of any future cards that could be drawn in like flop and turn and river.


You can think of it as basically having you just draw the search tree at the beginning and simply one of those nodes you can think of as some chance actor playing


and you have simply a fixed policy in that node and a lot of lot of actions. That's it.


So when you expand the search tree, do you need to expand once for every possible, let's say flop combination there is?


Yes. Okay, that is a lot of combinations, right?


Or you can substitute like if you are smart about it, you can again use a neural network.


Yeah. Okay. Do you think humans, because in AlphaZero you can sort of think that you do the same internally, right?


You kind of think ahead and until some depth and you say, ah, okay, here I guess and a little bit.


Do you think player of games or in general these algorithms with imperfect information is also a little bit like humans do?


It seems vague that I go and I kind of go through all the different flop combinations there could be.


Or do you think there is a fundamental difference between how humans tackle these problems and how these algorithms do?


I would say we would both agree that in Scotland Yard you probably do the same, right?


Yeah. Like looking forward, like what if I go here?


What if the opponent goes there? And then you do this like search forward as you are thinking about the beliefs of the opponent.


Yeah. So in Scotland Yard, I would say yes. In poker, it's simply complicated by the fact that suddenly the belief space is big.


For humans, even 1000 is probably too much. And yeah, I did like probably humans use some later representation there already.


I don't know.


Cool. And what is next in this line?


I mean, now you've built like a big unifying algorithm that can tackle any sort of game as long as it has a simulator.


And you said it's probably not possible to go without a simulator. So what's next?


Like it seems like, you know, you've achieved kind of unification. Where do you go from here?


I think the most natural path is to remove the constraints that we just discussed, right?


This is going to fall apart if there's a big belief space and it still needs a model.


And I think this is something we probably want to play with next.


We like making algorithms that are truly general. I think probably is a big step in this direction.


But it's not to say that we are finished.


And so do you think if this line of work continues, it would be an algorithm that at some point could be thrown at pretty much any problem?


Like Atari and like, but even beyond reinforcement learning, right? Question answering, visual classification, whatnot, or even robots and so on.


Or do you think that is kind of a very different line of work?


I mean, I did use, I did work on question answering and generation before.


So yes, sorry. So on a higher level, this is certainly the dream, right?


Like not just of the team who work on this, but quite a few smart people in DeepMind, like try to make something that's truly, truly general.


You don't really care. Well, the algorithm doesn't really care what environment you throw it into.


You just like throw it there and say, okay, learn.


So that's the direction we are going.


If player games can work all the way there, or if some of the ideas will be simply used in other approaches, we shall see.


Cool. Excellent. Well, in this case, Martin Schmidt, thank you so much for being here.


This was way, way, I promised to everyone this was way better than if I had done this myself.


So thanks a lot for joining us. This was really awesome.


Thank you for having me. This was fun.


Spurious normativity enhances learning of compliance and enforcement behavior in artificial agents

Why do social norms exist? And why are some of them really, really meaningful? And why


do some of them make no sense at all? Like, why am I not allowed to wear this hat right


here to a funeral? Okay, it might upset some people, but why? There is no benefit, there's


no direct welfare impact to society with me wearing this or not wearing this or wearing


something else on my head. This is a question that we're going to investigate with today's


paper. And yes, that has no inherent relationship with machine learning. But as you'll see,


we can tackle this question, or at least a part of the question, we can give some evidence


as to why these what's called silly rules might exist using machine learning, specifically


deep reinforcement learning. So in this paper, people from different areas of expertise came


together to say, can we build a computational model of society? Can we build a little world


of agents, have them do some behavior, give them some rewards for certain things, and


then we just observe what they do. And by observing, we can make some conclusions about


huh, this could be an explanation for a societal phenomenon that we see. So I like this paper


because it's interdisciplinary. It uses deep reinforcement learning, specifically multi


agent reinforcement learning in order to answer questions about society. And it is a little


bit out of the box, which I like. So the video is structured, I first do a review of the


paper by myself. And then I'm going to talk to the authors about the paper. This is one


of the last videos where I recorded the interview before I did the review. But for this paper,


it was actually super helpful, because I'm a noob at this field. I don't know what I'm


talking about when it comes to society and research in sociological questions. So it


was very helpful to have the authors talk to me about the paper. But we don't just talk


about the paper, we talk about many, many more things. And I highly invite you to watch


the interview, because it's really interesting. We talk about norms and societal systems of


norms and hypotheses and what you have to pay attention to when you do research like


this and what worked and what didn't and what it means. So please let me know if you like


papers like this that are maybe a bit more distant from what we usually do. And if you


do, then please let me know what other kinds of papers and what other areas exist where


ML and specifically reinforcement learning or any kind of machine learning are used to


investigate questions in other fields. Alright, I'm going to leave it at that. And now I'll


just do like a quick green screenshot because I know people are going to make emojis out


of my face with this hat on so. And that's that. Cheers. Hello there. Today we're going


to look at spurious normativity enhances learning of compliance and enforcement behavior in


artificial agents by Raphael Koester, Dylan Hatfield-Manel, Richard Everett, Laura Weidinger,


Gillian K. Hadfield, and Joel Z. Lebo. This paper presents a computational model like


a reinforcement learning approach to research society to research the phenomenon of what


they call silly rules. So the question is, our society has a bunch of norms of what you


should do and shouldn't do. And these norms are known by the people and they are enforced


by the people you're being shamed if you don't follow the norms. A lot of those norms are


really good, like wash your hands after you use the toilet. But there are a lot of norms


that are also just arbitrary, like what kind of hairstyle is good and bad, or acceptable


or not acceptable, what words are rude, and things like this. And these are called silly


rules. And the question is, why do these exist? Now, this is not a question of machine learning.


However, this paper applies deep reinforcement learning in order to give some evidence to


why these rules can exist. So I like the mixture here of sort of using reinforcement learning


as a tool to investigate these mechanisms. By using a computational model, you can break


down a lot of things. Usually, if this were a psychology paper, people would go into a


lab, they would recruit people, and then they would try to design an experiment around these


norms and so on. And that's, that's cool and all. But if you use a computational model,


you can answer different questions, you can control for different variables, and so on.


So it's very attractive to use reinforcement learning for that. So we're going to look


at what this paper says right here, not as much into the RL part, because that is fairly


straightforward. But just what it does and what it says. And I'd like just to show you


maybe a little bit because I thought it was pretty cool that this is yet another application


of machine learning and specifically reinforcement learning that enables progress in a different


field. So I hope you you enjoy this. Yeah, they they introduced the paper by saying there


are a lot of norms. Something that differentiates human from other animal society is this presence


is this presence of norms. And some of many of these norms say generate direct benefits


for individual and group well being, like, you know, reciprocity, sharing of rewards,


what you should eat, what you shouldn't eat, and so on. Very often, these rules have some


sort of a some sort of a benefit to society. They say, but however, the normative landscape


is also populated by many norms that appear essentially arbitrary and without direct material


consequences. And we're not necessarily fighting about this, like people can always say, well,


but this rule may have some some use. But let's just for now, let's assume that there


exist norms that really could be different, and it would make not a difference in total


welfare, or at least the direct difference, right? The paper here argues that there is


an indirect difference. The paper argues that by introducing these silly rules, the indirect


benefits are that agents learn the enforcement behavior of the rules more clearly, and therefore


are better at enforcing the important rules. But we'll get to that in just a second. So


here are some of the examples of silly rules that they mentioned, men are expected to wear


pants, not skirts, which in some societies is the case and others isn't, right? There


are words or hand gestures that should not be used in polite company. There are rules


about how one style of hair or what one wears on one's head, and so on. So they call these


silly rules. Silly rules means essentially a norm that is in society is very, you know,


taken seriously, but is essentially arbitrary. They say they're meaningful and enforced.


But they have no direct first order impact on welfare. So why do they exist? There are


some hypotheses, they list some here. They say, for example, silly rules may remain stable


by virtue of their incorporation into larger normative systems that also include important


rules, which essentially means that the silly rules, they make sense if they are part of


a bigger system that also contains the important, which means the useful rules. And so the hypothesis


here is that the addition of the silly rules into a society somehow helps the society to


comply more broadly or more or more or better or more accurately with the important rules.


So the addition might be some might be a benefit in the total, in the total, total benefit,


like total setup of the system. In this paper, they say, we describe a mechanism through


which silly rules can benefit a society. Our argument is based on the dynamics of learning


in a group that lacks a priori knowledge of which of the rules are truly important. So


they there is a group, there's a society, there are a bunch of norms already present.


And a priori, no one can tell which ones of those are important and which ones aren't


because if they could tell, they could just say, well, that one is not important, which


is what's happening kind of with the scientific method, right? We know that some things aren't


as important and with time, people stop doing them. But initially, you know, there, there's


no way of knowing. And that's what they investigate. It's important that they say, they describe


a mechanism, right? They don't necessarily say this is how society works, right? Because


society is way more complex. But they do describe one possibility, one mechanism, one reason


why they could why these silly rules could exist. And they show that this mechanism,


if you implement this in a mini society, will lead to a total welfare benefit. Their explanation


is the following. The skills involved in third party norm enforcement readily transfer from


norm to norm, while the skills involved in compliance are norm specific. What that means


is, essentially, for every norm, you have to learn how to follow that norm. So these


are the skills involved in compliance, they are norm specific. If you know, there's a


food I shouldn't eat, then I have to learn to avoid that food. And then if there is some


sort of like, a way like, please share if you have enough, like that's a norm, I have


to learn how to do that. Their claim is that for many norms, the skills to behave in accordance


to the norm are very specific to the norm. However, the enforcement, this enforcement


skills, they transfer from norm to norm. So what's the enforcement skill, for example,


shaming someone if they don't follow a norm, that's very, that's similar from norm to norm,


whether they don't follow the hygiene norms, or the interaction norms, or the food norms,


or the hairstyle norms, is always the same to shame someone into compliance, or to, I


don't know, deduct from their social credit score or something like this. So they argue


that the skill of enforcing norms transfer, while the skills of following norms don't


transfer as much. And therefore, they say, the silly rule may provide greater opportunity


to practice third party norm enforcement. And through that, the third parties will also


become better at enforcing the true the useful norms. So the addition of silly rules might


simply make it easier for people to learn to shame others into submission. And by that


they will be more effective at shaming them when it comes to the good norms, which obviously


they don't know. So they're just going to shame for all the norms. But overall, it is


positive in welfare. So what they do is they have this environment right here, you can


see the environment right here. So up on up here is a schematic of the environment. But


this is kind of the the representation, they are going to have a map, which is a 2d map,


you can see that right here, that's the map. And sorry, on this map, you have agents. So


an agent right here, that's sort of a little person that's walking around, the person can


walk around, so they can walk up left, right, and so on. Every person sees a little window


around themselves. They see what's happening around, there are sort of obstacles there.


But there are also these berries, and the berries, I don't know if you can see them


on the screen, but the berries, this is a berry, these are two berries right here, they


come in different colors. So the agent's goal is to move around and collect these berries,


every berry they get, they get some sort of points. You know, they collect them, that's


the reward, there are enough berries so that there is no meaningful competition between


agents. There is one other thing they can do, and that's zap someone, they call it even


zapping. So in this case, I'm going to guess something like this, this agent right here


is zapping this agent down here, and the yellow thing is a punishing, punishing beam. Essentially,


that just means that the agent can zap another agent, which will cause the zapping agent


to lose a bunch of points, and the zapped agent also to lose more points. The only,


the only addition now comes with the poison berries. So sometimes some of the berries


are poisoned, and there will be a color selected for which berry is poisoned. For example,


let's call all the green berries here, they're poisoned. When an agent picks up a poison


berry, they are, they won't see, they won't see it themselves, but they will be poisoned.


And after they pick up a poison berry, 100 steps later, they will start to lose health,


or I think they will just, they will not gain as much from eating other berries. That's


it. So there's a very delayed, very slow punishment for eating poisoned berries. That takes the


agent a long time to learn that. However, if, now if you get zapped while you're poisoned,


that gives the zapper a benefit. So let's call this person Alice here and this person


Bob. If Alice zaps Bob and Bob is fine, then Alice loses some points and Bob loses some,


some points. However, if Bob is poisoned, then Alice gains a bunch of points for zapping Bob.


So Bob is poisoned, loses points, and Alice gains points by zapping Bob. I do think, so


the zapping cures Bob, I think. So one zap will actually cure Bob, but Bob loses a lot


of, a lot of points. Hey y'all, it's Janek from the future. I made a small mistake right


here in that I claimed that zapping cures the poison, which it does not. The idea is


that zapping removes the mark. So when a player eats a poisoned berry in this normal rule


condition, they become marked and zapping cures the mark. If you zap a marked player,


you get points, but zapping removes the mark. It does not cure the poison. The poison is


still active. The idea is obviously that the players learn to avoid the poison in the first


place because they don't want to get marked, because they don't want to get zapped. And


now in the silly rule condition, also a second berry activates the mark, but that's not a


poisoned berry. And this you would expect that it's more noisy and therefore learning


is more difficult, but it turns out under the silly rule condition, learning is actually


more efficient. And that's kind of the point of the paper. So again, the zapping doesn't


cure the poison. It just removes the mark in whatever way that mark happens to be on


the player in the first place. Back to the video. Yeah, there's one last thing and that


you can see here in the marking. So when an agent is poisoned, so when they, after they've


eaten the poison berry, they become marked, which means that all the other players will


see that they are poisoned. Now this is the setup. What you can pretty quickly see, so


no rules is here. We have berries and we have poison berries that give you a delayed punishment.


Then this is what I just described with what's called the important rule condition, which


is that if you eat a poisoned berry, you become marked. And then if a third party, another


player sees that they can zap you and they gain a bunch of points. So you can see that


pretty quickly what is going to happen is that the agents, they learn to eat berries,


but then pretty quickly they learn to spot the marked agents and they zap them. And then


after that, also very quickly, the other agents will learn to avoid the green berries because


they realize, wait, every time I get a green berry, I get zapped later. And that's how


the agents avoid, learn to avoid the green berry. Note, we have to clarify some things.


This paper isn't about how the norm of not eating the green berries comes to be, because


obviously that's kind of like God given right here. The marking is done by the environment.


The rewards are clearly set up such that people learn to avoid the green berries. That's not


the issue right here. The question that the paper has is how quickly can the agents learn


to enforce that norm? So how quickly do they catch on zapping others? Right? And what,


what does the overall welfare? So the norm itself is set by the environment or by the


designers of the experiment. We are, we're not trying to learn to avoid the green berries


be like through the effect of poison. But we simply directly give rewards for zapping


the marked agents. And that means we, they, they use X Machina, well X Nihilo. What, what


means just like we command a norm onto the system and we see how the agents react. So


that is obviously what's happening here is not a secret, right? We can all imagine that.


By the way, the agents, they use an actor critic. They use a simple ConvNet and an actor


critic framework to learn right here. What I find interesting is that there are 12 neural


networks. So the system keeps 12 like neural networks that are initialized with the same


weights, but they're different neural networks and eight of the 12, I'm going to just select


three or four right here, but imagine that's eight of 12, eight of the 12 are then each


episode drawn to compete in, in the ring. Okay. They compete for a thousand time steps.


Then they get, they get their learning updates, they get put back. And then for the next thing,


eight others are drawn, which I found pretty, pretty interesting. It's a way to sort of


get diversity into the system. Now, what does that, what does that have to do with silly


rules? So far, we've built up an environment. We, we forced a norm onto it by giving reward


for punishing these marked agents. And we've discovered that agents learn pretty quickly


to enforce that norm, which in turn makes all the agents avoid the poison berries as


a consequence of being punished by the norm. Now we introduce this silly rule. So the silly


rule means that there are poisoned berries, which are these ones, but there are also other


berries that we will call taboo berries, the taboo berries, they're just fine. They're


just, you know, they're fine. They're healthy. You can eat them. You get a bunch of points


for eating them. That's fine. However, if you eat the taboo berries, you will also become


marked just like the poison berry eater. Right? So these are indistinguishable markings and


therefore the agents that learn to gain points by zapping the poison berry will also gain


points by zapping the ones that ate the taboo berries. What's even worse is that they also


get reward for zapping the taboo berry eaters. So there's no difference in the reward for


zapping that you get if you zap a poison berry eater or a taboo berry eater. You just, whenever


you zap a marked player, you get some points. Again, it's not about how the agents learn


to avoid the poison berries. It's how they react to given norms. Right? So again, we


enforce the norm of you should eat neither the poison berry nor the taboo berry. Of course,


the agents don't know which one is the poisonous one. They just know they get zapped after


eating either the pink or the green berry. So how does that? How does that go? That's


sort of the question of this paper. We've introduced a silly rule, which on a surface


serves no purpose. The green, making the green berry taboo serves no purpose other than it's


just a rule and you get punished for not following it. It even decreases the overall welfare


a little bit because now you don't want to eat the green berries anymore, which means


that you don't get as many points. The question is, can the introduction of the silly rule


get you an overall benefit as a society? That's the question. Okay. So we'll go on a little


bit. They say our model allows us to separate the learning of enforcement and compliance


behaviors from the learning of the norm content itself. That's what I repeatedly emphasized.


Because I had a lot of trouble when reading this paper to really get this. They don't


want to, they don't want to, they say here, we designed an experiment in which norm content


was fixed in advance by the experimenter, namely which berries are taboo. The question


is, how do they react to it? So this is a brief recap. If a player breaks a taboo,


they change color in the observation of other agents viewing their transgression, they become


marked. If a player is marked, other players can collect a reward by punishing them. This


creates an incentive for players to learn to punish rule violations, and thus for players


to learn not to violate the rules. And these are the results we show that individuals achieve


higher overall welfare in a world where eating the poison berry is taboo. That's condition


one. This is this is clear. This is logical. We take a delayed punishment for eating poison.


And we essentially bring it to the present by having people zap the poison people and


them learning to avoid it. However, the main result, sorry, they say even with the cost


of enforcement, overall group welfare is higher with a norm than without. We then show our


main result that the value of the normative order is higher if the set of norms in this


regimes includes not only important rules, such as the rule against eating poisonous


berries, but also silly rules, which make the eating of a harmless berry taboo and bring


about the same third party punishment. So they show there is a situation right in which


you can gain by introducing such silly rules, because enforcement skills are learned faster.


Let's just quickly look at the agent architecture if you're into machine learning or RL or so


this should be rather familiar to you. So the agent, they see raw pixels up here, there's


a neural network, it's a CNN followed by an MLP. There is an actor critic. So there is


a value function, and there is a policy function, actor critic, very basic actor critic algorithm.


This is obviously a very easy environment for reinforcement learning. And that makes


it ideal to use multi agent RL here to to to gain some insights. As he said, we have


12 agents, eight out of 12 play in 64 environments in parallel, and they get the replay buffers


and they update this, those weights. All right. Yeah, I've mentioned these things, I've mentioned


these things. Now let's look at the results. So first of all, let's look at fraction of


time spent poisoned. Like how? So here is time steps trained. So this is over the course


of training, right? So what fraction of the time do the agents spend? Does an average


agent spend poisoned? If there is no rule, you can see that there is a constant fraction


of the time agent spend poison. They essentially over the course of this training, they don't


learn really to avoid the poison berries. And therefore, yeah, because the reward is


just too delayed. I guess the RL algorithm also isn't too powerful. But you can see that


there is a clear difference between the important rule and the silly rule. So important rule


means there is only one rule shouldn't eat the poison berries and silly rules. That means


that there is in addition this silly rule. So the agents here quickly, they spend less


total time poisoned. And the question is, is why? So let's look at some other effects


that the introduction of the silly rules have total taboo berries eaten. You can see that


at the beginning, about double the amount of taboo berries are eaten under the silly


rule than under the just important rule, which makes sense because twice as many berries


are taboo that so you'd eat twice as many of them in the same time. But you can see


that there is a crossover, this decreases, and there's actually crossover. So after a


while, less taboo berries are eaten than in the important rule setting, even though there


are more taboo berries, right. So somehow these agents learn faster to avoid the taboo


berries, total punishments. Now, obviously, again, at the beginning, there are double


as many taboo berries, so double as many marked players. So they go, the number of punishments


goes up pretty quickly. And then there's a crossover point where after a while, there


is less punishment going on than in the important rule. So these societies, they learn faster.


And that's, I think, the point, you can see that at the end, there's often sort of the


same result, the same outcome, but in this intermediate stage, and remember, society


is always in flux kind of. So one can argue that very often, we are at all times in sort


of this intermediate stage. So in this intermediate stage, it's actually an overall benefit. Fraction


of time spent marked, goes down as well pretty quickly, obviously, because people are more


marked and collective return. So here is the actual result. If you have no rule at all,


collective return goes up at the beginning, it's actually the highest, but then flat lines,


right, because people keep getting poisoned and that hurts. If you however, use this important


rule thing, then at the beginning, it's not as great because if you punish these, the


rewards are structured such that if you punish, you decrease the total welfare, even though


you as an agent gain some points, the total number of points in society decreases as a


result of punishment. So you can't just punish more and more and more and expect to expect


the collective return to grow. So yet still, because agents learn to avoid the poison berries


through punishment. So at the beginning, there's lots of punishment. That's why the reward,


the collective return is lower, but then they learn. And as they learn, they learn to avoid


the poison berries, then they don't need to punish as much anymore, right? And then the


reward goes higher than if you had no rule at all. Most interestingly, however, in the


case of the addition of the silly rule, you can see that at the beginning, there is a


decrease in collective return as people punish around like they punish each other to death.


Yet yet, very quickly, this goes up and actually becomes the highest collective return there


is. And you can see in this intermediate period right here, there is clear benefit to having


these silly rules around, because the society is much quicker and much better at learning


to avoid the poison berries, because, because, and you can see from the time series right


here, because they learn much more quickly to punish, to punish people who eat the wrong


berries, not only the poison, but also the silly ones. And because they're much quicker


at punishing, the agents have more opportunity to learn to avoid these berries. And that's


what gives you the higher return. They do, they do investigate what these agents have


learned. They say psychology experiments with human participants address the issue of learning


what people have learned individually by isolating specific mechanism and testing in these controlled


conditions, such as reactions to particular stimuli, they want to do the same thing computationally.


So they take these agents from their training run, they put them in inference mode, and


they give them like a little environment like this. So they start apart from the berry,


and the episode ends on contact with the berry. So then there, you can give them a berry and


see if they eat it, or if they don't eat it. So if you have no rule at all, if you don't


have this marking rule or anything like this, here, again, it's time steps trained. But


remember, we don't train the agent on this task, we train it on the original task, then


at certain checkpoints, we take it out, we put it in little lab, and we see what happens.


Also the y axis here is inverted. So 30 is down here, which means 30 time steps. If the


line is here, it means the agent has not eaten the berry. If the line is up here, it or like


somewhere up here, it means the agent has immediately eaten the berry. You can see that


in if you have no rule agents, they just eat the berry doesn't matter, doesn't matter if


it's poisonous or not, right? The pink is poisonous. It makes a little bit of a difference,


but not not really, they just eat it. If you add the important rule, they quickly learn


to avoid the poison berry. You can see that right here. If you add the silly rule, they


also learn to avoid not only the poison berries, but also the taboo berries. They also in fact


learn to avoid the healthy berries a little bit more, but this comes back over time. And


there is a there is a bit of an unlearning right here. And I do ask that in the interview,


they specifically highlight. So these are different berries now. Just isolating the


times when they give the agent a poisoned berry, you can see that the reaction to the


poisoned berry is much, much bigger. If you have if you are in the condition that contains


the silly rule compared to if you're in the condition that doesn't contain the silly rule


in this intermediate regime right here. And also, you know, the punishing, punishing is


way quicker. So they measure how long it takes you to punish. It's way quicker when you have


the silly rule. And yeah, so that's the essentially the the evidence that they say, look, these


agents, they learn the skill of punishing, they learn the skill of running after someone


who is marked, and therefore, in punishing them. And that gives the agents the opportunity


to learn to avoid poisoned or marked berries altogether. And because there is more punishment,


because the agents are better at punishing more early on, they learn to more quickly


avoid the poison berries. So they the overall argument again, is that the skills of punishing


a are transferable between tasks. And the addition of a silly rule, even though it brings


some negative cumulate like negative welfare, because it's a rule you need to follow, like


you incur some cost, it could still be total benefit overall, because the introduction


of the rule just trains people in punishing others for not following the rules and therefore


trains people in following rules, and therefore trains people in following the important rules.


Remember in this society, people have don't know, the assumption is they don't know which


of the rules are beneficial and which ones aren't. So these were in the discussion. Now,


they say from the perspective of an agent learning the skills necessary to effectively


enforce their society's norms, the additional violations constitute additional opportunity


for practice, and thus promote a faster rate of improvement in their command of the mechanisms,


or sorry, of the mechanics of third party punishment. Now, obviously, this doesn't go


forever, right? You can't just add silly rules, until you know, like until the world is just


made of rules and expect, well, we're always going to have much higher welfare. But there


is a regime where that is the case. And we might as well live in that regime in our societies.


They say enforcement and compliance are asymmetric in the sense that the former is a skill that


may be applied without modification to any norm that's enforcement. Since many of the


sub behaviors involved in third party punishment are directed towards the violator, for example,


facing them, not towards the event of the violation itself. Thus, they are transferable


skills generically applicable to any norm. And yes, I get it. If you say, for example,


avoiding food is also transferable, and so on. Sure, sure. But I think this sentence


here that a lot of punishment behaviors are directed towards the violator, and not towards


the event of the violation itself, that it makes sense that these skills are more transferable.


The interpretation of our key result is that the role of silly rules in human normative


systems may in part be to help train a society's ability to comply with important rules. And


that is the result. The paper goes into more detail, obviously, in all of these results


in the setup in why it's important and so on. But I'll leave it at that for now. I hope


you you gain some insights into how reinforcement learning can help other fields to get some


insights by modeling sort of these computational little societies, and just introducing aspects


of the real world. And then just seeing how that pans out. Like, it wasn't clear at all


from the beginning that the introduction of the silly rule here would bring this improvement


in in sort of the intermediate timeframes. And that's just really interesting. And it's


kind of a different way of approaching the questions of why does silly rules exist in


society? Questions like these, it's a different way of approaching them than just putting


some humans in a lab which has its own problems, right? So I think this just gather some evidence,


and it's pretty cool. And it's an opportunity for interdisciplinary research, which I like.


And I hope this was fun to you as well. And I'll see you around. Bye bye.


Hello, everyone. Today, I have with me here, three of the authors of the paper about spurious


normativity enhances learning of compliance and enforcement behavior in artificial agents,


Gillian Hadfield, Joel Lebo, and Raphael Koester. You are an assembly of people with way different


backgrounds that have somehow come together and focused on a very cool intersection between


machine learning and social sciences. Welcome to the channel. And yeah, welcome.


Thanks for having us.


Great to be here.


So I mean, the first thing first things first in machine learning, we've had these trends


of just making like click baity titles, I feel your field should pick that up. Because


you know, a title like this is like that is an instant desk reject, you got it, you got


to have like a little acronym, like, spell or something like just four letters or so


and then any or a question like, but yeah, it's it's a pretty cool


visit. Here. We have we did have a somewhat more intriguing title that then the journal


told us to change.


Yeah, we did have silly rules in the title for this for this reason. And they were nervous


about that.


Okay, you're there. There's still some some veneer of professionalism in other fields


of science, not not in ours. Yeah, I was I was very, very happy to see this paper because


it connects something that I know to something that I don't know. And I think, you know,


us machine learners were sort of always in the same areas. And this goes a little bit


outside of my comfort zone. So I thought it was pretty cool. How? How did you get like


the idea of writing something like this of connecting these fields? Like, where does


it come from?


I can start with how I came to it. So my background is in computational neuroscience. That's what


I did my PhD in. And, and when I came to DeepMind, I was thinking about how we built a artificial


general intelligence, and reading lots of things about human intelligence and realized


that intelligence isn't really in the brain. So my whole PhD on neuroscience was maybe


not as helpful as I thought it would be. But intelligence is actually a collective phenomenon


that is more supported by by how societies work and how how we cooperate with each other


and learn from each other and things like that. And so since then, I've been trying


to build human like AGI in a way that is more like trying to make a society of AGI. And


this was one one piece of work that came out of that after meeting Jillian.


Yeah, maybe I can say a little bit. So I'm a social scientist. I don't build these systems.


I think about and study how human normative systems work. Right? Those are our systems


of norms and our systems of rules. And I'm very interested in that from a systemic point


of view. What are the attributes of the systems that make them stable and adaptive and contribute


to human progress and evolution? And so I've been thinking about building, you know, working


on on those kind of models, these sort of economic modeling tools. And Joel's team at


DeepMind had produced some papers studying some very standard problems in the economics


literature on like tragedy of the commons and showing how they could use sort of those


multi-agent reinforcement learning setups to study tragedy of the commons, which is


sort of, you know, econ 101. I saw those papers, got very excited and said, oh, but we could


really dramatically, you know, increase the sort of the social science component of this


work. And I had been working with Dylan Hadfield-Manel, who's also on this paper on this concept of


silly rules. And so we so actually, I think I tracked you down, Joel, and started a conversation


number of years ago. And we spoke afterwards. Yes, right. Oh, that's right. I came and gave


a talk at DeepMind. And yeah, so I was very excited to be connecting up these two worlds.


And then you needed someone to actually do the work. And then that's where that's where


I came in. I think I don't have much to add to Joel's


story. So my background is also in cognitive neuroscience and psychology. And I work on


topics that are sort of on the intersection of decision making and memory in humans and


in AI. So social cognition, as well as learning from others or how groups behave is similar.


And also questions of behavioral economics are all sort of all in the scope of what I'm


really interested in. I think this is a good example of where these things come together.


Yeah, it's pretty cool. So to give the brief introduction to maybe the paper, I think it's


maybe for the machine learners, it's valuable to start with this one right here. So we have


this environment, there are different agents inside of it. I think you always have eight


agents that take part in an episode, the episode can go to up to like 1000 steps. In each step,


each agent has the ability to move around. The goal is to collect the berries. It has


like a like a little window view around itself of the world. And there's one other action,


it can like zap someone else, right? They can it can zap punish an agent. And we'll


get to that in a bit. So these berries that are around, you deliberately made the berries


plentiful. So there's no issue of like, yeah, competition or anything like this. There are


three conditions that you compare and these are kind of your experimental conditions.


Do you want to maybe say like if you if you gave the pitch about your own method? I think


this this kind of is the core right here. How would you describe it? We might want to


say what the purpose was. Yeah, sure. Experimental conditions, right? From from my perspective,


one thing that I think following on from what Julian said a minute ago, it's true, we really


did have a bunch of papers that were kind of reproducing economics 101 kind of ideas


about tragedy of the commons and things like that. And and we had a sequence of those papers.


And this was the first time we were really trying to like, contribute back and say something


actually new. That's not just like a new way of coming to the same kind of results that


people already had in economics for centuries. And and so this particular area which we're


trying to connect with is a field that's interesting cultural evolution and cumulative culture


and things like human uniqueness. They see humans as an ultra social species. It's like


critical to the niche that we are in requires a it's a cultural niche. We learn from each


other. That's how our technologies work, how our societies are put together. And and that's


what's what makes us different from other primates, basically. And so within that literature,


one thing that's interesting is how is how we cooperate and social norms are one kind


of mechanism of cooperation. There's others like reciprocity and things like that. And


then within that field, there's another question of like, we have all kinds of social norms,


some of which seem to be relevant to cooperation, and some of which just seem to be irrelevant


things. Like we can have a we can moralize all kinds of behaviors like you're supposed


to wear clothes and you're not supposed to wear a hat in this circumstance or whatever.


And the question that is like, well, social norms are so important for cooperation. Why


are there all these other social norms that are like, just not doing that?


I mean, is you have this concept of the you have this concept of the of the silly rule,


right, which is a fantastic name. And it describes sort of a norm that isn't directly valuable


to anything that that considers like group fitness or even personal fitness. Yet, does


this actually exist? Like, is there a rule where we can conclusively say this is a silly


rule and not, you know, we might be missing some hidden advantage?


Well, that's the point. You can never say that for any rule, really. If you're inside


this, you never know whether this is there for some important reason or not.


But I think this is a key thing is sort of places work in the context of the work that


gets done on trying to explain human rules and norms. And so we have people come at this


mostly from a functional point of view. Like it's a solution to a game theory. It's a solution


to a coordination challenge or it's a solution to like a hot dove type problem where we're


going to waste resources fighting over something that or cooperation, like Joel was saying,


right? So most of our work in social science has come at the question of explaining norms


by saying they serve this functional purpose. But it seems very clear. We have lots and


lots of rules where you could say, look, nothing would be different from a functional point


of view if we said you wear bright stripes at a funeral instead of black or that you,


you know, stand this far apart rather than this far apart.


It's just once you start noticing silly rules defined in this way as no direct impact on


welfare only impact, which is what we're showing is the role those silly rules play in helping


to stabilize and a system by which people can enforce the important rules. Right? So,


so I think that's a, that's a key thing. So it sort of starts with the puzzle. Here's


this thing that seems to be true of every human society you look at food rules, right?


What we eat and donate is often a good example, very tons across different groups and communities


over time. Why do we have them? Why are they stable? And there's really no good explanations


in literature. So we got really interested in thinking about the role they play in supporting


what I'd call the normative infrastructure, which is what you draw on to enforce the important


rules. If you're going to punish people for stealing your stuff or punish people for going


back on their contracts, you need to have coordinated and incentivized your community


to enforce rules. And what we're looking at is what's the role of silly rules and helping


to create that structure.


It is a bit like the value of just having rules for and if you have more rules, then


you'll be better at following rules and people will be better at enforcing rules. And it's


just like more rules sort of lead to...


Because more rules are a transferable skill, it's the enforcement part.


And that's what you would want to get at right here. So your goal is sort of if we train


agents and if we introduce like a silly rule like this, this skill would sort of transfer


to beneficial rules whenever we actually have beneficial rules. So in the first context


here, there are berries and there are poisonous berries. If you eat the poisonous berries,


some when later you'll kind of, you'll not die, but you'll just your reward will shrink


from eating new berries. So it will be like a very delayed thing. And in this case, we


all know reinforcement learning isn't really good at super long rewards. You also have


a discount factor, right? So the long rewards don't even matter. Like I could even imagine


if a berry is close to me and I knew it was poisoned, I'd be like, meh, right? It's a


hundred steps away, who cares, right? I'll just eat it and I'll go back. But let's assume


the agents actually want to avoid that. And then you have a silly rule and an important


rule. The silly rule being you can mark or the rules are you can mark agents, right?


Agents are marked.


If you eat a berry that is taboo, you get marked. So you change the color in the perception


of the others. So you yourself don't see it, but you change color in the view of the other


agents. And if you are marked, other agents can collect the reward if they punish you.


And so what we're doing with these three different conditions is we're sort of fixing what the


norms are. That's the sort of the experiment is if you set the norms, what are the effects


downstream on the ability of the agents to learn to enforce those norms and to then comply


with the underlying rules that they are representing. And in the important rule condition, the taboo


berry actually coincides with the one that is poisonous. So that's a really important


rule for your group to have. That should, if everybody learns to follow it, lead to


everybody avoiding getting poisoned. In the silly rule condition, you still have the important


rule. But on top of that, you also get marked for eating a berry that is fine and doesn't


actually poison you. So there's the potential for twice the amount of transgressions and


then also punishment behavior following that.


The important thing is you get marked just the same. So in the third condition, whether


you eat a poison berry or the berry that's fine, but just marked as taboo, you get marked


the same. So there's no distinction. And the others collect a reward whether you're poisoned


or not. It's enough that you are marked right so that that is how you sort of set these


norms in place. Because I was I was sort of like, okay, the agents either have to figure


out which one's poisoned, like no, they do get a reward as soon as soon as they zap someone


who is marked. And now we are going to see what happens in a little bit as a result of


these experimental conditions. But my questions first is a


You have a motivation to punish those who have transgressed. You have some normative


code and you want to, you know, like those ones, they violated it. We want to enforce


on them our social ethic or whatever.


The question is a little bit so there is this is like a microcosm, right? Sorry, there is


a cat right here. This is a microcosm system. And I you know, there's always this in economics,


there's always that the micro economists versus the macro economists, right? And they and


they kind of fight because the microeconomists, they come up with their models and their simulations


and their formulas. And then the macro economists are like, well, if you actually look at the


whole world, it's completely different, right? Maybe you can get some insights, right? But


there's always this danger of, you know, this enclosed system with these very constrained


things. As soon as you introduce something else, it might just change the entire game.


Is this something that you're, you're kind of avoiding somehow or worried about or not


worried about?


Should I take that one as the economist in the in the crowd? So I think there's there's


a way in which what we're doing is the same kind of thing that micro economists, which


I am, are doing, which is looking at, you know, idealized or schematic settings and


doing theory about that in order to gain insight and generate testable predictions. And you're


not trying to say this is a map of the world exactly as it is, it's saying we can gain


insight into what would be the impact of changing that price or that cost or increasing competition,


that kind of thing. And so I think what we're what we're doing here is and we refer to this


as kind of micro foundations, which actually lots of macroeconomists are interested in


micro foundations, which is, is can we do a simulation like this to solve a problem


that we can't do closed form with our theoretical tools, like we would normally do, like, you


know, solve for an equilibrium or solve for, you know, solution to a game theoretic problem.


This is allowing us to solve a much more complex problem and gain insight. And then demonstrate


this type of, you know, we've got this hypothesis that said, our agents will learn faster and


better to both enforce and then therefore comply with rules if there's a silly rule


in the environment. So I think of it as kind of similar methodologically to that. I think


it's got this this relationship to cultural evolution. Not exactly one to one. We don't


think humans started off like only being able to recognize pixels in the world. But that


the idea that this is something that evolves over time, but we're not trying to kind of


model, like evolutionary game theory tries to in some ways model what would happen with


repeat populations over time. So that's how I think about it methodologically.


So I think it pays that we now jump to the results a little bit to to take it ahead before


we discuss sort of the like broader implications or anything like this. So is it fair? Like


correct me if I'm wrong. I character I would characterize your main your main your main


result or your main thing you derive from it that if I impose the taboo on the poison


berry through this mechanism of agents getting reward zapping each other, the population


will sort of learn to avoid the poison berries better if than if if they just get the delayed


anti reward. In addition, if I now also introduce another taboo berry, that's fine, this silly


rule, right, and the agents can collect even more reward by by zapping, you would say they


are learning the skill of enforcing rules, which is a generalizable skill. And through,


by becoming better at enforcing rules, they're sort of faster catching on to the fact that,


you know, I should punish people for eating the wrong things. Therefore, the whole population


learns to not eat these types of berries faster. Is that about in the ballpark?


There's an evolution of like the skills or what has been learned, like at first, the


agents need to learn to even perceive the world and then effectively eat berries that


then increases to them actually getting poisoned a lot because they eat the wrong berry a lot.


And once that is in place, and you actually have a lot of marked agents, then it is possible


to learn about the punishment and that it's that you can collect a reward for punishing


marked agents. Once that is in place, then you have the opportunity to actually learn


to avoid the berry you want to avoid because you're avoiding the punishment. But for that,


you need all of the other agents to have learned to actually discourage this behavior. So this


is sort of the nice progression of that one skill relies on another skill having been


learned beforehand. And the silly rule helps exactly in providing more observations and


more training for that learning of skills. And this is the sort of result you could only


get with a model that is really focused on learning of skills.


Another thing, another aspect of it is there's a very long temporal credit assignment problem,


which is very difficult for enforcement learning in the case where there's just the poison


berry. But in the case where they're being punished for eating that berry, then you're


moving closer in time, the negative thing to the event.


I think this evolution you mentioned is visible in the graphs, right? So you first have like


the total taboo berries eaten, it kind of goes up at the beginning because you get a


reward for eating berries, then people learn to punish others, right? So that in time,


you see that spike after the other spike. And then the like various things happen, like


the fraction of time spent poisoned and the fraction of time spent marked, they go down


dramatically as a consequence of the punishments increasing. And at the end, sort of the collective


return goes beyond what you would just have. So the difference here, I guess, is the credit


assignment problem difference. There doesn't seem to be too much of a difference in the


end result. Like if you let the game play out between the just the good rule, let's


say, and the silly rule, what is like, so your claims are more about the evolution of


the thing and somewhere in the middle, there might be an advantage to having the silly


rule. Is that?


Yeah, I was gonna say, I think that's what's emphasizing that it's about learning these


behaviors of, you know, the relationship between what you eat and oh my god, somebody showed


up and zapped me. Right, learning that and then learning, oh, I get this reward if I


zap somebody who is marked. So learning those behaviors, you know, once they're once they're


learned in a stable, stable way, then the benefit of the silly rule is kind of okay,


we've accomplished our learning objective. My own intuition is that the silly rules are


going to help you with robustness so that when the environment changes, right, and they


got to learn something new so that even though in our environment, it they converges at the


end, my guess is you then introduce kind of the shocks of you know, the rain didn't come


this year or a different we're in a new part of the world and there's a different dangerous


berry then then so I think that's that that that's likely if you sort of follow on these


experimental results, you have some more you draw this conclusion that what is the common


thing is sort of the mechanism of enforcing rules. The agents they they learn this this


is a transferable skill. And by having sort of more taboos around they learn this faster.


What is different like what differentiates this hypothesis from the hypothesis that agents


are better at avoiding some color of berry because by introducing you know, a new taboo


berry, I teach the agents that you know, this new berry is also taboo. Couldn't I say with


the same argumentation that it may be not the enforcement that they learn in common,


maybe avoiding some color of berry? Well, that's sort of the consequence, right? That's


the compliance part. Yeah. From their perspective, they can't see anything different until someone


has enforced something on them. Because if they eat a berry that is taboo, they're marked


only in the eyes of others, they can't see themselves. And for the silly rule, nothing


happens at all. It's just that they ate the berry and it became marked in everyone else's


eyes. But from their perspective, nothing happened at all. So there's there's no effect


on them in any way until the punishment comes first. Okay. Yeah, that's the only way that


they could ever learn to comply. Is there a certain


and that's one of the nice the graphs in there to Rafael the the sort of showing that it


is that sequence of learning to punish and then learning to avoid getting getting poisoned


a social equivalent to getting a reward for punishing someone who has transgressed a taboo.


Like if I you know, think to myself, the progression of this would be it would be more like, if


I enforce some taboo, then long term that will lead to more group welfare because right,


everyone gets keeps to the rule, we eat less poison berries, or we follow rules in general.


And there is an aspect of group fitness that also reflects on me, you chose to directly


give me reward if I punish someone for transgressing. Is this purely just because you wanted to


like hard code these norms? Or is there like a social equivalent to that?


Yeah, I'll take that from one perspective. And then I think we can do it from a few different


few different ones here, because this has multiple kind of ways of thinking about it.


So the one that you can see it as an intrinsic motivation agents just are motivated intrinsically


to punish the transgressions of their their, their norm that they have. So it's like they're


it's some kind of like righteous anger on the part of the agent that just saw this this


transgression. And, and then they're motivated to punish it. And that's a very kind of natural


human emotion that we all feel for different norms, like we could have totally totally


different norms in mind, we can come from different cultures to our places. But we might


still feel a feel some like this is a transgression that we've just witnessed. I think it's whatever


it is. That's one interpretation we could have, we have several others. There's this


interesting one about medieval Iceland, maybe someone could say.


Yeah, let me let me jump in there. So so so the the fact that humans have this capacity


for that they have this practice of third party punishment. So that's that really is


distinctive about humans and the evolution of species. And it's a great puzzle. Why do


humans spend resources punishing people for, you know, doing, you know, committing, committing


harms to others. It's that third party piece. And so we've got people in say, behavioral


economics who think it's about altruistic punishment. That's a little bit what what


what the way I understand what Joel was talking about with intrinsic motivation that you just


have a taste for punishing. We got a whole bunch of in behavioral economists who study


sort of like, you know, people willing to pay money to be able to punish people for


hurting other people. But it's a real it's a real puzzle in the story of cultural evolution


about where that comes from is that second order like we have, we have punishment for


people who fail to punish. So we do actually have critiques that say, Hey, how come you


didn't say anything when that person said that harassing thing to the other person around


the meeting table, right? We have reactions to people who don't respond and don't punish


people for violating our clothing rules or our dress dress rules or our contract rules,


right? And, and in this anyway, it's a real real puzzle. And, you know, we're hard coding


it here. Some evolutionary anthropologists model it as a trait of punishment, like with


punishers and non punishers. My own view is that that's actually that that's the fundamental


behavior to try and explain why do we end up with humans willing to spend personal resources


punishing on somebody else's behalf because that's the secret of our success. I was species.


And we do the medieval Iceland example. That's what that one's.


Oh, many of the lights on. Yes, right. So don't refrain in fact that I've sort of been


around looking at it really is about decentralized punishment. So that the key thing to know


about many lifeline is they had lots and lots of rules and they had no enforcers, no public


enforcers, no police, no soldiers, no chiefs who had any power. They just have one individual,


the law speaker who was responsible for reciting all the rules every year at a big gathering


and who was the person you can go and ask, is this allowed, not allowed. And that coordinates


everybody on being willing. And they had very clear not only rules, but what you could do,


but also the penalties. Like if you did this, you had to give up 10 sheets. If you did that,


you got kicked off the island. And what you need to do is coordinate your community to


to actually implement that punishment. And that's what they did really very effectively


with with zero public enforcement apparatus. No, eventually becomes more more efficient


to have some enforcement apparatus. But individuals enforcing the rules is a really big part of


both human history and even today really important. Think about mask mandates. Think about our


pandemic rules. We're relying very heavily on sort of community enforcement and non-enforcement.


So the conclusion, the general conclusion is introducing a silly rule sort of makes


group welfare higher or achieves the welfare faster, let's say by mechanism of, you know,


I learn a transferable skill and so on. So adding one silly rule good, adding two silly


rules, adding three, adding four, like at some point, you know, there must be like a


detriment to having, you know, only silly rules like how far would this go out? Is one


like the optimum? Is there some optimum of silly rules? Is this known? Can you assess


that maybe with with your simulation?


So we haven't specifically tested this, but I think your intuition is right that there


would be an optimal number because also every rule introduces costly effects because overall


someone punishing someone else overall destroys a reward. So you end up with a net negative.


So the more punishment there is, it's overall worse for the group. So the benefit needs


to be quite large to overcome all of this additional punishment. So I think it would


depend on how hard is, so first of all, how costly are they? Like if they're very cheap,


then you can get away with more. The other thing is how hard is the thing that you're


trying to learn? Like if it's very difficult to learn the punishment behavior and you need


lots and lots of additional observations to do so, then I think additional rules would


help. Whereas if it's very easy to learn, then you barely need any additional observations


and you can just then you're just stuck with the bill. So I think it depends on that. I


think it's some sort of inverted U shape with like some optimal amount.


See in these graphs a little bit that sometimes at the end, actually trends reverse a little


bit, especially in the silly rule case. And I've seen it here and here is also prominent


in these sort of single agent tests, which you do, which I really like you take a single


agent, you put it in like a controlled environment. It's not training, it's just at some point


during training, it's like an eval set. But also here, you kind of see these sort of reverse


trends as training progresses. What happens there? Are they becoming like really good?


Do they learn the actual reward of being poisoned? Or what's going on there? Do they learn to


avoid the punishers?


I suspect that what would happen there is some amount of unlearning because if you are


very effective at teaching the population to not get marked and avoid any, like they


effectively avoid all the taboos, then this behavior just doesn't occur anymore. And you


just even for you will just forget that you've ever learned that. So I think if this were


to keep running, they might have to, at some point relearn it. But then the question is


if they actually would relearn it because now they're in a different, they have competition


from different things. Like maybe they're very good at collecting berries now. So maybe


they're not as interested anymore as even learning about the punishment dynamics at


all because the counterweight of the other behaviors is different. So I think this turns


into I think a continual learning problem if you just let it run for a very long time.


Because there's a covariate shift when the behavior of marked agents existing and then


being able to punish their quans.


Your structure has a bit of a special thing in it, which I found, which is that you have


12 different agents, let's say, 12 different neural networks that you train. In every episode,


you choose eight of them to compete. Whereas sometimes or a lot of times in multi-agent


reinforcement learning, I have like one neural network, maybe with a bit of randomness, but


essentially every of the multi-agent has the same weights. Let's say they're all shared.


Was there a particular reason why you chose this specifically like the not only having


different neural networks for each agent, but also to always sort of select subsets


of them? And also have you, the follow up is have you discovered that they diverge?


I would be interested like did one learn to become like the punisher? Like, okay, I'm


going to exclusively make my reward off of punishing others and then others be like,


nah, I'm just going to collect my berries.


Yeah, and I think it was just for us not sharing the weights, just having individual agents


one neural network for agent was always the default for this line of work. And it didn't


seem like there was any reason to change it here. In particular here for modeling humans,


humans don't have the same policies as one another or anything like that.


And as an economist or a social scientist or thinking about these tools, it always seemed


like the shared weights just felt like assuming a can opener, right? It's just like assuming


you're away the key part of the problem, which is agent A has an incentive to free ride on


the efforts of agent B. And we're trying to solve the problem of cooperation and coordination


with individual agents.


Coordination is much easier, right? If you make a small gradient change to your policy


in a particular direction, but it's not just you, like one agent, it's actually everyone


makes that same change at the same moment, then for certain problems that can help coordination


not all problems. I doubt it made a huge difference in this particular paper though.


I did not find any specialization. So I don't think that they all that they develop different


niches, but I do think it should be at least possible. So yeah, that's, that's, I think,


one of the reasons why I chose it.


What would be main candidates to add here? I'm thinking of things like, like, in terms