[Music] yeah you have Ken loss on B and then you would I think you also have Ken loss on Z which like Z you restricted to piece the marginal of that you restrict it to be some Gaussian or something so in a sense you cannot put infinite information in there so both of them are in a sense information regular lies it's it's really more like an adversarial autoencoder which we didn't cover in a lecture so it's kind of like a VA yi but instead of like a care loss you use again loss so it is more it is but like a it is very much like a Nathan Coe


model that is training with again it could it wouldn't like that applies to everything that we would go over today there are holes in all of them oh you mean why they don't use a Vee I think it's probably the mapping from A to B that he wouldn't do well like you wouldn't do it in like a visually appealing way otherwise I think for Z they could actually use a VAD type of loss but it would not be a VAD because there's still this thing that is [Laughter] yes no it's not it's not that fair comparison I would say though like if

L9 Semi-Supervised Learning and Unsupervised Distribution Alignment -- CS294-158-SP20 UC Berkeley


your data set is small like Dan training is usually pretty fast as well anyways but that's off topic so just like operationally what does that mean if we go through like one cycle of that well cycle consistency loss so we get some image from some from a source domain and then we would randomly sample AZ for my mapping to be because remember that the mapping from A to B is also stochastic so B could take up a lot of different forms and I'm going to generate a noise source that dictate what it is so this is what I'm going to samples and then


there will be a set of mappings that go through so the mapping from A to B now it takes a and annoy sauce for B that gives me B and then from this BN a I can try to guess what was the the Z that has generated my original a so that is my encoder to guess Z of a and then finally I would plug in the B they're generated and the Z they are generated and from these two I get back I get back this a prime which supposedly should be close to my original sample so that's all good like it's fairly easy to implement like


just a small surgery on cycle again how a does it you so the full the first thing that you would want to run is essentially you would want to first give Z as the additional input to the mapping which they call stochastic cycle again I guess so that is without changing the loss function like without introducing the encoder it's what this column that we are looking at and and then the test here is we simple and add app for my Gwangju of data here and then I will fit it through my cycle again but with different Z terms so this is


imagine this is coming from Z 1 Z 2 Z 3 Z 4 so this is surprising right because we ocean aliy we went through this argument of like how just changing cycle games to make you to have to take in Z wouldn't actually make use of Z like it would just ignore it but what we are seeing here is actually different right so you give it an H mask and it actually generate diverse samples for you so that's interesting like if we look at like this shoe like apparently there are all different colors and even though we do the Augmented one like using the new


loss function and the encoder I would say they look probably about the same the same kind of diversity but it is kind of interesting like why why does why does cycle again walk especially like this is highly contrasting with the analysis that we just went through so if I especially if I take a black shoe and then map it to a semantic mask and then map it back then if I get a Y shoe which is a point to here is something that could happen if I get a y shoes back I'm going to incur huge l1 loss because black and white are


just to end of the spectrum in your color space so so that is somewhat puzzling like what what is this actually doing like if if it can generate this diverse samples then that means it's not optimizing its cycle gain loss well but it is optimizing its cycle gain loss as well so the very interesting thing here is that cycle again when you go from a high dimension like a high entropy mapping like RGB images to a low entropy one like a semantic mask it can actually hide information in some kind of high frequency pattern so this is what we are


this is what we are seeing here so like like going back to the black shoe exam like when you map from a black shoe to his H pattern like it would give you like seem a plot seemingly plausible H patterns and then as some high-frequency noise to it that they know that this is coming from a black shoes and then you can look at the rough shape of that pattern and then it also reached the high frequency noise that's encoded in there to say oh this should be black and that's how it manages to still do the cycle consistency right so I


consistently get the same color back by hiding imperceptible information in some of my mask on my edge and that's pretty interesting and the way that you can show that it's doing that is by essentially constructing an experiment where you so so this is quite domain a this is domain B you get the B out and then you try to sample different Z that you try to fix that B and then you try to sample different Z's so if this is coming from a cycle gain loss then you will see that the mask itself even though seemingly it doesn't in cold


color information it's implicitly encoding color information so if I take this mask that's coming from my model I will have color information hidden in it in such a way that when I sample different Z's you always get the same output and that's how is able to still satisfy the cycle consistency constraint and so and then if you do the Augmented cycle gain loss what you will see is that the mask looked basically the same but there are seemingly less information in there so when you sample different random noise or Z you actually get


different color of shoes back I don't remember that's a that's good to check but like I like I guess like us as we have discussed like if Z is very powerful and potentially getting Co too much so I think I think there would be a balance there and this is kind of doing the cycle walk this is somewhat interesting some are similar to what we had discussed so maybe from A to B and then B to a a to be while I cycle through different noise sauce and then if you do this kind of random walk in an auto augmented cycle again


you can see that even though the mask stays relatively the same the over appearances some of the other color texture does change over time whereas if you train it with the original cycle getting lost you will just get the same pairs repeated again and again that's a somewhat interesting a in my opinion relatively simple and arrogant extension to cycle gain that helped you to deal with stochastic mappings any questions on that before we move on so the next set of questions are can we do better so so far we have covered two


learning principles one is marginal matching and then the other is psycho consistency and I guess it's a good question that whether those are the all of the invariances that we can rely on or are there additional learning signals that we can derive from it it's a good open problem and if we step back and think about this whole problem it's really aligned to distributions we found knowing what's inside was really difficult if we if we think about the categorical distribution that has even probabilities like it's just impossible


to align them because we treat them as pure black paths like all values are the same as each other so one idea that we can move forward from this point is we can look inside a random variable it's we can say this image it's not just a huge a high dimensional random variable to me like I can actually look inside and see what's in there and then maybe use that to help us and for this kind of high dimensional a and B like they typically have certain structures in them that that could be leveraged and as people have pointed out


like when you use a continent and patch phase discriminator in a cycle game they're kind of implicitly employing some of this inductive bias already but I think we there are cases where we can push this even further so the best example that I could find is in NLP for that so let's say ptomaine a is all English sentences and domain B or French sentences then we can imagine that like I can get a random sentence from or English sentences and a random sentence from all French sentences they might have the same empirical frequency but


they might be totally semantically unrelated which is likely to happen so it's like what consistency wooden also rule out that either this is just basically going back to the problems of when you have distributions to have uniform densities nothing could help but what we do know is that each sentence is made up of words and it's very unlikely that in those two totally semantically unrelated sentences they would have words that have same kind of statistics so I'm using the term statistics loosely here I'm going to say more about like what we


can do with this so basically what the exercise that we have gone through is instead of thinking of it as distribution alignment between sentences in different languages if we are allowed to look inside like what's in between each random variable and look at their sub components and do some influence on the sub components they can help us circumvent the problem of long enough learning signal so in the case of NLP the sub components are the words and the large and the larger higher dimensional random variables are sentences or


paragraphs and so that's interesting so like now what we can do is we can for one we can first of all align the words like we can think of ways that you can do distribution alignment on words and even more we can think of we can make use of how different words occur together so let's say the word eye is most likely to be followed by M and because these two things Co occur most frequently in within this large random variable sent a sentence so what is the thing that we can make use of this kind of co-occurrence to the six of sub


components well we have learned one of them what you back in two lectures ago probably so just a recap on sip sip gram water back it's really simple so basically always trying to do is is trying to say given one was the world in a sentence I'm going to say that others other words in this sentence is more likely to occur than every other things in my corpus and in practice you wouldn't sum over all of your dictionary you would do some negative sampling to optimize this but essentially the end we saw this certain vector that described


if two vectors are close together in that vector spaced and they're more likely to occur in a sentence and if you train a very very large model on a lot of text data then they capture how different words are likely to occur together so let's give Graham and what's really interesting is that this kind of woodwork method exhibits really interesting vector calculus so this is again a recap slice what we can look at is that if we look at the direction from a country to his capital the vector is actually relatively similar across a lot


of these different pairs and we might we might be able to ask like so based on this kind of vector if the vector calculus makes sense then does it mean that we can say the vector representation of those words are distributed in a certain manner and more importantly if similar cap vector calculus holds true for all languages meaning I train a word embedding for English let's say on the on the left and they also training for embedding for Italian if they exhibit the same vector calculus meaning all different in


bearings in them placed together in a such way that you could do the kind of country to capital translation then that's a really strong inductive bias for us and if that holds true then we can possibly align words by similarly by just like uncovering some kind of affine or linear transformation that align these two things together so very surprisingly it's actually true so for the virtual vac that we use let's say in fast text if you train it on one language if your trailer multiple languages and then these embedding space they are only a


rotational weight so you're going to learn a rotation matrix that rotate another language into your space and then the results would be both this this is a graphics that grab from a Facebook blog post that Illustrated really well so you basically get this embedding space that that exhibit similar relative structure and then so what you can do but the absolute location is undefined so what you can do is you can just learn a way to align them together and then after the rotation one point in the embedding space would be very likely to


exhibit the same word but in different languages so this this totally blew my mind that this could work yeah so so initially so it was with the citations here you can basically use a small dictionary play a language to language dictionary you can use a small dictionary to learn the alignment so this it was still supervised but the search space is much smaller like instead of going from each word map through a new net and then after another word you would be like every word is already presented by some embedding now


I'm only learning the rotation that is used across all embeddings but then like so basically a couple data points is enough to to specify that yes I don't know probably two mm I would I would guess like I've been totally uneducated guess I like I'm not gonna NLP cousin it could be pretty big yeah so that's pretty interesting so that's what happened up until like fari before 2017 ish is people can like you can align these two embedding spy is basically examples like I can just go to you like French English dictionary and


then look up a couple Wars and then you use that to to align these two embeddings that's really cool then you can do that yes what apparently they are scaled the same or less similar enough [Laughter] this recent paper that proposed a way that you can oh actually another thing that I forgot to mention is so no actually this is it so basically that's a supervised way to align this wording batting like so that's that's really interesting that you can just capture that by a simple rotation well probably not simple but a rotation and what this


work has done is to show that you can actually do that with really good performance in an unsupervised way so now I have two embedding space and then you are lying them without any training signal so the way that is done is actually basically just using the principle of marginal matching so you would similarly apply so basically you your rotation each of each possible rotation basically specify a mapping and then you're going to say after the mapping my marginal distributions to match and then they just train that with


ever Cyril training so like again like a loss to spur make sure that the marginal is specified is the margin always matched and then after they do that but one of the results of possibly do to gain training is usually no very robust and high precision so after they do that they have they have found a rotation that roughly aligned to distributions and then after they have that they would select some top pairs of high-frequency words in those graph alignment and then they would assume that they are actually gwangju for linemen and then you would


use those as like actual pairs to to solve for the exact rotation and apparently this works really well so that's some of the data that you get from there's some additional tricks in terms of embedding nearest neighbor that I didn't go into so but this is the results that they have and they're comparing that with cost lingo supervision and without any supervision which is their own method and it's really surprisingly they could get you competitive performance with using ground truth data of actual pairs so this again like this is not as complex


as translating whole sentences this is only translating words but I still see this as very impressive that this can work at all and become very competitive with supervised methods so then the next part of this is again another paper from facebook is now you can actually lavish all three of the core principles that we have covered so far you can look at so they use world level alignment meaning they started from what we just look at the end supervised level the unsupervised world level alignment so this is you're not


just looking at sentence level you're looking inside sentence to sub component level statistics and then they also use a monolingual language models to make sure what you translate actually looks like a real sentence so that basically you can see that as marginal matching and then they also have this thing called black translation which is another variant of cycle consistency so you translate it from English to French and then French back to English you get you should get back the same sentence and this is a paper that essentially


utilized all these three methods and then from there I think I think they get say of the art and supervised machine translation results that are I think ten you know I don't remember the precise results but that were widely beyond previous stay of the arts and these are some of the Appalachian of showing how many training sentences they use to in order to surpass this kind of system so you can see that you would need probably somewhere between half a million data in order to for it to surpass the flat line which is the unsupervised machine


translation results so that's that for that method and so again like these none of the kind of principles that we have talked about so far a bullet proof but I think what's really interesting is you can seemingly extract training signal out of nowhere by just carefully considering the problems of what are the invariants that you can exploit and especially in NLP like really thinking about how thinking about it knock at a random variable level but really look inside each random variable and what are some additional core current statistics that


>> I'm going to talk about AI for Large Imperfect Information Games, in particular, on how emitted AI that beat top humans in no-limit poker. Okay. So, for starters, this talk is going to be about imperfect-information games in general. I'm not going to talk about perfect-information games like chess or Go, it will be applicable to poker, but also more generally, any strategic interaction that involves hidden information, for example, security interactions or negotiations. I think this is really important for bringing AI into the real world,

AI for Imperfect-Information Games: Beating Top Humans in No-Limit Poker


because the truth is most real-world strategic interactions involve some amount of hidden information. So, when it comes to these games, poker has served as the primary benchmark challenge going back decades. In fact, if you look at the original papers on game theory, pretty much the only application they talk about is poker, because it's so accurately, it captures the challenge of hidden information. Particularly, there's a variant of poker called heads-up no-limit Texas hold'em that has emerged as the primary benchmark for these games.


A heads-up no-limit Texas hold 'em is a massive game. It has about 10 to the 161 different decision points. It is also the most popular variant of poker in the world. For example, no-limit Texas hold'em is the game that is played at The World Series of Poker main events. Every year the winner is determined by Heads of No-limit Texas Hold 'Em. It's also featured in popular movies about poker. For example, Casino Royale and Rounders. In some ways, you could argue it's the purest form of poker. It's subjective, but it is a very strategic game,


whether you win or lose, it's entirely up to your skill. It's not up to the other players at the table, except for your own opponent, I guess. So, there's no kingmaker effects for example, and no pro AI has been able to beat top humans in this game. That isn't till 2017. So, in 2017, we organized something called the Brains vs AI Challenge. We created an AI called the Libratus, which we played against four of the world's best heads-up no-limit Texas hold'em specialists in the world. These are all people that make about seven figures per


year playing this game online. As we played 120,000 hands of poker over the course of 20 days, and there was a $200,000 prize pool divided among the pros to incentivize them to play their best. So, they weren't risking money, but how much money they want, depended on how well they did relative to the other players. So, obviously, if you're familiar with poker, you might not have heard of these pros. So, I wanted to say a word about how strong these pros are, because it really is important to play against the top players.


Unfortunately, there are no objective rankings of professional poker players. But like I said, these are all players that make millions of dollars a year. In fact, here's a question from the poker subreddit, where somebody was asking, ''How good are these players that we were playing against?'' Somebody responded, ''These players will absolutely trounce all the 2,000 heroes that you might have heard of. The heroes from 2000s would be division three college players. Well, whereas these guys are all star caliber pros.''


So, this is a pretty accurate description I would say. There is a big scale difference between the a pros that you see on ESPN, and these guys who actually play this game for. The guys you see on ESPN are basically celebrities. These guys are the guys that actually make a living playing this game. The final result is that Libratus beat the humans in this game by a lot. The victory margin was 147 mbb/game, which is a measurement of win rate and poker, which, unless you are an actual poker player doesn't mean much,


but to give you some perspective, this is about three times the win rate of a top pro versus an average pro. It was statistically significant at about four standard deviations, and each human lost individually to the AI. This was a big surprise to everybody. In fact, when we announced the competition, there was a betting market on the outcome, because it's the poker world, and obviously, like to gamble on these things. When we first announced that we're going to do this competition, the betting odds were four to one against us.


In fact, even after we won on the first day, the betting odds were still two to one against us. I think I was until like the third day that the betting odds were even, and by the eighth day, you couldn't even bet on the outcome of the competition anymore. You could just bet on how much each human would lose on each individual day, because it was clear at that point that this AI is going to win. In fact, even if you asked us, we were not very confident that we would win. I put our odds at about like 60 percent,


maybe 65, but I didn't think we would have a lot victories like this. Actually, after this competition, we did another competition against these Chinese pros. So, basically, somebody called Kai-Fu Lee in China called us and he said, ''We would like you to do another competition in China against Chinese players. We will broadcast it, it would be a lot of fun.'' We were like, ''Well, why should we do this? Because we just played against the top humans. These Chinese players not as good.'' He said that he would pay us.


So, we said, ''Okay, great.'' So, we played 36,000 hands against six Chinese players. We beat them by even more than we beat the top humans in America. That was actually a huge hit in China. It was watched live by millions of people during that competition. They had really nice production where you could see a poster like this. It was way better than what we did in America. All right. So, why are imperfect-information games so hard? After all, we have AIs that can beat humans in games like chess, we have AIs that beat humans in Go.


In fact, you might have heard recently about AlphaZero which can beat humans. Well, it's essentially superhuman in chess, Go, and shogi, all using the same algorithm. So, what is it about imperfect information games that are so difficult? One of the major challenges, not the only one, but one of the major ones, is that in an imperfect-information game, the optimal strategy for a subgame, for part of the game, cannot be determined in isolation. It cannot be determined using information in just that subgame alone.


So, let me show you what I mean. Before I get to that, deep learning has taken a lot of credit recently for a lot of the breakthroughs in AI. Actually, all AI did not use any deep learning, no deep learning at all. But I would also argue that a big reason for why all these AIs are superhuman in various games like chess, Go, backgammon even, is because they use real-time planning. The planning component is huge. In AlphaGo, for example, use Monte-Carlo Tree Search, in Deep Blue, it used Alpha-beta pruning. So, in fact, if you look at


AlphaZero, without real-time planning, I guess this is washed out, but it ends up being right around there without Monte-Carlo Tree Search during real-time. Top human performance is right around here. So, in fact, without Monte-Carlo Tree Search, AlphaZero is not superhuman. The tree search gets you 2,000 ELO addition. So, real-time planning is really important, not just in Go, but also in poker, it turns out. This is actually the key breakthrough that allowed us to be top humans is figuring out how to do real-time planning.


But it turns out that in poker, it ends up being way harder which is where it gets you right now. So in perfect-information games, you take some action, your opponent takes some action, you find yourself in a particularly subgame. Now, you can forget about everything that came before, all the other situations you did not encountered. The only thing that matters is the situation that you're in, and the situations that can be reached from this point on. So in perfect-information games, so for example, if I were to show you this chess board,


you don't have to know how we ended up in this situation, you don't have to know about the Sicilian defense of the Queen's gambit. You can just look at this board, and if you're white, you can say, ''Okay, well, if I do a search, I can see if I move my white queen there, then it's checkmate, and the game is over. So, I should just do that. You don't have to know anything about the strategy of chess. But in imperfect-information games, if you take some action, and your opponent takes some action, and you find yourself in a particularly sub-game,


now some other sub game that you are not in, and in fact, you might not even be able to reach from this point on, can affect what the optimal strategy is for the sub-game that you are in. This is counter-intuitive, but I'm going to give you a concrete example in a little bit that illustrates this. Now, before I get to that, I want to talk a little bit about what our goal is in these games. Our goal is to find a Nash equilibrium which in-two player zero-sum games, is the same thing as a min-max equilibrium. I won't get too technical


about the definition, but basically, in a two-player zero-sum game, if you're playing the Nash equilibrium, you are guaranteed to not lose an expectation. Now, it's not always easy to find a Nash equilibrium, but it's always guaranteed to exist and a finite two-player zero-sum game. So, for example, in rock, paper, scissors, the Nash equilibrium is to this mix randomly between rock, paper, and scissors, with equal probability, because if you do that, then no matter what your opponent does, you will not lose an expectation.


Now, in rock, paper, scissors, that also means you're not going to win an expectation, but in a complicated game like poker where there's a lot of sub-optimal actions that aren't actually played in the Nash equilibrium, it's likely that your opponent will make mistakes and you will end up in practice winning as well. Yes. >> How important is it going to be to play a game So, if I compare this to say, heads up or not. If I got a heads up, if I got to sort of thinking about like this will go about seven players.


>> That is a great question. So, I'll get to this, let's talk about this now. In poker, it doesn't really matter. So, in poker, if you were to use these same techniques for six player poker, you would almost certainly win. That said in general, poker is a special game because, I don't know if you play poker but two special things about poker. One is, it's really hard to collaborate with other players. So, you can't say, "Hey, let's team up against this other person at the table." In fact, if you try to do that,


that'll be against the rules of poker. The other thing that's unique about poker is that people fold in the game. So, even if you have six players at the start of the game, it very quickly comes down to two players because people fold. So, you can use these techniques that are only guaranteed for two-player zero-sum games and it will just work in six player poker. But a big challenge is extending these techniques to other games that do allow for collaboration. In that, we don't really have a good approach for those games yet.


So, for now, I'm just going to assume that we're working in the two-player zero-sum setting and it does extend in some cases to other situations as well. So, our goal is to find an approximate Nash equilibrium. We're going to measure performance in terms of exploitability. You can think of it as, distance from a Nash equilibrium, it's how well we would do against a worst-case adversary relative to if we had played a Nash equilibrium instead. So, how exploitable we are? I would argue that exploitability is actually extremely important and has


been overlooked in the AI community as a whole. I think two recent man-machine matches actually really highlight this. One is the OpenAI, one versus one Dota2 Matches that you might have heard about, and the other is Fan Hui versus AlphaGo. In the OpenAI Matches, they made this AI that was able to beat top humans in one versus one Dota2 over three games. But after they won against the top human, they actually opened it up to the public and they invited random mediocre players to play against it to see if they could find any weaknesses.


In fact, pretty quickly within a few thousand games, weak players were able to find certain tricks that they could basically fool the AI and figured out how to exploit it and beat it. Also, in Fan Hui versus AlphaGo, so they famously beat Fan Hui 5-0. But then after they published the Nature paper, they invited him to play several more matches against it to see if he could find out any weaknesses in the AI. In fact, he was able to find weaknesses where he was able to consistently beat the AI and they had to patch this before they released on the Nature.


So, I think what this really demonstrates is that it's not enough to beat top humans in three or five or even 10 games. You really have to be able to consistently beat top humans, especially if you want to deploy an AI into the real world. If you're Microsoft and you're trying to deploy this products with real users, there's millions or billions of them, if there's a weakness, they're going to find it. But with the [inaudible] , we played the top humans not just in three or five hands of poker, we played them in 120,000 hands of


poker over the course of 20 days. That whole time, all four players working as a team to try to exploit the AI in any way they could find. In fact, actually, I had lunch with one of the players, just a couple months ago. He said that the thing they found most shocking about the competition is that, at the end of each day, we gave them a log of all the hands that were played and we told them what the bot had on each hand that was played. This is big because in poker, a big part of the game is actually keeping your strategy hidden.


If you fold, your opponents does see what cards you have. In fact, even if you don't fold but you lose the hand, you still don't show your cards are. So, you only see your opponent's hand about 20 percent, 25 percent of the time. So, like poker players will sometimes even call, just to see what their opponent had. But here, we're just giving them that information. We're telling them what the bot had on every single hand that it played. So, they didn't have to worry about that part all, and they found it absolutely


amazing that they could not figure out how to exploit the AI, even though we were showing them the hands that the bot was playing every single time and the bot strategy wasn't really changing that much between days. All right, so I think exploitability is extremely important. I think has been overlooked by the AI community, and this is telling that the imperfect information game solving community has focused on throughout its existence. All right, so now, I want to get to the example of why imperfect information games are hard.


I'm going to talk about a simple example that I call a Coin Toss. It starts with the coin flip. So, the coin is flipped that lands heads or tails with 50-50 probability. Player one is going to observe the outcome of the coin toss. Player two is not. So, after this coin lands, Player one has a choice. They can either sell the coin or they can choose play. We'll say, if they choose sell this to some separate subgame, the details of which are not important. The only thing that's important is the expected value.


So, we'll say, if the coin landed heads, then the coin was lucky and they can sell it for 0.50 cents. On the other hand, if the coin landed it tails, we'll say it's unlucky and Player one loses 0.50 cents by selling it. On the other hand, they could choose play, and if they choose play, then it leads to Player two, and Player two has to then guess how the coin landed without having observed how it actually landed. So, if they guess correctly that is Player two guesses heads and the coin actually landed heads,


then Player one is going to lose one dollar and Player two is going to gain one dollar. Here, the payoffs are shown for Player one because this is a two-player zero-sum game. So, Player two just receives the opposite payoff. Now, on the other hand, if Player two guesses incorrectly that is they guess tails and the coin actually landed heads, then Player one gains one dollar and Player two loses one dollar. You can see there's a dotted line between the two players, two nodes this signifies that Player two is in what's called an information set.


This means that Player two because they did not observe how the coin landed, they do not know which of those two states they were actually in. So, why do you imagine that you are Player two in this game and yeah, so why do you imagine that you are Player two in this game, you've just observed Player one chooses play action and so you know that you are in this imperfect information subgame. So, what should you do? Should you guess heads or should you guess tails? But one option is to just always guess heads. But if you do that, that's


obviously a really bad strategy because now Player two can just sell the coin when it lands heads and get 0.50 cents, and choose play when the coin lands tails and gain a dollar. So, on average they're getting 0.75 cents. On the other hand, you could always choose tails, but that's also a really bad idea because now Player two can choose play when the coin lands heads and gain a dollar and choose sell when the coin lands in tails and lose 0.50 cents but it's better than losing a dollar. So, on average, they're still getting


0.25 cents in this game. So, it turns out that the optimal strategy is to mix. It's to guess heads with 25 percent probability and tails with 75 percent probability. If you do that, then no matter what Player one does, the best they can do is just break-even, get on average zero dollar in this game. So, this is the Nash equilibrium strategy for Player two in this game, at least for this subgame. But now, let's say we change the game a little bit. Let's say we changed the payoff for the sell action. So, now, an expectation Player one


loses 0.50 cents for choosing sell when the coin lands heads, and gains 0.50 cents for choosing sell when the coin lands tails. Well, it's pretty easy to see that as Player two, your strategy in this subgame should now change as well. Now, you should be guessing heads with 75 percent probability and tails with 25 percent probability. But you can see what's happened here is that, by changing the expected value of the sell action, we have affected what the optimal strategy is in the play subgame. Even though the sell action is not


part of the play subgame and in fact, it's not even on the path leading to the play subgame. So, this is something that happens in imperfect information games. It does not happen in perfect information games. In perfect information games, if you wanted to determine the optimal strategy in subgame, you only need to look at that subgame by itself. But in imperfect information games, you have to look at the game as a whole. So, you can think of like perfect information games is a special case where you don't have to worry about all this stuff.


Imperfect information games are the more general case where this is a problem. So, what do we do? Well, it turns out that we don't actually have to know the strategy for the entire game as a whole. I mentioned that this sell action leads to a subgame, where both players might take actions. But you don't have to worry about that, the only that really matters for determining the optimal strategy in this play subgame, is the expected value of Player one choosing sell. So, what we can do is try to estimate what that value is to Player one,


and if we have that, then we can determine the optimal strategy in the play subgame. So, that's what we actually did in [inaudible]. We also have a theorem that says, "If this estimate is within delta of the true Nash equilibrium value, then we can solve for the play subgame and get within delta of the Nash Equilibrium." So, in the [inaudible] , we actually do this. We have this massive game which is simply way too large to solve upfront. So, we come up with a really good strategy just for the early part of the game,


and we estimate what the optimal strategy is and what the expected values are in the later parts of the game. Now, when we're actually playing, we find ourselves in a particular subgame, we come up with a much better strategy for that particular subgame using information about the expected values from the other subgames. Then, we repeat this process, we just come up with a really good strategy for that early parts that are coming up and just estimate how to play in the later parts. We find ourselves in early subgame,


we again compute a much better strategy and that particular subgame using information about the expected values of the other subgames. That's called nested subgame solving. This was the key breakthrough that allowed us to be top humans. So, when I, yes? >> Just [inaudible]. >> Yes, that's a great question. So, actually when we do this, this is sort of a general, how we would do this in general. But in poker, we solved the first two, there's four betting rounds. So, we solve the first two, with a pre-computed strategy.


Because it's like each round grows exponentially in size. So, the first two rounds are actually pretty small. We got to the end of the second betting round, that's when we applied Subgame Solving. So, we came up with a much better strategy for the remainder of the game. We abstracted the bets. So, we want to consider all the 20,000 different bet sizes, we would just consider a small fraction of them. Then each time the opponent acted, each time they've made a bet, then we would solve a new subgame for that bet size.


So, we would apply this recursive solving thing every time the opponent made an action beyond the second betting round. So, when I mention this idea of what's called Safe Subgame Solving, where we use the expected values from the other subgames, people always ask about this thing called Unsafe Subgame Solving, which is the more intuitive approach to doing this. The idea here is well, why don't we just estimate what the opponent strategy is? Let's say we can sort of like we played a bunch of hands against them or we can estimate what


the Nash equilibrium is for them, and we figured out, well, they should be choosing play 80 percent of the time when the coin lands heads and play 30 percent of the time when the coin lands tails. Let's say, just for example. Now, if we assume that the opponent's playing this strategy, can we then reason about the distribution of states that we might be in and then solve optimally using that distribution. It turns out that doesn't work. So, let me give you an example of what this would look like. When the coin lands either heads or tails,


we reason that we're in one of these states with 50-50 probability. Now if we observe player one choose play, we would say, okay, well, in a Nash equilibrium, we would expect player one to choose play 80 percent of the time if we were in the left state, 30 percent of the time when we're in the tail state. So, we update our belief about what state we're in using Bayes rule, and now we can reason that we're in that left state with 73 percent probability, and in that right state with 27 percent probability. Now, we would just say, well, if


we assume this distribution is correct, then the optimal strategy is to always choose heads. But if we've already established that's a really bad idea, because now the opponent can simply shift to selling the coin when it lands heads and choose and play with the coin lands tails. So, the problem with this approach is that we're making an assumption about how the opponent is playing. If it were true, like if this distribution were true that they were choosing play with 80 percent probability in heads and 30 percent probability in tails,


then yes, we can apply this reasoning. But the opponent strategy can always change. They can always change adversarially to us. Yes? >> There's one thing that I've always been interested in, when you play the Nash or when you play against the opponent. It seems like they're not going to shift. Even if you're playing the wrong strategy, they wouldn't exploit it off immediately. They have to learn to exploit it. I guess it's safe to definitely model the Nash, but I am curious about this intermediate space where


you'd play against how they've been playing in the past, recognize that you need to shift in some way because they may shift as well. >> So, yeah, that's a great question. We actually did not, so one of the interesting things about humans playing poker, is that they're actually really good at exploiting. They are phenomenal at it. Way better than computers are currently. So, we actually did a competition against them in 2015 where we lost, and we would sometimes change the bot that they were playing against between days.


Within 50 hands of playing, they could figure out how the bot had changed. So yes, if you can make an AI that could figure out how to do this better than humans, then that that might be valuable. But we're playing against really talented humans and we didn't think that we could beat them at that game, but then also why bother playing that game? Why bother trying to play that mind game if we can just approximate a Nash equilibrium and guarantee that we're going to win. So, I would argue that in the two player zero-sum game,


if you want to beat top humans in a two-player zero-sum game, the best approach is to just approximate the Nash equilibrium because now, no matter what they're going to do, you're going to beat them. Now, I would argue that if your objectives are different, so for example, if you really want to beat up on a weak player and exploit them, then yeah, you don't want to necessarily play Nash equilibrium. You want to adapt to their weaknesses. This is challenging to do correctly, because if you try to adapt to a weak players weaknesses,


you never know if they're just fooling you. Like if you're playing rock, paper, scissors against somebody and they throw rocks three times in a row, and you say, well, he's clearly an idiot who's throwing rock every single time, I'm going to throw paper next time, they could just throw scissors. So, there's no safe way that, except in special cases, there's no safe way to do that kind of opponent exploitation and still guarantee that you're going to beat top humans expectation. So, I think that is an excellent avenue for future research,


but I think that in the two-player zero-sum setting, where we're just trying to beat top humans, I think this is the better way to go about it. So, Unsafe Subgame Solving, is very risky for this reason because if you make an assumption about how the opponent is playing, they can always shift to a different strategy and take advantage of that. Now, that said, it turns out that this actually works, yes, so we must account for the opponent's ability to adapt. Now, that said in practice, Unsafe Subgame Solving works unusually well in poker.


It turns out that if you just approximate what the Nash Equilibrium strategy is and then assume that the opponent is playing that, and apply Subgame solving in this way, that actually works really well in this domain. But we have found situations where this does not work well, and I think in more general settings, it would not do well. So, we actually use this in a few situations in Libratus. But in general, I would not recommend doing this. Unless the domain is specially structured that it would work. So, Safe Subgame Solving,


the idea is instead that we're just going to estimate what the expected value is for the opponent's Subgames, for the opponent's actions for different Subgames, and use that information to determine the optimal strategy for the Subgame that we're in. Now, this works if your expected values are perfect, but if they're not perfect you're obviously not going to compute an exact Nash equilibrium. So, it turns out that there's room for improvement here. By the way, this idea has been around for awhile. It's was first introduced in 2014.


It was never really used in practice because it didn't actually give you good results in practice. Because you don't have perfect estimates. But what we came up with, is a way to dramatically improve the performance without giving up any theoretical guarantees. With this thing called Reach Subgame Solving. So, here's an example of how this works. This is going to get a little tricky, so if you have any questions in the next few slides please let me know. So, let's say that we have this slightly larger game now.


It's still basically the same structure, there's a coin flip that only player one observes. Player one takes a sequence of actions, they eventually end up in this choice between selling the coin or playing, choosing play. Now, if they choose play, player two has to guess how the coin landed. Well, let's say your estimates are off in this game. Let's say we estimate that for choosing Sell, they will get an expected value of minus one regardless of which state they're in. Well, the best that we can do is just guess


50-50 between heads and tails and guarantee that they get just an expected value zero, for choosing play. But maybe we can use information about the earlier actions to improve upon this. So, maybe there is this earlier action that player one could have chosen, if the coin landed heads, where they could have gotten expected value of 0.5. Well, that means that in the Nash equilibrium, they would choose that action and get expected value of 0.5, and in the other case, they would come down here and choose play and get it fixed value of zero.


So, they're getting an average of 0.25 in this game. But we can now shift our strategy as player two, to ensure we get tails more often, which guarantees that player one now gets negative 0.5 in this case. In the heads case that means they will get to 0.5 for choosing play, but that doesn't really matter because they're already getting 0.5 for this earlier deviate action. So, we're not really giving up anything in this situation, we're just making ourselves better off. Because they would never gets to the situation


where they would choose the 0.5 anyway. This seems really intuitive but there's a problem with this, which is really subtle. I'm going to have to go to a bigger game, which is going to get even more complicated to really illustrate it. So, here is this bigger game. It's still pretty similar, that a coin that lands heads or tails with 50-50 probability. Player one in both cases now, let's say has this deviate action. In the heads case, they can get 0.5, and at tails case, they get minus 0.5. Or they can choose to continue,


in which case we run into this chance node. This chance node is public. It just leads to two different Subgames, so both players observe the outcome of this chance node. It just leads to see different situations that are strategically identical. It's an irrelevant chance node, but it is a chance node. Then after this chance node, player one let's say, chooses play and we estimate the expected value of them choosing play is now zero. So, let's say, we were player two in this game, we observe player one choose play.


Which means that we are in one of these two different situations. Either the coin landed heads, they choose to continue and let's say we observed that the chance node end up going left, and then they chose play. Or the coin landed tails, player one chose to continue. We observe the chance node going left and they choose play. So, we're in either this situation or this situation. Well, we observed that they have this deviate action of where they could've gotten expected value 0.5, if the coin landed heads. So, maybe we would say,


we say, okay, well, we can increase the expected value for this action to one and lower it to minus one in this case, for example, by always guessing tails. That is okay because since this situation is only encountered 50 percent of the time, the expected value for this action is now just 0.5, and so that matches the deviant actions, so we're not giving up anything. Does anybody see the problem with this? All right. The problem is, if that chance node had gone the other way, if it had gone right, we would apply this same exact reasoning.


We would say, okay, well, we can increase this expected value to one, because we're all encountering the situation half the time, so this expected value goes up to 0.5, and now the opponent is getting expected value zero, we're not giving up anything. But if we apply this reasoning regardless of which way this chance node goes, then what that means is our strategy is to always guess tails in both situations. So, in reality, it means that the expected value in this case is one and in this case is one, which means that the expected value is


actually one, it's not 0.5. So, now player one could be better off by choosing to continue instead of choosing this deviate action. So, what this illustrates is that when you are doing this Subgame solving, you're doing this real-time reasoning, you can't look at the expected values of what we call the Blueprint Strategy, the pre-computed strategy. You have to think about what the expected values would have been if we had entered that Subgame and applied Subgame solving there too. So, that makes things that way more complicated.


But fortunately, with Reach subgame solving, by the way, two prior papers had actually discussed this idea of, okay, we're encountering this situation, let's just increase the expected value for here, because they could have gotten an expected value earlier on and missed this problem that you have to consider about all the subgames that people could end up in. So, two prior papers published about this and they both got it wrong, and our paper, NIPS 2017 recognized this problem and actually came up with a fix that


allows you to do this Reach subgame solving, while still guaranteeing that the points, your exploitability is not going to go up. The basic idea is to just only increase the expected value for both of these situations by 0.5. The actual details get a little bit more complicated, but the aren't too important for this talk. But the idea is you just increase the expected values by less depending on how many subgames they could end up in. You have question? >> Well, I was just wondering, I know this is really simple thing, you can't hold it.


You don't think if it's just wrong versus this weird like if you only increase the values for the left of the public chance node. >> Yeah, so let's see where the situation- >> You set those first to explain it? >> Yeah. >> It seems like this is still correct. It's just weird because you're saying if nature flips a coin heads, then I'm going to do this weird reaction, and if it's tails, then I'm not. >> So, I'll say this. If you were actually only increasing the expected value to one in this situation, and keeping the expected value at zero


in this situation, that's totally fine. >> Okay. >> But you have to think about what would have happened. Imagine this from player one's perspective. If we would increase the expected value to one in this situation because this chance node went left, and we would have increased this expected value to one if this chance node are gone, right, Then what player one is thinking is that if they're in this situation they're thinking that, if I choose this action, regardless of which way this chance node goes, I'm getting expected value of one.


>> Yeah, yeah. If our algorithms actually did. >> Yeah. So, that's what I'm saying is that, you have to think about what your algorithm would have done in all these other situations that didn't come up as player two, yeah. >> Okay. >> Okay. So, that's Reach subgame solving. Yeah, so the idea is for off path actions, we have to consider how we would have applied subgame solving and all these other situations that didn't come up. We have a solution for it, which is basically to split that what we call slack among the different subgames.


We have a theorem that says, "Reach subgame solving will not increase the exploitability of a safe subgame solving's technique." If there is, these earlier actions where the opponent could have chosen a higher expected value, then it will actually improve performance relative to traditional subgame solving. In terms of actual, yes. >> [inaudible]. >> Yeah. >> Then you'll have to know the expected values for the subgames that are on the path from there. >> That's correct. So, you look at all the path, yeah, yes.


You look at all the situations where the opponent could have gone off path. >> Yeah. >> You have to know the fixed values for those. Yeah. >> I understand that these are values you put. So, is it correct when you're learning? So, it seems that you're allowed to do the updating some of these expected values, but then you are making this per all [inaudible]. >> Yes, so maybe I should have clarified this earlier. So, I'm assuming that we're basically run an algorithm that approximates a Nash equilibrium for the entire game,


and that's where these expected values are coming from. So, we have this a rough approximation of what the Nash equilibrium is for the entire game, and that's giving us expected values for all these different actions, but they're not perfect. It's like with AlphaZero for example. AlphaZero gives you policy and values for all the different states that you might be in, but that's obviously not perfect and you can improve upon it by doing Monte-Carlo tree search in real-time. >> What do you mean formally by the safe technique to exploitability.


So, you assume that [inaudible] is off by a certain fixed data? >> That's a great question. So, by safe subgames solving, I mean that there is some exploitability, our strategy, this pre-computed strategy that we have is exploitable by some amount. >> Assuming that all of these are correct or they have-? >> Well, I'm just saying that we've run this. Let's just say we've run a traditional reinforcement learning algorithm on the entire game, no real-time playing, just pre-computed strategy, that strategy that we have now


is exploitable by some amount. I am saying that we would like to improve upon this in practice by doing real-time planning. But we want to guarantee that by doing real-time planning, we're at least not going to increase the exploitability of our strategy relative to what we had before. Now, in practice, it ends up being way lower exploitability, but we want to guarantee. We can't really guarantee that it's going to decrease in most situations, but we want at least guarantee that's not going to increase. So, that is what I mean by safe subgame solving.


>> The purple like estimates from your pre-computed suboptimal value function? >> Yeah, so those are the values that we've estimated based on our prec-omputed strategy for the game. >> We can use those to compute the red basically? >> Yeah. >> Okay. >> So, the red is the real-time planning expected values. >> So, what's the right procedure? >> For time reasons, I decided to not really talk about how we're actually doing all this computation of the strategy. We use something called counterfactual rep minimization,


which converges to an approximate solution and one over square root T time. So, if you do T iterations, you will get within one over square root T of the Nash equilibrium. Okay. >> Like in terms of the numbers things then. >> It is also, yes. So, it's linear in the number of information sets. >> So, like terabytes? >> Well, okay. So, with the broadest, we actually used about several terabytes of data and we used about millions of core hours of computation time. >> In real-time? >> In real time, no. In real time, it was lower.


>> [inaudible] . >> So, that was for the pre-computed strategy. For real time, it ended up being we used about 1,000 cores, so about 50 nodes, and the memory is actually really small. It was probably less than 100 megabytes. We all actually figured out how to improve upon this massively, which I'll get to in a little bit. >> Is it a 100 like per core or per? >> No, it's the whole thing, 100 megabytes for the whole thing. The actual game, when you're solving from the turn onwards, like third betting rounds at the end of the game,


the actual size of the game is pretty small, but because you have to compute a solution equilibrium for it, it takes a lot of computational power. >> So, was there a limit on how much time you have to make a decision? >> Yeah, we ended up doing it in less than 20 seconds or so. There wasn't like an official time limit, but we gave them some guidelines on how long it would take on average. We also didn't get the humans the time on it. So, if they wanted to take 10 minutes for a decision then that was fine with us.


I will also, when we're thinking about the time on the thing. So, one of the interesting challenges of poker is that, you don't want to give it away timing tells, right? So, if it takes you two seconds to make a decision, then the opponent might think you have a really easy decision whereas, if you take two minutes, and it might be a difficult decision, they can figure out what hand you have. So, if you're playing, if you look at the World Series of Poker, it gets really annoying because, at the final table,


they all take the same amount of time for every single decision. So, they take like two minutes for every single decision, even if it's a really trivial one. We didn't want the humans to have to do this because it would've taken forever and would pissed them off, would have pissed us off, so we told them flat out that the bot is not going to look at timing tells. So, if they took two seconds to make a decision, that's totally fine, we won't change anything. But we can't make them also do that for the bot, rather like if the bot took


two seconds versus two minutes, they would pick up on that and they can't not pick up on that, right? So, we had to make the bot take the same amount of time for every single decision you made to prevent that. So, that's also why it ends up taking longer to do this thing. There's a lot of decisions that are easy, but we can't make that obvious. All right. So, experiments on medium-size games. So, it turns out that our reach subgame solving technique that I just described, that's about three times, it's about three times less exploitable.


Then, prior, safe subgame solving techniques, and nested subgame solving, this idea of applying subgame solving repeatedly as you go down the game tree. That is 12 times less exploitable than the prior state of the art, which is to just say, well, if the guy bet $60, and we have in our precomputed solution, a strategy for if he had bet $50, then we'll round it, and treat it as if you had bet $50 instead. That was the previous state of the art. So, this is 12 times less exploitable than that in Heads-up No-Limit Texas Hold'em.


Sorry, in smaller versions of Heads up No-limit Texas Hold'em. Okay. So, that is one reason for why imperfect information games are hard. There is a second reason that I wanted to get to you, and I think I still have time to do it. This is more recent research. The second reason is that states don't have well defined values in imperfect-information games. So, let me show you what I mean here. In a perfect-information game, and it's single agent settings, if you take smashing, your opponent takes an action, you find yourself in


a particular decision point is particularly subgame. You don't solve to the end of that subgame. Right? You do what's called depth limited reasoning. So, the remainder of the game is too large, so you come up with a strategy for the next several actions. Then, once you get to a certain depth limit, you say,okay, I've looked far enough, I'm just going to substitute a value for this leaf node, and say the value of this arrangement of pieces on the board looks like player one, white wins with 60 percent probability.


So, you assign a value to that state, and then you solve the depth limited subgame using those values at the leaf nodes. It turns out that this does not work in imperfect-information games. I can give you a really simple example of why. This is a game that I called Rock-Paper-Scissors+, is exactly like rock-paper-scissors, except if either player throws, what is it? Scissors? Yes. If other player throws scissors, then the winner gets two points, and the loser loses two points. That's just to break the symmetry of the game.


So, the next equilibrium in this game is to throw rock and paper with 40 percent probability, each, and scissors with 20 percent probability. Now, imagine that we're trying to do a depth-limited version, a depth-limited solving of this game as player one. So, we look one move ahead, and then we're going to substitute the Nash equilibrium value at each of those states, where instead of going to the end of the game. This is the depth-limited subgame. It's really easy to see that if we were to try to solve this depth limited subgame,


there is no way that we're going to find the optimal strategy of 40 percent, 40 percent, 20 percent. Right? There's no just not enough information in this depth-limited subgame to find the Nash Equilibrium. Why is that? Well, it turns out the reason is because we are essentially assuming that beyond this decision point, player two is going to play the Nash equilibrium strategy. Right? That's where we got the 000 from, this is assuming that if we had chosen scissors, and player two plays the Nash equilibrium strategy beyond this point,


this expected value is zero.. But in reality, player two strategy beyond the depth-limit depends on what our strategy is above the depth-limit. If we choose rock, 80 percent of the time, player two's strategy isn't going to be to play the Nash equilibrium, it's going to be choosing paper, 100 percent of the time. So, this is what the state values will look like or if we were to choose paper, 80 percent of time, then they would switch to always choosing scissors, and this is what the state values would look like.


So, in an imperfect-information game, the values of the states of the depth-limit depend on what our policy is, in the earlier parts of the game. So, how do we deal with this? Well, one option is to just actually make the state values dependent on our policy, and say, "Okay. Well, the value of a state is a function of the description of that state, and our policy for the entire game." Well, that is theoretically correct to the promise that's extremely expensive, I mean absurdly expensive. The other option is something called, well,


we actually metaphor about this AI called DeepStack. They condition the value of a state on the belief distribution of both players at that state. So, they said, okay. Well, at this decision point, I'm not going to condition on the strategy for the early part of the game, I'm going to look at all the different states that I might be in, and the probability that I believe I'm in each of those states, which in this case is 80 percent, 10 percent, 10 percent. Then, this game, it ends up being the same exact thing as just


conditioning on the policy but, in general that's not the case. The problem is that this is still extremely expensive. So DeepStack for example, in Heads up No-limit Texas Hold'em, use 1.5 million core hours of computation, and could not be prior top AIs. The other problem is that the technique currently does not scale to larger games, basically games where you have more states in an information set. This, you can get by with this in Heads-up No-Limits Texas Hold'em, because it's only about, in any single decision point,


every single information set, there's only about 1,000 different states in that information set. But in a game like Five Card Draw for example, there could be 5 billion or in a game like Stratego, that could be 10th to the 20th or something. So, this would not work in those larger games. So, we do instead, is this paper that we just had except it's NIPS for 2018 called a depth-limited solving, we use this in an AI, we created called Modicum, and let me walk you through the algorithm here. The idea is, instead of assuming


that there is a single value at this depth-limit, we're actually going to let the opponent choose between multiple different values for these states. We create these different values in an iterative process. So, we start off by assuming that player two, beyond the depth-limit, is going to play the Nash equilibrium strategy that we precomputed. Then, we solve this depth-limited subgame, which in that case means, let say, we solve it, and we say our strategy is going to be one-third, one-third, one-third probability.


Now, we're going to calculate a player two best response. So, we can say, okay. Well, if our strategy here is to play one-third, one-third, one-third, player two could exploit us by always choosing rock. Now, we add that best response to the set of strategies that player two can choose at the depth-limit. So, now, we're going to solve this depth-limited subgame again, and we're going to say, add this depth-limit, now player two can choose. They can either choose the Nash equilibrium strategy that we had before or they can


choose this best response that we just calculated, which was always play rock. They make this decision because all of these states, sharing information set, they can't say, "Okay, well, in this state, I'm going to choose this policy, and this state I'm going choose this policy." They have to make the same decision at each of these different states that share an information set. So, we solve this depth-limited subgame again. Then, we again calculate a player two best response to that strategy, and then we add that strategy to the set of


best responses that player two can choose at the depth-limit. We can repeat this process as many times as we want. Now, some details about this technique, it might seem like this is really expensive, and may not get good performance because we can't add like a million different strategies for player two to choose that the debt limit, but it turns out that because they are making this choice separately at each information set, they're essentially able to, even if we only give them a choice between 10 different strategies,


beyond the depth limit, if they are making that choice individually at 100 different information sets, they're actually choosing between 10 to the 100 different strategies for the entire remainder of the game. So, it actually grows very quickly. The other thing for player one, I talked about what player two does, so player two is choosing between these different strategies that they could play for the remainder of the game, player one, we're going to assume is playing the approximate Nash equilibrium. Player one is us, we're going


assume that weren't playing according to the approximate Nash equilibrium strategy for the remainder of the game. The set of player two's strategies is precomputed it's not determined in real time. They could do it in real time, it would be too expensive in practice. Okay. So, this is what performance looks like if we do this depth-limited solving thing. So, on the x-axis here, we have the number of values per leaf node that the opponent can choose between at the depth limit, and on the y-axis, we have exploitability measured in milligrams per game.


This is a simplified version of No-Limit Access Hold'em, that only has two betting rounds instead of four. You can see, if we only assume, that if we assume that each state has a unique value, which is essentially the Nash equilibrium value like we would in a perfect information game, exploitability is extremely high. But as we add more strategies for the appointed to choose between at the depth limit, exploitability drops off very quickly. In fact, with 16 different choices, were essentially at Nash equilibrium.


So, the beautiful thing about this. Yes. >> So why is that have one. [inaudible] blue will be the same.. >> No. Because actually blue, sorry, so for blue here, it's not doing any real-time reasoning, it's doing this like, if they had bet $60, I'm going to round that to $50. So, red is actually doing real-time reasoning, but it's assuming that each value has a well defined unique value. >> So, [inaudible] generation or frequency or something, so what [inaudible]. >> There are a lot of similarities between this


and things like double oracle methods and things like that. Yes. All right. So, in terms of head-to-head performance, the really cool thing about this technique is that it allows us to make a really strong poker AI using very few resources. To give you some perspective, we had this bot called Tartanian8 that we made in 2016, which won the Annual Computer Poker Competition, which is competition among poker AIs. It used two million core hours of computation, 18 terabytes of memory, and there's no real-time reasoning.


We have Slumbot which won the, that wasn't us, this is a different bot, that won the 2018 competition, used 250,000 core hours, two terabytes of memory, no real-time reasoning. Modicum which is the bot that uses its depth limited solving, uses just 700 core hours, 16 gigabytes of memory, plus real-time with a 4-core CPU in under 20 seconds per hand, and it beats both of those other bots. So, to put this in perspective even further, the broadest, which is the AI that we played against the top humans, used millions of core.


I think it was something like two million or five million core hours, probably it's 20 terabytes of memory, and played in real-time using 1,000 cores. So, we're able to get what is essentially probably superhuman, we haven't actually tested against humans but I'm pretty sure this is a superhuman poker AI. We're able to get basically superhuman performance using the resources in a laptop. In fact, since I published this paper, I've just put another paper on archive where we figured out how to make this three times faster.


So, it could probably run on a smartphone now. Yeah. >> Where is the [inaudible] you can just run. >> It turns out that there is a huge amount of variance in poker. >>Yes. >> Because we're doing real-time reasoning and we're taking 20 seconds per hand, the variance is massive. In fact, we actually, we train this using 700 core hours, it took us like a million core hours to actually compute all these results. So, this has been a problem in the entire field, that the variance is just absurd. So, this is a graph of head-to-head performance.


Here we have the Libratus, which beats up humans. Here is Modicum which is the AI we just created that uses way fewer resources. Here are some other benchmark bots. A bot from 2016, a bot from 2014. Here is DeepStack, which is a bought from the University of Alberta, which actually has very low exploitability. But in terms of head-to-head performance didn't end up being that strong. It also uses this real-time reasoning as well. Though a different form of it. All right so the key takeaways, yes. >> You said that one doesn't have,


it has low exploitability but it's not that strong? >> In terms of head-to-head performance it's not as strong. So, in terms of head-to-head performance it actually doesn't beat prior benchmark bots. >> Yes, I guess that's curious to me, because if you're not exploitable. Okay so-. >> When I say low exploitability, I mean just relative to the past bots. So, the exploitability is still, it could be extremely high, we actually don't know. We can't calculate exploitability exactly and heads up no limit takes it whole so we don't know.


But it appears that it has lower exploitability compared to the absurdly high exploitability of the previous bots Yes. So, key takeaways. In real-time planning, you always have to consider how the opponent can adapt to changes in your policy. That is something that is really important in imperfect permission games. Perfect-information games you can mostly ignore that but not completely. Imperfect-information subgame cannot be solved in isolation. States and imperfect-information games do not have well-defined values.


All right. So, I have done some other work that I did not discuss. One thing I did not discuss, well I guess I talked about it briefly is how we actually solve these games. We use an algorithm called counterfactual regret minimization, which has been around for about 10 years now. Works extremely well, even though the theoretical guarantee on convergence is only 1 over square root t. I just had a paper that I released, where I developed a new form of CFR which beats the prior state-of-the-art by a factor of three.


So, I'm really excited about that, and that's going to be used in all the future research. I have some work on pruning in CFR. So, it turns out that CFR you end up exploring the entire game tree which is a big waste, because a lot of actions are suboptimal and you don't want to waste time coming up with what you should do if you play a really crappy poker hand, because in reality you just fold it right away. So, I have this pruning technique that provably reduces to computing and memory costs of running CFR asymptotically.


In practice, it speeds things up by an order of magnitude of two. I also have a paper on determining the optimal actions in a continuous action space. So, one of the interesting things about no limits exist hold them is that you have this continuous action space where you can choose to bet any amount between $100 to $20,000. The truth is that, what we do right now is just domain-specific abstraction techniques, where we say okay, well, you probably just want to bet either half the pot or one times the pot or two times the pot,


and it doesn't really matter if you are betting 0.6 times the pot or 0.5 times the pot. But that relies on domain knowledge that we know the optimal bet fractions are roughly in that range. So, it'll be nice to have an algorithm that doesn't rely on domain knowledge that can actually determine the bets to use without any human knowledge. So, that's where this 2014 paper does, and hopefully we'll have some follow-up work on that in the future. For future directions. So, I mentioned before, a lot of this work has been looking at poker as a domain,


and that's because it takes a lot of infrastructure to actually build up to do experiments on large games. So, we have a lot of expertise that has been developed over the years on how to run these techniques efficiently on a game like poker. And if we wanted to test on another large game, it would take years to build up that expertise on how to do experiments on those other games. There's also no other really good, we can't really compare performance to other bots in other games, because there are no other games where


there are bots that are competitive. But I would love to move beyond poker, and a few different directions with that. One is, we have these techniques for perfect information games like AlphaZero, and we have these techniques for imperfect-information games like poker, and it'll be nice to bridge the gap and find a single algorithm that works really well in all these different games. Another thing that I'm really interested in is going beyond two-player zero-sum games. So, I mentioned that if you try to move on


to general-sum games it's a bunch of theoretical challenges the pop-up. So, right now, we don't know how to cope with those challenges. But dealing with general-sum games is really important because most real-world situations are general-sum and not not zero-sum, except for like maybe military interactions or security. So, in particular working out something like negotiation, I think is a really interesting line of research. In general moving things more towards real-world domains, I think we're at the point right now where we


can actually start bringing these techniques into the real world, and I think that's going to be a really interesting line of research as well. All right. So, I'll stop there and I'll take some last minute questions thank you. >> Yes so, I guess, in the real world often you don't know the rules of the game, you don't know them in advance, [inaudible] situation. >> Right. >> But you do observe the outcomes of players. Have you thought about trying to, what works with race when you go into a situation where you don't know


the rules of the game but you can observe the outcomes of players? >> That's a good question so yes. So, all of our work assumes that we have an accurate model of the world or the game that's being played. I think a lot of these techniques will carry over if you want to try to figure out the structure of the game as you go. There was actually a paper from another student at CMU recently on this problem. So, people are starting to look in this direction, I have not. But it's something that I think is very interesting.


It is also a way harder problem, because to figure out what information your opponent knows or figuring out what they could know is a really difficult challenge. I'm not sure how to go about that. Yes. >> I think in many applications in something like reinforcement learning. So, right now I can imagine splitting the environment into the portion that which I can model that will be like Chess moves, and then I can set the parts that I don't necessarily want to model, because I'm very risk averse. So, that could become like adversarial moves,


and I to be robust. Have you thought about how well other techniques would apply, or completely out of the ballpark. >> To be honest, I never really considered that direction for the research. I think there's a lot of potential here. This is an area of research that I think has been overlooked by a lot of people. So, it's actually been a very small community that has been working on the space, and I think people are starting to appreciate that can be applied to a lot of different things. We're starting to see papers on how


to apply something like counterfactual regret minimization to more mainstream reinforcement learning topics. So, I think there was a paper from Berkeley recently on regret minimization in single-agent settings. So, I think there is definitely potential to extend the research and to the more traditional reinforcement learning settings, but I have not looked into that yet. >> [inaudible] a little bit, is there anything I'm learning? So, in the real world one of the things that's really difficult is, I don't know the rules of the game


and I really have no idea what my problems hails are often. Is that something that people have looked at? Trying to basically think about simultaneously trying to improve my own hails trying to get them in model of what's going on with my opponent. >> Yes. I guess that would be kind of a subset of the previous case we discussed, where it will be like, maybe you know the structure of the game but you don't know what the opponent's payoffs. I think this has been looked at in the game theory community, but more in simple cases not


[Music] [Music] from the earliest writings of men we know that the human race has been comprised of the haves and the have-nots when I was a kid back during the Great Depression I was obsessed with a desire to know what invisible something separated the haves from the have-nots being a have nots I wanted to know why so few manage to be well-off financially in a country where success is available to everyone for example in checking the Statistical Abstract of the United States published by the Bureau of the census I discovered just lately

Change Your Life in 19 Minutes with Earl Nightingale


that only 10% of the men in this country 65 years of age and older have incomes of $6,000 or more a year more than 80 percent of all men 65 or older have incomes under four thousand a year only seven point six percent of incomes between seven and ten thousand a year and only three point seven percent have incomes of ten thousand a year or more a man starts his working career in his 20s often earlier he's fortunate in that he lives in the free world he has better than 40 years to make the grade financially in the richest country on


earth yet according with his statistics only about ten out of a hundred will be financially secure by the time 65 rolls around and only about four men out of a hundred will be financially comfortable now why let me tell you how to find out for yourself conduct your own survey start down the street in your neighborhood on any Saturday or Sunday and ask the man of every house two questions the first question is what are you doing at the present time to increase your income now that is how much do you want to earn


and when you've evaluated the blanks there you get in response to that question asked question number 2 which goes how much money are you planning to be worth at age 65 and when the silence becomes too unnerving thank him and move on to the next house ask 50 men a hundred a thousand until you're completely convinced that the reason men don't make more money during their working lives and the reason they're not financially independent by the time they're sixty-five is simply that they seldom if ever do any constructive


thinking on either subject it's that simple unfortunately the reason is so easy to earn far more money than the average man earns in this country is that so few so very few are going about it the right way this is a race without enough contestants to mother about the few who are really in the race can all be winners some will finish ahead of the others but even the man who finishes last in this race will be financially secure most people more than 90% aren't even in the race to prove it ask yourself the two survey questions


up until the time you started listening to this message what were your plans for increasing your income how much do you want to earn and how much money had you decided to be worth by the time you're 65 you see people who earn large incomes aren't lucky and they're not crooks as those without money are so fond of pretending nor are they end up with more brains or talent necessarily than their friends and neighbors nor are they privy to our cult secrets and only a very few were lucky enough to have had rich fathers or grandfather's most of the


people earning the big incomes today started the same way you and I did and most other people the only difference between the men who earned big incomes and those who earn small incomes is that those earning big incomes decided to earn more they're the people who made it their business to earn more you see a woman who does not think about baking an apple pie for dinner tonight will never think of looking up the recipe for apple pie without the decision for pie there's no motivation for checking out the recipe a man who


does not think about driving his car to st. Louis Missouri or Nacogdoches Texas will never get roadmaps which show how to get the same go as in Nacogdoches and a man who never decides to earn more money will never think of learning how of looking up the rules burning more money you see people do what they make up their minds to do so get rid of the ancient superstition once and for all that people who are in big money are special people are lucky or get the breaks or had money to begin with or knew someone or are smarter or


anything else these are alibis they can all be disproved a thousand times the reason there are so many of these alibis around is that men who failed to make the grade financially are seldom honest enough to just admit that they really didn't try and keep trying so in order to justify their failure in order to remain seated they dream up and passed along these old alibis we're all self-made but only the successful will admit it I want that occasion to visit Charleston South Carolina had never been there before so I hired a taxi to drive me


around the historic old town I particularly wanted to see the battery with that famous shot was fired on Fort Sumter along this beautiful drive some of Charleston's oldest and finest homes without over the bay I commented to my cab driver on what lovely homes they were and he said yes some of those homes have 40 rooms and then he thought a moment he said and every one of them is owned by a crook this is how the have-nots justify themselves and their lot in life I didn't say anything because I didn't feel I was entitled to


advise him or try to straighten out his thinking this is a free country where as long as he doesn't hurt others everyone has the inalienable right to be just as wrong as he wants to be as Thomas R Lounsbury the American scholar and educator put it we must view with profound respect the infinite capacity of the human mind to resist the inroads of useful knowledge my taxi driver and men and women like him all over the world have been kidding themselves and holding themselves down and refusing the bounty and abundance of the world for


centuries knowledge is available to everyone we can either listen to those qualified to teach us or we can go along with those ancient stumbling blocks we get from people who don't know any more than we do the truth incidentally about those homes along that beautiful Drive is that they were built by the men and women who made the largest contribution to the city of Charleston in just a moment I'm going to give you the formula for getting rich but before I do I want to remind you of something before a jet pilot begins his takeoff from an airport


he carefully goes over a checklist item by item he does this not only because it's required by law but because he cannot afford to trust so important a job to his memory alone he has another checklist that he goes over just as carefully before he begins his lap down at his destination he does this without fail every time he takes off and every time he lands well I think living successfully is as important as flying an airplane and because of this I think each of us needs a checklist too and that's why there's one included with


this cassette we need a checklist to go over item by item before we take off in the morning and before we drop off to sleep every night so I want to recommend that you have fixed the checklist be a bathroom mirror stare at it as you brush your teeth in the morning and stare at it again as you prepare for bed at night go over each item and as you do think of what each item represents and here's number one it's the formula for getting rich it also explains why you're in your present position whether you're earning


six thousand a year or sixteen thousand or sixty thousand or six hundred thousand it applies to every adult whether he's employed or unemployed it applies to the richest man and to the forest and every person in between and here it is our rewards in life will always be an exact proportion to our contribution our service now that's what the formula means as the first item on your checklist memorize it our rewards in life will always be in exact proportion to our contribution our service listen to it think about it


until you know it emotionally as well as intellectually it might give you some slight feeling of superiority to realize that there's probably not another man within a mile of where you live who knows it you can add it as a question on your survey if you want proof of that if you want it in another form here it is as it applies to a man's job it's the same thing really the same thing applies but you can express it differently the money you're paid by the company you work for will always be in direct ratio to then


what you do your ability to do it and the degree of difficulty involved in replacing you maybe you want to write the formula down in both of its forms and think about it until it's as much a part of you as your name the reason it isn't spelled out on your checklist is because you might not want everyone to know what you're up to the checklist is valuable only to a person who knows what the words really indicate all right you've got the formula as you think about it its meaning will become clearer to you with


the formula there are two rules which must be applied to properly use it this formula together with the two rules is your recipe or your roadmap to earning all the money you really want now let's take a look at item number two on your checklist and I'm serious about your putting this checklist on your mirror you'll notice it's pressure sensitive item number two the gold mine the pulitzer prize-winning playwright Archibald MacLeish and is play the secret of freedom wrote the only thing about a man that is a man is his mind


everything else you can find in a pig or a horse strong words aren't they but as long as you live you will never hear a truer statement the key to every human being success lies in his mind the goldmine between his ears one idea can make you rich a lot of good ideas can move your steadily upward in the work you do and ideas are free and just think there's nothing now being done commercially that will not be done better much better in the years ahead next years homes and most of what's in them will be better than this year's


next year's cars will be better next year's manufacturing distributing marketing and selling and advertising should be better nothing is now being done as well as it must be done in the future and every innovation every new improvement will be somebody's brainchild now what your especially how many good ideas have you come up with during the past year if you continue on as you have in the past where would you be and what will you be earning say a year from now five years from now every day of our lives we walk


or drive buy more opportunity than we could develop in a lifetime in 50 lifetimes back in the twentieth Sinclair Lewis wrote that you can kidnap a man blindfold and take him to any city in the country with a couple of notable exceptions put him in a chair in the downtown area take off his blindfold and he could sit there a week and not be able to tell you what town he's in the streets are all alike the buildings are all alike the businesses all look alike this is still largely true today the reason for this being that most business


men in this country are playing a game called follow the follower if a man goes into business no matter what line it happens to be the first thing he does is make certain that his place of business outside and inside looks exactly like every other place of business of that type in the country do you know why it's because he's been playing copycats since he was a year old and does it without thinking about it for the same reason kids dress alike in school he wants to be one of the gang he doesn't necessarily examine all the


business establishments in his field and pattern his on the one outstanding example the one that inspires him the one that he can really believe in he just does what everybody else in his business is doing and by this simple process he guarantees his own mediocrity whose drum are you marching to if indeed you're marching to anyone's and why remember whatever you now do for a living will be done differently quite differently a few years from now never in the history of mankind have the opportunities for all of us been so


great but the great majority of people will be the beneficiaries of progress not those who bring it about which group you've rather belong to if you want to be a contributor you're not just a beneficiary here's the first rule it appears on your checklist as the gold mind so think think deliberately and with a purpose use the goldmine between your ears begin by thinking of a special time every day back during the Depression in New York lumber dealer was growing rich while other lumber dealers were going broke when asked how he did


it he said every evening when I get home I closed myself up in a quiet room sit in a comfortable chair and ask myself how will my business be conducted ten years from now then I try to do it now instead of competing with every other lumber dealer which is what they were doing he was creating he was doing the very thing man was designed to do the very thing man does best a company growing at the rate of 10% a year will double its size in less than eight years but a man can improve his effectiveness 50% or a hundred percent a year or more


the experts tell us that every one of us says with an in deep reservoir sir rebuttal - even genius that he habitually fails to use well let's begin now to reach into these deep rich areas of pure net profit and use more more of our real abilities let's think here's the best way I've found to make yourself think start getting up a little earlier than you're accustomed to right off the bat this gives you extra time that 95% of the men in this country are not using it all one power earlier a day gives you six and a half extra 40-hour


weeks a year but at this time in the morning take a refreshing shower address get yourself a fresh hot cup of coffee if you're a coffee man and then sit down to a clean sheet of paper at the top of the paper write your financial goal now this is the amount of money per year you intend to earn soon incidentally you might like to keep this to yourself - it's nobody's business with yours then start to think think about your goal and what it'll mean to you and your family then see how many ideas you can come up with to help you reach that goal ideas


to improve what you now do for a living ways of increasing your contribution to match your income goal you know jobs don't have futures people do no matter what line of work you may be in there is within it more than enough opportunity to last a lifetime you don't have to think of brand new ideas or revolutionary new ways of doing things although you well might come up with them think of ways of improving what is now being done if you are to increase your income by the amount you've specified you must find ways of


increasing your contribution your service and the heat of this is to be found in your mind in that goldmine between your ears try for five ideas every morning and write them down and save those sheets of paper in a special idea file many perhaps most of your ideas will be worthless but some of them will be very good a few will be excellent and every once in a while you will come up with something really outstanding you see five ideas a day is 25 a week if you don't think on weekends that's more than a thousand ideas a year


one idea can get you to that income you shooting for the law of averages swing so far in your favor you just can't miss try to develop a sense of expectancy that is try to hold the feeling that the goal you're shooting for is a sure thing and that it's only a matter of time before it's realized you know Henry Ford didn't start making cars until he was 45 a friend of mine started a new company at 65 he's still going strong and his new company has sales of better than 300 million dollars a year it's almost never


too late try not to think of things outside of your own line of work or whatever it is you're most interested in to think well and profitably you must discipline your thinking keep it on course controlled he put in one field specialized now for the final item on your checklist it appears as the word and the word is attitude attitude has been called the most important word in the language William James put it this way he said the greatest discovery of my generation is that human beings can alter their lives by altering their


attitudes of mind now this is something to think about can alter their lives by altering their attitudes of mind it's another way of saying we become what we think about look at it this way you're a total environment if you've been an adult for any appreciable period of time your total environment is a reflection of you as a person the house or the neighborhood in which you live the car you drive the clothes you wear the job you do the people with whom you regularly associate your total environment is an exact and merciless


mirror of you as a human being how have you feel your environment can stand some improvement you have only to improve your attitude and your world will gradually change to reflect the changing prison is how to change your attitude beginning now begin to act as would the person you most want to become now that is if you were already in possession of the goal you're shooting for how would you conduct yourself at all of your affairs we'll do it now and tomorrow and the next day begin now how to act the part of the person you most