Continuous Adaptation in Multi-agent Competitive Environments

In a multi-agent competitive environment , we would expect an agent who can quickly adapt to environmental changes may have a higher probability to survive and beat other agents . In this paper , to discuss whether the adaptation capability can help a learning agent to improve its competitiveness in a multi-agent environment , we construct a simplified baseball game scenario to develop and evaluate the adaptation capability of learning agents . Our baseball game scenario is modeled as a two-player zero-sum stochastic game with only the final reward . We propose a modified Deep CFR algorithm to learn the strategies of agents in a half-inning game . We also propose a strategy adaptation mechanism that continuously updates strategies based on the anticipation of the opponent ’ s strategy in the inference time . We form several teams , with different teams adopting the same adaptation mechanism but different initial strategies , trying to analyze ( 1 ) whether an adaptation mechanism can help in increasing the winning percentage and ( 2 ) what kind of initial strategies can help a team to get a higher winning percentage . The experimental results show that the winning percentage can be increased for the team with an initial strategy learned from the modified Deep CFR algorithm . Nevertheless , those teams with deterministic initial strategies actually become less competitive . 1 INTRODUCTION . Reinforcement learning has been successfully employed to solve various kinds of decision making problems , such as game playing ( Silver et al. , 2017 ; 2018 ) , robotics ( Levine et al. , 2016 ) , and operation management ( Han et al. , 2017 ) . An RL method typically finds the optimal strategy through the interactions with a stationary environment , which is usually modeled as an MDP process . Nevertheless , in the real world , there could be multiple learning agents in the same scenario . The interactions among these learning agents may make the environment no longer stationary from the standpoint of each individual agent . Besides , if these agents are in a competitive relationship , we would expect an agent who can quickly adapt to environmental changes may have a higher probability to beat other agents . To discuss whether the adaptation capability can help a learning agent to improve its competitiveness in a multi-agent environment , we choose a simplified baseball game as the scenario to develop and evaluate the adaptation capability of learning agents . A lot of games , like Chess ( Silver et al. , 2018 ) , Go ( Silver et al. , 2017 ) and Atari games ( Bellemare et al. , 2013 ; Mnih et al. , 2015 ; Wang et al. , 2016 ) , only try to find the optimal action at each state . In comparison , some complicated games , like baseball and basketball games , need to take into account not only the current status but also the opponent ’ s possible strategies . Moreover , the opponent ’ s strategy is typically time-varying . Hence , players should not only determine what to do under different situations , but also need to dynamically adjust their strategies based on the observations from the opponent ’ s past actions . In a typical baseball game , for example , two competitive teams play against each other based on their pre-determined initial strategies at the beginning of the game . As the game proceeds , the teams continuously update their strategies based on the actions of their opponents , trying to win the game . However , baseball games are inherently highly uncertain while the number of interactions between the pitcher and the batters is rather limited . It is very tricky for the teams to properly adjust their strategies based on a small number of interactions . In this work , to discuss the adaptation issue in a multi-agent competitive environment , we intentionally construct a simplified baseball game scenario based on the MLB data on Statcast Search ( MLB ) . We make the following assumptions to simplify the problem . ( The simple baseball rule is shown on Appendix D. ) 1 . We only focus on batting and pitching in our game scenario and treat fielding as the environment which is based on the MLB data on Statcast Search . 2 . We assume there are only one pitcher & batter in each team , rather than the typical 9-batter case . 3 . We assume the pitcher and the batter have the same performance in pitching and batting across different teams . The only difference is their strategies . Besides , they always perform normally and we do not consider the case of abnormal performance in all games . 4 . Both the pitcher and the batter have only five possible actions , as explained in the next section . Based on the above simplifications , we manually form 13 teams , with different teams adopting different playing strategies . We also propose a modified version of the Deep CFR ( Counterfactual Regret Minimization ) ( Brown et al. , 2019 ) algorithm to learn the strategies of the batter and pitcher . In total , we form 14 teams to analyze their adaptation capabilities . In our simulation , each of these 14 teams plays the best-of-three games against every other team for many series . At the beginning of each series , each team plays based on its initial strategies . As the game proceeds , each team follows the same strategy adaptation mechanism based on the observed actions of its opponent . In our simulation , there are only three games at most for each pair of teams to adjust their strategies . We then analyze the following two main issues about strategy adaptation . 1 . With a small number of observations ( three games at most for each pair of teams ) , can the adaptation mechanism help in increasing the winning percentage ? . 2 . If two competitive teams adopt the same adaptation mechanism during the game , what kind of initial strategies can help a team to get a higher winning percentage ? . 2 BACKGROUNDS . 2.1 SCENARIO DESCRIPTION . We first explain how we define a simplified baseball game for the analysis of strategy adaptation in a multi-agent scenario . Even though all the following discussions are based on the baseball game scenario , the deduced conclusions can be useful for similar multi-agent problems as well . In every play of the baseball game , the pitcher aims at an expected target location and selects a pitch type . On the other hand , the batter looks for specific types of pitches appearing in some preferred attack zones . Both the pitcher and the batter select their actions for each pitch and , depending on the result of their actions , the game proceeds to the next state . Specifically , we treat our baseball game scenario as a two-player zero-sum stochastic game ( multi-agent MDP ) ( Bowling , 2000 ) with only the final reward ( win or lose ) . We first mention two examples to explain players ’ different strategies under different situations . Situation 1 : 0 out , bases empty , 3 ball , 1 strike Under this situation , the batter tends to be more selective or may even wait for a walk . On the other side , the pitcher tends to pitch a fastball to the middle of the strike zone to avoid a walk . However , if the pitcher has pitched to the middle too often , the batter may look for pitches in this zone to hit . This will increase the probability of solid contact . Situation 2 : 1 out , bases full , 3 ball , 1 strike Under this situation , the batter tends to be more aggressive since it is a good chance to score . On the other hand , the pitcher might take advantage of the batter ’ s aggressiveness and pitch to the corner to fool the batter . However , if the pitcher has pitched to the corner too often , the batter may shrink his attack zone to avoid being fooled . In Figure 1 , we illustrate the flow of our baseball game . The pitcher and the batter are the two learning agents to be trained to play against each other . Here , the Pitching Control represents the pitcher ’ s ability in ball control . As the pitcher aims to pitch at a target location , we model his Pitching Control as a Gaussian distribution centered at the target location with a manually pre-determined variance . The actual pitched location is then sampled from this distribution . On the other hand , the Batting Result Distribution P ( results|zones , type , ab ) models the probability of the possible batting results . Here , we classify all the possible batting results into 32 categories , listed in Appendix C , with respect to different zones ( represented by different numbers in Figure 2 ) , pitch type , and the batter ’ s action ab . To model this Batting Result Distribution , we first obtain the distribution P ( results|zones , type ) based on the MLB data on Statcast Search . After that , we heuristically convert the distribution P ( results|zones , type ) into the distribution P ( results|zones , type , ab ) for every possible action ab according to the athletic ability of the batter . Since the average MLB fastball travels at 95 mph , reaching home plate in just 0.4 second , it is very difficult for human beings to react within such a short period . Hence , the batter typically focus on a specific zone and a specific pitch type to swing in order to increase the probability of solid contact . Generally , the smaller the batter sits on the hitting zone , the more chance he can make a solid contact if the ball goes into the pre-selected zone . In Figure 1 , the fielding distribution models the probability of the next state when a batting result occurs at a certain state . Again , this distribution is modeled based on the MLB data on Statcast Search . In our baseball game , we denote each state s by a 6-tuple vector : ( inning , runs , outs , runners on , strike , ball ) . To simplify the game , both the pitcher and the batter have only five possible actions , as listed in Table 1 . These actions are defined based on the pitcher ’ s Pitch Location and the batter ’ s Attack Zone , as illustrated in Figure 2 . Here , we assume the pitcher only aims at three possible spots ( the green dots ) on pitcher ’ s Pitch Location and only has two types of pitch , fastball or curveball . Due to the uncertainty in pitch control , the pitched ball actually spreads around these three spots . On the other hand , based on the pitch type and the location of the pitched ball , the batter has five possible actions . Table 1 : Possible actions . 2.2 RELATED WORK . Humans are able to learn new skills quickly based on the past experience . It is necessary for artificial agents to do the same . Meta-learning , also known as “ learning to learn ” , aims to train a model on a variety of learning tasks so that it can solve new learning tasks or adapt to the new environments rapidly with minimal training samples . Meta-learning has been used to learn high-level information of a model such as learning optimizers for deep networks ( Ravi & Larochelle , 2017 ; Andrychowicz et al. , 2016 ; Li & Malik , 2016 ) , learning task embeddings ( Vinyals et al. , 2016 ; Snell et al. , 2017 ) , and learning to learn implicitly via RL ( Duan et al. , 2016 ; Wang et al. , 2016 ) . Especially , model-agnostic meta-learning ( MAML ) ( Finn et al. , 2017b ) aims to find a set of highly adaptable parameters that can be quickly adapted to the new task . The goal of quick adaptation to the new environments for meta-learning is similar to our work , but meta-learning discusses only the case of single learning agent , instead of the two-agent case . On the other hand , Maruan et al . ( Al-Shedivat et al. , 2018 ) construct an adversarial multi-agent environment , RoboSumo , allowing the agent to continuously adapt to the changes of the opponent ’ s strategies . However , their approach considers only one-way adaptation and the opponent is not allowed to anticipate the learning agent ’ s strategy . This is different from our scenario in which both teams adapt their strategies in the game . When it comes to the multi-agent environments , multi-agent reinforcement learning ( MARL ) has been widely applied in various applications , such as transportation ( Fernandez-Gauna et al. , 2015 ) , social sciences ( Leibo et al. , 2017 ) , resource management ( Hussin et al. , 2015 ) , and controlling a group of autonomous vehicles ( Hung & Givigi , 2017 ) . MARL has an issue of instability of the training process . To deal with the instability issue , MADDPG ( Lowe et al. , 2017 ) and M3DDPG ( Li et al. , 2019 ) have been proposed which adopt a centralized critic within the actor-critic learning framework to reduce the variance of policy gradient results . However , these methods have been designed for deterministic policies only . These methods do not perform as well in our baseball game scenario . Opponent modeling ( Zhang & Lesser , 2010 ; Foerster et al. , 2017b ) is another method in which each agent can explore the opponent ’ s strategy . Foerster et al . ( Foerster et al. , 2017b ) propose a learning method , named Learning with Opponent-Learning Awareness ( LOLA ) , to consider the learning processes of other agents . Their method has successfully enabled the cooperation of two players in repeated prisoner ’ s dilemma games . In this paper , we further extend the discussion to the learning issue of multiple agents in a competitive game scenario .

Summary: The paper studies adaptation of agent policies in a simplified baseball game, which is designed as a zero-sum two-agent game between a batter (B) and a pitcher (P), each of which has 5 discreet actions. The introduced game is fully observable but stochastic, which the authors argue is a challenging setup. The authors propose a Bayesian-style adaptation of the agent strategies (where each agent models the probability of the actions of the opponent by computing the posterior give a prior and evidence from the past observations), which seems to be computable analytically, from an initialization learned with counterfactual regret minimization (CFR) that approximates the Nash of the considered game.