Title: CONTINUOUS ADAPTATION IN MULTI-AGENT COMPETITIVE ENVIRONMENTS

Abstract: In a multi-agent competitive environment, we would expect an agent who can quickly adapt to environmental changes may have a higher probability to survive and beat other agents. In this paper, to discuss whether the adaptation capability can help a learning agent to improve its competitiveness in a multi-agent environment, we construct a simplified baseball game scenario to develop and evaluate the adaptation capability of learning agents. Our baseball game scenario is modeled as a two-player zero-sum stochastic game with only the final reward. We propose a modified Deep CFR algorithm to learn the strategies of agents in a half-inning game. We also propose a strategy adaptation mechanism that continuously updates strategies based on the anticipation of the opponent's strategy in the inference time. We form several teams, with different teams adopting the same adaptation mechanism but different initial strategies, trying to analyze (1) whether an adaptation mechanism can help in increasing the winning percentage and ( 2) what kind of initial strategies can help a team to get a higher winning percentage. The experimental results show that the winning percentage can be increased for the team with an initial strategy learned from the modified Deep CFR algorithm. Nevertheless, those teams with deterministic initial strategies actually become less competitive.

1. INTRODUCTION:

Reinforcement learning has been successfully employed to solve various kinds of decision making problems, such as game playing (Silver et al., 2017; 2018) , robotics (Levine et al., 2016) , and operation management (Han et al., 2017) . An RL method typically finds the optimal strategy through the interactions with a stationary environment, which is usually modeled as an MDP process. Nevertheless, in the real world, there could be multiple learning agents in the same scenario. The interactions among these learning agents may make the environment no longer stationary from the standpoint of each individual agent. Besides, if these agents are in a competitive relationship, we would expect an agent who can quickly adapt to environmental changes may have a higher probability to beat other agents. To discuss whether the adaptation capability can help a learning agent to improve its competitiveness in a multi-agent environment, we choose a simplified baseball game as the scenario to develop and evaluate the adaptation capability of learning agents.

A lot of games, like Chess (Silver et al., 2018) , Go (Silver et al., 2017) and Atari games (Bellemare et al., 2013; Mnih et al., 2015; Wang et al., 2016) , only try to find the optimal action at each state. In comparison, some complicated games, like baseball and basketball games, need to take into account not only the current status but also the opponent's possible strategies. Moreover, the opponent's strategy is typically time-varying. Hence, players should not only determine what to do under different situations, but also need to dynamically adjust their strategies based on the observations from the opponent's past actions. In a typical baseball game, for example, two competitive teams play against each other based on their pre-determined initial strategies at the beginning of the game. As the game proceeds, the teams continuously update their strategies based on the actions of their opponents, trying to win the game. However, baseball games are inherently highly uncertain while the number of interactions between the pitcher and the batters is rather limited. It is very tricky for the teams to properly adjust their strategies based on a small number of interactions.

In this work, to discuss the adaptation issue in a multi-agent competitive environment, we intentionally construct a simplified baseball game scenario based on the MLB data on Statcast Search (MLB). We make the following assumptions to simplify the problem. (The simple baseball rule is shown on Appendix D.) 1. We only focus on batting and pitching in our game scenario and treat fielding as the environment which is based on the MLB data on Statcast Search. 2. We assume there are only one pitcher & batter in each team, rather than the typical 9-batter case. 3. We assume the pitcher and the batter have the same performance in pitching and batting across different teams. The only difference is their strategies. Besides, they always perform normally and we do not consider the case of abnormal performance in all games. 4. Both the pitcher and the batter have only five possible actions, as explained in the next section.

Based on the above simplifications, we manually form 13 teams, with different teams adopting different playing strategies. We also propose a modified version of the Deep CFR (Counterfactual Regret Minimization) (Brown et al., 2019) algorithm to learn the strategies of the batter and pitcher. In total, we form 14 teams to analyze their adaptation capabilities.

In our simulation, each of these 14 teams plays the best-of-three games against every other team for many series. At the beginning of each series, each team plays based on its initial strategies. As the game proceeds, each team follows the same strategy adaptation mechanism based on the observed actions of its opponent. In our simulation, there are only three games at most for each pair of teams to adjust their strategies. We then analyze the following two main issues about strategy adaptation.

1. With a small number of observations (three games at most for each pair of teams), can the adaptation mechanism help in increasing the winning percentage? 2. If two competitive teams adopt the same adaptation mechanism during the game, what kind of initial strategies can help a team to get a higher winning percentage?

2. BACKGROUNDS:



2.1. SCENARIO DESCRIPTION:

We first explain how we define a simplified baseball game for the analysis of strategy adaptation in a multi-agent scenario. Even though all the following discussions are based on the baseball game scenario, the deduced conclusions can be useful for similar multi-agent problems as well.

In every play of the baseball game, the pitcher aims at an expected target location and selects a pitch type. On the other hand, the batter looks for specific types of pitches appearing in some preferred attack zones. Both the pitcher and the batter select their actions for each pitch and, depending on the result of their actions, the game proceeds to the next state. Specifically, we treat our baseball game scenario as a two-player zero-sum stochastic game (multi-agent MDP) (Bowling, 2000) with only the final reward (win or lose).

We first mention two examples to explain players' different strategies under different situations. Situation 1: 0 out, bases empty, 3 ball, 1 strike Under this situation, the batter tends to be more selective or may even wait for a walk. On the other side, the pitcher tends to pitch a fastball to the middle of the strike zone to avoid a walk. However, if the pitcher has pitched to the middle too often, the batter may look for pitches in this zone to hit. This will increase the probability of solid contact. Situation 2: 1 out, bases full, 3 ball, 1 strike Under this situation, the batter tends to be more aggressive since it is a good chance to score. On the other hand, the pitcher might take advantage of the batter's aggressiveness and pitch to the corner to fool the batter. However, if the pitcher has pitched to the corner too often, the batter may shrink his attack zone to avoid being fooled.

In Figure 1 , we illustrate the flow of our baseball game. The pitcher and the batter are the two learning agents to be trained to play against each other. Here, the Pitching Control represents the pitcher's ability in ball control. As the pitcher aims to pitch at a target location, we model his Pitching Control as a Gaussian distribution centered at the target location with a manually pre-determined variance. The actual pitched location is then sampled from this distribution. After that, we heuristically convert the distribution P (results|zones, type) into the distribution P (results|zones, type, a b ) for every possible action a b according to the athletic ability of the batter. Since the average MLB fastball travels at 95 mph, reaching home plate in just 0.4 second, it is very difficult for human beings to react within such a short period. Hence, the batter typically focus on a specific zone and a specific pitch type to swing in order to increase the probability of solid contact. Generally, the smaller the batter sits on the hitting zone, the more chance he can make a solid contact if the ball goes into the pre-selected zone. In Figure 1 , the fielding distribution models the probability of the next state when a batting result occurs at a certain state. Again, this distribution is modeled based on the MLB data on Statcast Search.

In our baseball game, we denote each state s by a 6-tuple vector: (inning, runs, outs, runners on, strike, ball). To simplify the game, both the pitcher and the batter have only five possible actions, as listed in Table 1 . These actions are defined based on the pitcher's Pitch Location and the batter's Attack Zone, as illustrated in Figure 2 . Here, we assume the pitcher only aims at three possible spots (the green dots) on pitcher's Pitch Location and only has two types of pitch, fastball or curveball. Due to the uncertainty in pitch control, the pitched ball actually spreads around these three spots. On the other hand, based on the pitch type and the location of the pitched ball, the batter has five possible actions. 

Pitcher's action:

Batter's action 

2.2. RELATED WORK:

Humans are able to learn new skills quickly based on the past experience. It is necessary for artificial agents to do the same. Meta-learning, also known as "learning to learn", aims to train a model on a variety of learning tasks so that it can solve new learning tasks or adapt to the new environments rapidly with minimal training samples. Meta-learning has been used to learn high-level information of a model such as learning optimizers for deep networks (Ravi & Larochelle, 2017; Andrychowicz et al., 2016; Li & Malik, 2016) , learning task embeddings (Vinyals et al., 2016; Snell et al., 2017) , and learning to learn implicitly via RL (Duan et al., 2016; Wang et al., 2016) . Especially, model-agnostic meta-learning (MAML) (Finn et al., 2017b) aims to find a set of highly adaptable parameters that can be quickly adapted to the new task. The goal of quick adaptation to the new environments for meta-learning is similar to our work, but meta-learning discusses only the case of single learning agent, instead of the two-agent case. On the other hand, Maruan et al. (Al-Shedivat et al., 2018) construct an adversarial multi-agent environment, RoboSumo, allowing the agent to continuously adapt to the changes of the opponent's strategies. However, their approach considers only one-way adaptation and the opponent is not allowed to anticipate the learning agent's strategy. This is different from our scenario in which both teams adapt their strategies in the game.

When it comes to the multi-agent environments, multi-agent reinforcement learning (MARL) has been widely applied in various applications, such as transportation (Fernandez-Gauna et al., 2015) , social sciences (Leibo et al., 2017) , resource management (Hussin et al., 2015) , and controlling a group of autonomous vehicles (Hung & Givigi, 2017) . MARL has an issue of instability of the training process. To deal with the instability issue, MADDPG (Lowe et al., 2017) and M3DDPG (Li et al., 2019) have been proposed which adopt a centralized critic within the actor-critic learning framework to reduce the variance of policy gradient results. However, these methods have been designed for deterministic policies only. These methods do not perform as well in our baseball game scenario. Opponent modeling (Zhang & Lesser, 2010; Foerster et al., 2017b) is another method in which each agent can explore the opponent's strategy. Foerster et al. (Foerster et al., 2017b) propose a learning method, named Learning with Opponent-Learning Awareness (LOLA), to consider the learning processes of other agents. Their method has successfully enabled the cooperation of two players in repeated prisoner's dilemma games. In this paper, we further extend the discussion to the learning issue of multiple agents in a competitive game scenario.

3. MODIFIED DEEP CFR ALGORITHM & STRATEGY ADAPTATION:



3.1. MODIFIED DEEP CFR ALGORITHM:

To study the impact of initial strategy over the strategy adaptation mechanism, we propose an algorithm based on the modification of the Deep CFR algorithm to learn the strategies of the batter and the pitcher. The Deep CFR algorithm is a state-of-the-art technique to solve the imperfect information game, especially the Poker game, by traversing the game tree and playing with the regret matching algorithm at each iteration to reach the Epsilon-equilibrium (Brown et al., 2019; Zinkevich et al., 2007) . However, due to the stochastic state transitions in our baseball game scenario, it is more efficient to learn the state-value function via temporal-difference (TD) learning (Sutton, 1988) than the tree searching method originally used in Deep CFR. Besides, baseball games are inherently highly uncertain and has only the final reward (win or lose). To simplify the problem, we learn the strategies of the pitcher and the batter in a half-inning and then apply the learned strategies for the whole game.

In a half-inning game, a state s is represented by a 5-tuple vector: (runs, outs, runners on, strike, ball). The batter's final reward V t is defined as expressed in Equation 1, while the pitcher's final reward is defined as -V t . We regard the final reward as "the state-value of the terminal state V t (s)" to avoid the possible confusion with the term "reward", which is to be discussed later.

V t (s) =      -20 • δ[runs] + 30 • δ[runs -1] + 90 • δ[runs -2] +180 • δ[runs -3] + 300 • δ[runs -4] + 400 • δ[runs -5] if runs ≤ 5 400 + (runs -5) • 30 if runs > 5 (1)

The above definition is based on the observation that the average number of runs per game in MLB is about 5 and the intuition that different number of runs in a half-inning would have different impacts on the final winning percentage.

Our algorithm alternatively estimates the payoff matrices and updates the two agents' strategies based on the modified Deep CFR algorithm. In two-player zero-sum games, it has been proven that if both agents play according to CFR at every iteration, then these two agents' average strategies will gradually converge to the Nash equilibrium as the number of iterations approaches infinity (Cesa-Bianchi & Lugosi, 2006; Zinkevich et al., 2007; Waugh, 2009) .

In more details, at the Iteration t, given the batter's strategy σ ′t-1 b , the pitcher's strategy σ ′t-1 p , and V t (s), a state-value network V (s|θ v ) is trained from scratch by using TD learning. Based on V (s|θ v ), we define the trained reward as

reward trained = V ′ -V (s|θ v ) where V ′ = V (s ′ |θ v ) s ′ = s terminal V t (s ′ ) s ′ = s terminal (2)

where s ′ is the next state, depending on the current state and the two agents' joint actions (a b , a p ).

Since this game only has one final reward, we manually choose a default reward reward def ault in order to improve the training efficiency. In our approach, the reward is defined as

EQUATION

where K 1 is a manually selected constant. Based on the above definitions, an action-value network Q(s, a b , a p |θ Q ) is trained from scratch to estimate the expected reward of the two agents' joint actions (a b , a p ). We can express Q(s, a b , a p |θ Q )) at the state s as a 5 × 5 payoff matrix with respect to pitcher's and batter's actions. With Q(s, a b , a p |θ Q ), the batter's instantaneous regrets r b can be estimated by the following equation:

r b (s, a b ) = E ap∼σ ′t-1 p (s) [Q(s, a b , a p |θ Q )] -E ap∼σ ′t-1 p (s),a b ∼σ ′t-1 b (s) [Q(s, a b , a p |θ Q )] (4)

The pitcher's instantaneous regrets r p is calculated in a similar way. We assume both agents know each other's strategies in order to improve the learning efficiency. The sharing of strategy information is feasible in real life since we can ask the pitcher and the batter in the same team to compete with each other to learn their individual strategies.

These instantaneous regrets r b (s, a b ) and r p (s, a p ) are converted to the new strategies σ t b , σ t p based on the following equation:

EQUATION

where A(s) denotes the actions available at the state s and r + (s, a) = max(r(s, a), 0). If Σ a ′ ∈A(s) r + (s, a ′ ) = 0, each action is assigned an equal probability.

Equation 5 comes from the modification of the Deep CFR algorithm. The original Deep CFR algorithm converts the agent's accumulated regret R into the strategy by using the regret matching algorithm at each iteration (Hart & Mas-Colell, 2000; Zinkevich et al., 2007; Brown et al., 2019) . Due to the heavy computational load of the learning process, we cannot afford too many iterations. Hence, we replace the accumulated regret R in the Deep CFR algorithm by the instantaneous regrets r. However, the replacement of R by r results in larger strategy variations at different iterations. To mitigate this problem, we define the new strategy based on the following equation

EQUATION

)

where σt-1 denotes the average strategy and K 2 is a manually selected constant to balance between σt-1 and σ t . At the next Iteration (t + 1), σ ′t b , σ ′t p are used to train a new value network V (s|θ v ). Meanwhile, σ t b , σ t p are accumulated respectively in their strategy memories M σ as encountering the state s, weighted by t as expressed below:

M σ (s, a) ← M σ (s, a) + σ t (s, a) • t (7)

The average strategy at Iteration t is then computed from the strategy memories, expressed as

EQUATION

where A(s) denotes the actions available at the state s. Note that the state-value function V (s|θ v ) highly depends on the agents' strategies, which have large variations at different iterations. Hence, V (s|θ v ) has to be retrained from scratch at every iteration.

3.2. STRATEGY ADAPTATION:

At the beginning of the baseball game, each team plays against each other with its initial strategies σ p and σ b for the pitcher and the batter, respectively. The payoff matrix Q(s, a b , a p |θ Q ) at each state is then learned based on σ p and σ b . The payoff matrix is used for a team to estimate the expected reward of the two agents' joint actions at any state. As the game proceeds, we propose a strategy adaptation mechanism to gradually modify the strategies. In the following paragraphs, we present the adaptation mechanism for the batter only. The adaptation mechanism for the pitcher can be deduced in an analogous way.

During the team's batting time, σ b is the batter's initial strategy and σ p is the prior assumption about the opponent pitcher's strategy. At each pitch, the batter anticipates what the pitcher will pitch based on the past observations over the pitcher's actions. In Equation 9, O(s) denotes all the past observations at the state s and σp (s, a p ) denotes the predicted probability of the pitcher's action a p at the state s. In our mechanism, we treat σp (s, a p ) as the posterior probability, conditioned on the past observations and the prior belief. That is,

EQUATION

In Equation 9, σ p (s, a p ) denotes the prior knowledge about the pitcher's action at the state s. Π

N (O(s)) i=1

π(o i (s)|s, a p ) denotes the likelihood function based on all the past observations at this state s for the pitcher's action a p . Here, N (O(s)) denotes the number of observations at the state s.

In addition to properly anticipate the pitcher's action, the batter should also know the expected reward of each action under different situations, which are expressed by Q(s, a b , a p |θ Q ). The batter can then obtain the advantageous strategy σb by calculating the instantaneous regret in 4 based on Q(s, a b , a p |θ Q ), σp (s, a), and σ b (s, a). After that, he can update his strategy σb based on 5. To gradually modify the strategy, the batter also takes into account the initial strategy σ b (s, a). The adapted strategy at the state s is then defined as

σ * b (s, a b ) = σ b (s, a b ) • (1 - N (O(s)) K 3 ) + σb (s, a b ) • N (O(s)) K 3 (10)

where K 3 is a manually selected constant. Equation 10 indicates that σ * b (s, a b ) depends more on σb as the number of observations N (O(s)) at this state s increases. This is similar to the batter's behavior in real-life baseball games.

As the batter's strategy is adaptively modified as described above, the strategy adaptation of the opponent pitcher is performed based on the batter's reaction in an analogous way. As the number of observations N (O(s)) reaches a pre-selected threshold N pitch , both σ b (s, a b ) and σ p (s, a p ) are updated based on the following equations:

EQUATION

)

where η is a manually determined learning rate. After the update, all the past observations are reset and the same strategy adaptation process starts again.

In our simulation, each team has its own strategy pair (σ p , σ b ) for its pitcher and its batter. For each team, either the pitcher's strategy or the batter's strategy is adaptively modified depending on whether the team is under pitching or batting. In our baseball game scenario, each team plays bestof-three games with every other team. That is, each team has at most three games to observe the opponent team.

4. EXPERIMENTAL RESULTS:



4.1. THE STRATEGY LEARNED FROM THE MODIFIED DEEP CFR ALGORITHM:

Figure 3 : The winning percentage of the learned strategies in the training process.

We train the pitcher and batter in a half-inning game by the modified Deep CFR algorithm, then we apply the learned strategy to the whole(nine-inning) game. At each iteration of the training process, the trained team plays the best-of-three game against every team of the Team-0 to Team-12 (listed on Table 3 ) for 200 series. The averaged winning percentage continuously increases as the iteration proceeds, as shown in Figure 3 . This implies that the learned strategy is getting closer to the Nash-equilibrium strategy if compared with the strategies of the other teams.

Table 2 lists the learned strategies for the pitcher and the batter at 24 different states based on the modified Deep CFR algorithm. Here, we attempt to observe the strategy differences with respect to different states. In Table 2 , the "count" denotes the number of balls and the number of strikes. For example, (2-1) means 2 balls and 1 strike. Each row in Table 2 represents the probability of each action with respect to a specific count for the pitcher and the batter, respectively. It is very interesting to observe that the learned strategy at some state is actually quite similar to real-life baseball strategies. For example, Case 1: Count of two strikes When there are already two strikes, the batter has less freedom to sit on any preferred zone or specific pitch type. As shown in Table 2 , for the batter, the probability of "any" is indeed quite high for these 2-strike cases, except the very special case of 3-2 (3 balls, 2 strikes). On the other hand, for the pitcher, the probability of curveC becomes higher because the pitcher has more freedom to pitch out of the strike zone to fool the batter.

Case 2: Count of 3-1 For this case (3 balls, 1 strike), we may consider two different situations. When the number of runners is smaller than or equal to 1 (Situation 1), the probability of fastM is quite high for the pitcher (0.98) because the pitcher wants to avoid a possible walk in this situation. On the other hand, the batter would have the freedom to reduce the attack zone to make a solid contact (the probability of (fastH+any) in Situation 1 is only 0.27 for the batter). However, when the number of runners are larger than 1 (Situation 2), a batter will be more aggressive since it is a good opportunity to score (the probability of (fastH+any) in Situation 2 for the batter is 0.44). On the other hand, the pitcher needs to pitch more carefully to prevent solid contact. (the probability of fastM (0.76) is lower than the probability (0.98) in Situation 1).

Table 2 : Strategies of the batter and the pitcher.

Situation1: number of runners≤ 1. Situation2: number of runners > 1.

In our simulation of competitive games, we form 14 teams, including 13 manually determined teams and 1 team trained by the modified Deep CFR algorithm. Each team has its own initial strategy and the payoff matrix. Each team plays the best-of-three games against every other team for 200 series.

In each series, each team has its own initial strategy at the beginning, followed by the adaptation mechanism mentioned in Section 3.2 to update its strategy based on the observations of its opponent.

In the left three columns of Table 3 , we list the 14 teams, together with the characteristics of the pitcher and the batter in each team. For Team-0 to Team-3, an active batter is more aggressive to swing, while a passive batter tends to wait or to choose the action fastM more often. On the other hand, an active pitcher tends to pitch to the middle, while a passive pitcher tends to pitch to the corner. For Team-4 to Team-7, both the pitcher and the batter keep choosing the same action. For example, the batter of Team-4 always chooses the action fastH and the pitcher always chooses the action fastS. Team-8 to Team-11 are intentionally formed to defeat Team-4 to Team-8. For example, the strategy of Team-8 is especially designed to defeat Team-6, while the strategy of Team-9 is designed to defeat Team-4. The strategies of Team 0 to Team 3 & Team 8 to Team 11 are listed in B.

In the two columns on the right of Table 3 , we list the averaged winning percentage (WP) of each team with respect to the other teams. For the teams with "specific tendency" (Team-4 to Team-11), some of them have higher WP, such as Team 5 and Team 9 in the without-adaptation domain. This indicates some strategies, like always pitch to the corner, can be very effective in winning the games. However, as the strategy adaptation mechanism is employed, the WP of most teams with "specific tendency" actually decreases. This is quite reasonable since the team with "specific tendency" strategy will restrict themselves to properly modify their strategies against various kinds of opponents and can be easily exploited by their opponents. On the other hand, it seems the the strategy learned from the modified Deep CFR algorithm for Team-13 can serve as a good initial strategy if we want to adopt the strategy adaptation mechanism.

In Table 4 , we compare the winning percentage among the first four teams (Team-0 to Team-3) in the without-adaptation domain. The simulation results are very similar to real-life baseball games: (1) Table 3 : Average Winning Percentage (WP) for each team in the without-adaptation domain (non) and the with-adaptation domain (adap.).

Table 4 : Winning Percentage (WP) among Team-0 to Team-3. The number stands for the WP of the corresponding team listed in the left column.

when the batter is active, a passive pitcher has a better chance to win than an active pitcher (Team-1 vs Team-0); (2) when the batter is passive, an active pitcher has a better chance to win than a passive pitcher (Team-2 vs Team-3); (3) when the pitcher is active, an active batter has a better chance to win than a passive batter (Team-0 vs Team-2); and (4) when the pitcher is passive, a passive batter has a better chance to win than an active batter (Team-3 vs Team-1).

Table 4 shows that none of these four teams can dominate the games. Team-1 beats Team-0, Team-0 beats Team-2, Team-2 beats Team-3, while Team-3 beats Team-1. This implies the importance of strategy adaptation. Besides, in Table 4 , we also show that all these four teams have better chances to beat Team-12, which adopts a random initial strategy. This implies that a team with a strategy is better than a team without any strategy.

When two teams, if named as home team and guest team, compete with each other, there are four possible combinations in strategy adaptation: (1) only the guest team adapts, (2) both teams do not adapt, (3) both teams adapt, and (4) only the home team adapts. In Table 5 , we list the competition results of these four different combinations when we choose Team-5 or Team-13 as the home team and choose each of the remaining teams as the guest team. Here, Team-5 represents those teams with a "specific tendency" strategy, while Team-13 represents the team with an initial strategy learned from the modified Deep CFR algorithm. Besides, in Table 5 , we classify Team-4 to Team-11 as Category 1, whose initial strategy has "specific tendency", while classify the other teams as Category 2. On the right three columns of Table 5 , "avg" represents the averaged WP value with respect to all the guest teams, "avg-1" represents the averaged WP with respect to Category-1 teams, and "avg-2" represents the averaged WP value with respect to Category-2 teams.

In Table 5 , we observe some interesting phenomena:

1. In terms of "avg", Team-5 has the best performance when both teams do not adapt, while has the worst performance when both teams adapt. 2. In terms of "avg1", the WP of Team-5 is roughly 50%, basically independent of the adaptation mechanism.

3. In terms of "avg2", the WP of Team-5 drops drastically from "No Adapt" to "Both Adapt". 4. For Team-13, the WP increases if the strategy adaptation mechanism is adopted (see "Both Adapt" and "Home Adapt"). 5. With strategy adaptation, the WP of Team-13 against any other team is higher than 50 %. This implies, in average, Team-13 always wins when strategy adaptation is used.

Based on the above observations, we have the following conclusions:

1. Those teams with a "specific tendency" strategy might have high winning percentage against some of the other teams when there is no strategy adaptation. However, as the adaptation mechanism is adopted, those teams' advantage tends to drop.

2. The team with an initial strategy learned from the modified Deep CFR algorithm benefits from the strategy adaptation mechanism.

Table 5: WP of Team-5 and Team-13 against the other teams for 2000 series.

WP of Team 5 against the other teams.

WP of Team 13 against the other teams.

5. CONCLUSION:

In this paper, we construct a simplified baseball game scenario to develop and evaluate the adaptation capability of learning agents. We are especially interested in what kinds of teams have a better chance to survive when there is strategy adaptation. We propose a modified Deep CFR algorithm to learn an initial strategies of the batter and pitcher. The experimental results indeed show that the team with an initial strategy learned from the modified Deep CFR algorithm is more favorable than a team with a deterministic initial strategy in our baseball game scenario.

In this work, since we only focus on the impact of strategies on Winning Percentage, the capabilities of the pitcher and the batter are fixed across different teams. In the future work, we would relax this constraint and both teams are required to anticipate the " capabilities and strategies " of the opponents during the game process. This setup will make the game scenario more realistic, and the behavior of the agents would be more similar to the behavior of human players in real life. Initialize strategies σ 0 b , σ 0 p , σ ′0 b , σ ′0 p with random strategies.

3:

Initialize Strategy memories M σ b , M σp .

4::

Choose final reward V t (s) and default reward reward def ault . Select constant K 1 and K 2 .

5:

for Iteration t=1 to T do 6:

V (s|θ V ), M σ b , M σp = LEARNVALUE(σ t-1 b , σ t-1 p , σ ′t-1 b , σ ′t-1 p , M σ b , M σp ) 7: Q(s, a b , a p |θ Q ) = LEARNQ(V (s|θ V ) 8:

Calculate both players' instantaneous regrets r b , r p respectively with equation 4.

9::

Convert the both players' instantaneous regrets r b , r p to the new strategies σ t b , σ t p respectively with equation 5. Calculate the new strategies σ ′t b , σ ′t p with equation 6.

12:

end for Initialize V (s|θ V ), Replay buffer D.

3::

for episode t=1 to K do 4:

Initialize s.

5::

repeat (for each state of episode) 6:

Accumulate σ b (s), σ p (s) respectively in their strategy memories, weighted by t with equation 7.

7:

Select actions according to σ ′ b , σ ′ p from s and observe the new state s ′ .

8::

Store transition (s, s ′ ) Into D.

Sample a random minibatch of N transitions (s i , s ′ i ) from D. Update the value function V (s|θ V ) by minimizing the loss: Initialize Q(s, a b , a p |θ Q ), Replay buffer D.

L = 1 N Σ i [(V (s i |θ V ) -V ′ ) 2 ] where V ′ = V (s ′ i |θ v ) s ′ i = s terminal V t (s ′ i ) s ′ i =

for episode t=1 to K do 4:

Randomly initialize s.

repeat (for each state of episode) 6:

Randomly select actions (a b , a p ) from s and observe the new state s ′ and reward according to equation 3. Set N pitch , η, K 3 .

3:

Initialize the observation memory O.

Prepare σ b (The batter's initial strategy), σ p (the prior assumption about the opponent pitcher's strategy), and payoff matrix Q(s, a b , a p |θ Q ).

repeat (for each state of episode) 6:

if N (O(s)) ≥ 1 then // N (O(s)) denotes the number of observations at the state s.

Table 6 : The strategy of Team 0 Table 7 : The strategy of Team 1

Table 8: The strategy of Team 2 Table 9 : The strategy of Team 3

Table 10: The strategy of Team 8 Table 11 : The strategy of Team 9

Table 12: The strategy of Team 10 Table 13 : The strategy of Team 11

:



Figures and Tables:

Figure 1: Flow of baseball game scenario.

Figure 2: Strike Zone.

memories M σ b , M σp respectively with equation 8.11:

Learning Value function1: function LEARNVALUE(σ b , σ p , σ ′ b , σ ′ p , M σ b , M σp ) 2:

s|θ V ), M σ b , M σp 14: end function In our experiment, The hyperparameters are chosen in the following:K 1 = 30, K 2 = 30, K 3 = 20, N pitch = 4, η = 0.1. Algorithm 3 Learning Q function 1: function LEARNQ(V (s|θ V ))2:

s, a b , a p , reward, s ′ ) into D.8:Sample a random minibatch of N transitions (s i , a b,i , a p,i , reward i , s′ i ) from D. Update the Q function Q(s, a b , a p |θ Q ) by minimizing the loss: L = 1 N Σ i [(Q(s i , a b,i , a p,i |θ Q )reward i ) 2 ] s, a b , a p |θ Q ) 13: end function A.2 ADAPTATION MECHANISMAlgorithm 4 Adaptation mechanism for the batter during a game 1: procedure A GAME 2:

Possible actions.

Table content: <table><tr><td>1. fastM</td><td>1. Wait</td></tr><tr><td>(fastball in middle)</td><td>(do not swing)</td></tr><tr><td>2. fastS</td><td>2. fastM</td></tr><tr><td>(fastball in shadow)</td><td>(sit on fast ball in middle)</td></tr><tr><td>3. curveM</td><td>3. fastH</td></tr><tr><td>(curveball in middle)</td><td>(sit on fastball in heart)</td></tr><tr><td>4. curveS</td><td>4. curveH</td></tr><tr><td colspan="2">(curveball in shadow) (sit on curveball in heart)</td></tr><tr><td>5. curveC</td><td>5. Any</td></tr><tr><td>(curveball in chase)</td><td>(hit any pitch in strike)</td></tr></table>

Peer review (iclr2020):
UPDATE: Thank you for the detailed response. I think the changes you have made to the paper have improved it, but there remains significant work to complete before the paper reaches its full potential. For example, Figure 3 is a useful additional insight but it provides no quantification of the variation between repeated runs nor a comparison to a suitable baseline approach. Similarly, the additional hyperparameter details in Appendix A are helpful to enable reproducability but without explaining how these values were chosen it is impossible to assess the rigor of the empirical evaluation. -- This paper proposes modification to Deep CFR and introduce a simplified baseball environment to evaluate the modifications. The rigour of and detail in documenting the empirical evaluation is not currently of the standard I would expect for publication at ICLR. I will detail suggestions for improvement below, but my most pressing concern for discussion in the rebuttal is with regard to the 2nd conclusion - If the team has learnt a Nash equilibrium as initial strategy, then how can performance of either team be improved by only one team further adapting their policy as shown in Table 5 lower avg2 by the difference between No Adapt (0.61) and Guest Adapt (0.51)? If I have interpreted the authors results correctly, this demonstrates that the initial policy is not a Nash equilibrium. Suggested Improvements: In the related work section, the narrative of the paper would be clearer if the authors introduced why the meta-learning literature is being reviewed. I would also recommend weakening the claim that *instability of the training process* is *the critical issue in MARL* to just noting it is *an issue* and request the authors justify the claim that this *is particularly severe in a competitive environment*. Why is this more of an issue in fully competitive environments than it is in general sum games? In Section 3.1 the authors claim they need to switch to TD learning from tree search due to stochastic state transitions but there are forms of search that can accommodate stochastic transitions so this claim needs to be removed. Perhaps the authors can motivate this change in another way? This section also introduces 2 further simplifications of the domain (1) learning strategies for half-innings and applying them to the whole game; (2) agents knowing their opponents true action chosen and not the noisy observation (e.g. pitchers target location instead of actual pitched location) and (3) *both agents know each other*s strategies* - these should be included in Section 2.1 when the domain is described. Section 3.1. closes by stating *In principle, their average strategies will gradually converge to the Nash-equilibrium strategy.* This is very weakly argued, to make such a claim the authors should provide evidence that their environment and modifications to Deep CFR meet the requirements of the theory where this guarantee was proven. Section 3.2. notes that the learning rate is *manually determined* but the precise methodology of tuning hyperparameters is not provided and no values of settings used for any hyperparameter are included in the paper. Without details of the methodolofy it is unclear if a rigorous empirical evaluation was performed and without the precise hyperparameter settings used the results are not reproducible. Section 4.1. notes *It is very interesting to observe that the learned Nash-equilibrium strategy is actually quite similar to real-life baseball strategies.* Setting aside the issue regarding whether a Nash equilibrium has been learnt, this is a subjective opinion not an rigorous empirical observation. The same applies to the comment *The simulation results are very similar to real-life baseball games* on page 9 Can you support these claims that the learnt strategy is similar to real-life by comparison to the data collected from the MLB Statcast? In Section 4.2. there is further imprecision in the discussion of results. For example *as the strategy adaption mechanism is employed, the WP of most teams … actually decreases.* As this mechanism is a core contribution of the paper, this evaluation needs to be more rigorous. Is there a statistically significant difference caused by using the proposed mechanism? All results presented should include quantification of variation as well as average values and the number of repeats these averages are taken from should be clearly documented. The inclusion of a limited subset of teams in empirical evaluations (e.g. Table 4 only including teams 0,1,2,3 and 12 and Table 5 only including teams 5 and 13) should be clearly justified. All references to papers published in conference proceedings or journals should cite the published version of the paper and not the arxiv version. All references should include the full publication venue and not abbreviations (e.g. Zhang and Lesser, 2010 is currently listed as just AAAI). References to online resources should include a note of the date accessed. Minor Comments: 1) The word *purpose* is used frequently in place of *propose* (e.g. line 7 of the abstract) 2) Page 2: *plays the-best-of-three game* -> plays the best-of-three games 3) Page 2: *that worth further study* -> that are worth further study 4) Page 3: *To simply the game* -> To simplify the game 5) Page 3: *listed in C* -> listed in Appendix C 6) Page 8 (and recurring): Initial speech marks are the wrong way around, this looks like a Latex error. 7) Page 10: *we would relief* -> we would relax 8) Page 10: *in the real life* -> in real life