Year: 2019

Abstract:
In a multi-agent competitive environment, we would expect an agent who can quickly adapt to environmental changes may have a higher probability to survive and beat other agents. In this paper, to discuss whether the adaptation capability can help a learning agent to improve its competitiveness in a multi-agent environment, we construct a simplified baseball game scenario to develop and evaluate the adaptation capability of learning agents. Our baseball game scenario is modeled as a two-player zero-sum stochastic game with only the final reward. We propose a modified Deep CFR algorithm to learn the strategies of agents in a half-inning game. We also propose a strategy adaptation mechanism that continuously updates strategies based on the anticipation of the opponent’s strategy in the inference time. We form several teams, with different teams adopting the same adaptation mechanism but different initial strategies, trying to analyze (1) whether an adaptation mechanism can help in increasing the winning percentage and (2) what kind of initial strategies can help a team to get a higher winning percentage. The experimental results show that the winning percentage can be increased for the team with an initial strategy learned from the modified Deep CFR algorithm. Nevertheless, those teams with deterministic initial strategies actually become less competitive.

text: In a multi-agent competitive environment, we would expect an agent who can quickly adapt to environmental changes may have a higher probability to survive and beat other agents. In this paper, to discuss whether the adaptation capability can help a learning agent to improve its competitiveness in a multi-agent environment, we construct a simplified baseball game scenario to develop and evaluate the adaptation capability of learning agents. Our baseball game scenario is modeled as a two-player zero-sum stochastic game with only the final reward. We propose a modified Deep CFR algorithm to learn the strategies of agents in a half-inning game. We also propose a strategy adaptation mechanism that continuously updates strategies based on the anticipation of the opponent’s strategy in the inference time. We form several teams, with different teams adopting the same adaptation mechanism but different initial strategies, trying to analyze (1) whether an adaptation mechanism can help in increasing the winning percentage and (2) what kind of initial strategies can help a team to get a higher winning percentage. The experimental results show that the winning percentage can be increased for the team with an initial strategy learned from the modified Deep CFR algorithm. Nevertheless, those teams with deterministic initial strategies actually become less competitive.

heading: 1 INTRODUCTION
text: Reinforcement learning has been successfully employed to solve various kinds of decision making problems, such as game playing (Silver et al., 2017; 2018), robotics (Levine et al., 2016), and operation management (Han et al., 2017). An RL method typically finds the optimal strategy through the interactions with a stationary environment, which is usually modeled as an MDP process. Nevertheless, in the real world, there could be multiple learning agents in the same scenario. The interactions among these learning agents may make the environment no longer stationary from the standpoint of each individual agent. Besides, if these agents are in a competitive relationship, we would expect an agent who can quickly adapt to environmental changes may have a higher probability to beat other agents. To discuss whether the adaptation capability can help a learning agent to improve its competitiveness in a multi-agent environment, we choose a simplified baseball game as the scenario to develop and evaluate the adaptation capability of learning agents.
A lot of games, like Chess (Silver et al., 2018), Go (Silver et al., 2017) and Atari games (Bellemare et al., 2013; Mnih et al., 2015; Wang et al., 2016), only try to find the optimal action at each state. In comparison, some complicated games, like baseball and basketball games, need to take into account not only the current status but also the opponent’s possible strategies. Moreover, the opponent’s strategy is typically time-varying. Hence, players should not only determine what to do under different situations, but also need to dynamically adjust their strategies based on the observations from the opponent’s past actions. In a typical baseball game, for example, two competitive teams play against each other based on their pre-determined initial strategies at the beginning of the game. As the game proceeds, the teams continuously update their strategies based on the actions of their opponents, trying to win the game. However, baseball games are inherently highly uncertain while the number of interactions between the pitcher and the batters is rather limited. It is very tricky for the teams to properly adjust their strategies based on a small number of interactions.
In this work, to discuss the adaptation issue in a multi-agent competitive environment, we intentionally construct a simplified baseball game scenario based on the MLB data on Statcast Search (MLB). We make the following assumptions to simplify the problem. (The simple baseball rule is shown on Appendix D.) 1. We only focus on batting and pitching in our game scenario and treat fielding as the environment which is based on the MLB data on Statcast Search. 2. We assume there are only one pitcher & batter in each team, rather than the typical 9-batter case. 3. We assume the pitcher and the batter have the same performance in pitching and batting across different teams. The only difference is their strategies. Besides, they always perform normally and we do not consider the case of abnormal performance in all games. 4. Both the pitcher and the batter have only five possible actions, as explained in the next section.
Based on the above simplifications, we manually form 13 teams, with different teams adopting different playing strategies. We also propose a modified version of the Deep CFR (Counterfactual Regret Minimization) (Brown et al., 2019) algorithm to learn the strategies of the batter and pitcher. In total, we form 14 teams to analyze their adaptation capabilities.
In our simulation, each of these 14 teams plays the best-of-three games against every other team for many series. At the beginning of each series, each team plays based on its initial strategies. As the game proceeds, each team follows the same strategy adaptation mechanism based on the observed actions of its opponent. In our simulation, there are only three games at most for each pair of teams to adjust their strategies. We then analyze the following two main issues about strategy adaptation.

heading: 1. With a small number of observations (three games at most for each pair of teams), can the adaptation mechanism help in increasing the winning percentage?

heading: 2. If two competitive teams adopt the same adaptation mechanism during the game, what kind of initial strategies can help a team to get a higher winning percentage?

heading: 2 BACKGROUNDS

heading: 2.1 SCENARIO DESCRIPTION
text: We first explain how we define a simplified baseball game for the analysis of strategy adaptation in a multi-agent scenario. Even though all the following discussions are based on the baseball game scenario, the deduced conclusions can be useful for similar multi-agent problems as well.
In every play of the baseball game, the pitcher aims at an expected target location and selects a pitch type. On the other hand, the batter looks for specific types of pitches appearing in some preferred attack zones. Both the pitcher and the batter select their actions for each pitch and, depending on the result of their actions, the game proceeds to the next state. Specifically, we treat our baseball game scenario as a two-player zero-sum stochastic game (multi-agent MDP) (Bowling, 2000) with only the final reward (win or lose).
We first mention two examples to explain players’ different strategies under different situations. Situation 1: 0 out, bases empty, 3 ball, 1 strike Under this situation, the batter tends to be more selective or may even wait for a walk. On the other side, the pitcher tends to pitch a fastball to the middle of the strike zone to avoid a walk. However, if the pitcher has pitched to the middle too often, the batter may look for pitches in this zone to hit. This will increase the probability of solid contact. Situation 2: 1 out, bases full, 3 ball, 1 strike Under this situation, the batter tends to be more aggressive since it is a good chance to score. On the other hand, the pitcher might take advantage of the batter’s aggressiveness and pitch to the corner to fool the batter. However, if the pitcher has pitched to the corner too often, the batter may shrink his attack zone to avoid being fooled.
In Figure 1, we illustrate the flow of our baseball game. The pitcher and the batter are the two learning agents to be trained to play against each other. Here, the Pitching Control represents the pitcher’s ability in ball control. As the pitcher aims to pitch at a target location, we model his Pitching Control as a Gaussian distribution centered at the target location with a manually pre-determined variance. The actual pitched location is then sampled from this distribution.
On the other hand, the Batting Result Distribution P (results|zones, type, ab) models the probability of the possible batting results. Here, we classify all the possible batting results into 32 categories, listed in Appendix C, with respect to different zones (represented by different numbers in Figure 2), pitch type, and the batter’s action ab. To model this Batting Result Distribution, we first obtain the distribution P (results|zones, type) based on the MLB data on Statcast Search. After that, we heuristically convert the distribution P (results|zones, type) into the distribution P (results|zones, type, ab) for every possible action ab according to the athletic ability of the batter. Since the average MLB fastball travels at 95 mph, reaching home plate in just 0.4 second, it is very difficult for human beings to react within such a short period. Hence, the batter typically focus on a specific zone and a specific pitch type to swing in order to increase the probability of solid contact. Generally, the smaller the batter sits on the hitting zone, the more chance he can make a solid contact if the ball goes into the pre-selected zone. In Figure 1, the fielding distribution models the probability of the next state when a batting result occurs at a certain state. Again, this distribution is modeled based on the MLB data on Statcast Search.
In our baseball game, we denote each state s by a 6-tuple vector: (inning, runs, outs, runners on, strike, ball). To simplify the game, both the pitcher and the batter have only five possible actions, as listed in Table 1. These actions are defined based on the pitcher’s Pitch Location and the batter’s Attack Zone, as illustrated in Figure 2. Here, we assume the pitcher only aims at three possible spots (the green dots) on pitcher’s Pitch Location and only has two types of pitch, fastball or curveball. Due to the uncertainty in pitch control, the pitched ball actually spreads around these three spots. On the other hand, based on the pitch type and the location of the pitched ball, the batter has five possible actions.
Table 1: Possible actions.

heading: 2.2 RELATED WORK
text: Humans are able to learn new skills quickly based on the past experience. It is necessary for artificial agents to do the same. Meta-learning, also known as “learning to learn”, aims to train a model on a variety of learning tasks so that it can solve new learning tasks or adapt to the new environments rapidly with minimal training samples. Meta-learning has been used to learn high-level information of a model such as learning optimizers for deep networks (Ravi & Larochelle, 2017;
Andrychowicz et al., 2016; Li & Malik, 2016), learning task embeddings (Vinyals et al., 2016; Snell et al., 2017), and learning to learn implicitly via RL (Duan et al., 2016; Wang et al., 2016). Especially, model-agnostic meta-learning (MAML) (Finn et al., 2017b) aims to find a set of highly adaptable parameters that can be quickly adapted to the new task. The goal of quick adaptation to the new environments for meta-learning is similar to our work, but meta-learning discusses only the case of single learning agent, instead of the two-agent case. On the other hand, Maruan et al. (Al-Shedivat et al., 2018) construct an adversarial multi-agent environment, RoboSumo, allowing the agent to continuously adapt to the changes of the opponent’s strategies. However, their approach considers only one-way adaptation and the opponent is not allowed to anticipate the learning agent’s strategy. This is different from our scenario in which both teams adapt their strategies in the game.
When it comes to the multi-agent environments, multi-agent reinforcement learning (MARL) has been widely applied in various applications, such as transportation (Fernandez-Gauna et al., 2015), social sciences (Leibo et al., 2017), resource management (Hussin et al., 2015), and controlling a group of autonomous vehicles (Hung & Givigi, 2017). MARL has an issue of instability of the training process. To deal with the instability issue, MADDPG (Lowe et al., 2017) and M3DDPG (Li et al., 2019) have been proposed which adopt a centralized critic within the actor-critic learning framework to reduce the variance of policy gradient results. However, these methods have been designed for deterministic policies only. These methods do not perform as well in our baseball game scenario. Opponent modeling (Zhang & Lesser, 2010; Foerster et al., 2017b) is another method in which each agent can explore the opponent’s strategy. Foerster et al. (Foerster et al., 2017b) propose a learning method, named Learning with Opponent-Learning Awareness (LOLA), to consider the learning processes of other agents. Their method has successfully enabled the cooperation of two players in repeated prisoner’s dilemma games. In this paper, we further extend the discussion to the learning issue of multiple agents in a competitive game scenario.

heading: 3 MODIFIED DEEP CFR ALGORITHM & STRATEGY ADAPTATION

heading: 3.1 MODIFIED DEEP CFR ALGORITHM
text: To study the impact of initial strategy over the strategy adaptation mechanism, we propose an algorithm based on the modification of the Deep CFR algorithm to learn the strategies of the batter and the pitcher. The Deep CFR algorithm is a state-of-the-art technique to solve the imperfect information game, especially the Poker game, by traversing the game tree and playing with the regret matching algorithm at each iteration to reach the Epsilon-equilibrium (Brown et al., 2019; Zinkevich et al., 2007). However, due to the stochastic state transitions in our baseball game scenario, it is more efficient to learn the state-value function via temporal-difference (TD) learning (Sutton, 1988) than the tree searching method originally used in Deep CFR. Besides, baseball games are inherently highly uncertain and has only the final reward (win or lose). To simplify the problem, we learn the strategies of the pitcher and the batter in a half-inning and then apply the learned strategies for the whole game.
In a half-inning game, a state s is represented by a 5-tuple vector: (runs, outs, runners on, strike, ball). The batter’s final reward Vt is defined as expressed in Equation 1, while the pitcher’s final reward is defined as −Vt. We regard the final reward as “the state-value of the terminal state Vt(s)” to avoid the possible confusion with the term “reward”, which is to be discussed later.
Vt(s) =

 
 
−20 · δ[runs] + 30 · δ[runs− 1] + 90 · δ[runs− 2] +180 · δ[runs− 3] + 300 · δ[runs− 4] + 400 · δ[runs− 5] if runs ≤ 5 400 + (runs− 5) · 30 if runs > 5
(1)
The above definition is based on the observation that the average number of runs per game in MLB is about 5 and the intuition that different number of runs in a half-inning would have different impacts on the final winning percentage.
Our algorithm alternatively estimates the payoff matrices and updates the two agents’ strategies based on the modified Deep CFR algorithm. In two-player zero-sum games, it has been proven that if both agents play according to CFR at every iteration, then these two agents’ average
strategies will gradually converge to the Nash equilibrium as the number of iterations approaches infinity(Cesa-Bianchi & Lugosi, 2006; Zinkevich et al., 2007; Waugh, 2009).
In more details, at the Iteration t, given the batter’s strategy σ′t−1b , the pitcher’s strategy σ ′t−1 p , and Vt(s), a state-value network V (s|θv) is trained from scratch by using TD learning. Based on V (s|θv), we define the trained reward as
rewardtrained = V ′ − V (s|θv) where V ′ =
{
V (s′|θv) s ′ 6= sterminal Vt(s ′) s′ = sterminal
(2)
where s′ is the next state, depending on the current state and the two agents’ joint actions (ab, ap).
Since this game only has one final reward, we manually choose a default reward rewarddefault in order to improve the training efficiency. In our approach, the reward is defined as
reward = [V ′ − V (s|θv)] · t
K1 + rewarddefault · (1 −
t
K1 ) (3)
where K1 is a manually selected constant. Based on the above definitions, an action-value network Q(s, ab, ap|θQ) is trained from scratch to estimate the expected reward of the two agents’ joint actions (ab, ap). We can express Q(s, ab, ap|θQ)) at the state s as a 5× 5 payoff matrix with respect to pitcher’s and batter’s actions. With Q(s, ab, ap|θQ), the batter’s instantaneous regrets rb can be estimated by the following equation:
rb(s, ab) = Eap∼σ′t−1p (s)[Q(s, ab, ap|θQ)]− Eap∼σ′t−1p (s),ab∼σ′t−1b (s) [Q(s, ab, ap|θQ)] (4)
The pitcher’s instantaneous regrets rp is calculated in a similar way. We assume both agents know each other’s strategies in order to improve the learning efficiency. The sharing of strategy information is feasible in real life since we can ask the pitcher and the batter in the same team to compete with each other to learn their individual strategies.
These instantaneous regrets rb(s, ab) and rp(s, ap) are converted to the new strategies σ t b, σ t p based on the following equation:
σt(s, a) = r+(s, a)
Σa′∈A(s) r+(s, a′) (5)
where A(s) denotes the actions available at the state s and r+(s, a) = max(r(s, a), 0). If Σa′∈A(s) r+(s, a ′) = 0, each action is assigned an equal probability.
Equation 5 comes from the modification of the Deep CFR algorithm. The original Deep CFR algorithm converts the agent’s accumulated regret R into the strategy by using the regret matching algorithm at each iteration (Hart & Mas-Colell, 2000; Zinkevich et al., 2007; Brown et al., 2019). Due to the heavy computational load of the learning process, we cannot afford too many iterations. Hence, we replace the accumulated regret R in the Deep CFR algorithm by the instantaneous regrets r. However, the replacement of R by r results in larger strategy variations at different iterations. To mitigate this problem, we define the new strategy based on the following equation
σ′t = σ̄t−1 · t
K2 + σt · (1−
t
K2 ) (6)
where σ̄t−1 denotes the average strategy and K2 is a manually selected constant to balance between σ̄t−1 and σt. At the next Iteration (t + 1), σ′tb , σ ′t p are used to train a new value network V (s|θv). Meanwhile, σtb, σ t p are accumulated respectively in their strategy memories Mσ as encountering the state s, weighted by t as expressed below:
Mσ(s, a)←Mσ(s, a) + σ t(s, a) · t (7)
The average strategy at Iteration t is then computed from the strategy memories, expressed as
σ̄t−1(s, a) = Mσ(s, a)
Σa∈A(s)Mσ(s, a) (8)
where A(s) denotes the actions available at the state s. Note that the state-value function V (s|θv) highly depends on the agents’ strategies, which have large variations at different iterations. Hence, V (s|θv) has to be retrained from scratch at every iteration.

heading: 3.2 STRATEGY ADAPTATION
text: At the beginning of the baseball game, each team plays against each other with its initial strategies σp and σb for the pitcher and the batter, respectively. The payoff matrix Q(s, ab, ap|θQ) at each state is then learned based on σp and σb. The payoff matrix is used for a team to estimate the expected reward of the two agents’ joint actions at any state. As the game proceeds, we propose a strategy adaptation mechanism to gradually modify the strategies. In the following paragraphs, we present the adaptation mechanism for the batter only. The adaptation mechanism for the pitcher can be deduced in an analogous way.
During the team’s batting time, σb is the batter’s initial strategy and σp is the prior assumption about the opponent pitcher’s strategy. At each pitch, the batter anticipates what the pitcher will pitch based on the past observations over the pitcher’s actions. In Equation 9, O(s) denotes all the past observations at the state s and σ̃p(s, ap) denotes the predicted probability of the pitcher’s action ap at the state s. In our mechanism, we treat σ̃p(s, ap) as the posterior probability, conditioned on the past observations and the prior belief. That is,
σ̃p(s, ap) = π(ap|s,O(s)) = Π
N(O(s)) i=1 π(oi(s)|s, ap)
π(O(s)) σp(s, ap) (9)
In Equation 9, σp(s, ap) denotes the prior knowledge about the pitcher’s action at the state s. Π N(O(s)) i=1 π(oi(s)|s, ap) denotes the likelihood function based on all the past observations at this state s for the pitcher’s action ap. Here, N(O(s)) denotes the number of observations at the state s.
In addition to properly anticipate the pitcher’s action, the batter should also know the expected reward of each action under different situations, which are expressed by Q(s, ab, ap|θQ). The batter can then obtain the advantageous strategy σ̃b by calculating the instantaneous regret in 4 based on Q(s, ab, ap|θQ), σ̃p(s, a), and σb(s, a). After that, he can update his strategy σ̃b based on 5. To gradually modify the strategy, the batter also takes into account the initial strategy σb(s, a). The adapted strategy at the state s is then defined as
σ∗b (s, ab) = σb(s, ab) · (1− N(O(s))
K3 ) + σ̃b(s, ab) ·
N(O(s))
K3 (10)
where K3 is a manually selected constant. Equation 10 indicates that σ ∗ b (s, ab) depends more on σ̃b as the number of observations N(O(s)) at this state s increases. This is similar to the batter’s behavior in real-life baseball games.
As the batter’s strategy is adaptively modified as described above, the strategy adaptation of the opponent pitcher is performed based on the batter’s reaction in an analogous way. As the number of observations N(O(s)) reaches a pre-selected threshold Npitch, both σb(s, ab) and σp(s, ap) are updated based on the following equations:
σp(s, ap)← σp(s, ap) · (1− η) + σ̃p(s, ap) · η (11)
σb(s, ab)← σb(s, ab) · (1− η) + σ̃b(s, ab) · η (12)
where η is a manually determined learning rate. After the update, all the past observations are reset and the same strategy adaptation process starts again.
In our simulation, each team has its own strategy pair (σp, σb) for its pitcher and its batter. For each team, either the pitcher’s strategy or the batter’s strategy is adaptively modified depending on whether the team is under pitching or batting. In our baseball game scenario, each team plays bestof-three games with every other team. That is, each team has at most three games to observe the opponent team.

heading: 4 EXPERIMENTAL RESULTS

heading: 4.1 THE STRATEGY LEARNED FROM THE MODIFIED DEEP CFR ALGORITHM
text: We train the pitcher and batter in a half-inning game by the modified Deep CFR algorithm, then we apply the learned strategy to the whole(nine-inning) game. At each iteration of the training process, the trained team plays the best-of-three game against every team of the Team-0 to Team-12 (listed on Table 3) for 200 series. The averaged winning percentage continuously increases as the iteration proceeds, as shown in Figure 3. This implies that the learned strategy is getting closer to the Nash-equilibrium strategy if compared with the strategies of the other teams.
Table 2 lists the learned strategies for the pitcher and the batter at 24 different states based on the modified Deep CFR algorithm. Here, we attempt to observe the strategy differences with respect to different states. In Table 2, the ”count” denotes the number of balls and the number of strikes. For example, (2-1) means 2 balls and 1 strike. Each row in Table 2 represents the probability of each action with respect to a specific count for the pitcher and the batter, respectively. It is very interesting to observe that the learned strategy at some state is actually quite similar to real-life baseball strategies. For example,
Case 1: Count of two strikes When there are already two strikes, the batter has less freedom to sit on any preferred zone or specific pitch type. As shown in Table 2, for the batter, the probability of ”any” is indeed quite high for these 2-strike cases, except the very special case of 3-2 (3 balls, 2 strikes). On the other hand, for the pitcher, the probability of curveC becomes higher because the pitcher has more freedom to pitch out of the strike zone to fool the batter.
Case 2: Count of 3-1 For this case (3 balls, 1 strike), we may consider two different situations. When the number of runners is smaller than or equal to 1 (Situation 1), the probability of fastM is quite high for the pitcher (0.98) because the pitcher wants to avoid a possible walk in this situation. On the other hand, the batter would have the freedom to reduce the attack zone to make a solid contact (the probability of (fastH+any) in Situation 1 is only 0.27 for the batter). However, when the number of runners are larger than 1 (Situation 2), a batter will be more aggressive since it is a good opportunity to score (the probability of (fastH+any) in Situation 2 for the batter is 0.44). On the other hand, the pitcher needs to pitch more carefully to prevent solid contact. (the probability of fastM (0.76) is lower than the probability (0.98) in Situation 1).

heading: 4.2 STRATEGY ADAPTATION
text: In our simulation of competitive games, we form 14 teams, including 13 manually determined teams and 1 team trained by the modified Deep CFR algorithm. Each team has its own initial strategy and the payoff matrix. Each team plays the best-of-three games against every other team for 200 series. In each series, each team has its own initial strategy at the beginning, followed by the adaptation mechanism mentioned in Section 3.2 to update its strategy based on the observations of its opponent.
In the left three columns of Table 3, we list the 14 teams, together with the characteristics of the pitcher and the batter in each team. For Team-0 to Team-3, an active batter is more aggressive to swing, while a passive batter tends to wait or to choose the action fastM more often. On the other hand, an active pitcher tends to pitch to the middle, while a passive pitcher tends to pitch to the corner. For Team-4 to Team-7, both the pitcher and the batter keep choosing the same action. For example, the batter of Team-4 always chooses the action fastH and the pitcher always chooses the action fastS. Team-8 to Team-11 are intentionally formed to defeat Team-4 to Team-8. For example, the strategy of Team-8 is especially designed to defeat Team-6, while the strategy of Team-9 is designed to defeat Team-4. The strategies of Team 0 to Team 3 & Team 8 to Team 11 are listed in B.
In the two columns on the right of Table 3, we list the averaged winning percentage (WP) of each team with respect to the other teams. For the teams with ”specific tendency” (Team-4 to Team11), some of them have higher WP, such as Team 5 and Team 9 in the without-adaptation domain. This indicates some strategies, like always pitch to the corner, can be very effective in winning the games. However, as the strategy adaptation mechanism is employed, the WP of most teams with ”specific tendency” actually decreases. This is quite reasonable since the team with ”specific tendency” strategy will restrict themselves to properly modify their strategies against various kinds of opponents and can be easily exploited by their opponents. On the other hand, it seems the the strategy learned from the modified Deep CFR algorithm for Team-13 can serve as a good initial strategy if we want to adopt the strategy adaptation mechanism.
In Table 4, we compare the winning percentage among the first four teams (Team-0 to Team-3) in the without-adaptation domain. The simulation results are very similar to real-life baseball games: (1)
when the batter is active, a passive pitcher has a better chance to win than an active pitcher (Team-1 vs Team-0); (2) when the batter is passive, an active pitcher has a better chance to win than a passive pitcher (Team-2 vs Team-3); (3) when the pitcher is active, an active batter has a better chance to win than a passive batter (Team-0 vs Team-2); and (4) when the pitcher is passive, a passive batter has a better chance to win than an active batter (Team-3 vs Team-1).
Table 4 shows that none of these four teams can dominate the games. Team-1 beats Team-0, Team-0 beats Team-2, Team-2 beats Team-3, while Team-3 beats Team-1. This implies the importance of strategy adaptation. Besides, in Table 4, we also show that all these four teams have better chances to beat Team-12, which adopts a random initial strategy. This implies that a team with a strategy is better than a team without any strategy.
When two teams, if named as home team and guest team, compete with each other, there are four possible combinations in strategy adaptation: (1) only the guest team adapts, (2) both teams do not adapt, (3) both teams adapt, and (4) only the home team adapts. In Table 5, we list the competition results of these four different combinations when we choose Team-5 or Team-13 as the home team and choose each of the remaining teams as the guest team. Here, Team-5 represents those teams with a ”specific tendency” strategy, while Team-13 represents the team with an initial strategy learned from the modified Deep CFR algorithm. Besides, in Table 5, we classify Team-4 to Team-11 as Category 1, whose initial strategy has ”specific tendency”, while classify the other teams as Category 2. On the right three columns of Table 5, ”avg” represents the averaged WP value with respect to all the guest teams, ”avg-1” represents the averaged WP with respect to Category-1 teams, and ”avg-2” represents the averaged WP value with respect to Category-2 teams.
In Table 5, we observe some interesting phenomena:
1. In terms of ”avg”, Team-5 has the best performance when both teams do not adapt, while has the worst performance when both teams adapt. 2. In terms of ”avg1”, the WP of Team-5 is roughly 50%, basically independent of the adaptation
mechanism. 3. In terms of ”avg2”, the WP of Team-5 drops drastically from ”No Adapt” to ”Both Adapt”. 4. For Team-13, the WP increases if the strategy adaptation mechanism is adopted (see ”Both Adapt” and ”Home Adapt”). 5. With strategy adaptation, the WP of Team-13 against any other team is higher than 50 %. This implies, in average, Team-13 always wins when strategy adaptation is used.
Based on the above observations, we have the following conclusions:

heading: 1. Those teams with a ”specific tendency” strategy might have high winning percentage against some of the other teams when there is no strategy adaptation. However, as the adaptation mechanism is adopted, those teams’ advantage tends to drop.

heading: 2. The team with an initial strategy learned from the modified Deep CFR algorithm benefits from the strategy adaptation mechanism.

heading: 5 CONCLUSION
text: In this paper, we construct a simplified baseball game scenario to develop and evaluate the adaptation capability of learning agents. We are especially interested in what kinds of teams have a better chance to survive when there is strategy adaptation. We propose a modified Deep CFR algorithm to learn an initial strategies of the batter and pitcher. The experimental results indeed show that the team with an initial strategy learned from the modified Deep CFR algorithm is more favorable than a team with a deterministic initial strategy in our baseball game scenario.
In this work, since we only focus on the impact of strategies on Winning Percentage, the capabilities of the pitcher and the batter are fixed across different teams. In the future work, we would relax this constraint and both teams are required to anticipate the ” capabilities and strategies ” of the opponents during the game process. This setup will make the game scenario more realistic, and the behavior of the agents would be more similar to the behavior of human players in real life.

heading: A PSEUDO CODE
text: A.1 MODIFIED DEEP CFR
Algorithm 1 Modified Deep CFR
1: function LEARNSTRATEGY 2: Initialize strategies σ0b , σ 0 p, σ ′0 b , σ ′0 p with random strategies. 3: Initialize Strategy memories Mσb ,Mσp . 4: Choose final reward Vt(s) and default reward rewarddefault . Select constant K1 and K2. 5: for Iteration t=1 to T do 6: V (s|θV ),Mσb ,Mσp = LEARNVALUE(σ t−1 b , σ t−1 p , σ ′t−1 b , σ ′t−1 p ,Mσb ,Mσp ) 7: Q(s, ab, ap|θQ) = LEARNQ(V (s|θV ) 8: Calculate both players’ instantaneous regrets rb, rp respectively with equation 4. 9: Convert the both players’ instantaneous regrets rb, rp to the new strategies σ t b, σ t p respec-
tively with equation 5.
10: Calculate the average strategy σ̄t−1b , σ̄ t−1 p from Strategy memories Mσb ,Mσp respec-
tively with equation 8.
11: Calculate the new strategies σ′tb , σ ′t p with equation 6. 12: end for 13: return σ̄Tb , σ̄ T p 14: end function
Algorithm 2 Learning Value function
1: function LEARNVALUE(σb, σp, σ ′ b, σ ′ p,Mσb ,Mσp ) 2: Initialize V (s|θV ), Replay buffer D. 3: for episode t=1 to K do 4: Initialize s. 5: repeat (for each state of episode) 6: Accumulate σb(s), σp(s) respectively in their strategy memories, weighted by t with
equation 7.
7: Select actions according to σ′b, σ ′ p from s and observe the new state s ′. 8: Store transition (s, s′) Into D. 9: Sample a random minibatch of N transitions (si, s ′
i) from D. Update the value function V (s|θV ) by minimizing the loss:
L = 1 N Σi[(V (si|θV )− V ′)2] where V ′ =
{
V (s′i|θv) s ′ i 6= sterminal Vt(s ′ i) s ′
i = sterminal

heading: 10: s← s′
text: 11: until s is terminal 12: end for 13: return V (s|θV ),Mσb ,Mσp 14: end function
In our experiment, The hyperparameters are chosen in the following: K1 = 30,K2 = 30,K3 = 20, Npitch = 4, η = 0.1.
Algorithm 3 Learning Q function
1: function LEARNQ(V (s|θV )) 2: Initialize Q(s, ab, ap|θQ), Replay buffer D. 3: for episode t=1 to K do 4: Randomly initialize s. 5: repeat (for each state of episode) 6: Randomly select actions (ab, ap) from s and observe the new state s
′ and reward according to equation 3.
7: Store transition (s, ab, ap, reward, s ′) into D. 8: Sample a random minibatch of N transitions (si, ab,i, ap,i, rewardi, s ′
i) from D. Update the Q function Q(s, ab, ap|θQ) by minimizing the loss: L = 1
N Σi[(Q(si, ab,i, ap,i|θQ)− rewardi) 2] 9: s← s′
10: until s is terminal 11: end for 12: return Q(s, ab, ap|θQ) 13: end function
A.2 ADAPTATION MECHANISM
Algorithm 4 Adaptation mechanism for the batter during a game
1: procedure A GAME 2: Set Npitch, η,K3. 3: Initialize the observation memory O. 4: Prepare σb(The batter’s initial strategy), σp(the prior assumption about the opponent pitcher’s strategy), and payoff matrix Q(s, ab, ap|θQ). 5: repeat (for each state of episode) 6: if N(O(s)) ≥ 1 then // N(O(s)) denotes the number of observations at the state s. 7: Anticipate the pitcher’s strategy σ̃p(s, a) based on the past observations in O(s)
according to equation 9.
8: Calculate the regret rb(s, ab) by 4 with Q(s, ab, ap|θQ), σ̃p(s, a), σb(s, a). 9: Convert rb(s, ab) to the strategy σ̃b(s, a) with equation 5.
10: end if 11: Calculate σ∗b (s, ab) according to equation 10. 12: We select action according to σ∗b (s, ab), and the opponent pitcher select his action. 13: The game proceeds to the new state s′. 14: Restore the observed pitch into O(s). 15: N(O(s))+ = 1. 16: if N(O(s)) == Npitch then 17: update σp(s, ap) by equation 11 18: update σb(s, ab) by equation 12 19: N(O(s)) = 0 20: Reset O(s). 21: end if 22: s← s′ 23: until s is terminal 24: end procedure

heading: B THE STRATEGIES OF TEAM 0 TO TEAM 3, AND TEAM 8 TO TEAM 11
text: During a game, these team’s strategies only depend on ”count”.

heading: C POSSIBLE BATTING RESULTS
text: We collect every piece of data of batters vs. pitchers from MLB data on Stacast Search in 2016. Then we classify all the possible batting results into 32 categories, as listed in Table 14, according to the batted ball information of “Event”, “Location”, “Type”, “Distance”, and “Exit Velocity” from collected data.
Batted Ball Type describes a batted ball, depending on how it comes off the bat and where in the field it lands. There are generally four descriptive categories: fly ball, pop-up, line drive, ground ball. Line drive is that a batted ball hit hard enough and low enough that it appears to travel in a relatively straight line. Pop-up is a specific type of fly ball that goes very high while not traveling very far laterally.

heading: D SIMPLE BASEBALL RULE
text: In a half-inning, the batting team scores a “run” if a player legally runs around the bases and touches home plate. On the other hand, the fielding(defense) team tries to prevent runs by getting batters or runners “out”. As soon as the fielding(defense) team gets three outs, the half-inning ends. There are usually nine innings in a baseball game, and two opposing teams take turn batting and fielding (defense). The team with the larger numbers of runs by the end of the game is the winner. The game proceeds when the pitcher in the fielding(defense) team throws the ball towards home plate and the batter decide whether to swing. A “strike” results when a batter swings and misses a pitch, does not swing at a pitch in the strike zone or hits a foul ball that is not caught. A “ball” results when a pitch misses the strike zone and the batter do not swing. A batter is out when getting three strikes, while is allowed to “walk” to the first base when getting four balls. There are lots of outcomes when the batter hit a ball into the field. In general, the batter aims to make solid contact to the pitch, while the pitcher aims to prevent the solid contact.

Review summaries:
The paper studies adaptation of agent policies in a simplified baseball game, which is designed as a zero-sum two-agent game between a batter (B) and a pitcher (P), each of which has 5 discreet actions. The introduced game is fully observable but stochastic, which the authors argue is a challenging setup. The authors propose a Bayesian-style adaptation of the agent strategies (where each agent models the probability of the actions of the opponent by computing the posterior give a prior and evidence from the past observations), which seems to be computable analytically, from an initialization learned with counterfactual regret minimization (CFR) that approximates the Nash of the considered game.