Continuous Adaptation in Multi-agent Competitive Environments
Authors: Kuei-Tso Lee
Sheng-Jyh Wang
Venue:  (2020)
Keywords: multi-agent environment
continuous adaptation
Nash equilibrium
deep counterfactual regret minimization
reinforcement learning
stochastic game
baseball

Abstract: In a multi-agent competitive environment, we would expect an agent who can quickly adapt to environmental changes may have a higher probability to survive and beat other agents. In this paper, to discuss whether the adaptation capability can help a learning agent to improve its competitiveness in a multi-agent environment, we construct a simplified baseball game scenario to develop and evaluate the adaptation capability of learning agents. Our baseball game scenario is modeled as a two-player zero-sum stochastic game with only the final reward. We purpose a modified Deep CFR algorithm to learn a strategy that approximates the Nash equilibrium strategy. We also form several teams, with different teams adopting different playing strategies, trying to analyze (1) whether an adaptation mechanism can help in increasing the winning percentage and (2) what kind of initial strategies can help a team to get a higher winning percentage. The experimental results show that the learned Nash-equilibrium strategy is very similar to real-life baseball game strategy. Besides, with the proposed strategy adaptation mechanism, the winning percentage can be increased for the team with a Nash-equilibrium initial strategy. Nevertheless, based on the same adaptation mechanism, those teams with deterministic initial strategies actually become less competitive.

Under review as a conference paper at ICLR 2020
CONTINUOUS ADAPTATION IN MULTI-AGENT COMPETITIVE ENVIRONMENTS
Anonymous authors Paper under double-blind review
ABSTRACT
In multi-agent competitive environment, we would expect agent who quickly adapt to environmental changes may have a higher probability to survive and beat other agents. In this paper, to discuss whether the adaptation capability can help a learning agent to improve its competitiveness in a multi-agent environ- ment, construct simplified baseball game scenario to develop and evaluate the adaptation capability of learning agents. Our baseball game scenario is mod- eled as a two-player zero-sum stochastic game with only the final reward. We propose a modified Deep CFR algorithm to learn the strategies of agents in a half-inning game. We also propose strategy adaptation mechanism that contin- uously updates strategies based on the anticipation of the opponent's 's strategy in the inference time. We form several teams, with different teams adopting the same adaptation mechanism but different initial strategies, trying analyze (1) whether an adaptation mechanism can help in increasing the winning percentage and (2) what kind of initial strategies can help team to get a higher winning percentage. The experimental results show that the winning percentage can be increased for the team with an initial strategy learned from the modified Deep CFR algorithm. Nevertheless, those teams with deterministic initial strategies actually become less competitive.
INTRODUCTION
Reinforcement learning has been successfully employed solve various kinds of decision making problems, such as game playing (Silver et al. 2017: 2018), robotics (Levine et all 2016), and oper- ation management (Han et al. 2017). An RL method typically finds the optimal strategy through the interactions with stationary environment, which is usually modeled as an MDP process. Neverthe- less, in the real world, there could be multiple learning agents in the same scenario. The interactions among these learning agents may make the environment no longer stationary from the standpoint of each individual agent. Besides, these agents are competitive relationship, we would expect an agent who quickly adapt to environmental changes may have a higher probability other agents. To discuss whether the adaptation capability can help learning agent to improve its competitiveness in multi-agent environment, choose simplified baseball game as the scenario to develop and evaluate the adaptation capability of learning agents.
beat
A lot of games, like Chess (Silver et all 2018h, Go (Silver et al. 2017) and Atari games (Bellemare et all 2013 Mnih et al. 2015; Wane et al. (016), only try to find the optimal action each state. comparison, some complicated games, like baseball and basketball games, need to take into account not only the current status but also the opponent's 's possible strategies. Morcover, the opponent's s strategy is typically do under different situations, but also need to dynamically adjust their strategies based on the obser- vations from the opponent's 's past actions. In a typical baseball game, for example, two competitive teams play against each other based on their pre-determined initial strategies at the beginning of the game. As the game proceeds, the teams continuously update their strategies based on the actions of their opponents, trying to win the game. However, baseball games inherently highly uncertain while the number of interactions between the pitcher and the batters is rather limited. It is very tricky for the teams properly adjust their strategies based on small number of interactions.
varying. Hence, players should not only determine what to

---

Under review as a conference paper at ICLR 2020
In this work, to discuss the adaptation issue in a multi-agent competitive environment, we inten- tionally construct a simplified baseball game scenario based on the MLB data on Statcast Search (MLB). We make the following assumptions simplify the problem. (The simple baseball rule is shown on Appendix(D] D
1. We only focus on batting and pitching in our game scenario and treat fielding the environment which is based on the MLB data on Statcast Search.
We assume there are only one pitcher & batter in each team, rather than the typical 9-batter case. 3. We assume the pitcher and the batter have the same performance in pitching and batting across different teams. The only difference is their strategies. Besides, they always perform normally and we do not consider the case of abnormal performance in all games.
Both the pitcher and the batter have only five possible actions, as explained the next section.
Based on the above simplifications, we manually form 13 teams, with different teams adopting different playing strategies. We also propose modified version of the Deep CFR (Counterfactual Regret Minimization) (Brown et all 2019) algorithm to learn the strategies of the batter and pitcher. In total, we form 14 teams to analyze their adaptation capabilities.
In our simulation, each of these 14 teams plays the best-of-three games against every other team for many series. At the beginning of each series, each team plays based on its initial strategies. As the game proceeds, each team follows the same strategy adaptation mechanism based the observed actions of its opponent. In our simulation, there are only three games at most for each pair of teams to adjust their strategies. We then analyze the following two main issues about strategy adaptation.
1. With a small number of observations (three games at most for each pair of teams), can the adaptation mechanism help in increasing the winning percentage?
2. If two competitive teams adopt the same adaptation mechanism during the game, what kind of initial strategies can help team to get higher winning percentage?
2 BACKGROUNDS
2.1 SCENARIO DESCRIPTION
We first explain how we define a simplified baseball game for the analysis of strategy adaptation in multi-agent scenario. Even though all the following discussions are based on the baseball game scenario, the deduced conclusions can be useful for similar multi-agent problems as well.
In every play of the baseball game, the pitcher aims at expected target location and selects pitch type. On the other hand, the batter looks for specific types of pitches appearing in some preferred attack zones. Both the pitcher and the batter select their actions for each pitch and, depending on the result of their actions, the game proceeds to the next state. Specifically, we treat our baseball game scenario two-player zero-sum stochastic game (multi-agent MDP) (Bowling 2000) with only the final reward (win or lose).
We first mention two examples explain players' different strategies under different situations. Situation 1: O out, bases empty, ball, strike Under this situation, the batter tends to be more selective or may wait for walk. On the other side, the pitcher tends to pitch a fastball to the middle of the strike zone to avoid a walk. However, if the pitcher has pitched the middle too often, the batter may look for pitches in this zone to hit. This will increase the probability of solid contact. Situation 2: out, bases full, 3 ball, 1 strike
Under this situation, the batter tends to be more aggressive since it is good chance to score. On the other hand, the pitcher might take advantage the batter's 's aggressiveness and pitch to the corner to fool the batter. However, if the pitcher has pitched to the corner too often, the batter may shrink his attack zone to avoid being fooled.
In Figure D illustrate the flow of our baseball game. The pitcher and the batter are the two learning agents to be trained to play against each other. Here, the Pitching Control represents the pitcher's s ability in ball control. As the pitcher aims to pitch at target location, model his Pitching Control as a Gaussian distribution centered at the target location with a manually pre-determined variance. The actual pitched location is then sampled from this distribution.
2

---

Under review as a conference paper at ICLR 2020
A Agents
Environment
Pitcher
action( (P)
Pitching
Type, Zone
Control
state:
Batter
action((B
Batting Result Distribution
Fielding
< Batting Result
Next state
Distribution
F state
Figure 1: Flow of baseball game scenario.
On the other hand, the Batting Result Distribution (iresults]zones type, as) models the prob- ability of the possible batting results. Here, we classify all the possible batting results into 32 categories, listed in Appendix [ with respect to different zones (represented by different num- bers in Figure 2), pitch type, and the batter's action ag- To model this Batting Result Distribution, presulls|2onnes type) based the MLB data Statcast Search.
first obtain the distribution After that, we heuristically convert the distribution P(results/ zones, type) into the distribution 2(resu zomes, type, as) for every possible action dş according the athletic ability of the bat- ter. Since the average ML.B fasthall travels at 95 mph, reaching home plate in just 0.4 second, is very difficult for human beings react within such a short period. Hence, the batter typically focus on a specific zone and a specific pitch type to swing in order to increase the probability of solid con- tact. Generally, the smaller the batter sits on the hitting zone, the more chance he can make solid contact if the ball goes into the pre-selected zone. In Figure []] the fielding distribution models the probability of the next state when a batting result occurs a certain state. Again, this distribution modeled based on the MLB data on Statcast Search.
In our baseball game, we denote each state 8 by 6-tuple vector: (inning, runs, outs, runners on, strike, ball). To simplify the game, both the pitcher and the batter have only five possible actions, as listed in Table 1. These actions are defined based on the pitcher's Pitch Location and the batter's Attack Zone, as illustrated in Figure 2. Here, we assume the pitcher only aims three possible spots (the green dots) on pitcher's Pitch Location and only has two types of pitch, fastball or curveball. Due to the uncertainty in pitch control, the pitched ball actually spreads around these three spots. On the other hand, based on the pitch type and the location of the pitched ball, the batter has five possible actions.
Pitcher's action
Batter's action
21
22
23
Strike Zone
1. fastM
1. Wait
Pitcher Pitch Location
(fastball in middle)
(do not swing)
11
12
13
M:middle
S:shadow
2. fastS
2. fastM
M
Cichase
(fastball shadow)
(sit on fast ball in middle)
24
14
4
6
16
26
Batter Attack Zone
3. curveM
3. fastH
Mcmiddle
(curveball in middle)
(sit on fastball in heart)
Htheart
4. curveS
4. curveH
17
18
19
Heart Shadow
(curveball in shadow)
(sit on curveball in heart)
Chase Waste
5. curveC
5. Any
27
C
29
30
(curveball in chase)
(hit any pitch in strike)
28
Figure 2: Strike Zone.
Table Possible actions.
2.2
RELATED WORK
Humans are able to learn new skills quickly based on the past experience. It is necessary for tificial agents to the same. Meta-learning, also known as "learning to learn", aims to train a model on variety of learning tasks that solve new learning tasks adapt to the new vironments rapidly with minimal training samples. Meta-learning has been used to learn high-level information of model such learning optimizers for deep networks (Ravi & Larochelle 2017
3

---

Under review as a conference paper at ICLR 2020
Andrvchowiczeco et a 2016 & Malik, 2016). learning task embeddings (Vinvals et al
2016
Snell et al., 2017), and learning to learn implicitly via RL (Duan et al. 2006: Wang et al., 2016). Especially, model-agnostic meta-learning (MAML) (Finn et all 2017b) aims to find set of highly adaptable parameters that can be quickly adapted to the new task. The goal of quick adaptation to the new environments for meta-learning is similar to work, but meta-learning discusses only the case of single learning agent, instead of the two-agent case. On the other hand, Maruan et al. [Al-Shedivat et al. 2018) construct an adversarial multi-agent environment, RoboSumo, allowing the agent to continuously adapt to the changes of the opponent's 's strategies. However, their approach considers only one- one-way adaptation and the opponent is not allowed to anticipate the learning agent's ss strategy. This is different from our scenario in which both teams adapt their strategies in the game.
When it comes the multi-agent environments, multi- aggei reinforcement learning (MARL) has been widely applied in various applications, such as transportation (Fernandez- et al 2015), social sciences (Leibo al. 2017). resource management (Hussin al., 2015). and controlling a group of autonomous vehicles (Hung & Givigi. 2017). MARL has an issue of instability of the training process. To deal with the instability issue, MADDPG (Lowe et all 2017) and M3DDPG (Li et al. 2019) have been proposed which adopt centralized critic within the actor-critic learning framework to reduce the variance of policy gradient results. However, these methods have been designed for deterministic policies only. These methods do not perform well in our baseball game scenario. Opponent modeling (Zhang & Lessei, 2010: Foerster et al_ 201 is another method in which each agent explore the opponent's s strategy. Foerster et (Foerster et all 201 7b) propose a learning method, named Learning with Opponent-Leariing Awareness (LOLA), to consider the learning processes of other agents. Their method has successfully enabled the cooperation of two players in repeated prisoner's 's dilemma games. In this paper, we further extend the discussion to the learning issue of multiple agents in a competitive game scenario.
3
MODIFIED DEEP CFR ALGORITHM & STRATEGY ADAPTATION
3.1 MODIFIED DEEP CFR ALGORITHM
To study the impact of initial strategy over the strategy adaptation mechanism, we propose algo- rithm based on the modification of the Deep CFR algorithm to learn the strategies of the batter and the pitcher. The Deep CFR algorithm is fatte-of-the-an technique solve the imperfect informa- tion game, especially the Poker game, by traversing the game tree and playing with the regret match- ing algorithm each iteration reach the Epsilon-equilibrium (Brown et all. 2019: Zinkevich all. 2007). However, due to the stochastic state transitions in our baseball game scenario, it is more effi- cient to learn the state-value function via temporal-differene (TD) learning (Sutton 1988) than the tree searching method originally used in Deep CFR. Besides, baseball games are inherently highly uncertain and has only the final reward (win or lose). To simplify the problem, we learn the strate- gies of the pitcher and the batter in a half-inning and then apply the learned strategies for the whole game.
In a half-inning game, state s is represented by a 5-tuple vector: (runs, outs, runners on, strike, ball). The batter's final reward V; defined as expressed in Equation [ while the pitcher's final reward is defined as - -V,, We regard the final reward as "the state-value of the terminal state Vs())" to avoid the possible confusion with the term "reward", which is to be discussed later.
20 8[runs] +30 8|runs 1] + 90 1000000rra 8[runs - 2]
+180 0[runs 3] + 300 ö|runs 4] + 400 6fruns 5]
Vi(s)
(1)
if runs < if runs
400 (runs 5) -00
The above definition is based on the observation that the average number of runs per game in MLB is about 5 and the intuition that different number of runs in half-inning would have different impacts on the final winning percentage.
Our algorithm alternatively estimates the payoff matrices and updates the two agents' strategies based on the modified Deep CFR algorithm. In two-player zero-sum games, it has been proven that if both agents play according to CFR every iteration, then these two agents' average
4

---

Iteration is then computed from the strategy memories, expressed as Under review as a conference paper at ICLR 2020
strategies will gradually converge to the Nash equilibrium as the number of iterations approaches infinity(Cesal SeaittyCesaBBaanchh & Lugosi 2006; Zinkevich et al. 2207:; Waugh. 2009).
In more details, at the Iteration t, given the batter's strategy o'' the pitcher's strategy o01 and (s), state-value network V (s()0,) is trained from scratch by using TD learning, Based on V O,)), we define the trained reward as
V' v'={
V()10,)) 8 + Sterminal Vi(s') 8 Sterminal
rewarderained = V (s))0)) where
(2)
where s' is the next state, depending on the current state and the two agents' joint actions (ab, Op)- Since this game only has one final reward, we manually choose default reward rewarddefautt in order to improve the training efficiency. In our approach, the reward is defined as
t K
+ rewardde reward- = IV' - v(s[0.)] K1
(3)
where K1 is a manually selected constant. Based the above definitions, an action-value network ap apl00) is trained from scratch to estimate the expected reward of the two agents' joint
Q(s, az. actions (as, ap). We can express (s, abZ ap (0)) at the state s as 5 5 payoff matrix with respect to pitcher's s and batter's actions. With (ss, ab. ap)0q), the batter's 's instantaneous regrets can be estimated by the following equation:
(4) The pitcher's instantancous regrets rp is calculated a similar way. We assume both agents know each other's strategies in order to improve the learning efficiency. The sharing of strategy informa- tion feasible in real life since we can ask the pitcher and the batter in the same team to compete with each other learn their individual strategies.
[s ab) Bapna (ss) [1(s as, ep10 E Bap "()),asno'' *(x) (Q)) as, ap)
These instantaneous regrets ro a&) and (s(s, are converted to the new strategies of of based on the following equation:
(+
d's Ed'EA(s) as)
(5)
where A(s) denotes the actions available the state and r+ (s. a) max(r(s, a), 0). If
A()) r+ (s, = 0, each action assigned an equal probability.
Equation 5 comes from the modification of the Deep CFR algorithm. The original Deep CFR al- gorithm converts the agent's accumulated regret R into the strategy by using the regret matching algorithm at each iteration (Hart & Mas-Colell, 2000 Zinkevich et al. 2007 Brown et al_ 2019). Due to the heavy computational load of the learning process, we cannot afford too many iterations. Hence, we replace the accumulated regret R in the Deep CFR algorithm by the instantaneous regrets However, the replacement of It by results in larger strategy variations at different iterations. To mitigate this problem, we define the new strategy based on the following equation
K2 + +o' (1'.1111
(6)
K₂
where 5-1 denotes the average strategy and K₂ a manually selected constant to balance between g--1 and of. At the next Iteration (I (L+ 1), of' of are used to train a new value network Vs)00,). Meanwhile, of of, are accumulated respectively in their strategy memories M, encountering the state 8, weighted by as expressed below:
(s. (s, + a)
The average strategy at
Mo(s,a)
8' (s.a) Eae (s,a)
(8)
where A(s) denotes the actions available at the state s. Note that the state-value function ((s highly depends on the agents' strategies, which have large variations at different iterations. Hence, (s,0,) has to be retrained from scratch every iteration.
5

---

Under review as a conference paper at ICLR 2020
3.2 STRATEGY ADAPTATION
At the beginning of the baseball game, each team plays against each other with its initial strategies Op and σ6, for the pitcher and the batter, respectively. The payoff matrix ((s, (b; Op (p)00) at each state is then learned based on Op and ob. The payoff matrix is used for a team to estimate the expected reward of the two agents' joint actions at any state. As the game proceeds, we propose strategy adaptation mechanism to gradually modify the strategies. In the following paragraphs, we present the adaptation mechanism for the batter only. The adaptation mechanism for the pitcher can be deduced in an analogous way.
During the team's batting time, o& is the batter's initial strategy and op is the prior assumption about the opponent pitcher's strategy. At each pitch, the batter anticipates what the pitcher will pitch based on the past observations over the pitcher's actions. observations at the state 8 and (s,s, (p)) denotes the predicted probability of the pitcher's action 4p at the state s. In our mechanism, we treat (p(s, ap) as the posterior probability, conditioned on the past observations and the prior belief. That is,
Equation g| ((s) denotes all the past
āp(8, ap) T (apl8, O())) ((OO) ((0;())1s,ap) op(s,0p)
(0(())
In Equation ap(s,ap) ap) denotes the prior knowledge about the pitcher's action at the state s. I1 N(((s)) (()(()) [s, ap) denotes the likelihood function based all the past observations this state s for the pitcher's s action Gp. Here, .N (O))) denotes the number of observations at the state s.
In addition to properly anticipate the pitcher's action, the batter should also know the expected reward each action under different situations, which are expressed by ((,, as ap100). The batter then obtain the advantageous strategy 5, by calculating the instantaneous regret in [ based on Q(s, ap. 0g (0). (s a), and (s(s, a). After that, he can update his strategy ōn based on 5 To gradually modify the strategy, the batter also takes into account the initial strategy an(s, a). The
adapted strategy at the state is then defined as
(s, 0s) = os(ss ab) (1 N(O(s)) + 5b ab) N(O(s))
(10)
K3
K3
where K3 manually selected constant. Equation I0 indicates that of (s, as) depends more on 5n as the number of observations (O(s))) at this state increases. This is similar to the batter's behavior in real-life baseball games.
As the batter's s strategy is adaptively modified as described above, the strategy adaptation of the opponent pitcher performed based on the batter's reaction in an analogous way. As the number of observations N (O))) reaches pre-selected threshold Npitch, both os(s, as) and op ap) are updated based on the following equations:
(s ap) (p)))up) (1-n) 7) + (s,
(11)
0b(s, ap) (s,s.ap) ab) - m) 5o ab)
(12)
where 7 is manually determined learning rate. After the update, all the past observations are reset and the same strategy adaptation process starts again.
In our simulation, each team has its own strategy pair (ap, σb) for its pitcher and its batter. For each team, either the pitcher's strategy or the batter's strategy adaptively modified depending on whether the team is under pitching batting. In our baseball game scenario, each team plays best- of-three games with every other team. That is, each team has at most three games to observe the opponent team.

---

Under review as a conference paper at ICLR 2020
4 EXPERIMENTAL RESULTS
4.1
THE STRATEGY LEARNED FROM THE MODIFIED DEEP CFR ALGORITHM
0.6
0.5
winning percentage
0.4
0.3
0.2
0.1
13 14 15 16 17 18 20
22 24
iteration
Figure 3: The winning percentage of the learned strategies in the training process.
We train the pitcher and batter in half-inning game by the modified Deep CFR algorithm, then we apply the learned strategy to the whole(nine-imning) game. At each iteration of the training process, the trained team plays the best-of-thre three game against every team of the Team-O to Team- 12 (listed on Table 3) for 200 series. The averaged winning percentage continuously increases as the iteration proceeds, as shown Figure This implies that the learned strategy is getting closer to the Nash-equilibrium strategy if compared with the strategies of the other teams.
Table 2 lists the learned strategies for the pitcher and the batter 24 different states based on the modified Deep CFR algorithm. Here, we attempt to observe the strategy differences with respect to different states. In Table 2 the "count" denotes the number of balls and the number of strikes. For example, (2-1) means 2 balls and I strike. Each row in Table 2 represents the probability of very
each action with respect to a specific count for the pitcher and the batter, respectively. interesting observe that the learned strategy some state is actually quite similar to real-life baseball strategies. For example,
Case 1:
Count of two strikes
When there are already two strikes, the batter has less freedom to sit on any preferred zone or specific pitch type. As shown in Table for the batter, the probability of ""an" is indeed quite high for these 2-strike cases, except the very special case of 3-2 (3 balls, 2 strikes). On the other hand, for the pitcher, the probability of curveC becomes higher because the pitcher has more freedom to pitch of the strike zone to fool the batter.
Case 2: Count of 3-1
For this case (3 balls, strike), we may consider two different situations. When the number of runners smaller than or equal to (Situation 1), the probability of fastM is quite high for the pitcher (0.98) because the pitcher wants to avoid possible walk in this situation. On the other hand, the batter would have the freedom to reduce the attack zone to make a solid contact (the probability of (fastH+any) in Situation 1 is only 0.27 for the batter). However, when the number of runners are larger than (Situation 2), a batter will be more aggressive since is a good opportunity to score (the probability of (fastH+any) in Situation 2 for the batter is 0.44). On the other hand, the pitcher needs to pitch more carefully to prevent solid contact. (the probability of fastM (0.76) is lower than the probability (0.98) in Situation 1).

---

Under review as a conference paper at ICLR 2020
Table 2: Strategies of the batter and the pitcher.
Situation:: number of runners Situation2: number of runners > 1.
Batter's strategy
Pitcher's strategy
Count
walt
fastM
fastH
curveH
any
Count
fastM
fastS
curveM curveS
curveC
0-0
0.25
0.28
0.47
0.00
0.00
0-0
0.27
0.25
0.33
0.12
0.04
1-0
0.18
0.21
0.52
0.09
0.00
1-0
0.23
0.27
0.32
0.16
0.02
0-1
0.20
0.27
0.49
0.04
0.00
0-1
0.31
0.27
0.32
0.08
0.03
1-1
0.21
0.28
0.47
0.04
0.00
1-1
0.28
0.33
0.28
0.09
0.02
2-1
0.29
0.24
0.43
0.04
0.00
2-1
0.19
0.27
0.34
0.18
0.02
Situation1
3-1
0.28
0.19
0.20
0.26
0.07
3-1
0.98
0.00
0.02
0.00
0.00
2-0
0.19
0.16
0.49
0.17
0.00
2-0
0.28
0.27
0.24
0.18
0.03
3-0
0.30
0.24
0.18
0.23
0.05
3-0
0.98
0.00
0.01
0.00
0.00
0-2
0.00
0.00
0.08
0.03
0.89
0-2
0.30
0.06
0.34
0.19
0.11
1-2
0.00
0.02
0.18
0.02
0.78
1-2
0.36
0.06
0.33
0.18
0.07
2-2
0.00
0.00
0.15
0.03
0.82
2-2
0.33
0.05
0.30
0.22
0.10
3-2
0.04
0.04
0.54
0.016
0.32
3-2
0.68
0.12
0.16
0.05
0.00
Count
wait
fastM
fasti1
curvell
any
Count
fastM
fastS
curveM
curve8
curvel
0-0
0.15
0.16
0.49
0.10
0.11
0-0
0.17
0.25
0.32
0.19
0.07
1-0
0.14
0.13
0.43
0.17
0.13
1-0
0.11
0.24
0.30
0.22
0.07
0-1
0.10
0.11
0.60
0.10
0.10
0-1
0.13
0.22
0..40
0.22
0.03
1-1
0.14
0.14
0.48
01166
0.08
1-1
0.13
0.30
0.29
0.23
0.05
2-1
0.13
0.12
0.46
0.18
0.12
2-1
0.17
0.19
0.30
0.28
0.06
Situation2
3-1
0.14
0.29
0.34
0.14
0.10
3-1
0.76
0.14
0.06
0.04
0.01
2-0
0.12
0.10
0.46
0.18
0.13
2-0
0.13
0.21
0.31
0.26
0.08
3-0
0.17
0.41
0.24
0.11
0.07
3-0
0.71
0.16
0.08
0.04
0.01
0-2
0.00
0.00
0.04
0.02
0.95
0-2
0.24
0.05
0.31
0.22
0.19
1-2
0.00
0.01
0.06
0.07
0.87
1-2
0.21
0.04
0.30
0.23
0.23
2-2
0.00
0.01
0.11
0.10
0.78
2-2
0.29
0.09
0.29
0.15
0.18
3-2
0.01
0.04
0.55
0.11
0.28
3-2
0.40
0.20
0.19
0.17
0.05
4.2 STRATEGY ADAPTATION
In our simulation of competitive games, we form 14 teams, including 13 manually determined teams and team trained by the modified Deep CFR algorithm. Each team has its own initial strategy and the payoff matrix. Each team plays the best-of-three games against every other team for 200 series. In each series, each team has its own initial strategy at the beginning, followed by the adaptation mechanism mentioned in Section 3.2 to update its strategy based on the observations its opponent.
In the left three columns of Table 3 we list the 14 teams, together with the characteristics of the pitcher and the batter in each team. For Team-0 to Team-3, an active batter is more aggressive to swing, while a passive batter tends to wait or to choose the action fastM more often. On the other hand, active pitcher tends to pitch to the middle, while passive pitcher tends to pitch to the corner. For Team-4 to Team-7, both the pitcher and the batter keep choosing the same action. For example, the batter of Team-4 always chooses the action fastH and the pitcher always chooses the action fastS. Team-8 to Team-11 are intentionally formed to defeat Team-4 to Team-8. For example, the strategy of Team-8 is especially designed to defeat Team-6, while the strategy of Team-9 is designed to defeat Team-4. The strategies of Team 0 to Team 3 & Team 8 to Team 11 are listed in
In the two columns on the right of Table 3 we list the averaged winning percentage (WP) of each team with respect to the other teams. For the teams with "specific tendency" (Team-4 to Team- 11), some of them have higher WP, such as Team and Team 9 in the without- adapout-ddaptation domain. This indicates some strategies, like always pitch to the corner, can be very effective in winning the games. However, as the strategy adaptation mechanism is employed, the WP of most teams with "specific tendency" actually decreases. This is quite reasonable since the team with "specific tendency" strategy will restrict themselves to properly modify their strategies against various kinds of opponents and can be easily exploited by their opponents. On the other hand, it seems the the strategy learned from the modified Deep CFR algorithm for Team- 13 can serve as a good initial strategy if we want to adopt the strategy adaptation mechanism.
In 4 we compare the winning percentage among the first four teams (Team-0 to Team-3) in the addhout-adaptation domain. The simulation results are very similar real-life baseball games: (1)
8

---

Under review as a conference paper at ICLR 2020
Table Average Winning Percentage (WP) for each team in the without-addaptation domain (non) and the with-adaptation domain (adap.).
The batter
The pitcher
WP(non)
WP((adp.))
Team
Active
Active
0.55
0.61
Team
Active
Passive
0.45
0.49
Team
Passive
Active
0.52
0.50
Team
Passive
Passive
0.41
0.55
Team
fastH
fastS
0.57
0.52
Team
fastH
curveM
0.60
0.41
Team
any
fastS
0.43
0.53
Team
any
curveM
0.49
0.30
Team
Exploit fastS
Exploit any
0.59
0.46
Team
Exploit fastS
Exploit fastil B
0.64
0.41
Team
Exploit curveM P
Exploit any
0.48
0.52
Team
Exploit curveM P
Exploit fastll
0.53
0.45
Team
random
random
0.20
0.48
Team
Trained with modified Deep CFR
0.54
0.71
Table Winning Percentage (WP) among Team-O Team-3. The number stands for the WP the corresponding team listed in the left column.
Team
0.47
0.67
Team 0.53
0.50
0.48
0.73
Team 0.33
0.50
0.66
0,90
Team
0.39
0.52
0.34
0.66
when the batter is active, a passive pitcher has a better chance to win than an active pitcher (Team-l vs Team-0); (2) when the batter is passive, active pitcher has better chance to win than passive pitcher (Team-2 vs Team-3); (3) when the pitcher active, an active batter has a better chance to win than passive batter (Team-O vs Team-2); and (4) when the pitcher is passive, passive batter has a better chance to win than an active batter (Team-3 vs Team-l).
shows that none of these four teams can dominate the games. Team-1 beats Team-O, Team-0 beats Team-2, Team-2 beats Team-3, while Team-3 beats Team-1. This implies the importance of strategy adaptation. Besides, in Table 4 we also show that all these four teams have better chances to beat Team-12, which adopts a random initial strategy. This implies that team with strategy is better than team without any strategy.
When two teams, if named as home team and guest team, compete with each other, there are four possible combinations in strategy adaptation: (1) only the guest team adapts, (2) both teams do not adapt, (3) both teams adapt, and (4) only the home team adapts. In Table 3] we list the competition results of these four different combinations when we choose Team-5 or Team-13 as the home team and choose each of the remaining teams as the guest team. Here, Team-5 represents those teams with a "specific tendency" strategy, while Team-13 represents the team with an initial strategy learned from the modified Deep CFR algorithm. Besides, in Table 5 we classify Team-4 to Team- as Category 1, whose initial strategy has "specific tendency"' while classify the other teams as Category 2. On the right three columns of T5 "avg" represents the averaged WP value with respect to all the guest teams, "avg-I" represents the averaged WP with respect to Category- teams, and "avg-2" represents the averaged WP value with respect Category-2 teams.
In Table we observe some interesting phenomena:
1. In terms of "avg", Team-5 has the best performance when both teams do not adapt, while has the worst performance when both teams adapt. 2. In terms of "avgl", the WP of Team-5 is roughly 50%, basically independent of the adaptation

---

Under review as a conference paper at ICLR 2020
mechanism.
In terms of aav2"" the WP of Team-5 drops drastically from "No Adapt" to "Both Adapt".
4. For Team-13, the WP increases if the strategy adaptation mechanism is adopted (see "Both Adapt" and "Home Adapt").
5. With strategy adaptation, the WP of Team-13 against any other team is higher than 50 %. This implies, in average, Team-13 always wins when strategy adaptation is used.
Based on the above observations, we have the following conclusions:
1. Those teams with a "specific tendency" strategy might have high winning percentage against some of the other teams when there is strategy adaptation. However, the adaptation mechanism is adopted, those teams' advantage tends to drop.
The team with an initial strategy learned from the modified Deep CFR algorithm benefits from the strategy adaptation mechanism.
Table 5: WP of Team-5 and Team-13 against the other teams for 2000 series.
WP Team against the other teams.
domain
3
11
12
8vg
avg2
GRest Adapt
0.69
0.56
0.76
0.57
0.78
0.39
0.93
0.27
0.70
0.08
0.77
0.30
0.84
0.35
0.57
0.53
0.63
No Adapt
0.75
11.77
1.66
10.69
10.98
0.49
0.80
0.10
0.83
0.38
0.34
0.05
0.93
0.70
0.61
0.550
0.75
Both Adapt
0.27
0.65
0.24
0.37
0.72
0.50
0.43
0.41
0.43
0.78
0.13
0.27
0.28
0.29
0.41
0.46
0.35
Adapt
WP of Team against the other teams.
domain
2
10
8:g
a121
avg2
Guest Adapt
0.37
0.43
0.63
0.53
0.43
0.63
0.61
0.76
0.62
0.50
0.61
0.69
0.77
0.31
0.56
0.61
0.51
No Adapt
0.46
0.57
0.57
0.67
0.47
0.31
0.65
0.46
0.44
0.32
0.67
0.57
0.90
0.50
0.54
0.49
0.61
Beth Adapt
0.62
0.58
0.77
0.66
0.71
0.72
0.50
0.84
0.75
0.58
0.82
0.76
0.88
0.49
0.71
0.75
0.67
Home Adapt
CONCLUSION
In this paper, we construct a simplified baseball game scenario to develop and evaluate the adaptation capability of learning agents. We especially interested in what kinds of teams have better chance to survive when there is strategy adaptation. We propose a modified Deep CFR algorithm to learn initial strategies of the batter and pitcher. The experimental results indeed show that the team with an initial strategy learned from the modified Deep CFR algorithm is more favorable than team with deterministic initial strategy in our baseball game scenario.
In this work, since we only focus on the impact of strategies on Winning Percentage, the capabilities of the pitcher and the batter fixed across different teams. In the future work, we would relax this constraint and both teams are required to anticipate the capabilities and strategies " of the opponents during the game process. This setup will make the game scenario more realistic, and the behavior of the agents would be more similar to the behavior of human players in real life.
REFERENCES
Maruan Al-Shedivat, Trapit Bansal, Yuri Burda, Ilya Sutskever, Igor Mordatch, and Pieter Abbeel. Continuous adaptation via meta-learning in nonstationary and competitive environments. Pro- ceedings of International Conference on Learning Reprexentations, 2018.
Marcin Andrychowicz, Denil Misha, Gomez Sergio, W Hoffman Matthew, Pfau David, and Schaul Tom. Learning to learn by gradient descent by gradient descent. In Advances in Neural Informa- fion Processing Systems, 2016.
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: an evaluation platform for general agents. Journal of Artifcial Intelligence Research, 2013.
10

---

Under review as a conference paper at ICLR 2020
M. Bowling, M.and Veloso. An analysis of stochastic game theory for multiagent reinforcement learning. Comput. Sci., Carnegie Mellon Univ., Pittsurgh,, PA, Tech. Rep., 2000.
Noam Brown, Adam Lerer, Sam Gross, and Tuomas Sandholm. Deep counterfactual regret. In Proceedings of the 36 th International Conference on Machine Learning. 2019.
N. BesaaBBancchi and G Lugosi. Prediction, learning, and games. Cambridge University Press, 2006.
Yan Duan, John Schulman, Xi Chen, L Bartlett Peter, and llya. Fast reinforcement learning via slow reinforcement learning. arXiv preprint, (arXiv:161 1tfi:1661..000 1..007991 2016.
B. Fernandez- Gauna, I. Etxeberria- Exxcberria-ggiriano, and M. Grama. Learning multirobot hose transportation and deployment by distributed round-robin q-learning PloS One, 2015.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta- leatallenning for fast adaptation of deep networksg. arXiv preprint, farXiv:1703.0340 2017b
Jakob N Foerster, Richard Y Chen, Maruan Al-Shedivat, Shimon Whiteson, and Pieter Abbeel. Learning with opponent-learning awareness. arXiv preprint, (arXiv:1 700.04436), 2017b.
Cai Han, Ren Kan, Zhang Weinan, and KleanthisMalialis. Real-time bidding by reinforcement learning display advertising. Proceedings of the Tenth ACM International Conference on Web Search and Data Mining. ACM, 2017.
S. Hart and A. Mas-Colell. simple adaptive procedure leading to correlated equilibrium. Econo- metrica, 2000.
S. M. Hung and S. N. Givigi. A q-learning approach to flocking with uavs in stochastic environ- ment. IEEE Transactions on Cybernetics, 2017.
M. Hussin, N. A. W. A. Hamid, and K. A. Kasmiran. Improving reliability in resource manage- ment through adaptive reinforcement learning for distributed systems. Journal of Parallel and Distributed Computing, 2015.
Peysakhovich Lazaridou and Baroni. Multi-agent cooperation and the emergence of (natural) lan- guage. arXiv preprint, (arXiv:1612.0)) 1222)))71222222 2016.
J. Z. Leibo, V. Zambaldi, M. Lanctot, J. Marecki, and T. Graepel. Multi-agent reinforcement learn- ing in sequential social dilemmas. Proceedings of the 16th International Conference on Au- tonomous Agents and Multiagent Systems, 2017.
Sergey Levine, Chelsea Levine, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. Journal of Machine Learning Research, 17, 2016.
Ke Li and Jitendra Malik. Learning to optimize. arXiv preprint, (arXiv: 1606001 160.00188)... 2016.
Shihui Li, Yi Wu, Xinyue Cui, Honghua Dong, Fei Fang, and Stuart Russell. Robust multi-agent reinforcement learning via minimax deep deterministic policy gradient. In AAAI Conference on Artificial Intelligence, 2019.
R. Lowe, Y. Wu, A. Tamar, J. Harb, O. P. Abbeel, and Mordatch. Multi-agent actor-critic for mixed weweeraaive-coppttttieeee environments. In Advances in Neural Information Processing Systems, 2017.
MLB. Statcast search. bttps::/baseballsass
mlb. wtwp:/atcttts_eeeeeeeeeeeee _eapssttteeerhhh_e
V. Mnih, K. Kavukcuoglu, D. Silver, A. Rusu, J. Veness, and M. G. Bellemare. Human-level control through deep reinforcement learning. Nature, 2015.
Sachin Ravi and Hugo Larochelle. Optimization as model for few-shot learning. In International Conference on Learning Representations (ICLR), 2017.
D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, and A. Guez. Mastering the game of go without human knowledge. Nature, 550:354-359, 2017.

---

Under review as a conference paper at ICLR 2020
D. Silver, T. Hubert, J. Schrittwieser, and I. Antonoglou. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362, 2018.
Jake Snell, Swersky Kevin, and S Zemel Richard. Prototypical networks for few-shot learning, arXiv preprint,
2017.
Richard Sutton. Learning to predict by the methods of temporal differences. Machine Learning, 3: 9 44, 1988.
Oriol Vinyals, Blundell Charles, and Lillicrap Tim. Matching networks for one shot learning. In Advances in Neural Information Processing Systems, 2016.
Z. Wang, Schaul, M. Hessel, H. Hasselt, M. Lanctot, and N. Freitas. Dueling network architec- tures for deep reinforcement learning. In International Conference on Machine Learning, 2016.
Waugh. Abstraction large extensive games. Master's thesis, University of Alberta, 2009.
Chongjie Zhang and Victor Lesser. Multi-agent learning with policy prediction. AAAI, 2010.
M. Zinkevich, M. Johanson, M. H. Bowling, and C. Piccione. Regret minimization in games with incomplete information. In Proceedings of the Annual Conference Neural Information Pro- cessing Systems, 2007.
12

---

Under review as a conference paper at ICLR 2020
A PSEUDO CODE
A.1 MODIFIED DEEP CFR
Algorithm Modified Deep CFR
1: function LEARNSTRATEGY
10 /0 with random strategies. 2: Initialize strategies og pp: 6b 0p
3: Initialize Strategy memories Mo.: Mop.
4: Choose final reward (s(s) and default reward rewarddefaute Select constant K₁ and K2. for Iteration
5:
T do
6:
1t-1 (ss Ma, Ma = LEARNVALUF(O o0 oop
May, May)
7:
((s, ab, apl02) = LEARRN(((((BBV)
8:
Calculate both players' instantaneous regrets rb, rp respectively with equation
9:
Convert the both players' instantaneous regrets rb; Tp to the new strategies o of respec- tively with equation
10:
Calculate the average strategy - from Strategy memories Ma Ma, respec- tively with equation Calculate the new strategies of of with equation 6
11:
12:
end for
13:
return of 00
14:
end function
Algorithm 2 Learning Value function
op o on Mo.. Ma,) 2: Initialize V(s(0v) @v). Replay buffer D.
1: function LEAAR
3: for episode t=1 to K do
4: Initialize s.
5: repeat (for each state of episode)
6:
Accumulate (s(s), (s))) respectively in their strategy memories, weighted by t with equation
7:
Select actions according %f of from and observe the state s'.
8:
Store transition (s, s') Into D. Sample a random minibatch of N transitions (8i, 84) from D. Update the value function (ss(0y: by minimizing the loss:
9:
L * [VV v()3] where V'= {
V(s(0,,)
8/ F Sterminal
Ve(s,) 8 Sterminal
10:
+ s'
until is terminal
12: end for
return V(s(0v), May, Mo
14: end function
In our experiment, The hyperparameters are chosen in the following: K1 30, K2 = =30, K3 20, Nptteh 4, 17 = 0.1.
13

---

Under review as a conference paper at ICLR 2020
Algorithm 3 Learning Q function
1:
function LEARNQ(V (s (vv)) Initialize (s, db. ap (00), Replay buffer D.
2:
3:
for episode t=1 K do
4:
Randomly initialize s.
5:
repeat (for each state episode)
6:
Randomly select actions (ab;ap) from and observe the state s' and reward according to equation B
7:
Store transition (s, db, ap, reward, s') into D.
8:
Sample a random minibatch of N transitions (5;, 0b,s, dp,i, reward, 8f) from D. Update the Q function ((s as, ap)dq) by minimizing the loss: [,[(Q(s; ab,5; pp.s(6p) - reward,)
9:
8
10:
until is terminal
11:
end for
12:
return (s,, ub. Up (Q)
13:
end function
A.2 ADAPTATION MECHANISM
Algorithm 4 Adaptation mechanism for the batter during a game
1: procedure A GAME
2:
Set Npitch 1), K3-
3: Initialize the observation memory O.
4: Prepare aTee batter's initial strategy), (hee prior assumption about the opponent pitcher's s strategy), and payoff matrix Q(s, ab, ap (0)) repeat (for each state of episode)
5:
6:
if (s))s)) 2 1 then // (O)(s)) denotes the number of observations at the state s. Anticipate the pitcher's strategy (s, based on the past observations in ((s) according to equation g
7:
8:
Calculate the regret i's (s, as) by with Q(s, as, (..00)).āppp a), (s)
9:
Convert (s(s, (b) the strategy (s(s,a) a) with equation end
10:
Calculate (s, 0&) according to equation] TO
12:
We select action according to og (s, as), and the opponent pitcher select his action. The game proceeds the new state s'.
14:
Restore the observed pitch into (s)).
N(O =
16:
if (s))) == Npitch then update (s, ap) by equation |II update (s, ab) by equation 2
17:
18:
19:
((O(s)) = 0
20:
Reset (s))
21:
end
22:
8'
23:
until is terminal
end procedure
B THE STRATEGIES OF TEAM TO TEAM 3, AND TEAM TO TEAM 11
During a game, these team's strategies only depend on "count".
14

---

Under review as a conference paper at ICLR 2020
Table 6: The strategy of Team 0
Table 7: The strategy of Team
Batter's streecyy
Picker's strilegy
Batter's
strateyy
Pliche''s straleyy
Cou
rd1
Eadt
Eadt
cevell
FT
Exps
Each1
Seb
nnecll camch cana
Down
vit
Batt
Bottl
ceell
Cou:
butt
Iwb
cunc
636
6.0
633
1.0
033
4.8
6:33
031
679
1.3
0:33
1/4
1119
1.0
1.00
0:39
0.8
0.30
811
6:31
1.4
4.8
0.80
677
8.0
0.3
14
0:31
1/4
6.1
1131
1.5
14
168
8.00
6.79
4.8
4.30
14
8.0
0.38
811
6:31
931
0.00
676
1.0
1.00
0119
1.0
0.78
4.8
8.30
8.31
138
0.03
1.48
1.50
116
6.0
038
1.5
0.43
0.1
1.8
1.5
1.5
0331
1/4
14
8M4
0:31
0.18
8.00
088
333
3.6
6.00
088
1.50
34
0008
1.30
0.39
4.8
1.50
36
8.0
0.38
1.38
0.0
0/8
0.00
1/8
1.00
083
6.0
1.2
0.80
033
8.8
630
1/8
1.00
043
1.4
6.0
038
1.80
180
13
033
811
0.0
138
33
0.8
0.0
1/8
0.50
1/8
639
118
6.0
1/8
1.5
0.0
0.9
1.20
180
34
620
033
135
630
1/4
Table 8: The strategy of Team 2
Table 9: The strategy of Team 3
Batter's stralery
Picker's snsitg
Batter's strategy
Pliche'ss strategy
Cau
rS1
Eact
Eart
Expt
[w=h1
Srb
necill cans canc
vil
butil
boll
cavil
Com
buSt
Iwc
nerch
exec
6:3
548
1.3
1.3
1.0
811
928
8.39
1/4
63:
011
1.30
830
0:33
1.8
8.5
0.33
111
633
138
33
8.18
0.00
9.28
6.50
048
14
6:3
1.4
1:31
1.9
14
0118
8.50
6.38
4.1
1.80
11
8.0
0.38
116
6.51
431
818
0.00
9.38
6330
0339
1.0
0.31
418
1.9
341
8.31
031
1.3
0.18
1.5
314
6.0
088
1.50
8.0
0:38
1.8
0.8
16
1.5
633
1/4
34
0:33
0/6
4.5
048
343
3.6
6.0
188
1.5
110
110
0.89
1.8
1.50
30
0.28
1.3
6.0
0/8
4.8
0.80
1/4
8:3
3.8
6.0
1.5
0.89
633
1.8
0.0
5/8
1.5
043
1.4
6.0
038
1.8
033
111
6.41
138
33
4.8
6.39
5/8
0.0
039
609
8.0
108
130
33
0/8
130
039
1.8
130
33
620
033
135
630
1/8
Table 10: The strategy of Team 8
Table 11: The strategy of Team 9
Hattee'ss stmingy
Plicher's stratigy
Batter's stradgy
Phcber's stritegy
Ceei
IaiN
fa:t
2x.
Liod
Bits
casch
cenvi (anveC
Each
Exil
Cxai
bucht
(nrvel)
canch
CaniC
64
8.00
1/8
1.0
8.8
1.00
8:1
444
8.31
149
1.3
6.33
8:1
1.8
0.3
128
8.00
049
011
0.8
8.78
6.51
1110
3.51
1.00
4/8
1.0
0.88
3.00
4/4
1.91
143
1.3
1.33
51
1.83
114
1.00
148
111
0.83
8.50
1.39
1.00
14
1.0
1/4
8.10
1.0
8.1
6.01
149
1.5
1.53
24
0.15
1.50
545
1.3
6.33
1/8
0.35
3.30
1.8
1.00
0.5
1.31
143
1.50
14
0.30
118
1.00
0/9
533
633
3/0
0.30
048
1.00
8.0
3/8
8.0
049
23
111
631
049
1.3
6.33
1.0
139
1.0
23
1.0
0/8
1.0
0/9
35
18
19)
100
18
189
32
1.0
0.0
0/8
1.0
1/0
52
041
1/6
1.0
089
1.0
Table 12: The strategy of Team 10
Table 13: The strategy of Team 11
Batter's striegt
Pltcher's straleyy
Batter's strategy
Plicher's strategy
Coad
butt
bult
canil
44y
Linal
eavill
187
Cam
bach
1.00
849
8:8
0.30
1/8
1.0
8:8
6.91
1/8
8:30
1
1.0
1.81
1/4
8.30
8.1
0.31
1.5
1.31
1159
6.89
6/8
4.03
1/8
1.00
8:6
6.33
438
1.20
1
14
1188
1.10
0.80
1/48
1.00
33
6.30
6.33
1.10
6.31
1150
6:3
1/6
0.33
3.10
1.0
6.3
1/8
8.30
144
1710
8.00
0.51
3.18
1.30
3.6
8.10
0.83
6331
1150
611
1.3
0
133
0.31
438
1.00
3.3
111
0.53
1.50
1.
1.10
8.0
1150
640
1.8
833
0/8
135
631
1/8
1.0
633
1.31
1/8
11
1.2
8.0
048
32
6.9
128
033
0/8
1.0
33
145
090
1/8
130
0/8
35
088
115
0.0
1/8
1.0
34
140
083
1/4
0.0
048
15

---

Under review as a conference paper at ICLR 2020
C POSSIBLE BATTING RESULTS
We collect every piece of data of batters vs. pitchers from MLB data on Stacast Search in 2016. Then we classify all the possible batting results into 32 categories, as listed Table 14, according to the batted ball information of "Event", "Location", "Type", "Distance", and "Exit Velocity" from collected data.
Batted Ball Type describes batted ball, depending how it comes off the bat and where in the field it lands. There are generally four descriptive categories: fly ball, pop-up, line drive, ground ball. Line drive is that a batted ball hit hard enough and low enough that it appears to travel in relatively straight line. Pop-up a specific type of fly ball that goes very high while not traveling very far laterally.
Table 14: The 32 categories of possible batting results
category
description
location
Hit
HR
irrelevant
3B
irrelevant
2
2B
hit into gap, far down the foul lines
2B
4
2B
hit into gap, not far
5
1B
Left field
1B
Center field
7
1B
Right field
8
1B
Infield
Out
ground ball
Pitcher
10
ground ball
Catcher
11
ground ball
First baseman
12
ground ball
Second baseman
13
ground ball
Third baseman
14
ground ball
Shortstop
15
line drive
Pitcher
16
line drive
Catcher
17
line drive
First baseman
18
line drive
Second baseman
19
line drive
Third baseman
20
line drive
Shortstop
21
fly ball
Left field
22
fly ball
Center field
23
fly ball
Right field
24
far fly ball
Left field
25
far fly ball
Center field
26
far fly ball
Right field
27
pop-up
irrelevant
do not hit into play
28
foul ball
29
swing and miss
30
called strike
31
ball
16

---

Under review as a conference paper at ICLR 2020
D SIMPLE BASEBALL RULE
In half-inning, the batting team scores a "run" i player legally runs around the bases and touches home plate. On the other hand, the fielding(defense) team tries prevent runs by getting batters or runners "out". As soon as the fielding(defense) team gets three outs, the half-inning ends. There usually nine innings (defense). The team with the larger numbers of runs by the end of the game is the winner. The game proceeds when the pitcher in the fieldingddefense team throws the ball towards home plate and the batter decide whether to swing. A "strike" results when a batter swings and misses a pitch, does not swing at pitch in the strike zone or hits a foul ball that is not caught. "ball" results when pitch misses the strike zone and the batter do not swing. A batter is out when getting three strikes, while allowed to "walk" to the first base when getting four balls. There are lots of outcomes when the batter hit ball into the field. In general, the batter aims to make solid contact to the pitch, while the pitcher aims to prevent the solid contact.
baseball game, and opposing teams take batting and fielding
17


Reviews:
[{"review_id": "r1eMQEs3FB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "UPDATE: Thank you for the detailed response. I think the changes you have made to the paper have improved it, but there remains significant work to complete before the paper reaches its full potential. For example, Figure 3 is a useful additional insight but it provides no quantification of the variation between repeated runs nor a comparison to a suitable baseline approach. Similarly, the additional hyperparameter details in Appendix A are helpful to enable reproducability but without explaining how these values were chosen it is impossible to assess the rigor of the empirical evaluation.\n\n--\nThis paper proposes modification to Deep CFR and introduce a simplified baseball environment to evaluate the modifications. The rigour of and detail in documenting the empirical evaluation is not currently of the standard I would expect for publication at ICLR. I will detail suggestions for improvement below, but my most pressing concern for discussion in the rebuttal is with regard to the 2nd conclusion - If the team has learnt a Nash equilibrium as initial strategy, then how can performance of either team be improved by only one team further adapting their policy as shown in Table 5 lower avg2 by the difference between No Adapt (0.61) and Guest Adapt (0.51)? If I have interpreted the authors results correctly, this demonstrates that the initial policy is not a Nash equilibrium.\n\nSuggested Improvements:\nIn the related work section, the narrative of the paper would be clearer if the authors introduced why the meta-learning literature is being reviewed. I would also recommend weakening the claim that \"instability of the training process\" is \"the critical issue in MARL\" to just noting it is \"an issue\" and request the authors justify the claim that this \"is particularly severe in a competitive environment\". Why is this more of an issue in fully competitive environments than it is in general sum games?\n\nIn Section 3.1 the authors claim they need to switch to TD learning from tree search due to stochastic state transitions but there are forms of search that can accommodate stochastic transitions so this claim needs to be removed. Perhaps the authors can motivate this change in another way? This section also introduces 2 further simplifications of the domain (1) learning strategies for half-innings and applying them to the whole game; (2) agents knowing their opponents true action chosen and not the noisy observation (e.g. pitchers target location instead of actual pitched location) and (3) \"both agents know each other's strategies\"  - these should be included in Section 2.1 when the domain is described.\n\nSection 3.1. closes by stating \"In principle, their average strategies will gradually converge to the Nash-equilibrium strategy.\" This is very weakly argued, to make such a claim the authors should provide evidence that their environment and modifications to Deep CFR meet the requirements of the theory where this guarantee was proven. \n\nSection 3.2. notes that the learning rate is \"manually determined\" but the precise methodology of tuning hyperparameters is not provided and no values of settings used for any hyperparameter are included in the paper. Without details of the methodolofy it is unclear if a rigorous empirical evaluation was performed and without the precise hyperparameter settings used the results are not reproducible.\n\nSection 4.1. notes \"It is very interesting to observe that the learned Nash-equilibrium strategy is actually quite similar to real-life baseball strategies.\" Setting aside the issue regarding whether a Nash equilibrium has been learnt, this is a subjective opinion not an rigorous empirical observation. The same applies to the comment \"The simulation results are very similar to real-life baseball games\" on page 9  Can you support these claims that the learnt strategy is similar to real-life by comparison to the data collected from the MLB Statcast?\n\nIn Section 4.2. there is further imprecision in the discussion of results. For example \"as the strategy adaption mechanism is employed, the WP of most teams … actually decreases.\" As this mechanism is a core contribution of the paper, this evaluation needs to be more rigorous. Is there a statistically significant difference caused by using the proposed mechanism? \n\nAll results presented should include quantification of variation as well as average values and the number of repeats these averages are taken from should be clearly documented. The inclusion of a limited subset of teams in empirical evaluations (e.g. Table 4 only including teams 0,1,2,3 and 12 and Table 5 only including teams 5 and 13) should be clearly justified.\n\nAll references to papers published in conference proceedings or journals should cite the published version of the paper and not the arxiv version. All references should include the full publication venue and not abbreviations (e.g. Zhang and Lesser, 2010 is currently listed as just AAAI). References to online resources should include a note of the date accessed.\n\nMinor Comments:\n1) The word \"purpose\" is used frequently in place of \"propose\" (e.g. line 7 of the abstract)\n2) Page 2: \"plays the-best-of-three game\" -> plays the best-of-three games\n3) Page 2: \"that worth further study\" -> that are worth further study\n4) Page 3: \"To simply the game\" -> To simplify the game\n5) Page 3: \"listed in C\" -> listed in Appendix C\n6) Page 8 (and recurring): Initial speech marks are the wrong way around, this looks like a Latex error.\n7) Page 10: \"we would relief\" -> we would relax\n8) Page 10: \"in the real life\" -> in real life\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BkxzF3NCtr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper studies adaptation of agent policies in a simplified baseball game, which is designed as a zero-sum two-agent game between a batter (B) and a pitcher (P), each of which has 5 discreet actions. The introduced game is fully observable but stochastic, which the authors argue is a challenging setup. The authors propose a Bayesian-style adaptation of the agent strategies (where each agent models the probability of the actions of the opponent by computing the posterior give a prior and evidence from the past observations), which seems to be computable analytically, from an initialization learned with counterfactual regret minimization (CFR) that approximates the Nash of the considered game.\n\nComments/Questions:\n\n1. Why do we need adaptation if CFR already learns Nash? It looks like one of the key differences between the proposed game/setup and many of the previous work is that it is a fully observable zero-sum game. The initialization learned by CFR already might come close to the Nash equilibrium. It's unclear why the agents need to play something different than Nash. Could the authors argue (preferably, formally theoretically or at least quantitatively) why adaptation is necessary?\n\n2. Related to point 1, in section 4.2, it looks like post-adaptation strategies turn out to be superior when playing against opponents that play Nash. I would like to understand why. Do opponents actually play Nash? Does the asymmetry of the game have to do something with this? There's virtually no analysis of the results in the paper, which significantly undermines any contribution.\n\n\nOther comments:\n\n1. I'm personally not familiar with baseball, and the paper doesn't really introduce the game. So, parts of the introduction and background that use baseball-specific terminology (paragraph 3) make no sense to readers unfamiliar with the game. It would be nice to have the game exemplified and explained along with the key simplifying assumptions.\n\n2. Writing can be significantly improved (and compressed!). There are typos throughout. Some phrases from the paper which meaning is really hard to parse (for example, \"To focus on the impact of strategy adaptation over the winning percentage <...>\").\n\nAlso, the last paragraph of the intro that describes the organization of the paper is unnecessary for conference submissions (it just takes spaces and no one reads it because it's easy to scroll through 10 pages to get a sense of the organization).", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies whether adopting strategy adaptation mechanisms helps players improve their performance in zero-sum stochastic games (in this case baseball). Moreover they study two questions in particular, a) whether adaptation techniques are helpful when faced with a small number of iterations and 2) what’s the effect of different initial strategies when both teams adopt the same adaptation technique. Reviewers expressed concerns regarding the fact that the author’s adaptation techniques improve upon initial strategies, which seems to indicate that their initial strategies were not Nash (despite the use of CFR). In the lack of theory of why this seems to happen at the current setup (and whether indeed the initial strategies are Nash and why do the improve), stronger empirical evidence from more rigorous experiments seem somewhat necessary for recommending acceptance of this paper.
Decision: Reject