Implicit Ensemble Training for Efficient and Robust Multiagent Reinforcement Learning
Authors: Macheng Shen
JONATHAN P HOW
Venue: Accepted by TMLR ()

Abstract: An important issue in competitive multiagent scenarios is the distribution mismatch between training and testing caused by variations in other agents' policies. As a result, policies optimized during training are typically sub-optimal (possibly very poor) in testing. Ensemble training is an effective approach for learning robust policies that avoid significant performance degradation when competing against previously unseen opponents. A large ensemble can improve diversity during the training, which leads to more robust learning. However, the computation and memory requirements increase linearly with respect to the ensemble size, which is not scalable as the ensemble size required for learning robust policy can be quite large. This paper proposes a novel parameterization of a policy ensemble based on a deep latent variable model with a multi-task network architecture, which represents an ensemble of policies implicitly within a single network. Our implicit ensemble training (IET) approach strikes a better trade-off between ensemble diversity and scalability compared to standard ensemble training. We demonstrate in several competitive multiagent scenarios in the board game and robotic domains that our new approach improves robustness against unseen adversarial opponents while achieving higher sample-efficiency and less computation.

Published in Transactions on Machine Learning Research (01/2023)
Implicit Ensemble Training for Efficient and Robust Multia- gent Reinforcement Learning
Macheng Shen
macshen@miis edu
Deportment of Mechanical Engineering Massachusetls Institute of Technology
Jonathan P. How
jhoo edu
Department Department of Aeronautics and Astronautics Massachusetts Institute of Technology
Reviewed on OpenReview: 5D X2FgrruuX 3F 3DTMLB)
https: o/penreveeee formm? id-
X5BTHLRX
Abstract
An important issue in competitive multiagent scenarios the distribution mismatch between training and testing caused by variations in other agents' policies. As result, policies op- timized during training are typically sub-optimal (possibly very poor) in testing. Ensemble training is an effective approach for lcarning robust policies that avoid significant perfor- mance degradation when competing against, previously unseen opponents. A large ensemble can improve diversity during the training, which leads to more robust learning. However, the computation and memory requirements increase linearly with respect to the ensemble size, which is not scalable the ensemble size required for learning robust policy can be quite large (Czarnecki al., 2020). This paper proposes novel parameterization of a policy en- semble based deep latent variable model with multi-task network architecture, which represents an ensemble of policies implicitly within single network. Our implicit ensemble training (IET)) approach strikes a better trade-off between ensemble diversity and scalability compared to standard ensemble training. We demonstrate several competitive multiagent scenarios the board game and robotic domains that our new approach improves robust- ness against unseen adversarial opponents while achieving higher sample-effccinncy and less computation.
1 Introduction
In competitive multiagent scenarios, agents learn concurrently (Lowe et al., 2017) and the learned policy of each agent depends on the joint policy of all the other learning agents. One challenge resulting from such multiagent learning is that the policy learned from training might not perform well in the testing envi- ronment where the opponents' policies could be significantly different from those the training opponents (distribution shift. between training and testing). Even worse, the testing opponents could be trained exploit the weakness of the policy learned from the training (Gleave et al., 2019).
One effective approach mitigate the performance degradation from training to testing is ensemble training, which has been applied in many previous works (Bansal et al., 2017; Lowe et al., 2017; Silver al., 2016; Jaderberg et al., 2019; Vinyals et al., 2019) and shown improved robustness against previously unseen opponents compared with training a single policy per agent. In ensemble training, each agent has multiple policies as in (Lowe et al., 2017; Jaderberg et al., 2019; Vinyals et al., 2019) or keeps multiple copies of previous policies as (Silver et al., 2016; Bansal al., 2017), from which one policy for each agent is sampled to play against each other. As result, each policy is optimized against a distribution of the other agents' policies, which effectively reduces the distribution shift bet training and testing (because

---

Published in Transactions on Machine Learning Research (01/2023)
single policy can be regarded single Dirac-delta distribution centered at point within the policy space). However, one drawback of applying this ensemble training technique is the significant increase in computation and memory consumption due to the learning and storage of multiple policies. Besides, the number of policies required for guaranteed policy strength improvement is a function of the non-transitivity dimension of the multiagent scenario, which could be on the order of lens for simple games such as Tic-Tac- Toe and more than thousands for complicated real-world games (Czarnecki al., 2020). As a result, this ensemble training approach could become intractable due to constraints on computational resources and/or end up learning weak policies due to insufficient policy diversity.
In this paper, we propose novel implicit ensemble training (IET) approach by formulating ensemble training as a multitask learning problem. 1 Instead of maintaining multiple policies explicitly with independent neural networks, our IET approach uses a multitask network architecture with a learnable conditional latent variable distribution to implicitly represent a policy ensemble. The multitask network architecture improves samplo-effcienncy via parameter sharing and the conditional latent variable captures the diversity of the policy.
Our contributions are: 1) we identify the cause of inefficiency in standard ensemble training: 2) we develop novel algorithm that extends ensemble training with latent variables and a multitask network, resulting in a better trade-off between policy diversity and scalability, which is demonstrated by improved policy robustness with fewer samples and less computation compared with standard ensemble training.
2 Preliminaries
2.1
Markov games
We use Markov Games (Littman, 1994) as the decision-making framework in our work. A Markov game for agents is defined by a set of states describing the possible configurations of all agents, a set of , ON for each agent. Each agent has a stochastic policy O₂ x A: + [0, 1}, and reward function 8 A R
actions , AN, and a set of observations OI....,
2.2 Multiagent reinforcement learning (MARL)
The objective each agent is to maximize its own cumulative reward R, = -' r'r with discount factor y and time horizon T (Lowe et al., 2017). As result, the learning problem is formulated as finding a joint policy (T,) ==1:N where each policy maximizes its expected cumulative reward,
J = Es B an [R a)l,
(1)
with nx being the transition dynamics and the subscript -i denotes the set (ili A í.j =
2.3 MARL with policy distribution
In practice, learning with single joint policy typically leads to over-fitting between policies and poor generalization to previously unseen policies. A variety of empirical studies (Lowe al., 2017; Silver et al., 2016; Jaderberg et al., 2019; Vinyals et al., 2019; Lanctot et al., 2017; Bansal et al., 2017; Vinitsky et al., 2020) and recent theoretical study (Czarnecki et al., 2020) suggest that it is necessary to maintain diverse set of policies for each agent to improve the strength of the joint policy learned via MARL. Therefore, instead of learning a single joint policy, we consider the following objective function that learns distribution of policies for each agent,
Jj a [R a));
(2)
where P is a joint distribution over the joint policy space II = I1 x 12... x II. Each agent is learning its own policy distribution II, to optimize its objective J subject to the joint distribution II.
ecode: http:r//gthhbb.cm//Meeengggge
2

---

Published in Transactions on Machine Learning Research (01/2023)
Note that the feasibility set of Eq. contains that of Eq. which is analogous to the relationship between mixed-strategy Nash Equilibrium and pure-strategy Nash Equilibrium (Nash et al., 1950). This relationship also suggests that Eq. 2 is a more general learning objective than Eq. 1.
One simple parametrization of ) a uniform distribution over a policy ensemble of fixed size (a set of independent policies, each parameterized by a neural network), such as in (Lowe et al., 2017; Bansal et al., 2017; Vinitsky al., 2020). Policy space response oracle (PSRO) (Lanctot et al., 2017) is another parametrization, with expanding set of policies. At each iteration, new policy produced by an oracle, via solving for the (approximate) best response against mixture of other agents' policies, is added to the policy ensemble. PSRO leads to better convergence towards Nash Equilibrium in poker games than self-play (single policy) and fictitious self-play (uniform distributed over previous policies). Despite the better convergence property, these ensemble approaches are not scalable because the computation increases lincarly with respect the ensemble size, which can be exponential with respect to the complexity of the multiagent interactions (Czarnecki al., 2020) guarantce policy improvement. This limitation motivates developing a new parametrization of policy distribution with better parameler-eflcieney, which leads to higher sample-efficincy for training.
3
New approach
In this section, we present a parameter-effiiiet parametrization of the policy distribution, which we call implicit ensemble. We start by discussing the relationship between uniform ensemble with latent- conditioned policy and then show how our implicit ensemble approach extends the uniform ensemble for higher parameter-efliiennc and policy diversity.
3.1 Generalization of ensemble training cenet--condiionee policy
Ensemble training with uniform distribution over fixed-sized set of policies is common approach, e.g. in (Lowe et al., 2017; Bansal et al., 2017; Vinitsky et al., 2020), for improving the robustness of policies in MARL. In ensemble training, each agent's ' policy π, is an ensemble of K sub-policies, with each denoted as T or x; [k) and parameterized by separate neural network parameters 0!th) The generative process for the ensemble training policy is expressed as
K 5(k - j),,
k: ~ uni((1, K)
Σ
K
==1
(3)
0 ~ 5(0. 0
a.10: ~ fe. (o:),
where ô is the Dirac-delta distribution. One interpretation of Eq. 3 is that n, is conditional policy on a uniform discrete latent random variable k;
lk f(or, 0000( ,....... ..... (K..,,,000 ((00k)) [g(ss
(4)
Eq. 4 suggests that ensemble training can be interpreted as a latent-conditioned policy (Haarnoja et al., 2018; Petangoda et al., 2019; Lee et al., 2019) conditioned on the discrete random variable & coming from uniform distribution. However, this parametrization is inefficient because only one out of the K sets of sub- policy parameters (0("). ( ..... o(*)] is activated per sampling of a₁ As a result, the rollout from executing set of sub-policy parameter cannot be used to optimize the rest K sets of sub-policy parameters, which reduces the sample efficiency by factor of K.
3.2 Implicit ensemble training
Our IET generalizes ensemble training via two steps: 1) relax the discrete random variable is E f1.2...., into continuous latent variable R& with learned distribution that adaptively captures the diversity
3

---

Published in Transactions on Machine Learning Research (01/2023)
Standard Ensemble
Implicit Ensemble
Policy ID unif[1, K
Shaping Network
Policy 1
Policy k
Policy
Ic
==
Share Modules
Latent-condittoned
policy network
Action
Action
Action
x( (o;
Activated path: Inactive path
Action
Figure 1: Connections between standard ensemble and our proposed implicit ensemble: In the standard ensemble, random index is sampled and the corresponding network is selected to output an action; while in our implicit ensemble (right), random Gaussian variable is sampled and encoded into a latent vector by a shaping network, which is then passed to a conditional policy network to output an action.
of the policy ensemble; 2) replace the K independent neural networks with unified modular network architecture with parameter that improves paraameter--lliiinnyy by knowledge sharing between sub-modules within the network. The continuous relaxation also makes the policy differentiable with respect to the latent variable, thus making it possible to synthesize new policies from the learned skills represented by the sub- modules within the modular network by perturbing the latent variable distribution.
The generative process of this implicit ensemble
((0,Ixx c/= = go(2), a: ~ (o;ic))
(5)
In Eq. 5, random Gaussian noise vector is sampled from the standard multivariate Gaussian distribution (sampled once at the beginning of each episode and remains fixed during the episode), then passed through shaping network parameterized by vé to output latent condition variable Finally, the action is sampled from a policy that is parameterized by é and conditioned on the latent condition variable c. Both & and ø are learnable parameters that are optimized end-to-end with respect to the reinforcement learning objective Eq. 2.
The implicit ensemble Eq. 5 is a more flexible parameterization than the ensemble training Eq. 3. In fact, the latter is special case of the former, where the distribution of the latent variable e collapses into discrete distribution (corresponding to the uniform discrete distribution of k), and the shared parameter ò is divided into K disjoint sets of parameters each of which is activated in the policy network when only one of the discrete values of e is drawn.
In contrast to the ensemble training formulation Eq. 3, where the ensemble size K a hyperparameter to control the diversity of the (ensemble) policy, Eq. does not contain this explicit hyperparameter. Instead, the diversity of the policy is captured adaptively by the learned latent distribution parameterized by th: complicated multi-modal distribution corresponds to high diversity, while simple uni-modal distribution corresponds to low diversity.
3.3 Model architecture
Fig. 1 shows the model architecture for implementing the IET. There are two major components within this architecture: 1) shaping network, corresponding to the mapping go(.) in Fq. 5 that transforms the Gaussian

---

Published in Transactions on Machine Learning Research (01/2023)
noise vector into the latent condition variable e; and 2) conditional policy network, corresponding to the parametrization hg.
The shaping network is responsible for adding divensity to the conditional policy for improving the robustness of the learned policy, and the conditional policy network improves srrammeter-fficinne via knowledge sharing. The detailed design of these two networks is presented in the following sections.
3.3.1 Shaping network
The shaping network takes Gaussian noise vector E R4 input and transforms z into a latent condition variable which modifies the standard multi-variate Gaussian distribution into a learned (complicated) distribution. We use a feedforward network with 2 fully-connected layers followed by a Softmax layer to parameterize this shaping network.
3.3.2 Multi-tasking network
Recent studies (Yang et al., 2020; Devin et al., 2017; Huang et al., 2020) found that modular architecture is parameder-effiient way of learning multi-tasking policies. In ensemble training, the multi-tasking require- ment naturally arises from the fact that each policy is optimized against the ensemble of policies of the other agents.
To improve the parameter-efficiency of ensemble training, we use the modular architecture proposed (Yang et al., 2020) as our conditional policy network for multi-tasking. We describe the detail of this modular network in the Appendix section. However, this specific network architecture not the only choice for effective knowledge sharing. We show in Section 4.6 that commonly used multi-head network is also effective for parameter-efficient ensemble training.
3.4 Model training
Algorithm shows the pseudocode of the IET. At the beginning of each environment episode, each agent samples its own latent vector 2; and then exccutes its conditional policy till the end of the episode. The latent vectors fixed within each episode that agents' policies consistent across time steps. The model parameters ti: and Φ; are trained through policy gradient update, which maximizes the long-term discounted cumulative reward objective for each agent.
Algorithm Implicit Ensemble Training
Require: Number of training episodes M, set of agents with index in (I,...,N Initialize network parameters vi, Φi, i NN,...,N) for each agent
for 1: M do
3: for i= 1 N do
4: Sample latent vector 2, ~ NN, ItxI.
5: end for
6: Rollout with each agent sampling its action via a, ~
7: Update network parameters vi; Φi with policy gradient
end for
return 2/1 di
Although Algorithm 1 shows that each agent independently samples its private latent vector, it is beneficial to share the latent vectors between agents under certain situations. For example, consider mixed cooperative- competitive scenario where two learss of agents compete against each other. Agents within the same team are incentivized to communicate their latent vectors or condition their policies on the same latent vector (which could be achieved without communication by using common random scod) to achieve effective coordination. However, since the focus of this paper on policy robustness, later sections of this work focus experimental evaluation of the proposed IET approach in competitive scenarios involving two agents.
5

---

Published in Transactions on Machine Learning Research (01/2023)
4 Evaluation
4.1 Scenarios
We evaluate our approach on two types of 2-player (which we refer to as the blue agent and the red agent hereafter) multiagent scenarios.
turn-based games implemented in the PettingZoo multiagent environment? (Terry et al., 2020) and the RLCard toolkit (Zha et al., 2019): Connect Four, Ledue Hold'em, and Texas Hold'en (Limit).
Board-game:
continuous problems modified from the robot racing scenarios in the Ro- boSchool environment': Ant and Cheetah, where we decompose each robot into front and rear parts, and assign opposite rewards to each part. As such, the front is learning to move forward while the rear is learning to move backward.
RoboSchool-Racer:
4.2 Baselines
We compare our approach with the following baselines:
1. Single Policy Training (SPT): standard multiagent training approach wherein each agent opti- mizes only one policy.
Simple Ensemble Training (SET): standard ensemble training approach wherein each agent optimizes ensemble of policies against each other. We ensemble size of 3 for each agent (an ensemble of 3 policies has been shown to improve robustness significantly in previous works (Lowe et al., 2017) and (Bansal et al., 2017)), which also increases the amount of computation by factor of 3.
Minimax Training (MT): achieves worst-case robustness through optimizing one's policy against the worst-case opponents, which is formulated as minimax optimization. We use the one-step approximation approach in (Li et al., 2019) with the Multiagent Actor-attention-critii (MAAC) algorithm (Tqbal & Sha, 2019)
4.3 Implementation detail
We used the RLlib (Liang et al., 2018) implementation of Proximal Policy Optimization (PPO) with a mini- batch size of 256 and a learning rate of 5 x 10-5 We use independent networks for the policy and the value function approximations and set the following hyperparameters for IET: L 10 for the latent condition variable dimension; H = 64 for the hidden layer dimension in the shaping network; n = 2 and = 2 for the number of layers and number of modules of the modular network; D = 64 and d = 64 for the embedding and module hidden dimension. For the other approaches, each of the policy and the value networks consists of two fully-connected layers with 256 hidden units.
In the simple ensemble training setting, each sub-policy within an ensemble only has probability of 1/3 to be selected for execution. II the same number of environment. rollouts is used to train the simple ensemble used for the other training settings, the simple ensemble policy will perform poorly due to insufficient training. Therefore, we roll out two additional environment simulations (but counted as one training step when training the simple ensemble) for a fair comparison al the cost of additional computational overhead.
To evaluate the robustness of the learned policies, we adopted a similar approach as in (Vinyals et al., 2019; Gleave et al., 2019) by training an independent exploiter agent. Specifically, we launched two concurrent threads, for the training, the other for the lesting, and repeated the following steps:
hhttps://xww.pe http:///www.ettngooo..tr

---

Published in Transactions on Machine Learning Research (01/2023)
1. Train the blue agent and the red agent in the training thread for one training epoch.
2. Copy the blue agent's policy to the testing thread and freeze it.
3. Train the red exploiter agent in the testing thread against the fixed blue agent.
As result, the red exploiter agent learns to exploit any weakness of the blue policy, and the corresponding reward is an informative indicator of the adversarial robustness of the blue policy.
4.4 Results and comparisons
This section discusses experimental results and compares baseline approaches with our proposed IET ap- proach.
4.4.1
Adversarial robustness
We investigate the adversarial robustness of the learned blue policy by training a red exploiter agent against the frozen blue agent policy.
Table 1 shows the average testing reward, the best testing reward, and the average reward gap between training and testing. We compare our IET approach with two baselines. One baseline is our IET approach with the input latent noise fixed as zero vector, and the other baseline is standard ensemble with 10 policies. For the IET setlings, the difficulty of balancing the relative strength between the blue and the red agents, sometimes (for some random sceds) the blue agent's 's training reward converges to near -1.0, and so does the testing reward. Since we are interested in investigating the robustness of the learned policy (whether strong training performance is sufficient to guarantee a strong testing performance), we filter out those seeds that lead to poor training performance (converged training reward lower than -0.5). There turned out to be 19 remaining valid seeds for each of the two IET settings, and the results are evaluated only across these valid seeds. As for the SET setting, we found that both the training and the testing reward are not very sensitive to random seeds, so we report the results across 4 random seeds. Our IET approach outperforms the baselines for both the mean testing reward and the best testing reward, especially for the besting testing reward where our IET reached near optimal testing reward within 4 out of the 19 valid seeds.
ran experiments across 25 random seeds. However, we found that due to
Table 1: Testing reward against exploiter and reward degradation
Connect Four
IET (fixed latent)
SET (10)
IET
Mean testing reward
-0.40 +. 0.08
-0.21 1 0.06
-0.06 0.14
Best testing reward
-0.16
-0.03
1.0
Mean baaining/testingg reward gap
0.70 + 0.18
0.17 + 0.08
0.42 + 0.14
4.5 Competition scores between training settings
To evaluate the out-of-distribution robustness, we also include the competition scores between the blue agent and the red exploiter agent, calculated as (Tblue - Tree))/2 for each pair of training settings in Tables 2. The blue agent uses the policy learned in the training thread, while the red agent uses the exploiter policy learned in the testing thread. As result, the diagonal scores are in favor of the red agent, because the red exploiter policy is trained against the corresponding blue policy, but not the other way around. The diagonal scores measure the adversarial robustness of the learned blue policy. In contrast, the off-diagonal scores mcasure the out-of-distribution robustness of the blue policy, because neither the blue policy nor the red policy has been trained against each other. However, drawing conclusions directly from Tables 2 about the relative strength of the policies learned by the three training scttings is difficult, because the compotition scores show that there is no single policy that can dominate all the rest policies. This non-transitive nature of real-world games has also been observed in previous work (Czarnecki al., 2020).
7

---

Published in Transactions on Machine Learning Research (01/2023)
Table 2: Competition scores between SPT, SET,, and IET. Higher score implies that the blue policy is stronger than the red policy. The last column lists the lowest scores over the columns for each row, where our IET achieves the highest scores among the training settings.
Connect Four
SPT
SET
TET
Lowest across columns
SPT
-1.0=0.0
-0.16++0.04
0.29++0.04
-1.0++0.0 -0.98++0.01
SET
0.35++0.01
-0.98++0.01
0.65++0.3
IET
1.0+0.0
1.040.0
-0.33++0004
-0.33=0.04
Leduc Hold'em
SPT
SET
IET
Lowest across columns
SPT
-0.3320.12 0.55
-0.3410.099
0.12+0.07
-0.63:0.112
SET
-0.04:0.1 0.11+0.09
-0.07:2.009
-0.07+0.09
IET
0.17:0.112
-0.03++0.09
-0.03:00.99
Texas Hold'em
SPT
SET
IET
Lowest across columns
SPT
-1.47=0.32
-0.1240.11
0.0++0.5
-1.47+0.32
SET
-0.6+0.27
-0.16++018
0.05+0.10
-0.6++0.27
IET
0.39:0.24
-0.0990.16 0.13#0.12
-0.09=:0116
To quantitatively evaluate the robustness of the learned blue policy, we provide two metrics: (1) Normalized score: we find the worst-case reward gap between blue and red mins, my) (Te, xT)], 11 SPT,SEE, IET) (higher is better) and normalize it to [0, 1], (2) Nash policy probability: approach proposed in (Balduzii et al., 2018) by solving for the Nash-Equilibrium (NE) (Holt & Roth, 2004) of the meta-game (Tuyls et al., 2020), which involves a row-player and a column-player, whose actions are selecting which row/column policy to execute from the three available policies. The NE of the meta-game is a pair of stochastic policies. The probability of a policy being selected by the meta-player measures the strength (robustness) of this policy.
follow the
We show these two metries in Table and Table 4. The results show that our IET approach consistently outperforms the other ones across all the scenarios.
Table 3: Normalized scores that measure the adversarial robustness.
Scenarios / Settings SPT SET IET
Connect Four
5e-3
le-2
0.33
Ledue Hold'em
0.18
0.46
0.48
Texas Hold'em
0.0 0.2
0.46
Table 4: Nash policy probability of the meta-player that measures the overall strength of the policy,
Scenarios Settings
SPT
SET
IET
Connect Four
0.0
0.38
0.62
Leduc Hold'em
0.0
0.41
0.59
Texas Hold'em
0.0
0.0
1.0
4.5.1 Comparison with the minimax robust approach
We also show in Table the comparison between the ensemble training approaches with the minimax robust learning approach (Li et al., 2019) which maximizes the worst-case reward to obtain adversarial robustness. We use the RoboSchool environment because the minimax approach requires differentiability with respect to the action (therefore continuous action environments).
As the complexity of the environments incrcases, we see that the scores bocome noisier due to the difficulty of the reinforcement learning algorithm to optimize the policy (indicated by the fact that fewer minimal scores across the column achieved at the diagonal). Again, we show the Nash meta-policy probability in
8

---

Published in Transactions on Machine Learning Research (01/2023)
Table 5: Competition scores between SPT, SET, IET, and MT. IET achieves the best adversarial robustness
in Ant as well as best out-oo-distribution robustness in both Ant and Cheetah.
Ant
SPT
SET
IET
MT
Lowest across columnss
SPT
8.42:0.26
26.2+0.82
1.95+0.54
24.4942.86
44.9922.86
SET
-35.311154
5.0911.0
-23.7.1..6
-24.5:1...03
-35.3:1:.54
IET
4.13.10.44
33.6411.03
15.210.38
37.710.81
4.13.10.84
MT
-5.337+1.73
-2.53+0.90
-13.85+1.44
-25.7+0..0
-25.7=0..0
Cheetah
SPT
SET
IET
MT
Lowest across columns
SPT
15.54440.68
-15.95=0.72
-7.16:10334
4.26::.22
-15.95=20.2
SET
-9.05:10.1
-12.6010.64
-1.76.10.41
6.45.0.29
-12.60.10.44
IET
-9.22:10.22
-11.8811..7
0.24.1037
8.16.10.2
-11.88.10.2
MT
-13.7110.59
-20.12-0.65
-4.02.1..50
4.84.00388
-20.12.10.65
Table 6, which suggests IET approach achieves the best robustness. The fact that the minimax robust approach does not work very well could be result of two reasons: 1) The minimax formulation may not be a good approach to achieve out-of-distribe which may lead to overly-conservative behaviors that fail to exploit the weaknesses of the opponent; 2) The one-step solution technique for the minimax problem proposed in (Li et al., 2019) is an approximate solution, which may find a sub-optimal solution due to the difficulty of selecting a suitable slep-size parameter, which has been reported in previous work (Sun al., 2021).
robustness since it optimizes for the worst-case reward,
Table 6: Nash meta-policy probability in the RoboSchool scenarios.
Scenarios / Settings SPT SET IET
MT
Ant
0.0
0.37
0.63
0.0
Cheetah
0.0
0.0
1.0
0.0
4.6 Ablation studies
To further understand the role of the shaping network and the multi-tasking network, we conduct two additional experiments: 1) We show that the shaping network learns non-trivial mapping from the Gaussian noise to the latent variable only when diversity is required through visualizing the mappings represented by the shaping networks learned in multiagent scenarios and compare those with that learned in single-agent scenario; 2) We show that we can improve the sample-efficiecy of the PSRO algorithm by sharing the intermediate layers within a aulti-tasking network, and the optimal performance is obtained at a medium level of sharing.
4.6.1 Policy diversity
To verify our conjecture that the robustness of our IET approach is result of IET's capability to present diverse set of policies, we show the histogram of the action probabilities (Fig. 3) in the Connect Four game, and the corresponding training and testing rewards (Fig. 2). Fig. 2a shows that the IET agent's testing reward increases significantly at around 60 million training steps. Fig. 3a and 3b show the action probability histograms of the IET agent at 15 million training steps, which is almost uni-modal. This lack of diversity explains the poor testing reward during the early training stage (0-60M steps). In contrast, Fig. 3c and 3d shows the histograms at 60 million training steps, which span much wider range of action probabilities. Specifically, Fig. 3c corresponds to the first time step of the game only, where the agent observation is always empty board (deterministic). Therefore, the wide range of action probabilities is a result of the policy conditioning on the latent noise. Fig. 3e and 3 shows the histogram al 75 million training steps, which is also modal. Notice that the corresponding testing reward shown in Fig. 2 near optimal. This result suggests that the IET agent has learned strong policy (Another evidence is that the probability of taking action 3, which corresponds placing the first chip in the middle of the board, is near 1.0. It is known that the optimal strategy in Connect Four for the first player is to place the chip in the middle.),
9

---

Published in Transactions on Machine Learning Research (01/2023)
which is in accordance with the fact that the optimal policy in Connect Four is deterministic policy. Our results are also in accordance with the findings in (Czarnecki et al., 2020) that this Connect Four game has a spinning top structure: when the policy is very weak (e.g. at 15M steps) or very strong (e.g. at 75M steps), there is little diversity; but within the intermodiate region of the policy strength spectrum (e.g. at 60M steps), there are many diverse policies; training against sufficiently diverse set of policies will lead to improved policy strength.
For comparison, Fig. 3g and Fig. 3h show the histograms of SET agent of ensemble size 10, where the SET agent's action probabilities only span few discrete values, implying less diversity compared against the IET agent at 60 million training steps. These results explain the superior testing reward achieved by the IET agent.
Connect Four
Connect Four
1.00
1.00
training reward
0.75
0.75
testing reward
0.50
0.50
0.25
0.25
Reward
Reward
0.00
0.00
-025
-0550
-0.0
-0755
training reward testing reward
-075
-1.00
-1.00
0
20
40
60
80
100
120
20
40
60
80
100
120
Million steps
Million steps
(a) IET agent reward
(b) SET (size 10) agent reward
Figure 2: Training and testing rewards of IET agent (left) and SET agent of ensemble size 10 in the Connect Four scenario.
4.6.2 Ablation on the levels of sharing
We investigale how the level of sharing within the multi-tasking network influences the sample-efficieccy As our approach is not restricted to the modular network architecture, for the convenience of ablation, we instead use a more intuitive multi-head multi-tasking network architecture consisting of L = 5 fully connected layers, where the first Leharing layers are shared and the last L - Leharing layers are independent for cach policy. We show in Fig. 4a, 4b, and 4d the exploitability and its area under the curve (AUC) of the joint policy when running the PSRO algorithm with the multi-tasking network of different sharing levels, With Leharing 0 (independent policies) corresponds to the standard PSRO, and Laharing 5 (identical policies) corresponds to self-play. We see that the best exploitability descent happens at Laharing = 2, while the two extremes (Twharing = 0,5) perform poorly. This observation suggests a trade-off between knowledge sharing (positive transfer) and loss of flexibility (negative transfer), which is commonly observed in multi- task learning. This ablation study also verifies our design purpose that the multi-tasking network in our implicit ensemble approach is responsible for improving the sample-efficiency via sharing parameters.
Table shows the scaling of network parameters and the representational power (policy diversity) of different approaches to parameterize the policy ensemble, where Kpollcy-net denotes the number of parameters of the policy network,, Khhaad and sharedd-eet denote the number of parameters of the policy head and shared network in the multi-head multi-task network, and N denotes the number of policies within the ensemble. For the standard simple ensemble approach that uses independent nelworks for each policy, the total nunber of parameters scales linearly w.r.t. the number of policy N. For the multi-head ensemble that uses independent heads with a shared base network to parameterize policies, the number of parameters also scales lincarly, where the scaling coefficient is Khead, For small Khead, the paraneter complexity is low. However, in this case, since the shared network will become the dominant part of this multi-head network, the diversity of the
10

---

Published in Transactions on Machine Learning Research (01/2023)
action probab Bay hesegrem
500
60-
action probabilty dappors
100
Action ricmng am
artinn protaddess vedempoe
meniumitty Pestrryom
action proball hrstouuam
probutsiity Trotoor
300
10-
200
on probubilty
Actin probabilty
(a) IET 15M training steps (empty board)
(b) IET 15M training steps
oction erobables vatoonam
ection arobutatly Fisteor
action grobablle ley sesprame
10
action probatedey tattggrrr
200
and aeckiyy Frstugg
action protuaiiiss instoguere
ext urchutikll Frotorr
Hoogas
100
1.-
0.0
0a
0.4
0.6
09
24
0.0
0.2
0.0
0.9
(c) IET 60M training steps (emply board)
(d) IET 60M training steps
indicially iiding
MO-
oction probab tey iattogamm
actioon probat liy stograas
360
200
Actionn andbaeiity ristogr am
ection brobutitly rittgg
Fistog
800
20
Ael protesedyy
Aetion protuctalyy
(e) LET 75M training steps (empty board)
(f) IET 75M training steps
100
action prosattey ddggro protuntes daagina oction probab foy attoorn gatogmam action dio sooppias
ection 6 ancaattyy Fistug am
00-
ection probatiiyy Pistogrin pr
Actioon ancowiiity rietog
180ggh
200
0.0
02
0.6
00
3.0
0.0
0.2
0
0.0
2.0
(g) SET (size 10) 60M training steps (empty board)
(h) SET (size 10) 60M training steps
Histogram of action probabilities corresponding to the rewards shown in Fig. 2 at snapshots of the IET agent (at 15, 60, 75 million training steps) and the SET agent of ensemble size 10 (at 60 million training steps): 3a, 3c, 3c and 3g show the histogram of action probabilities at the first time step of the game (empty board as in Connect Four); 3b, 3d, 3f and 3h show the histogram action probabilities accumulated across all the time steps of the game. IET agent's policy at 60M training steps covers a wide range of action probabilities.
Figure 3:
11

---

Published in Transactions on Machine Learning Research (01/2023)
tencanaless an sharing
poin afe
42-
tee cedtel sutte clanig wet-3
shaiing trvetid
Eco
fox seaees shering
0.0
8 g-435
pour shating
2.15
350
Sharma
(a) Exploitability of FSA
(b) AUC of FSA
(c) Exploitability of LH
(d) AUC of LH
Figure 4: Exploitability (interpreted distance from Nash Equilibrium (Lanctot et al., 2017)) and the area under the curve (AUC) of the joint policies learned through running PSRO with the multi-task network of different sharing levels. The optimal exploitability descent happens at medium sharing levels (Lsharing = 2). Scenarios First Sealed Auction (FSA) and Ledue Hold'em (LH) implementation from OpenSpiel lattps://gither www/deepmind//oeeeeeeeeeeeeeeeee hwtaapmiindoopnnsppee sppeel.
policy ensemble parameterized by this multi-head network is compromised. In contrast, for large Kheead the policy diversity is improved, but the parameter complexity scales poorly. For the implicit ensemble that uses a single shared network with a latent random variable to parameterize policy distribution, the paramcter complexity constant. The policy diversity arises from the randomness of the continuous latent variable. As result, the implicit ensemble can represent a continuous distribution of policies.
Table 7: Parameter complexity and policy diversity corresponding to various approaches of parameterizing policy ensemble
Approaches
Simple Ensemble
Multi-head Ensemble
Implicit Ensemble
Parameter complexity Krolicy--net (Linear) Kehead + Kahaard-not (Linear)
K Kashared-eet (Constant)
Policy diversity
N distinct policies
N distinct policies
Continuous distribution of policies
5
Related works
This work is at the intersection of robust MARL and multi-task RL.
5.1 Robust MARL
Ensemble training for robust MARL has been studied in many previous works such as (Lowe et al., 2017: Bansal et al., 2017; Jaderberg et al., 2017: 2019; Vinyals et al., 2019). These approaches focus on improving the robustness of the learned policies, regardless of the associated increase of computational complexity. In contrast, our work focuses on improving the efficiency of ensemble training without sacrificing robustness. Another approach for robust MARL is minimax policy optimization, in which each agent optimizes its policy assuming all the other agents and the environment dynamics are adversarial (see (Zhang el. al., 2020; Li al., 2019)). One difficulty with this approach is the requirement of solving the nested minimax optimization, which is typically approximated by optimizing the inner minimization for step only (Li et al., 2019). In addition, the minimax formulation tends to result in overly-conservaaive policies because of the pessimistic assumption about the other agents.
5.2 Multi-task Reinforcement learning
Our work is related to multi-task reinforcement learning (MTRL). MTRL aims to improve the learning efficiency by knowledge sharing across tasks (D (D'Eramo et al., 2019). Recent works (Huang et al., 2020; Yang et al., 2020; Devin et al., 2017) found that the modular policy network is an elfective archilecture for improving parameter--fficiencyy via sharing of learned modules. However, that MTRL work focuses on solving the single-agent multi-task problem. In contrast, our work leverages the multi-tasking network architecture combined with latent conditional policy learning (Haarnoja et al., 2018; Lee al., 2019) improve the efficiency of ensemble-based robust MARL. The innovation of our approach is re-formulating
12

---

Published in Transactions on Machine Learning Research (01/2023)
ensemble training multi-task learning while using latent variable to preserve policy diversity which is essential for robust MART.
5.3 regllai-re Reinforcement Learning
At first glance, approach has similarities to entropy regularized RL with Gaussian policy (Haarnoja et al., 2017; Wei ct al., 2018), which has the benefit of improved exploration. The key difference is that entropy regularized RL samples a random Gaussian noise at each time step, while our approach samples a policy from a contimuous policy space by sampling a latent Gaussian noise at the beginning of each episode and keeps it fixed across the whole episode. This difference analogous to the difference between a stochastic behavioral strategy and mixed strategy (Walker & Wooders, 2006) in game theory. Although adding randomness at each time step is helpful for smoothing the RL. learning objective to avoid local minimum (Ahmed et. al., 2019), it is not effective randomizing over the policy for creating diversity, which enables our IET approach to improve policy robustness.
5.4 Stochastic Neural Network
Stochastic neural networks such as networks with dropout layers (Gal & Ghahramani, 2016) and Osband et al. (2016) is another effcient way of parameterizing policy distribution. Compared with these approaches, our TET parametrization is more flexible and expressive. The dropout approach in (Gal & Ghahramani, 2016), for example, involves the dropout rate as a hyper-parameter, which determincs the diversity of the policy ensemble. As we discussed earlier, the required policy diversity for learning robust policy distribution in real-world games also evolves with the training stage and policy strength, which makes it difficult to select fixed dropout rate beforchand. In contrast, approach enables adaptive policy diversity that is arguably more suitable for learning in these settings.
6 Conclusions
This paper proposes IET, an implicit ensemble training approach that effectively reduces the computational complexity of ensemble training while maintaining the policy diversity for learning robust multiagent policies. In contrast to previous ensemble training approaches that roquire optimizing multiple policy networks, our IET approach optimizes a single shared network, which requires less compulation and memory. Numerical results show that our approach improves both the learning efficiency and the robustness of the learned policy,
We identify two promising future directions to extend training. Our current approach does not impose any regularization the learned representation of the latent conditional vector, which sometimes leads to mode collapse. A mutual information regularizer might be helpful to avoid mode collapse. 2) Our approach can be straightforwardly extend to scenarios where two teams of agents compete with each other. The latent vector can be shared across agents within the same team which acts as shared signal for team coordination to naturally emerge.
work: 1) Improving the stability of our IET
References
Zafarali Ahmed, Nicolas Le Roux, Mohammad Norouzi, and Dale Schuurmans. Understanding the impact of entropy on policy optimization. In International conference on machine learning, pp. 151 160. PMLR, 2019.
David Balduzzi, Karl Tuyls, Julien Pérolat, and Thore Graepel. Re-evaluating evaluation. abs/1806.02643,, 2018. URL http://aaxiv...g/bb///a// 02886.02643..
CoRR,
Trapit Bansal, Jakub Pachocki, Szymon Sidor, Tly Sutskever, and Igor Mordatch. Emergent complexity via multi-agent competition. arXiv preprint arXiv:1710.037//6 2017.
Wojeicch Marian Czarnecki, Gauthier Gidel, Brendan Traccy, Karl Tuyls, Shayegan Omidshafici, David Balduzzi, and Max Jaderberg. Real world games look like spinning tops. arXiv preprint or orXiv:2004.09463 2020.
13

---

Published in Transactions on Machine Learning Research (01/2023)
Carlo D'Eramo, Davide Tatco, Andrea Bonarini, Marcello Restelli, and Jan Peters. Sharing knowledge in multi-task deep reinforcement. learning. In International Conference on Learning Represenlations, 2019.
Coline Devin, Abhishek Gupta, Trevor Darrell, Pieter Abbeel, and Sergey Levine. Learning modular neural network policies for multi-task and multi-robot transfer. In 2017 IEEE international conference on robotics and aulomation (ICRA), pp. 2169 2176. IEEE, 2017.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp- 1050-1059. PMLR, 2016.
Adam Gleave, Michael Dennis, Cody Wild, Neel Kant, Sergey Levine, and Stuart Russell. Adversarial policies: Attacking deep reinforcement learning. arXiv preprint arXiv:1955.16111 2019.
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International Conference on Machine Learning, pp. 1852-1361. PMLR, 2017.
Tuomas Haarnoja, Kristian Hartikainen, Pieter Abbeel, and Sergey Levine. Latent space policies for hierar- chical reinforcement learning. In International Conference on Machine Learning, pp. 1851 1860. PMLR, 2018.
Charles A Holt and Alvin E Roth. The nash equilibrium: A perspective. Proceedings of the National Academy of Sciences, 101 (12):3999-44002 2004.
Wenlong Huang, Igor Mordatch, and Deepak Pathak. One policy to control them all: Shared modular policies for agent-agnostic control. In International Conference on Machine Learning, pp. 4455-4464. PMLR, 2020.
Shariq Iqbal and Fei Sha. Actor-attention-caitte for multi-agent reinforcement learning. In International Conference Machine Learning, pp. 2961-2970. PMTR, 2019.
Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, lain Dunning, Karen Simonyan, et al. Population based training of neural networks. arXiv preprinl ar 1 .09846, 2017.
Max Jaderberg, Wojciech Czarnecki, lain Dunning, Luke Marris, Guy Lever, Antonio Castaneda, Charles Beattie, Neil Rabinowitz, Ari Morcos, Avraham Ruderman, et al. Human-level performance in 3D multi- player games with population- reinforcement learning. Science, 364(6443):859--665 865, 2019.
Mare Lanciot, Vinicius Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Pérolat, David Silver, and Thore Graepel. A unified game-theoretic approach to multiagent reinforcement learning, anXiv preprint arXio:1711.008: 2017.
Alex X Lee, Anusha Nagabandi, Pieter Abbeel, and Sergey Levine. Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model. arXiv preprint arXiv:1907.00058, 2019.
Shihui Li, Yi Wu, Xinyue Cui, Honghua Dong, Fei Fang, and Stuart Russell. Robust multi-agent reinforce- ment learning via minimax deep deterministic policy gradient. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 4213 4213-4220, 2019.
Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg, Joseph Gonzalez, Michael Jordan, and Ion Stoica. Rllib: Abstractions for distributed reinforcement learning, In Internalional Conference on Machine Learning, pp. 3053 3062. PMLR, 2018.
Michael L Littman. Markov games as a framework for multi-agent reinforcement learning. In Machine learning proceedings 1994, pp. 157 163. Elsevier, 1994.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Picter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed copperative-compelttve environments. Xiv preprint arXio:1 106.0227, 2017.
14

---

Published in Transactions on Machine Learning Research (01/2023)
John F Nash et al. Equilibrium points in n-person games. Proceedings of the national academy of sciences, 36(1):48 49, 1950.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. Advonces in newral information processing systems, 29, 2016.
Janith C Petangoda, Sergio Pascual-Diaz, Vincent Adam, Peter Vranex, and Jordi Grau-Moya. Disentangled skill embeddings for reinforcement learning, arXiv preprint arXit:1906. 09223, 2019.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Toannis Antonoglou, Veda Panneershelvam,, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484 489, 2016.
Chuangchuang Sun, Dong-Ki Kim, and Jonathan P How. Romax: Certifiably robust deep multiagent reinforcement learning via convex relaxation. arXiv preprint ar arXiv:200000679,, 2021.
Justin K Terry, Benjamin Black, Ananth Hari, Luis Santos, Clemens Dieffendahl, Niall L. Williams, Yashas Lokesh, Caroline Horsch, and Praveen Ravi. Pettingzoo: Gym for multi-agent reinforcement learning. arXiv preprint (rXtc:2009.14470 2020.
Karl Tuyls, Julien Perolat, Marc Lanctot, Edward Hughes, Richard Everctt, Joel Leibo, Csaba Szepesvári, and Thore Graepel. Bounds and dynamics for empirical game theoretic analysis. Aulonomous Agents and Multi-Agenu Systems, 34(1):1-30) 30, 2020.
Eugene Vinitsky, Yuqing Du, Kanaad Parvate, Kathy Jang, Pieter Abbeel, and Alexandre Bayen. Robust reinforcement learning using adversarial populations. arXiv preprint ar Xrx:20008. 01825, 2020.
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):350-23 354, 2019.
Mark Walker and John Wooders. Mixed strategy equilibrium. Retrieved December, 19:2006, 2006.
Ermo Wei, Drew Wicke, David Freelan, and Sean Luke. Multiagent soft q-learning. arXiv preprint 1rXiv:1804.000 008117 2018.
Ruihan Yang, Huazhe Xu, Yi Wu, and Xiaolong Wang. Multi-task reinforcement learning with soft modu- larization. arxiv preprint ar Xrv:::0003, 2020.
Daochen Zha, Kwei-Herng Lai, Yuanpu Cao, Songyi Huang, Ruzhe Wei, Junyu Guo, and Xia Hu. Rleard: A toolkit for reinforcement learning in card games. arXiv preprint xrXiv:1100.00 910.04336, 2019.
Kaiqing Zhang, Tao Sun, Yunzhe Tao, Sahika Genc, Sumil Mallya, and Tamer Basar. Robust multi-agent reinforcement learning with model uncertainty. Advances in Neural Information Processing Systems, 33, 2020.
A Appendix A
We use the soft modular network proposed in (Yang et al., 2020) as the multi-tasking sub-network in our IET approach. The modular network is composed of base network and routing network. The base network is a modular network that has n layers, and each layer has m modules. Each module is a feedforward network, and the input and the output are d-dimension vectors. The routing network takes the observation and the latent condition variable as inputs, and it then outputs the n normalized weight vectors, one for cach layer, for weighting the modules of the base policy network. The weight vectors w E Rm" calculated as
-== = (F H (c))). +1 w (a (Wip' (F (c))))
(6)
w = Softmax(p'),
15

---

Published in Transactions on Machine Learning Research (01/2023)
where F and H. are the embedding layers, which map the observation vector and the latent condition variable into D-dimension embeddings. a is the activation function (we use the ReLU activation). W,, and Wa fully-connected layers of size Dxm* and R= respectively.
The base network takes the observation vector as input and outputs policy logits value, with the following relationship between the i-th module in the j ++1 layer's input and the k-th module in the j-th layer's output,
f = Σ w (o (w7 (w?ft))
m
(7)
where w/ E Raxd is the learnable module parameters.
16


Reviews:
[{"review_id": "5xyBZWRUFj", "reviewer": "Reviewer_Cc15", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n* The introduction to MARL was well-written\n* The problem statement was quite clear and well-motivated -- I agree that we need more compute-efficient ensembles\n* The approach chosen seems simple to implement\n## Weaknesses\n\n1) Everything in your problem framing suggests that IET shouldn't help training time performance: a single policy as as expressive for best-response. IET should *only* help for notions of robustness and generalization. However, your results show that IET does significantly boost performance during training (perhaps more so than at test time...). This can be explained by the large differences between the network architecture of IET and the baseline methods. Ideally, all methods should use the same architecture. Since the modular architecture used by IET is inherently multi-task, perhaps keeping the conditioning variable constant for baseline methods would be sufficient.\n\n2) Prior work has shown the general difficulty in learning diverse sets of policies, even in the single-agent regime (e.g. \"Diversity is all you need\"). I'm highly credulous that merely conditioning on a random variable will yield policies that are diverse wrt that random without any explicit objective pushing it to do so (e.g. the mutual information between the random variable and the resultant behavior). The only evidence you show for policy diversity is the increased performance (which could have other causes e.g. 1) ) and the T-SNE plots of the random variable in different games. Interpreting the complexity of T-SNE images is already a fraught undertaking, but even if we assume the conditioning variable is more diverse in harder environments, that isn't what we care about. We care about policy diversity: the extent to which a change in the conditioning yields a meaningful change in behavior. Without such evidence, how can we be sure that your explanation of the performance improvements is correct?\n\n3) The experimental results could be better formatted. The top and bottom rows of Figures 2-4 should be redundant, since one player's performance fully determines the performance of the other player, so the bottom rows should be removed. I'm a bit confused why they aren't perfectly symmetric e.g shouldn't Figure 3a bottom just be the negation of Figure 3a top? The axes aren't shared across columns, making cross-method comparison difficult -- why not just have a single figure comparing IET to both baselines?\n\n4) How does the 4.6.2 tie into the rest of the paper? Cross player parameter sharing is another way to reduce computation, but it seems orthogonal to IET and I'd suggest removing it.\n\n5) Ablate aspects of your approach. Specifically, IET without modular networks and IET without the shaping network. Choices for the input noise (why Gaussian, not categorical?) and shaping network output activation should at least be explained and potentially ablated.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors introduce a more compute-efficient way to induce policy diversity in agents and show that these agents show increased robustness in the face of novel players in multi-agents games (MARL). Specifically, they introduce implicit ensemble training (IET), which parameterize an ensemble as a goal-conditioned network with the goal coming from a source of external noise. They compare this approach to single policy agents, simple ensembles (i.e. N discrete networks), and adversarial trained agents on several two player adversarial games. They also analyze the effect of cross-player parameter sharing, though how this connects to their primary contribution is somewhat unclear.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "level of criticality: ** -> **** (least to most critically important for acceptance)\n\n**** Evidence that IET helps test-time performance when controlling for train-time performance\n\n*** Evidence that IET yields a diverse set of policies, at least comparable to explicit ensemble methods.\n\n** Replace 4.6.2. with ablations of IET\n\n** Remove the T-SNE stuff or explain how your interpretation is justified\n\n** Improved experimental presentation as per 3)", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tNt2dnA5gS", "reviewer": "Reviewer_obGm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- (Simplicity of the Approach) The paper proposes a simple and reasonable approach to achieve robustness to distribution mismatches while preserving sample and computation efficiency;\n- (Experimental Analysis) The experimental analysis includes various domains and a rich set of metrics, illustrations, and ablations;\n- (Clarity of the Presentation) The paper is well-written and easy to follow;\n- (Related Work) The paper does a good job in relating the presented contributions with previous works.\n\nWeaknesses\n- (Significance of the Experiments) The statistical significance of the reported results is mostly unclear. The text does not explicitly state the number of seeds employed in the experiments, as well as the meaning of shaded areas in the plots, performance ranges in the tables;\n- (Clarity of the Results) The experimental results are not always strong. The experimental section tries to convey too many information instead of focusing on a few crucial results.\n\nGeneral Comment\n\nThe paper tackles a relevant problem with a very reasonable approach, by borrowing ideas from multi-task learning to improve the efficiency of ensemble training. Having read the first three sections I was sold on the value of the methodology and the benefits it can provide. However, the experimental section is slightly underwhelming, as it struggles to clearly validate the promises of the previous sections, perhaps trying to convey too many information. While I am not an expert in the multi-agent domain, I think this paper provides a valuable contribution overall, even if the paper could be improved in a couple of aspects (more details below).\n\nMinor Comments\n- In Figure 2, it is not immediately clear what is changing between the top row and the bottom row of plots;\n- There is a duplicated reference for (Czarnecki, 2020).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses the problem of robustness to distribution mismatches between training and testing in multi-agent settings. Previous works demonstrated the effectiveness of ensemble training to overcome the distribution mismatch, but the computation and memory complexity scale linearly with the size of the ensemble. This paper proposes an implicit ensemble training approach, called IET, in which a single policy architecture is conditioned on a latent variable to produce diverse behaviors, effectively simulating an ensemble of policies. This simple procedure is evaluated on a series of board games and continuous control tasks, in which IET shows robustness to distribution mismatches and improved sample efficiency thanks to parameters sharing.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Critical\n\n- (Significance of the Results) I encourage the authors to detail the number of seeds employed in the experiments, and to comment the statistical significance of the reported results.\n\nSomewhat Critical\n\n- (Main Experimental Result) To my understanding, the main message the paper is trying to convey is that IET could provide similar robustness w.r.t. distribution mismatches as training with large ensembles, while avoiding the computation and memory blow-up. However, the experimental analysis only provides a comparison with a rather small ensemble (of three policies). I would suggest the authors to center their experimental section on a main experiment that compares the robustness achieved by IET with standard ensembles of increasing size. It would be especially crucial to show how many policies are needed to match the robustness of IET, while detailing the computation and memory requirement of the explicit ensemble.\n\nStrengthening the Work\n\n- (Analysis of the End-to-End Training) I am not completely convinced it is safe to train the diverse module end-to-end. What is preventing the diverse module to collapse to a Dirac delta, especially if the game admits a pure strategy equilibrium? I think that a deeper discussion on the pros/cons of end-to-end training could nicely complement the presentation of the IET approach.\n\n- (Minmax Comparison) I was actually expecting the comparison against the MT baseline to act as a sanity check, i.e., to show that IET is not far off the robustness that can be achieved by an \"ideal\" method, while being much more efficient in practice. Instead, the comparison against MT is quite unclear, and it seems to underline the flaws of the baseline rather than the merits of IET. I would suggest the author to provide a comparison in a much simpler domain where MT can perform well, or to move the comparison to the appendix to free some space for more interesting results.", "broader_impact_concerns": "I do not believe this paper requires to explicitly address the potential ethical implications of the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Tqu4Q8rkN7", "reviewer": "Reviewer_mj1G", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n●\tA novel solution to a real problem in ensemble methods is proposed. By proposing a single model that represents different agents dependent on the sampled latent variable, they are able to mitigate the major issue of computation scaling linearly with individual models for each ensemble agent.\n\n●\tMuch of the paper is well-written and also well-motivated.\n\n●\tThere has been a good initial attempt at the evaluation. For example, I believe the selection of environments is reasonably complex and I found the ablation studies, in particular on the levels of sharing, to be thought-provoking for the usage of single-models in ensemble methods.\n\n**Weaknesses**\n\n●\tIn section 3.4 there is discussion on how the latent vector can be used in the mixed cooperative-competitive setting in which two teams of agents compete against each other. However, there is no example of this in the evaluation section. In my opinion, if it could be shown that the IET method is able to produce effective coordination between agents in a team then this would be greatly increase the benefit and appeal provided by IET.\n\n●\tThere are aspects of the conditional variable that are unclear to me. Mainly, the sampling of the latent vector and the outputted latent condition variable do not seem to have any relevance to the observations of the environment nor the opponent that is being trained against. Understandably, I may not be fully grasping what is happening - but my instinct would be that the conditioning variable that essentially dictates how the agent is going to play in the environment should be influenced by the opponent in some way. Please could you clarify where I may be incorrect, or how the conditioning variable is influenced by the opponent?\n\n●\tIn my opinion, the related work and how this work fits in misses two critical areas. \n\n●\tFirstly, the discussion with respect to PSRO methods is limited and does not mention any examples of extensions to PSRO that aim to deal with scalability problems. For example - P2SRO [McAleer et al., 2020], NAC [Feng et al., 2021]. In addition, I view PSRO style methods as the major competitor to this approach in terms of performance and would expect there to, in general, to have been a wider discussion to PSRO methods that are not also just focused on improved scalability - Rectified PSRO [Balduzzi et al., 2019], Diverse PSRO [Nieves et al., 2021] JPSRO [Marris et al., 2021]\n\n●\tSecondly, NeuPL [Liu et al., 2022] was introduced recently and I am surprised this has not been discussed by this work. NeuPL additionally utilises a single conditional model that conditions on the strategy of the opponent in order to improve the transfer of skills between agents. It would be great to hear the authors thoughts on how IET improves over NeuPL and also maybe falls short of NeuPL. \n\n●\tIn my opinion, the evaluation section is not extensive enough in terms of baselines. \n\n●\tFirstly, a similar weakness to the one above but I think the major competitor to IET is PSRO methods. It would be great to see an environment where the amount of PSRO policies needed becomes intractable, whilst IET is able to handle the problem due to its ability to represent a larger amount of policies more efficiently.\n\n●\tSimilarly again, it would be great to see a comparison to NeuPL if feasible.\n\n●\tIn terms of the Simple Ensemble Training, the setup seems rather basic. For example, an ensemble size of 3 feels small in comparison to the amount of diversity that I believe IET can show, but also why is uniform sampling over the ensemble utilised? Is not using something like the Nash distribution over the policies more standard, and potentially a fairer comparison?\n\n●\tPersonally, I found figures 2,3,4 confusing to understand most notably due to their poor legends.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the topic ensemble training, which is an effective approach for learning robust policies that avoid significant performance degradation when competing against previously unseen opponents, since a large ensemble can improve diversity during the training, which leads to more robust learning. This paper proposes a parameterization trick of a policy ensemble based on a deep latent variable model with a multi-task network architecture, and demonstrate in several competitive multiagent scenarios in the board game and robotic domains that our new approach improves robustness against unseen adversarial opponents while achieving higher sample-efficiency and less computation. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "●\tImprovements to the related work section as mentioned in the weaknesses.\n●\tUpdate figures 2,3,4 for improved clarity about what each line represents and make it clearer why these figures show good performance of IET.\n●\tImprovements to the baselines used as mentioned in the weaknesses.\n●\tExperiment looking at team settings in mixed competitive cooperative environments. \n", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies how to address distribution robustness for multi-agent RL (MARL) setting against novel opponents, and proposes to replace network ensembles, which could be computationally expensive to scale, with a single latent conditioned policy with external noise. 

Two reviewers recommended "leaning accept" while one reviewer recommended "leaning reject". A "leaning accept" also expresses the decision to be borderline. Given the recommendations, my proposal is "Accept", mainly for:

- this is a simple easy-to-understand/apply method with clear audience (MARL). other papers may take this method as future baselines.
- reviewers appear satisfied with the efforts in the rebuttal: e.g. Reviewer Cc15's many feedback were addressed with new experiments (e.g. Jacobian norm), Reviewer mj1G's feedback on lack of larger ensembles as baselines was addressed with new experiment with 5/10 ensembles and additional related work were discussed, and Reviewer obGm's concern on statistical significance is partially addressed. 

However, this "Accept" is borderline, and there are worries about the work. Please provide these two additional revisions:
- statistical significance of the results: the results are quite noisy and the whole validity of method/domain seems to be questioned by Reviewer obGm recommending "leaning reject" as well as Reviewer Cc15 recommending "leaning accept" with a careful note. Could you follow up with additional experiments to engage in convincing these two more about statistical significance of the results? e.g. even running on more seeds or qualify the claims. 
- lack of discussion on literatures on efficient ensemble: Replacing ensembles with a stochastic network is not novel. It has been debated/studied in many papers on Bayesian deep learning (sometimes applied to sparse-reward exploration problems in RL), e.g. dropout [1] vs bootstrapped DQN [2]. These papers and other work should be referenced and added to the related work section. 

[1] Gal, Yarin, and Zoubin Ghahramani. "Dropout as a bayesian approximation: Representing model uncertainty in deep learning." international conference on machine learning. PMLR, 2016.

[2] Osband, Ian, et al. "Deep exploration via bootstrapped DQN." Advances in neural information processing systems 29 (2016).

Decision: Accept (poster)