Joint-Predictive Representations for Multi-Agent Reinforcement Learning
Authors: Mingxiao Feng
Wengang Zhou
Yaodong Yang
Houqiang Li
Venue: Submitted to ICLR 2023 (2023)

Abstract: The recent advances in reinforcement learning have demonstrated the effectiveness of vision-based self-supervised learning (SSL). However, the main efforts on this direction have been paid on single-agent setting, making multi-agent reinforcement learning~(MARL) lags thus far. There are two significant obstacles that prevent applying off-the-shelf SSL approaches with MARL on a partially observable multi-agent system : (a) each agent only gets a partial observation, and (b) previous SSL approaches only take consistent temporal representations into account, while ignoring the characterization that captures the interaction and fusion among agents. In this paper, we propose \textbf{M}ulti-\textbf{A}gent  \textbf{Jo}int-Predictive \textbf{R}epresentations~(MAJOR), a novel framework to explore self-supervised learning on cooperative MARL. Specifically, we treat the latent representations of local observations of all agents as the sequence of masked contexts of the global state, and we then learn effective representations by predicting the future latent representations for each agent with the help of the agent-level information interactions in a joint transition model. We have conducted extensive experiments on wide-range MARL environments, including both vision-based and state-based scenarios, and show that our proposed MAJOR achieves superior asymptotic performance and sample efficiency against other state-of-the-art methods.

Under review as a conference paper at ICLR 2023
JOINT-PREDICTIVE REPRESENTATIONS FOR MULTI- AGENT REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
The recent advances in reinforcement learning have demonstrated the effective- ness of vision-based seff-supervised learning (SSL). However, the main efforts on this direction have been paid on singlc-agent setting, making multi-agent re- inforcement learning (MARL. lags thus far. There are two significant obstacles that prevent applying off-the-shelf SSL. approaches with MARL a partially observable multi-agent system (a) each agent only gets a partial observation, and (b) previous SSL approaches only take consistent temporal representations into account, while ignoring the characterization that captures the interaction and fusion among agents. In this paper, we propose Multi-Agen Representations (MAJOR), a novel framework to explore self- superviee learning cooperative MARL. Specifically, servations of all agents as the sequence of masked contexts of the global state, and we then learn effective representations by predicting the future latent representa- tions for each agent with the help of the agent-level information interactions in a joint transition model. We have conducted extensive experiments on wide-range MARL environments, including both vision-based and state-based scenarios, and show that our proposed MAJOR achieves superior asymptotic performance and sample efficiency against other state-of-the-ard methods.
treat the latent representations of local ob-
INTRODUCTION
Representation learning has played an important role in recent developments of reinforcement learn- ing (RI.) algorithms. Especially self-supervised representation learning (SSL.) has attracted more and more attention due to its success in other research fields (He et al., 2019; Devlin et al., 2018; 88; Liu et al., 2019; Lan et al., 2019). Recently, numerous works(Srinivas et al., 2020; Zhu et al., 2022; Yarats et al., 2021; Schwarzer et al., 2021a; Yu et al., 2022) have borrowed insights from differ- ent areas and attempted to use SSL-based auxiliary tasks to learn more effective representations of RL and thus improve the empirical performance. Through augmentation for inputs, it can conduct multiple views for building SSL learning objectives, allowing the agent to improve data efficiency and generalization obtain better task-related representations. Moreover, many proper auxiliary self-supervision priors proposed predictive representations with the help of an additional learned dynamic model, which is utilized to encourage the representations to be temporally predictive and consistent.
However, when meeting partially observable multi-agent systems, it is challenging to apply such self-supervision priors to learn compact and informative feature representations in multi-agent rein- forcement learning (MARL). The critical obstacle to learning effective representations in MARL is that agents in partially observable multi-agent systems only have access to their observations, which means that other agents' behavior influences each agent's observations. As a result, independently building representation priors for each agent may be failed due to imperfect information. Further- more, in the MARL context, it is more important to focus on the representations that embody the interaction and fusion between agents in the environment but not temporal representations for each agent. In other words, it is necessary to learn the representations that can take the other agents into account.
In this work, we propose novel representation learning framework for MARL, named Multi-Agent Joint-Predietive Representations (MAJOR), which trains better representations for MARL by forc-

---

Under review as a conference paper at ICLR 2023
ing representations to be temporally predictive and consistent for all agents at the same timestep. We posit that in one timestep, latent representations of local observations of all agents can be treated as the sequence of masked contexts of the global state that can predict the future latent represen- tations for each agent with the representations of corresponding actions. Accordingly, we construct joint transition model the bridge connect all agents and implement the interaction of their individual information. The joint transition model treats the encoded representations of individual observations and actions as a sequence and attempts to predict future representations latent space. Meanwhile, we can also get another view of the subsequent timestep representations by feeding sampled observations into the encoder. In this way, we build the SSL objective by enforcing con- sisteney across different perspectives of each observation. Besides, our proposed framework is plug-and-play module for almost common-used MARL methods.
Additionally, to maximize its power, we implement an instantiation of MAJOR on the basis of the recently proposed MARL algorithm, named Multi-Agent Transformer (MAT, (Wen et al., 2022)), which solves MARL issues via sequential updating mechanism and encoder-decoder architec- ture. In MAT, the encoder seeks to extract post-interaction representations from observations; the decoder then uses them to generate actions sequentially by cross-attention mechanism. Moreover, our proposed MAJOR can employ both representations generated from the encoder and decoder, and the gradient derived from our representation learning objective can be back-propagated both the encoder and decoder.
To evaluate proposed algorithm, construct extensive experiments several common-used cooperative MARL benchmarks, including vision- and state-based environments in discrete and continuous scenarios against current state-of-the-art baselines such HAPPO, MAPPO, and MAT. Results demonstrate its state-of-the-art performance across all tested tasks.
2 BACKGROUND
2.1 DEC-POMDP
Cooperative MARL problems are often modeled by decentralized Partially Observable Markov De- cision Processes Decc-POMDPs, (Olichock & Amato, 2016)) (W, S, {A,). T, , R, Q, O,7). Here, N V= 1....... is the set of agents, S is a set of states, A X,A, is the set of joint actions, T is set of conditional transition probabilities between states, T(s, a, s') = P (s' s, a). .. R S x + R is the reward function, @ x,O; is set of observations for agent f2 is set of conditional obser- vation probabilities Ω (s', a. o) = P (o 8', a), and 7 E [0, 1] is the discount factor. At each time step, each agent takes an action a,, and the state is updated based on the transition function (using the current state and the joint action). Each agent observes an observation based on the observation function (s', a, o) (using the next state and the joint action) and reward is generated for the entire team based on the reward function R(s, a). The goal is to maximize the expected cumulative reward over a finite or infinite number of steps.
2.2 MULTI-AGENT TRANSFORMER
Multi-Agent Transformer (MAT, Wen al. (2022)) effectively casts cooperative MARL into Se- quential Modeling (SM) problems wherein the task is to map the observation sequence of agents the optimal action sequence of agents. Its sequential update scheme built on the Multi-Agent Advantage Decomposition Theorem (Kuba et al., 2021) and MeteroggneeousAAgen Proximal Policy Optimization (HAPPO, Kuba al. (2022)). The lemma provides an intuition guiding the choice of incrementally improving actions, and HAPPO fully leverages the lemma to implement multi-agent trust-region learning with monotonic improvement guarantee. Unfortunately. HAPPO requests the sequential update scheme in the permutation for agents' orders, meaning that HAPPO cannot be run in parallel. To address the drawback of HAPPO, MAT produces Transformer-basel implementation for multi-agent trust-region learning.
Concretely, MAT maintains an encoder structure where the encoder maps an input sequence of tokens latent representations. Then the decoder generates sequence of desired outputs in an ruto-regressive manner wherein, at each step of inference, the Transformer takes all previously generated tokens as the input. In other words, MAT' treats team of agents sequence, thus
2

---

Under review as a conference paper at ICLR 2023
implementing the sequence-modee modeling paradigm for MARL. The encoder takes sequence of ob- servations - (o"'..... ..... o²n) in arbitrary order and passes them through L computational blocks. Each of these blocks has self-attention mechanism{ Sathanismmaaswaaii et al., 2017) and a multi-layer percep- tron (MLP), well residual connections to prevent gradient vanishing and network degradation as depth increases. Thus we can obtain the encoding of the observations as â containing interrela- tionships among agents. Feeding the representations into the value head (an MLP), denoted as fa- will get value estimations. The encoder's learning objective is to minimize the individual version of empirical Bellman error by:
MA enteereder (0:) Tn
R(s.a)) +Va (0/T1) Vol (ott)) )]
where o' is the target network, which is nondifferentiable and updated every few epochs. Mean- while, the decoder passes the embedding joint action to sequence of decoding blocks. Crucially, the decoding block replaces the encoder's self-attention mechanism with masked self-attention mechanism: i... the attention of the action to be generated in the current step is computed only among previously computed agents' actions. The output of the last decoder block is sequence of representations of the joint actions. The same as the value head, this is fed into a policy head (also an MLP), denoted as fe. which outputs the policy Tig (atm ōtt a (2:m-1)) Besides, the decoder's learning objective is minimize the clipping PPO objective proposed in HAPPO of
MAT L
(Ol, az)
1
min (r.- (0) clip (r7T (0), 1 A.)
(2)
T'
m=1 t=
where r(r((e) "6" (a/m 10 10 $1-m1)
and A₁ is an estimation of the joint advantage function,
Radd (a:m of..0 a. f1:m 1)
c.g. GAE(Schulman et al., 2015).
OUR METHOD
3.1 CONNECTION BETWEEN JOINT TRANSITION MODEL AND SSL IN MARL
To encourage the observations and actions representations to integrate agent-level relationships and interactions, we exploit a multi-agent world model in latent space as the bridge to connect both observation representations and action representations of overall agents. Naturally, in the formulation of Dec-POMDP, one can feed the global state with joint actions into the world model T to predict the next state, i.e. implement T (8, a') P (s' 8, a). In this work, we learn transformer-based joint transition model T as an approximate version of the multi-agent world model, whose inputs are joint observation representations and joint action representations, and its outputs are the joint observation representations in the next timestep.
described
On the one hand, to achieve the goal of implementing information fusion among agents, it is better to use individual observations rather than global state as input. As a result, will enforce the learned joint transition model to (a) reconstruct the global state from individual observations, (b) predict the future state of the next timestep, and (c) implement the observation mapping functions for cach agent. In this way, the joint transition model must infer the influences caused by others and try to integrate all the imperfect information. As a result, executing consistency across different views of individual observations can lead to better representations generated from encoder networks.
On the other hand, it is necessary to emphasize that we treat the individual observations as a sequence of masked contexts of global state in the joint transition model. Based on the definition of Dec-POMDP, observations are the outputs of the mapping function; in other orders. individual observations can be seen as multiple views of the global state. This property revealed by Dec- POMDP's formulation in cooperative MARL inspires us to propose self-supervised prior tailored MARL. We use the joint transition model as a Masked Language Model (MLM, Devlin al. (2018); Lewis et al. (2019)) under core components in MARL, which is general in all MARL settings.
Becauue the algorithm makes assumptions about the order of the agents and don't 't change the original order, we subsequently omit the superseript about the agents' order for the sake of simplicity of presentation.
3

---

Under review as a conference paper at ICLR 2023
3.2 MULTI-AGENT
REPRESENTATIONS
Multi-Agent Joint-Predictive Representations(MAJORR is an auxiliary objective to promote repre- sentation learning in MARL and is generally applicable different MARL algorithms, e.g., MAT and MAPPO. The core process in the joint transition model of MAJOR is to implicitly recon- struct the global state and then infer the future observation representation of each agent. This mechanism enables the better use of agent-cross information when learning observation and action representations, further enhancing the understanding of MARL agents for individual messages. The overall framework of MAJOR is shown in Figure 1. We will introduce the components shown in the framework the following subsections.
Observation Encoder
Value Head
4.
Action Decoder
daTneeeer e.fo
Policy Head
Joint Transition Model
woving
Exponantial Lot.
Online Projection
Onl Prediction
PeelicN
Exponential roving oig.
Stop gradient
Target Observation
Target
Ft-1ESK
Encoder
Projection
An illustration of the full MAJOR method. representations from the online observation encoder are used in the action decoder, value head, and policy head. They will also be utilized for the prediction of future representations from the target observation encoder via the joint transition model. The target observation encoder and projection head are defined as an exponential moving average of their online version and are not updated via gradient descent. For brevity, we illustrate the iterative K K-steps results described in Equation 6 for the joint transition model totally in one loop.
Figure 1:
We use MAT's encoder & and decoder 6, excluding the fi- nal MLP individual, as the online observation encoder and action encoder, respectively. Taking the observation sequence of arbitrary order as input, the online observation encoder applies a self- attention mechanism and obtains post-interaction representations of agents, as o, Similarly to online observation encoder, the online action encoder accepts both the origin action sequence a and observation representations fat and the output action representations à, through the cross- attention mechanism. We employ these representations with a goal that motivates them to forecast future observation representations up to given temporal offset K, iteratively. Following prior work (Schwarzer et al., 2021a; Zhu et al., 2022; Yu et al., 202 2022), we utilize another observa- tion encoder for the encoding of original observations. This target encoder has the same architecture as the online observation encoder, and its parameters are an exponential moving average (EMA) of the online observation encoder parameters. Denoting the target observation encoder ø and the momentum coefficient as [0, L), the update scheme of the target observation encoder is:
Encoding observations and actions:
- r + - 7
Joint Transition Model: We construct the predictive version of future observation representations using a transformer-basee joint transition model ग. The architecture of the joint transition model is similar to the online encoder, but (a) contains L' Multi-Heas Head masks and (b) takes the combined sequence of observations and action representations as input tokens and then only outputs the sequence of observation representations of the subsequent timestep.
(MHSA) layers without
4

---

Under review as a conference paper at ICLR 2023
We obtain the t-th observation and action representations by feeding the origin observation sequence and the action sequence into the online observation encoder and the action encoder. Then the input tokens of the latent joint transition model can be mathematically represented as
a | ai ...... of" a
(4)
The process of passing the token sequence through the / th layer of the joint transition model can be mathematically described as:
h' MHSA (LN (z')) + w'; = FFN (LN (h')) + h²'
(5)
Here, LN and FFN denote the LayerNorm and the Feed-Forward Network mentioned in (Vaswani et al., 2017). Note that if the permutation order is known, one can also add agent ids' embedding and positional embedding on æ. And we only select the odd elements of the output tokens of the joint transition model as the corresponding predictive results for the latent future representations inferred from previous observation and action representations. Furthermore, in the k-th step of generating future representations where k 2, K, we use the joint transition model output representations instead of the online observation encoders as the input latent observation tokens. The process mentioned above can be denoted as
ōi+1 - T (ō;, āe), where ō - Φ (or). and à; a (Ôr. a.)
(6)
Ōi+k - (0.+R-1, 0x4k-1) where 0++k-1 -
4k: = 2, 2,...,
The joint transition model and prediction loss operate in the latent space, thus avoiding pixel-based reconstruction objectives and making MAJOR robust for vision-based and state-based MARL set- tings. the following, we elaborate on the predictive loss between the prediction results and the corresponding targets.
Motivated by the success of BYO (Grill et al., 2020) in SSL and sample-efficient RI((chwarzer al., 2021 a; Yu al., 2021b; 2022), compute the future prediction loss of MAJOR by calculating the cosine similarities between the predicted and observed representations. Concretely, from the outputs of the joint transition model, i.e. the sequence of observation rep- resentations set Ōt+1=+K, we use a projection head g and a prediction head q to obtain the final sequence of predictions result in ÿt-1:t+K - ((9(0 head 9 (i.e. follows the same EMA update strategy the target observation encoder) to process the encoded results of original observations, which is denoted as Wt+1:t+K & g(0++1=+k) where Oc+1æt- K A c(0++1:++)) Here, we apply a stop-gradient operation as illustrated Figure/ to avoid model collapse, following BYOL. Finally, MAJOR's 's objective is to enforce the final predic- tion result in Ut+1:t+K to as close to its corresponding target yt+1:t+K. And we construct the following cosine similarities between the normalized predictions and the target projections overall agents and the offset timesteps:
Prediction Loss:
Then we utilize target projection
1
n
Vith Tittalll
Fitk Di x
CMAIOR 6
(OELK Qc(KK-1)
Kn
Total learning objective: The proposed MAJOR is auxiliary task that optimized in conjunction with MAT. Therefore, the overall loss function is:
Liotal MMAT encoder 6.fo
+ -
dMATdecoder
MAJOR
(8)
where M :MATencoder MAToceder and MAAIRR are the MAT losses mentioned above and our pro- 0.0
oo f.
posed joint-predictive representations learning objective, respectively. À is hyperparameter for balancing the items. It is worth noting that, unlike other suggested SSL algorithms in CV and RI,, MAJOR can be employed with or without data augmentation, especially in situations where data augmentation is unavailable or counterproductive. Moreover, MAJOR mainly focuses on capturing the relationships among agents via the joint transition model. The proposed framework can also be transferred to other MARL algorithms that follow the centralized training decentralized execution (CTDE) paradigm, such MAPPO(Yu et al., 202 202la/HAPPOOKube et al., 2022) and QMIX(Rashii et al., 201 (OLE et al., 2020), etc. Here, we provide the pseudocode of MAJOR in Algo- rithm 1.
5

---

Under review as a conference paper at ICLR 2023
Algorithm Multi-Agent Joint-Predictive Representations
1: Input: Stepsize number of agents episodes K, steps per episode T.
2: Initialize: Observation encoder Φ. Action decoder O, Value head fs. Policy head So. Replay buffer B, Joint transition model T, Online projection head g, Online prediction head q, Target observation Encoder Φ. Target projection head g.
3: while Training do
4: Sample minibatch (ot, ae, Tt. OL+1) B.
5: Calculate CMAAee (o:) with Equation (1).
Lo.fe
6:
Calculate @MATDecude (òc, a.) with Equation (2).
7:
Sample another sequential minibatch (Our4 Qn:tKK1)) ~ B.
8:
Get pnojection/prediction representations Dt++1:t+K K and 1t+144 K with Equation (6).
9:
Calculate CMAJOR (OL:+ Qt.+KK1) with Equation (7)
10:
Update the encoder/decoder, value/policy head, joint transition model, and online projec- tion/prediction head by minimizing frotal with gradient descent according to Equation (8).
11:
Update target observation encoder and target projection head with Equation (3)
12:
end while
3.3 IMPLEMENT DETAILS FOR MAJOR
In practice, we define a hyperparameter H to control the update frequency of MAJOR's 's loss, w.r.t. MAT's loss. In other words, we update MAJOR's learning objective every certain number of it- erations of the original MAT optimization strategy. Concretely, we sample unique batch of B' samples from the trajectories collected using the latest policy. Each sample consists of temporal set of observations and the corresponding actions, that is, (Ora+K ar.t4 K). For the projection and prediction head, we do not use BatchNorm Layer and replace ReLU with GELU activation units, which is different from what B BYOL did. In addition, our joint transition model's 's architecture is the same as the encoder used in MAT. In vision-based settings, we use three convolutional layers with a ReLU layer after each convolutional layer, which is the same as DQN's, as the feature extractor in all algorithms. The full hyperparameters of MAJOR can be found in Appendix F.
4 RELATED WORK
4.1 OVERVIEW OF REPRESENTATION LEARNING AND SULESSUPERVISED LEARNING
Self-supervised learning empowers us exploit a variety of labels that come with the data for free. With self-supervised learning, we can utilize inexpensive unlabeled data and establish the learning objectives properly from designed pretexts gain supervision from the data itself. SSL has been developed in CV and NLP areas and can be divided into the various self-supervise pretexts in the literature into four broad (Erisies et al., 2022): Masked Prediction, Transformation Predic- tion, Instance Discrimination, and Clustering. (1) Masked Prediction methods (Mikolov et al., 2013: Baevski et al., 2020; Pathak et al., 2016; Hu et al., 2020) mask a portion of word tokens or image pixels from the input sentence or image and train the model to predict the masked components to ob- tain effective representations. Etemad, 2020; Xu et al., 2019) apply transformation that maps from canonical views to alternative views and trains the model to predict what transformation has been applied. In)tane Discrimi- nation methods (Velickovic et al., 2019; Chen et al., 2020; He et al., 2019; Tian et al., 2020) apply some transformation process in one instance to obtain multiple views of it and attempt to formalize the contrastive instance discrimination. (4) And Clustering methods (Caron et al., 2018; 2020; Zhan et al., 2020; Alwassel et al., 2020) focus on dividing the training data into several groups with high intra-group similarity and low inter-group similarity. We recommend readers read (Ericsson et al., 2022) to get more information.
(2) Transformation Prediction methods (Gidaris et al., 2018; Sarkar &
4.2 SELF-SUPERVISED LEARNING AND REINFORCEMENT LEARNING
There exist substantial works taking advantage of SSL techniques to promote representation learn- ing in RL. A popular approach is to jointly learn policy learning objectives and auxiliary objectives.

---

Under review as a conference paper at ICLR 2023
As for constructing auxiliary SSL objective, the primary way is to build multiple views of the same input through masked-latent reconstruction or dynamic models with augmentations. For instance, (Srinivas et al., 2020; Zhu al., 2022) tried to extract high-level features from pixels using trastive learning and performed off-policy control on top of the extracted features. (Schwarzer et al., 2021a; Yu et al., b021b; 2022; Zhang et al., 2021) leverage a dynamic model to obtain a predicted version of the subsequent observation and then use contrastive learning to enforce consistency be- tween the raw future observation and the prediction version of it in latent space. Another alternative way of obtaining good representations is to pre-train the observation encoder to learn effective rep- resentations before policy learning( Yarats et al., 2021; Stooke et al., 2021; Schwarzer et al., 202 bb; Yang & Nachum, 2021; Campos et al., 2021).
4.3 MULTI-AGENT REINFORCEMENT LEARNING AND REPRESENTATION LEARNING
Recent works (Ych et al., 2019; Sun et al., 2019; Zhan et al., 2018; Yuan et al., 2021) study the representation learning for developing models that predict trajectories over all agents at once but not under the MARL context. And as far as we know, only few works consider promoting representa- tion in the MARL context. The most similar works are (Shang et al., 2021) and (Zhang al., 2022). (Shang et al., 2021) task each agent to predict its future location, arriving at agent-centric predic- tive objective to be combined in their proposed agent-centric attention module in the football game. (Zhang et al., 2022) is model-based MARL method that proposed graph-assister predictive state representation learning framework that leverages the agent connectivity graphs to aggregate local representations computed by each agent. Note that the SSL prior proposed in (Shang et al., 2021) only be used in football-like environments and is not flexible. Additionally, our method aims to build general plugin for model-free MARL approaches so that model-based MARL methods are not directly comparable to our method. We focus on the auxiliary task line in this work.
5 EXPERIMENTS
We consider a series of MARL benchmarks to evaluate MAJOR, including vision-based and state- based MARL benchmarks. All hyperparameters settings can be found in Appendix G.
5.1 SETUP
Vision-based MARL Environments. Representation Learning and SSL, techniques show their strength in vasion-based RI. environments, such as DMControl and Atari. To evaluate whether our proposed MAJOR is also powerful in such vision-based MARL settings, we run it on three physics- based cooperative tasks in Multi-Ageni Quadcopter Control(dubbed MAQC) (Panerati et al., 2021). MAQC is an open-source, OpenAl Gym-like qulliiquadcopter simulator that provides vision-based observations and multi-agent controlling interfaces. Observations include video frames from the per- spective of each drone (toward the positive direction of the local x-axis) for the RGB (E (61xx5544) depth, and segmentation (E IR 161x11x11 views. The action of drones is continuous velocity and the magnitude of the velocity.
Here, we briefly introduce the two scenarios, named Flock and LeaderFollower in MAQC. Denote i-th agent's ' xyz coordinates as x = (x, y, 2). individual reward as ri, team reward is R = >> 1-1 Ti and then: (i In Flock, the first agent should keep its position with a predefined location (e.g.. g.. p) as close as possible, and i-th agents(i > 1) need to track (i - 1)-th agent's 's latitude, i.e., T1 -Ipp- x1 x112, r, = (y: 11-1)² Vi = 2, ...... n (ii) The goal of LeaderFollower in MAQC is to train the follower drones to track the leader drone, and the leader drone needs to keep its position with
predefined position as close as possible, i.e., r1 -Ipp-- 1ip-xx X 12, r; = - _ - 21 vi = 2, 2,..., n. An overview of MAQC is shown in Figure 2. Note that in experiments we only use the RGB information provided by the simulator.
State-based MARL Environments. Since our method can be used not only in vision-based but also in state- based scenarios in MARL environments, we also test our method in state-based cooperative MARL scenarios. Our evaluation includes StarCraftII Multi-Agent Challenge (SMAC) (Samvelyan et al., 2019), Multi-Agent MuJoCo (de Witt et al., 2020), and Google Research Football (Kurach al., 2020), in which the baselines compared show the current oftt--ofthhe-rt performance.
7

---

Under review as a conference paper at ICLR 2023
Depth View
Segnentation
Figure 2: Overview of Multi-Agent Quadcopter Control environment and the observation for the quadrotor. quadrotor (i) an easy-to-understand mobile robot platform whose (ii) control can be framed as continuous states and actions problem but, beyond I-dimension, (ii) it adds the complexity that many candidate policies lead to unrecoverable states, violating the assumption of the existence of a stationary state distribution on the entailed Markov chain.
Baselines and evaluations. We select MAPPO, HAPPO, and MAT as the compared baselines because they are general both continuous and discrete MARL environments. Moreover, they have achieved state-of-the-ar performance on common-used MARL benchmarks. We keep the hyperparameters proposed in their papers for fair comparison of all algorithms. We run five random seeds to get the mean and standard deviation of the evaluation metrics. We calculate the winning rate SMAC and the playing scores in Google Football. For other environments, we use episode rewards as the evaluation metric. Additionally, we only select three super-hard difficulty scenarios in the SMAC environment. And we play four drones the MAQC simulator to test the performance of candidates be compared. As for Multi-Agen MuJoCo, we enforce agents can only observe the state of their joints and bodies but not their immediate neighbor's joints and bodies.
5.2 RESULTS AND ANALYSIS
Multi-Agent Quadcopter Control. Figure shows the performance of MAJOR, compared with strong baselines in the MAQC environment. The three tasks require drones (agents) to Iearn to fly to a specific goal and keep a lower distance from others by controlling the velocity vector. The results in Flock and LeaderFollwer demonstrates the strength of MAJOR since it conducts the joint transition model and implicitly captures the relationships of euclidean distance via the learned joint transition model that for the goal of these two scenarios, being closed with other drones, MAJOR can inject more helpful information into networks thus bring significant improvement w.r.t other algorithms. Moreover, the results also illustrate that it is not enough learn effective representations from high- dimensional image-based observations only through policy optimization. So there are still many challenges in applying MARI, algorithms in vision-based environments. And we firmly believe that our work will bring some inspiration for basedd MARL studies.
Tlock
LeudetPolIceer
2010
Stage
Figure 3: Results on Multi-Agent Quadcopter Control. MAJOR outperforms prior methods on two of three tasks, and all methods cannot work on the Meetup.
SMAC, MA MuJoCo and Google Football. We compare MAJOR with the SOTA MARL methods for typical state-based cooperative MARL. benchmarks. According to Figure 4b, Figure 4a, and Figure 4c, MAJOR also shows superior performance, which means that even though the observations are non-visual signals, MAJOR still embodies the interaction and fusion between agents through the joint transition model in the latent space. This can be significantly shown in Multi-Agent MuloCo. To cooperate, the robot agents must attempt to infer information from other joints and bodies. Other compared methods only considered the feature encoding process from the perspective of policy optimization. Still, MAJOR can take into account other agents' positions and velocity by the extra self-supervised representation learning task, which makes it natural to achieve better performance.
8

---

Under review as a conference paper at ICLR 2023
Besides, algorithm can also improve MAT's sample efficiency in Google Football and SMAC benchmarks.
Exl-Agent aalecheetah
SwI-Aqent
6x1-2aeent
EnviDoomans: 204pa
Inclenomens abape
(a) Multi-Agent MuloCo 27m ve. 30m
10902
6n Vo. O2
(b) StarCrafill Multi Agent Challenge
acadeny www.teottttttac...c............e www.eeeetttaac....ooooo.o...ccce easy
acacdezry and shoot with keepez
acaderyy 3_ Liich keoppee
(c) Google Research Football
Figure 4: Performance for MAJOR, MAT, HAPPO, and MAPPO on (a) Multi-Agent MuJoCo, (b) StarCraftII Multi-Agent Challenge, and (c) Google Research Football. MAT outperforms compared current state-of-the thettrr baselines in common cooperative MARL benchmarks.
CONCLUSION
In this paper. we introduce Multi-Agent Joint-Predictive Reprssentation(MAJJRR)) a self- supervised representation learning algorithm designed to improve the data efficiency of MARL algorithms. MAJOR treats the individual observations as masked sequence and learns the repre- sentations that are jointly temporally predictive and consistent across different views overall agents, by implicitly reconstructing the global state and directly predicting representations of observations produced by joint transition model and target encoder. Experimental results both vision- based and state-based cooperative MARL benchmarks(i.e. Multi-agent Quadcopter Control, SMAC, Multi-agent MuJoCo, and Google Football) show MAJOR's 's state-of-the-an performance. Based on the established connection between MARL and SSL, in the future, we will consider incorporating cycle-consistent predictive or other powerful SSL priors into MAJOR and combining offline repre- sentation pretraining with task task-specific finetuning in the paradigm.
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian lbarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng. Do as can and not as say: Grounding language robotic affordances. In arXiv preprint arXiv:2204.0166), 2022.
Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, al. Solving rubik's cube with robot hand. arXiv preprint arXiv:1910.07113, 2019.
Humam Alwassel, Dhruv Mahajan, Bruno Korbar, Lorenzo Torresani, Bernard Ghanem, and
Du Tran. Self-supervised learning by cross-modal audio-video clustering. Advances in Neural Information Processing Systems, 33:9758-9777 9770, 2020.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A frame- work for self-supervised learning of speech representations. Advances in Neural Information Processing Systems, 33:12449-12460, 2020.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Christopher Hesse, Rafal Józefowiez, Scott Gray, Catherine Olsson, Jakub W. Pachocki, Michael Petrov, Henrique Pondé de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota with large scale deep reinforcement learning. ArXiv, abs/1911 1ess/1922.0680, 2019.
Víctor Campos, Pablo Sprechmann, Steven Hansen, Andre Barreto, Steven Kapturowski, Alex Vitvitskyi, Adria Puigdomenech Badia, and Charles Blundell. Beyond fine-tuning: Transferring behavior in reinforcement learning. arXiv preprint arXiv: 200.:2/02.11. 135155 2021.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for un- supervised learning of visual features. In Proceedings of the European conference on computer vision (ECCV), pp. 132-149, 2018
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33:9912-9924, 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.057099 2020.
Yuanpei Chen, Yaodong Yang, Tianhao Wu, Shengjie Wang, Xidong Feng, Jiechuang Jiang, Stephen Mareus McAleer, Hao Dong, Zongging Lu, and Song-Chun Zhu. Towards human-level bimanual dexterous manipulation with reinforcement learning. arXiv preprint arXiv:2206.0880 2022.
Christian Schröder de Witt, Bei Peng, Plexre-Alex Kamienny, Philip H. S. Torr. Wendelin Böhmer, and Shimon Whiteson. Deep multi-agent reinforcement learning for decentralized con- tinuous cooperative control. CORR, abs/2003.06709 2020. URL https: http:///axvvvv/o///// abs/2003. 06709.
Matt Deitke, Winson Han, Alvaro Herrasti, Aniruddha Kembhavi, Eric Kolve, Roozbeh Mottaghi, Jordi Salvador, Dustin Schwenk, Eli VanderBilt, Matthew Wallingford, al. Robothor: An open Aimulation-to-rewt embodied ai platform. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3164-3174, 2020.
Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Jordi Salvador, Kiana Ehsani, Winson Han, Eric Kolve, Ali Farhadi, Aniruddha Kembhavi, and Roozbeh Mottaghi. Procthor: Large- scale embodied ai using procedural generation. ArXiv, abs/2206.066 069944 2022.
10

---

Under review as a conference paper at ICLR 2023
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprins arXiv:1810.048800 04800, 2018.
Linus Ericsson, Henry Gouk, Chen Change Loy, and Timothy M Hospedales. Self-supervised rep- resentation learning: Introduction, advances, and challenges. IEEE Signal Processing Magazine, 39(3):42-00 62, 2022.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint arXi:: 2018.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bemardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-super learning. Advances in neural information processing systems, 33:21271-21284 2020.
Nicklas Hansen and Xiaolong Wang. Generalization in reinforcement learning by soft data aug- mentation. In 2021 IEEE International Conference Robotics and Automation (ICRA), pp- 13611 13611-13617. IEEE, 2021.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. arXiv preprint arXiv:1911.05722, 2019.
Siyi Hu, Fengda Zhu, Xiaojun Chang, and Xiaodan Liang. Updet: Universal multi-agent reinforce- ment learning via policy decoupling with transformers. arXiv preprint arXiv:2101.08001 2021.
Ziniu Hu, Yuxiao Dong, Kuansan Wang. Kai- Wei Chang, and Yizhou Sun. Gpt-gnn: Generative pre-training of graph neural networks. In Proceedings of the 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2020.
Jakub Grudzien Kuba, Muning Wen, Linghui Meng, Haifeng Zhang, David Mguni, Jun Wang, Yaodong Yang, et al. Settling the variance of multi-agent policy gradients. Advances in Neu- ral Information Processing Systems, 11113358-147701 2021.
Jakub Grudzien Kuba, Ruiqing Chen, Muning Wen, Ying Wen, Fanglei Sun, Jun Wang, and Yaodong Yang. Trust region policy optimisation in multi-agent reinforcement learning. In International Conference Learning Representations, 2022. URL http:: http::////pppeeeee//ee/e / forum?
Gd=EcGGFkNTxdJ.
Karol Kurach, Anton Raichuk, Piotr Stanczyk, Michal Zajac, Olivier Bachem, Lasse Espeholt, Car- los Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet, and Sylvain Gelly. Google research football: A novel reinforcement learning environment. ArXiv, abs/1907. 11100, 2020.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv: 1992,, 2019.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequene pre- training for natural language generation, translation, and comprehension. arXiv preprint arXiv: 134611, 2019.
Michael Littman and Richard S Sutton. Predictive representations of state. Advances in neural information processing systems, 14, 2001.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 1907 2019.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed representa- tions of words and phrases and their compositionality. In Proceedings of the 26th International Conference on Neural Information Processing Systems Volume 2, NIPS' 13, pp. 31:1111111111 11-31199 Red Hook, NY, USA, 2013. Curran Associates Inc.

---

Under review as a conference paper at ICLR 2023
Frans A Oliehoek and Christopher Amato, A concise introduction to decentralized POMDPs. Springer, 2016.
Jacopo Panerati, Hehui Zheng, SiQi Zhou, James Xu, Amanda Prorok, and Angela P. Schoellig. Learning fly-a a gym environment with pybullet physies for reinforcement learning of multi- agent quadcopter control. In 2021 IEEE/RS] International Conference on Intelligent Robots and Systems (IROS), 2021.
Deepak Pathak, Philipp Krähenbühl, Jeff Donahue, Trevor Darrell, and Alexei Efros. Context en- coders: Feature learning by inpainting. In Computer Vision and Pattern Recognition (CVPR), 2016.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforce- ment learning. In International conference on machine learning, pp. 4295-4304, PMLR, 2018.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philiph H. S. Torr, Jakob Foerster, and Shimon Whiteson. The StarCraft Multi-Agent Challenge. CoRR, abs/1902.04043 2019.
Pritam Sarkar and Ali Etemad. Self-supervised learning for ecg-based emotion recognition. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3217-3221. IEEE, 2020.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint wrXiv:1506.02438, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Max Schwarzer, Ankesh Anand, Rishab Goel, Devon Hjelm, Aaron Courville, and Philip Bach- man. Data-efficient reinforcement learning with self- eeffprrdictive representations. In Interna- tional Conference on Learning Representations, 2021a. URL https::/ hettteewwwweeeeeee.e.........
forum?id-uCQfP UwRaU.
Max Schwarzer, Nitarshan Rajkumar, Michael Noukhoviteh, Ankesh Anand, Laurent Charlin, R De- von Hjelm, Philip Bachman, and Aaron C Courville. Pretraining representations for data-efficient reinforcement learning. Advances in Neural Information Processing Systems, 34:12646 12696-1669,, 2021b.
Shital Shah, Debadeepta Dey, Chris Lovett, and Ashish Kapoor. Airsim: High-fidelity visual and physical simulation for autonomous vehicles. In Field and service robotics, pp. 621-635. Springer, 2018
Wenling Shang. Lasse Espeholt, Anton Raichuk, and Tim Salimans. Agent-centric representations for multi-agent reinforcement learning. arXiv preprint 2xxx::204400444 2021.
Aravind Srinivas, Michael Laskin, and Picter Abbeel. Curl: Contrastive unsupervised representa- tions for reinforcement learning, international conference on machine learning, 2020.
Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement learning. In International Conference on Machine Learning, pp. 9870-9879. PMLR. 2021.
Chen Sun, Per Karlsson, Jiajun Wu, Joshua Tenenbaum, and Kevin Murphy. Stochastic prediction of multi-agent interactions from partial observations. arXiv preprint arXiv:
2019.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 6827-6839, Curran Associates, Inc., 2020. URL ht https: / /ttpe//eeeeeeecoec///eep/ neurips paper/2020/F111 http://cea...ee11110000025533330
12

---

Under review as a conference paper at ICLR 2023
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL https: /tttps:/prroee http://poooeeennneeeeeeeee mew.rdiiiiinnnnnnn..eeeeee.eee.e https//urr.s...cpaaa//////////// wwtpp/pprr/11///ieeeee wtt.eeeeeeeeeeeeeeeeeeeeeeee
Papeesaaaappp.pppp pdf.
Petar Velickovic, William Fedus, William L Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. ICLR (Poster), 2(3):4, 2019.
Oriol Vinyals, Igor Babuschkin, Wojeiech Czamecki, Michaël Mathieu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782)350-3544 2019.
Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. Qplex: Duplex dueling
multi-agent q-learning, arXiv preprint arXiv:2008.0100 2020.
Minrui Wang, Ming Feng, Wen gang Zhou, and Houqiang Li. Stabilizing voltage in power distribu- tion networks via multi-agent reinforcement learning with transformer. Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2022.
Muning Wen, Jakub Grudzien Kuba, Runji Lin, Weinan Zhang, Ying Wen, Jun Wang, and Yaodong Yang. Multi-agent reinforcement learning is a sequence modeling problem. arXiv preprint arXiv:2205. 14953, 2022.
Dejing Xu, Jun Xiao, Zhou Zhao, Jian Shao, Di Xie, and Yueting Zhuang. Self-supervised spa- tiotemporal learning via video clip order prediction. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 10334-10343, 2019.
Mengjiao Yang and Ofir Nachum. Representation matters:
offline pretraining for sequential decision making. In International Conference on Machine Learning, pp. 11784-11794, PMLR, 2021.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Reinforcement learning with pro- totypical representations. ICML, 2021.
Raymond A Yeh, Alexander Schwing, Jonathan Huang, and Kevin Murphy. Diverse generation for multi-agent sports games. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4610-4619, 2019.
Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative multi-agent games, 2021a.
Tao Yu, Cuiling Lan, Wenjun Zeng, Mingxiao Feng, Zhizheng Zhang, and Zhibo Chen. Playvirtual: Augmenting cycle-consistent virtual trajectories for reinforcement learning. Advances in Neural Information Processing Systems, 34, 2021b.
Tao Yu, Zhizheng Zhang, Cuiling Lan, Zhibo Chen, and Yan Lu. Mask-based latent reconstruction for reinforcement leamning arXiv preprint arXiv:2201. 2022.
Ye Yuan, Xinshuo Weng, Yanglan Ou, and Kris Kitani. Agentformer: Agent-aware transformers for socio-temporal multi-agent forecasting. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision (ICCV), 2021.
Eric Zhan, Stephan Zheng, Yisong Yue, Long Sha, and Patrick Lucey. Generative multi-agent behavioral cloning. arXiv, 2018.
Xiaohang Zhan, Jiahao Xie, Ziwei Liu, Yew-Soon Ong, and Chen Change Loy. Online deep clus- tering for unsupervised representation learning. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 6688-6697, 2020.
Amy Zhang, Rowan Thomas McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learn- ing invariant representations for reinforcement learning without reconstruction. In International Conference on Learning Representations, 2021. URL https: /ppeneevieww net forum? 2F==2FCWDKRRED.
13

---

Under review as a conference paper at ICLR 2023
Zhi Zhang, Zhuoran Yang. Han Liu, Pratap Tokekar, and Furong Huang. Reinforcement learning under a multi-agent predictive state representation model: Method and theory. In International Conference Learning Representations, 2022. pttpttps://ppeeeeieeee wtw.ee/wweet/orrrrree/e
id=PLDOnF oVm4.
Jinhua Zhu, Yingce Xia, Lijun Wu, Jiajun Deng, Wengang Zhou, Tao Qin, Tie-Yan Liu, and Houqiang Li. Masked contrastive representation learning for reinforcement learning. IEEE Trans- actions on Pattern Analysis and Machine Intelligence, pp. 1-1, 2022. doi: 10.1109/PAMM.202. 3176413.
14

---

Under review as a conference paper at ICLR 2023
A FURTHER DISCUSSION ON MAJOR
A.1 PERFORMANCE OF MAJOR
It is worth noting that the state-based observations in these environments usually contain the key attributes demonstrating the position, velocity, and hp of the agents, which means that there are no irrelevance or noisy messages in the vector. Thus, the neural network easily capture useful information, and selected baselines will achieve strong performance close to the upper bound with the help of a powerful policy optimization mechanism. So our proposed representation learning task can only take slight improvements w.r.f the original baseline. In other words, for state-based observations, their representations are more effective than vision-based observations through the learning process of MARL methods.
However, in contrast to low-dimensional state-based vectors, learning representations from high- dimensional image- based observations only through policy optimization are more complicated. And our experimental results of MAQC show a significant improvement compared with baselines, which provides the benefit of extra representation learning in current MARL methods for vision-based environments.
In this work, we mainly focus extending SSL for MARI. to improve efficiency, especially in vision-based multi-agent environments. And the results of wide-range benchmarks can demonstrate the generalization of our proposed method for both vision- and state-based scenarios.
A.2 MOTIVATION OF DESIGNING MAJOR FOR MARL
Firstly, we would like to emphasize that the goal of incorporating SSL in the single-agent RL method is to learn more effective representations when fixing the encoder and the number of dimensions of latent space, either visual inputs or state- baste-bsed inputs. The more informative and rich representations contained in latent variable, the more effective it is to perform policy optimization. However, in practice, vision-based inputs usually include many noisy and irrelevant components rather than state-based inputs, which brings more obstacles to capturing informative features. Meanwhile, state- based inputs are often designed with critical elements that reflect the physical attributes of the agent with only a few distractions. For example, when we train the same RL algorithm (e.g.. SAC) on MuJoCo, it is accessible to extract features from the physical vector(i.e., velocity and position, etc.) MLP encoder and get high scores. but is more challenge use the CNN encoder obtain good representations and achieve the same performance in DeepMind Control Suite(i.e., visual MuloCo). Thus we believe that the benefit of applying SSL to both single- and multi-agent RL. more visible on high-dimensional visual space inputs.
Secondly, as stated in the original paper, we would like to clarify that our work is not simply applied SSL to existing MARL methods. It is worth noting that the representation learning challenge under the MARL context is more difficult than the SARL situation since local observations will be affected by the change in the ego agent's policy and other agents' behavior. As result, simply applying SSL priors for each agent may fail due to imperfect information. Furthermore, we believe that our work, which concerned embodying the interaction and fusion among agents in the environment rather than temporal representations for each agent, is not simply to apply SSL. to MARL. And our work is motivated by incorporating agent-level interactions to make representations that can take the other agents into account to be more effective. Due to the time limitation of the rebuttal, we will attempt apply the common-used SSL technique in SARL, named SPR, on MAT to verify whether simply doing the combination will work.
Thirdly, we believe that the vision-based multi-agent system will be paid more and more attention in the future days. It is no doubt that real-works scenarios, such as 2D/3D robotics and eSports games, are almost accepting visual signals but not state- vectors. Recently, many works are proposed in the community. Shah et al. (2018); Panerati et al. (2021) present simulators for control- ling drones to visual signals collected with drone cameras. Chen et al. (2022) presents a bimanual dexterous manipulation benchmark (Bi-DexHands) according to the cognitive science literature for comprehensive reinforcement learning research. A similar job that predates Chen et al. (2022) is OpenAl's 's is generated by built-in sensors. Moreover, many robotic navigation benchmarks are presented with more complex visual inputs, such as Deitke et al. (2020; 2022); Ahn et al. (2022), etc. Meanwhile,
robot hand Akkaya et al. (2019), which solves Rubik's 's Cube and whose input
15

---

Under review as a conference paper at ICLR 2023
in eSports, the famous OpenAl Five (Berner et al., 2019) and AlphaStar (Vinyals et al., 2019) also use human-like images as inputs. These developments indicate that more and more research will be studied multi-agent environments based on visual inputs. In other words, these 2D/3D robotics and eSports areas provide powerful platforms for researchers to solve real-world problems using MARL. And research based on these areas will also have greater potential to be applied actual industrial scenarios. However, current research on MARL seems to be lagging behind. The obser- vations of the common-used benchmarks(e.g g. SMAC, Google Research Football, MAMuJoCo) are all based on hand- designed physical values. These environments seem too simplistic compared with the real-world applications mentioned above. Thus it is meaningful make an effort explore the idea of representation learning in MARL for vision-based multi-agent systems rather than toy environments. And we believe our work will be useful to the community.
A.3 FEASIBILITY ANALYSIS OF MAJOR
Note that it may not work without any assumptions in Dec-POMDP with imperfect observation mapping functions, i.e. all agent's observations may not contain all the information in the state due to the partial observation issue. In this work, we build MAJOR MAT, and MAT utilities the local observations to estimate the expected team return under a cooperative MARL context. The MAT's encoder with the value head is used to approximate the global value function and get good performance, illustrating that it can recover the state and then estimate the team return from the sequence of partial observations. Thus we can assume that all agent's observations in the evaluated Dec-POMDP benchmarks can also contain all the state information, leading to our work becoming practical. At the same time, in our experimental results, we find that MAJOR loss in all experiments reduced rapidly, meaning that our method can work on common-used Dec-POMDP benchmarks without pointed concerns.
As mentioned in the related work section of the original paper, Zhang et al. (2022) takes a study of learning dynamic model under model-based MARL in Dec-POMDP, which is similar to the pointed issue. The work proposed a graph- based version adaptation of Predictive State Representa- tions (Littman& & Sutton, 2001) to infer the predictive representations and uses the representations as the input of a policy optimization algorithm. Even though the method is not directly comparable to our model-free way with the auxiliary task, Zhang et al. (2022) has implicitly proven the feasibility of predicting future representations from individual historical observation. We recommend readers to read Zhang et al. (2022) for more information. Furthermore, we believe that adding historical trajectories of agents information and agent-level interactions simultaneously in MAT and MAJOR to make them more powerful in our future work.
address the issue much possible. And will involve both temporal
A.4 FAIRNESS OF COMPARISON BETWEEN MAT AND CTDE METHODS
There is no doubt that MAT follows the CTCE(centralized training and centralized execution) pat- tern. But MAT only takes partial observations rather than a global state in the training and execution phases, while CTDE approaches must depend on the global state. However, in many vision-based scenarios, the system state is enormous compared with observations since it has use an extensive view to cover all agents' views to reflect the system(but most of its messages may be unnecessary). Therefore, the huge-volume state may become the bottleneck of all CTDE MARL methods. Be- cause MAJOR is more effective when applied to vision- based environments, comparing MAT with typical CTDE methods will not result in many unfair issues, respectively.
B MORE EXPERIMENTAL RESULTS AND ANALYSIS
B. TRAINING CURVES FOR Loss FUNCTION IN MAJOR AND MAT
Here we put the training curve of MAJOR loss and V loss in the four-drone Flock of MAQC and give the qualitative analysis of MAJOR. As shown in Figure 5, the MAJOR loss reduces rapidly, meaning that the MAT's encoder and the joint transition model can account for the information of other agents with the help of the designed auxiliary task. Meanwhile, as shown in Figure 6, the global value function approximation also benefits from the compact representation of observations
16

---

Under review as a conference paper at ICLR 2023
Training Curve in Flock
Training Curve in Flock
4.2
3.5
1.0
2.5
0.3
3AJOR
2.0
1.6
0.2
6.1
0.8
0.3
8.0
0.5
1.2
0.4
1-6
6.0
Environmen:
steps
conment steps
Figure 5: Training curve for MAJOR's s loss.
Figure 6: Training curves for value loss.
Flock
SeaderFolloo Pollower
-42
Kmmd
-15
-122
-20
-122
142
-35
-182
-232
-43
0.0
0u6
0.0
0.0
3.4
steps
Ice
Environment
stepa
lc6
Figure 7: Ablation study of forwarding steps K for the joint transition model where H
as training progresses since the MAJOR's 's value loss is lower than MAT's The more accurately the value function fits, the better the policy optimization effect. And we would like to infer that the non-convergence policy causes the rapid overfitting issue of MAJOR's loss at the beginning of the training phase. Similar results can be found in other scenarios.
B.2 ABLATION STUDY
Here we give more details about the ablation study. In our method, the critical hyperparameters predefined K and H. meaning the number of forward steps for the joint transition model and the updating frequency for MAJOR's learning objective, respectively. Figure 7 shows the results of K at 21, 4) in the four-drone Flock and LeaderFollower of MAQC. We find that extended dynamics modeling consistently improves performance up to roughly. A larger K (c.g., 8) does not bring further improvement since the network cannot predict long-term future representations as the accumulation of single-step prediction. And Figure 8 illustrates that updating frequency can also influence the efficiency of MAT. In our opinion, lazy updating for MAJOR will make representations mismatch with the corresponding policy gradient, thus bringing a negative impact on MAT.
C INCORPORATING MAJOR ON OTHER MARL ALGORIHTMS
In this section, start by introducing how to incorporate our proposed auxiliary task into MARL algorithms following the CTDE pattern, such as MAPPO and HAPPO, And then we will show the results of the modified version of MAPPO/HAPPO on four-drone Flock in MAQC.
As stated in Sec. 1, the benefit of combining MAT with MAJOR is that it can employ both represen- tations generated from MAT's encoder and decoder. Thus, the gradient derived from our auxiliary learning objective can be back- propagated to both the encoder and decoder simultaneously. Contrary to MAT, common-used MARI algorithms(e.g. MAPPO, HAPPO) under the CTDE framework usu-
17

---

Under review as a conference paper at ICLR 2023
Plock
LeaderFol
_pdate_freq-]
updete__feeq-1 sppdetin_.Triigeee
spdate_Fraqu
-15
Rewaxd
20
-18
-12
-222
8.0
0.2
0.4
0.5
0.0
1.0
0.0
6.2
0.4
2-4
0.2
1.0
Rnyel ronment attepa
Environment:
stepa
Figure 8: Ablation study of updating frequency H of MAJOR's s learning objective where K
ally take the global state as the input for the critic network, so we need to modify the pipeline of the proposed auxiliary task.
The following figure demonstrates the combined version of MAPPO. Since MAPPO's critic net- work does not use individual observations, we can only incorporate MAJOR in the actor and make the gradient of auxiliary loss proportionate through the actor network. We are supposing the archi- tecture the actor network contains more than two MLPs after the encoder. We are supposing architecture of the actor network contains more than two MLPs after the encoder. In other words, the pipeline of the actor network is:
CNVMMLP(paaammteriiie by 0) - 6 + MLPapaarameterized by の) + by w) + a
â - MLP Mprapaaamererriz
Actor's ObsEncoder
Actor's ActEncoder
Policy Head
MAPPOactor f A,00
0c-11+K-1
Oc+1it+K
Joint Transition Model
Exponential noving avg.
Onl ine Projection
OnL ine Prediction
It+is+
Exponential soving avg.
MAJOR 40
Ocolit4K
Target Actor's
Target
gradient
ObsEncoder
Projection
Figure 9: illustration of incorporating MAJOR into MAPPO.
Thus, given the intermediate outputs of the actor network, can obtain the observation and action representations. The subsequent pipeline similar to Figure 1. As shown Figure 9, the differences between MAPPO + MAJOR and MAT + MAJOR are:
We treat MAPPO's actor network as the sequential combination of observation en- coder(dubbed ObsEncoder) and action encoder(dubbed ActEncoder) followed by the pol- icy head. Alternatively, you can use separate network as ActEncoder outside the actor network.
The actor's ActEncoder only takes the observations' representations input.
18

---

Under review as a conference paper at ICLR 2023
Leade:Follower
-10
20
23
30
35
IAP20
MOPPO
HAPPO-MA.TOR
NAD2O-MAJOR
0.00
0.25
0.50
0,15
1.00
2.25
1:50
1:75
2,00
Enviro rnvironment
stepa
Figure 10: Results for combining MAJOR and MAPPO/HAPPO in LeaderFollow.
We feed action representations generated from the actor's ActEncoder rather than the exe- cuted actions in sampled transitions into the joint-transition model.
Note that recently there has been some literature(Hu al., 2021; Wang et al., 2022) utilizing Trans- former models as a feature extractor for single observations, which contain a sequence of various entities. So applying MAJOR on MAPPO/HAPPO will be similar to the shown pipeline in Figure 1. In this situation, we treat the global state as a sequence of observations or elements in the system. And the joint transition model accepts representations from both actor and critic networks.
We have also evaluated the performance of the modified version of MAJOR the four-drone Lead- erFollower of MAQC. The result in Figure 10 illustrates that MAJOR can also improve both MAPPO and HАPPO. But the improvement is not significant because MAJOR can only help the actor net- work to learn more impact observation representations. Still, does not benefit critic network, which makes no improvements for value estimation.
D EXTENDED BACKGROUND
D.1 EXISTING METHODS IN MARL
We will now give quick overview of two cutting-edge MARL Both based Prox- imal Policy Optimization (PPO, Schulman et al. (2017)), an RL approach known for its simplicity and stability performance.
MАPPO (Yu et al., 2021a) was the first and most straightforward technique for implementing PPO MARL. provides all agents with the same set of parameters and updates the shared policy based on the aggregated trajectories of the agents. In detail, at iteration & + 1, it optimizes the policy parameter 0x41 by maximizing the clip objective of
Ty (a' ())
Te (a'
0)
Ame, (o. )
Bo-pay aoate
min
Ano, (o, a), clip
11
To, (a
To, (a)
where the clip operator (if required) trims the input value to keep inside the interval [1 - + c)- Enforcing parameter sharing, the other hand, is analogous imposing restriction 0² 02, Vi, E N on the joint policy space, which might result in a suboptimal conclusion that is exponentially worse (Kuba et al., 2022). This encourages the development of more principled heterogeneous-agent trust-region approaches, such as HAPPO.
HAPPO of the SOTA algorithms that completely exploits Multi-Agent Advantage Decompo- sition (Kuba et al., 2021) to provide multi-agent trust-region learning with monotonic improvement guarantees. During update, the agents choose permutation i1 random, and then, in the
19

---

Under review as a conference paper at ICLR 2023
Online Encoder(f)
Online Projection(g)
Onl ine Projection(q)
Augg-
exponential moving avg.
exponential moving avg.
C BYOL
f.9
Aug
Stop Gradient
Target Encoder(
Target
Profection(
Figure 11: An illustration of the framework of BYOL.
sequence in which the permutation was chosen, each agent im picks nhe T that maximizes the aim of
E
min (r (x/m) A (o, aftmm))) clip (r (nTM) = Atun (o, a fnmm ))]
Stim-1
al
where r(rrm)) n' (a'm lo) (a'm (0m= o). is worth noting that the expectation is placed over the uewlyeupdated prior agents' policies, i.e., Tee 11::m-1; this reflects intuitive understanding that, cording to Theorem (1), the agent im responds to its preceding agents f1 However, one dis-
advantage of HAPPO is that agent policies must adhere to the sequential updating strategy in the permutation, preventing it from being executed in parallel.
D.2 BYOL-STYLE AUXILIARY OBJECTIVE
BYOL (Grill et al., 2020) is a powerful self-supervised representation learning algorithm that en- forces the similarity of representations of the same visual observation across varied data augmen- tation. Figure shows the pipeline. BYOL has an online branch and a momentum branch. The momentum branch is utilized to compute stable goal for learning representations (Chen et al., 2020; He et al., 2019). BYOL is made up of an online encoder f. momentum encoder f. an online projection head g, a momentum projection head g. and prediction head q. The momentum encoder and projection head have the same design as the equivalent online networks and are updated by an exponential moving average (EMA) of the online weights (see Equation 3 in the main manuscript). The prediction head is only employed in the online branch, making BYOL's design asymmetric. BYOL initially generates two views and of from an image using image augmentations. The on- line branch outputs representation
f(v) and projection = g(v). and the momentum branch outputs y' = =((') and momentum version of projection z' = g(z'). BYOL then uses a prediction head q to fit z' from 2, i... q(z) + z'. Finally, BYOL minimizes the similarity loss between q(2) and a stop-gradient target sg(z').
0(=) "[q(2) |g (2')llz
8g (=')
C = =1((2) sg (3')11 22
Recent research brings BYOL-style learning objectives vision-based RL for learning effective state representations and demonstrates promising performance (Hansen & Wang. 2021; Schwarzer al., 202 Yu et al., 202 2022; Yarats et al., 2021), inspired by the success BYOL in learning visual representations. Future state prediction (Schwarzer et al., 2021a), cycle-consistent dynamics prediction (Yu et al., 2021b), prototype representation learning (Yarats et al., 2021), and invariant representation learning (Hansen & Wang, 2021) are examples of B YOL-style learning. These studies also suggest that supervising/regulariingg anticipated representations space is more successful than in the representation of original pixel space. Furthermore, BYOL-style auxiliary losses are often trained with data augmentation since it may easily yield two BOL views. SPR Schwarzer al. (2021a) and Play Virtual (Yu et al., 2021b), for example, random crop and random intensity to enter observations in Atari games. The proposed MAJOR can also be classified as B YOL-sttle auxiliary objectives.
the BYOL's projected latent
20

---

Under review as a conference paper at ICLR 2023
D.3 THE TRANSFORMER MODEL AND THE ATTENTION MECHANISM
Transformer (Vaswani et al., 2017) was created originally for machine translation jobs (e.g.. input English, output French). It has encoder-decoder structure in which the encoder maps input sequence of tokens to latent representations and then the decoder generates a sequence of desired outputs in an auto-regressive manner, with the Transformer taking all previously generated tokens as input at each step of inference. The scaled dot-prot dot-product attention, which captures the interrelationship of input sequences, is a critical component of the Transformer. The attention function is written
OK"
Attention (Q, V) softmax vilk V, where the Q K, V corresponds to the vector of
queries, keys and values, which can be learned during training, and the da represent the dimension K. V share the same of parameters.
of and K. Sclf-attentions refer cases when Q,
In practice the MAT attention mechanism encodes observations and actions using weight ma- trix generated by multiplying the embedded queries, well as keys. The embedded values are multiplied by the weight matrix to output representations. While the encoder's unmasked attention employs a complete weight matrix extract the interrelationships between agents, the decoder's masked attentions capture sequential actions with triangular matrices. With the properly masked at- tention mechanism, the decoder can safely output the policy agent-by-agent. recommend readers give look at Figure 2, and Algorithm in Wen et al. (2022) to get more details about MAT.
E DYNAMIC PROCESS AND SOURCE CODE OF MAJOR
Please refer to https: //w.nymooooouus/////// 4open. www.scce/e/eaaa...
F ADDITIONAL HYPER-PARAMETER IN MAJOR
Table I: Hyperparameters used for MAJOR.
Hyperparameter
Value
Number of prediction steps K [MAJRR update frequency H
Auxiliary batch size for MAJOR B' Weight for MAJOR loss À
128
1
Hidden units in projection/prediction head Encoder MEA
512
0.01 SMAC and Google Football
0.05 MA MoJoCo and MAQC
EMA update frequency
Number of blocks for joint transition model
Number of heads for joint transition model
1
G HYPER-PARAMETER SETTINGS FOR EXPERIMENTS
During experiments, the implementations of baseline methods are consistent with their official repositories, all hyper-parameters left unchanged the origin best-performing status. The hyper- parameters adopted for different algorithms and tasks are listed in Table 2-8. In particular, the ppo epochs and ppo clip across different SMAC scenarios are unified 10 and 0.05 respectively.
21

---

Under review as a conference paper at ICLR 2023
Table 2: Common hyper-parameters used for MAJOR, MAT, MAPPO and HAPPO in SMAC domain.
aaper-parameters
value
spper-parameters
value
hyper-parameter
value
critic Ir
5e-4
actor Ir
5e-4
use gae
True
gain
0.01
optim eps
le-5
batch size
3200
training threads
16
num mini-batch
rollout threads
32
entropy coef
0.01
max grad norm
10
episode length
100
optimizer
Adam
hidden layer dim
64
use huber loss
True
Table 3: Different hyper-paramette used for MAJOR and MAT in the SMAC domain.
Maps
ppo epochs
ppo clip
num blocks
num heads
stacked frames
steps
7
27m vs 30m
5
0.2
1
1c7
0.99
MMM2
5
0.05
1
1e7
0.99
6h vs 8z
15
0.05
1
le7
0.99
Table 4: Differrntt dlyer-pprameeee used for MAPPO and HAPPO in the SMAC domain.
Maps
ppo epuchs
ppo clip
hidden leyer
stacked frames
network
steps
"MPPO
THAPPO
27m 30m
5
0.2
2
1c7
0.99
0.95
MMM2
5
0.2
2
rnn
1e7
0.99
0.95
8z
2
mlp
0.95
Table Common hyper-parameters used for all methods in the Multi-Agent MuloCo and Multi- Agent Quadcopter Control domain.
pper-parameters
value
hyyer-parameters
value
Hlyper-parameter
value
gamma
0.99
steps
le7
stacked frames
1
gain
0.01
optim eps
1e-5
batch size
4000
training threads
16
num mini-batch
40
rollout threads
40
entropy coef
0.001
max grad norm
0.5
episode length
100
optimizer
Adam
hidden layer dim
64
huber loss
True
Table 6: Different hyper-parameter used in the Multi-Agent MuJoCo and Multi-Agent Quadcopter
Control domain.
Maps
MAJOR
MAT
MAPPO
НАРPО
critic Ir
5e-5
5e-5
5e-3
5e-3
actor Ir
5e-5
5e-5
5e-6
5e-6
ppo epochs
10
10
5
5
ppo clip
0.05
0.05
0.2
0.2
num hidden layer
2
num blocks
/
num head
/
22

---

Under review as a conference paper at ICLR 2023
Table 7: Common hyper-parameters used for all methods the Google Research Football domain.
saper--arameters
value
paper-parameters
value
hyper- prper-parameters
value
critic Ir
5e-4
actor Ir
5e-4
gamma
0.99
gain
0.01
optim eps
le-5
batch size
4000
training threads
16
num mini-batch
rollout threads
20
entropy coef
0.01
max grad norm
0.5
episode length
200
optimizer
Adam
hidden layer dim
64
stacked frames
Table 8: Different hyper- parameter used the Google Research Football domain.
Maps
MAJOR
MAT
MAPPO
HАPPO
ppo epochs
10
10
5
5
ppo clip
0.05
0.05
0.2
0.2
num hidden layer
/
num blocks
1
/
num head
1
/
23


Reviews:
[{"review_id": "z4Mv_4PHee8", "reviewer": "Reviewer_patt", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes Multi-Agent Joint-Predictive Representations (MAJOR), which applys the self-supervised learning (SSL) technique to MARL, trying to improve the learning efficiency. Specifically, MAJOR builds a transformer-based transition model, which takes all agents observations and actions as inputs and predicts the observations of the next $K$ steps in the latent space. Experimental results on four MARL benchmarks show the SSL auxiliary task could slightly improve the performance of the underlying algorithms.\n\n", "summary_of_the_review": "This paper simply applies SSL to existing MARL methods and shows the experimental results. But the motivation is not very clear. Besides, the experimental results are not significant and the writing of the paper could also be improved. So, in its current form, the reviewer holds the point that the paper is below the acceptance threshold.", "strength_and_weaknesses": "**Strengths**:\n   * Studying how to learning efficient state abstraction in MARL is an interesting and open problem.\n   * The paper conducts extensive experiments on wide-range MARL environments.\n\n**Weaknesses**: \n  * The performance improvement of MAJOR is minor compared with the baselines. The training curves of different algorithms are overlapped and thus the improvements are not significant.\n  * The benefit of applying SSL to vision-based single-agent RL algorithms is that the policy is learned in the transition-irrelevance or policy irrelevance low-dimensional latent space rather than the original high-dimensional visual space. The paper simply applies SSL to existing MARL methods and shows the experimental results. But the motivation is not very clear. Why applying SSL to MARL is beneficial especially for state-based inputs? \n  * Predicting future observations simply based on all agent's current observations may also be inaccurate. In POMDP, all agent's observations may not contain all the information in the state due to the partial observation issue. \n  * Minor:\n    * \"And we define the following mean squared error between the normalized predictions and target projections\". 'mean squared error' should be 'cosine similarities'.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "cr0XSkfRYL", "reviewer": "Reviewer_yv1U", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper describes Multi-Agent Joint-Predictive representations (MAJOR), which is a self-supervised learning mechanism that allows the MARL system to learn policies in a data-efficient manner.\nIn MAJOR, observations obtained by individual agents are treated as a masked sequence for representation learning, i.e., masked prediction. \nThe learned representation should be jointly temporally predictive and consistent across different views overall agents.\nThe performance  is evaluated through both visionbased\nand state-based cooperative MARL benchmarks.\nThe empirical results show that MAJOR outperforms pre-existing methods. ", "summary_of_the_review": "The proposed method is new and improves the performance of MARL.\nThough further discussion is expected, the paper has sufficient strength.\nThe proposal is insightful and interesting. ", "strength_and_weaknesses": "Strength:\n-The empirical result solidly shows the performance of the proposed method.\n-The proposal is based on the recent progress and success in SSL, and is insightful.\n\nWeakness:\n-The qualitative evaluation and discussion are limited.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "TMwnuyhvJX_", "reviewer": "Reviewer_hLyZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a self-supervised learning approach for MARL which builds on multi-agent transformers to improve results in several domains. This approach consists of learning a centralized world model as an auxiliary task for learning better representations.", "summary_of_the_review": "I like the fact that the proposed approach is simple, builds on existing work without adding an overly complicated architecture, and is effective. My main hesitation with the paper is that it overcomplicates the exposition of a relatively simple idea and the presentation in the experimental section could use some work. However, these concerns are relatively minor.", "strength_and_weaknesses": "# Strengths\n* The main idea is straightforward and improves results.\n* The proposed approach is well motivated.\n\n# Weaknesses\n* The writing could be more clear. \n* Improvements over MAT (which MAJOR builds on) are marginal in many environments.\n* Baselines outside of MAT are methods which learn decentralized policies which seems an unfair comparison (unless my interpretation of the proposed approach as being fully centralized is incorrect - see questions).\n\n# Questions\n* From the description of the multi-agent transformer, it seems that observations from *all* agents are passed into a single network which computes actions. Given this description, the approach would be fully centralized, correct?\n* The authors state that the proposed framework could be applied to CTDE methods, though it is not clear to me how that is the case since the representation learning objective requires sharing information across agents.. Can the authors provide an example of how this may be accomplished in practice?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "GGR9z2ESrCw", "reviewer": "Reviewer_wdZd", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to integrate an SSL auxiliary objective with standard model-free MARL methods (specifically MAT in this paper) to reap the same benefits as CV and single-agent RL has seen with these types of approaches. Because each agent in a typical MARL setup receives a partial observation of the world, in order to apply the SSL auxiliary objective, they treat the individual observations as a sequence of masked views of the global state. Concretely, they use a joint transition model that inputs the individual agent observations and outputs the future observation representation for each agent, which implicitly requires a reconstruction of the global state. The joint-predictive auxiliary objective they introduce on top of MAT is BYOL-like prediction loss. The authors note that a similar objective can be applied to other model-free MARL methods. The new approach, MAJOR, is evaluated on a number of vision- and state-based multi-agent settings. Performance gains in sample efficiency are reported over MAT and other model-free MARL methods across the various settings. ", "summary_of_the_review": "* Overall the paper is well-written and provides a meaningful contribution to the field of MARL. There are some confusing and grammatically incorrect sentences here-and-there that should be corrected, and the claim of the module being plug-and-play is unsubstantiated (and if it were to be substantiated it would make up for the modest improvements in most of the tasks), but it is likely this paper will be useful to the research community in its present state. ", "strength_and_weaknesses": "Strengths: \n\n* The problem is well-motivated; there is a lack of MARL approaches that take advantage of SSL, and there are a number of challenges associated with doing so. \n\n* The proposed objective is a very natural incorporation of a BYOL-like SSL objective into an existing MARL approach. Moreover, the approach appears to be relatively applicable to other model-free MARL methods.\n\n* The writing is generally clear and gets the main ideas and implementation details across. Figure 1 (the illustration of the full MAJOR method) is also a very useful figure for understanding the gist of the method. \n\n* The method shows improvements in sample efficiency (and in some, final convergence in alloted training time) across the board on numerous and diverse multi-agent environments. \n\nWeaknesses: \n\n* Given that other baselines are used apart from MAT in the evaluation section, and that the authors proclaim the approach is “a plug-and-play module”, it would have been really nice to see the loss incorporated into all of the baselines to really drive home this point. Without this, it is less convincing that the module is indeed as simple as “plug-and-play”.\n\n* The methodology section is quite dense, and very few clarifying breathers are given while the stream of details are being revealed. This required me to jump back and forth quite a bit between the more conceptual explanations and the more technical explanations.\n\n* A lot of the intuition/understanding may require familiarity with previous works such as MAT and especially BYOL.\n\n* The results are nice, but the performance improvements (even in terms of sample complexity) are quite modest given the involvement of the approach. Especially for this reason it would have, again, been nice to have seen this module applied to multiple approaches. \n\n* Some grammatical issues / typos (there are others too—I suggest a careful rereading at the sentence-level):\n\n  * First sentence of 3.1 is very hard to parse (run-on sentence and probably grammatically incorrect): “We start with the intuition that encourages the observations and actions representations integrating other agents’ information and embodying agent-level relationships and interactions should reduce the non-stationary during the learning process, thus improving the data efficiency of MARL”\n\n  * Second paragraph 3.1 “since this way can execute” (not proper grammar)\n\n  * First sentence of 3.2 doesn’t make sense: “Multi-Agent Joint-Predictive Representations(MAJOR) is an auxiliary objective to promote the learned representations from the latent space of the sequential observation and action embeddings” (promote what in the learned representations?)\n\n  * Grammar and spelling at top of experiments: “We consider wide-rage MARL benchmarks for evaluating MAJOR and compared MARL algorithms” (”wide-rage” and “and compared”; maybe something like “a wide ranging set of” and “in comparison to relevant SOTA” is meant?). \n\n  * Confusing phrase at the end of “Multi-Agent Quadcopter Control” paragraph in 5.2: “our proposed representation learning framework shows its strength if the underlying approaches don’t work at all” (unclear what is meant here).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject