O-MAPL: Offline Multi-agent Preference Learning
Authors: The Viet Bui
Tien Anh Mai
Thanh Hong Nguyen
Venue: ICML 2025 poster (2025)
Keywords: Multi-agent Reinforcement Learning
Preference Learning
Primary area: reinforcement_learning->multiagent
TL;DR: Offline Multi-agent Preference Learning

Abstract: Inferring reward functions from demonstrations is a key challenge in reinforcement learning (RL), particularly in multi-agent RL (MARL). The large joint state-action spaces and intricate inter-agent interactions in MARL make inferring the joint reward function especially challenging. While prior studies in single-agent settings have explored ways to recover reward functions and expert policies from human preference feedback, such studies in MARL remain limited. Existing methods typically combine two separate stages, supervised reward learning, and standard MARL algorithms, leading to unstable training processes. In this work, we exploit the inherent connection between reward functions and Q functions in cooperative MARL to introduce a novel end-to-end preference-based learning framework.
Our framework is supported by a carefully designed multi-agent value decomposition strategy that enhances training efficiency. Extensive experiments on two state-of-the-art benchmarks, SMAC and MAMuJoCo, using preference data generated by both rule-based and large language model approaches demonstrate that our algorithm consistently outperforms existing methods across various tasks.

O-MAPL: Offline Multi-agent Preference Learning
The Viet Bui
Tien Mai
Hong Thanh Nguyen 2
Abstract
by RL algorithms, which might find ways to achieve high expected returns by inducing unexpected or undesirable be- haviors (Hadfield-Menell et al., 2017; Turner et al., 2020).
Inferring reward functions from demonstrations is key challenge in reinforcement learning (RL., particularly in multi-agent RL (MARL), where large joint state-action spaces and complex inter- agent interactions complicate the task. While prior single-agent studies have explored recov- ering reward functions and policies from human preferences, similar work in MARL is limited. Ex- isting methods often involve separate stages of su- pervised reward learning and MARL algorithms, leading to unstable training. In this work, we intro- duce a novel end-to-end preference-buee based Iearn- ing framework for cooperative MARL,, leveraging the underlying connection between reward func- tions and soft Q-functions. Our approach uses desiggee multi-agent value decomposi- tion strategy to improve training efficiency. Ex- tensive experiments on SMAC and MAMuloCo benchmarks show that our algorithm outperforms existing methods across various tasks.
To address these challenges, many RL studies have relaxed the reward structure by using sparse rewards, where agents receive feedback periodically (Arjona-Medina et al., 2019; Ren et al., 2021: Zhang et al., 2024b). While this reduces the need for dense reward signals, it is often insufficient train effective agents in complex domains. Alternatively, im- itation learning (IL) has been explored, where agents learn to mimic an expert's 's policy from demonstrations, without explicit reward signals (Ho & Ermon, 2016; Fu et al., 2017; Garg et al., 2021; Mai et al., 2024). However, achieving expert-level performance with IL requires a large amount of expert data, which can be costly and difficult obtain.
A recent promising approach is to train agents using human preference data, more resource-efffcient form of feed- back called Reinforcement Learning from Human Feedback (RLHF). This allows agents to learn behaviors aligned with human intentions. RLHF has proven effective in both single- agent control (Christiano et al., 2017: Mukherjee al., 2024; Lee et al., 2021: Shin et al., 2023; Hejna & Sadigh, 2024) and natural language tasks (Stiennon et al., 2020; Ouyang et al., 2022; Rafailov et al., 2024). However, RLHF in multi-agent environments is still underexplored, as simply extending single-agen methods is insufficient due to the complex interdependencies between agents' policies.
1. Introduction
Reinforcement learning (RL) has been instrumental in a wide range of decision making tasks, where agents gradu- ally learn to operate effectively through interactions with their enviromment (Levine et al., 2016; Silver et al., 2017: Kalashnikov et al., 2018; Haydari & Yılmaz, 2020). Typ- ically, when an agent takes an action, it receives feedback in the form of reward signals, enabling it to adjust or revise its action plan (i.e., policy). However, designing an appro- priate reward function is a significant challenge in many real-world domains. While essential for training successful RL agents, reward design often requires extensive instru- mentation or engineering (Yahya et al., 2017; Schenck & Fox, 2017; Peng et al., 2020; Yu et al., 2020; Zhu et al., 2020). Moreover, such reward functions can be exploited
Only a few recent studies have developed preference-based RL algorithms for multi-agent settings (Kang al., 2024; Zhang et al., 2024a), typically using a two-phase learning framework: first, preference data trains reward model, and then the policy is optimized. However, this approach has two main drawbacks: (i) it requires large preference datasets to cover the state and action spaces, and (ii) misalignment between the two phases can degrade policy quality.
In this work, investigate multi-agent preference- based RL (PbRL), focusing on the offline learning setting where agents do not interact with the environment but instead have access to an offline dataset of pairwise trajectory preferences. Unlike previous studies in multi-agent PbRL, we propose an end-to-end learning approach that directly trains agents' policies from preference data, without relying on an explicit reward model. Our main contributions are as follows:
Equal contribution 'Singapore Management University, Sin- gapore UUniversity Oregon Eugene, Oregon, United States. Correspondence to: Tien Mai
Proceedings of the nnnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

O-MAPL: Offline Multi-agent Preference Learning
First, we introduce new algorithm, O-MAPL (Offline Multi-Agent Preference Learning) for multi-agent PbRL.. O- MAPL exploits the inherent relationship between the reward and the soft-Q functions in MaxEnt RL (Garg et al., 2021; 2023) to directly learn the soft Q-function from preference data, rather than recovering the reward function explicitly. Once the Q-function learned, the optimal policy can be derived. This one-phase learning process carried out un- der the centralized training with decentralized execution (CTDE) paradigm (Oliehoek et al., 2008; Kraemer & Baner- jee, 2016), allowing effective training of local policies.
approaches (Meng et al., 2023; Tseng al., 2022).
Preference-based reinforcement learning (PbRL.. Early works developed general frameworks using linear approximations Bayesian models to incorporate human feedback on policies, trajectories, and state/action pairwise comparisons into policy learning (Fürnkranz et al., 2012; Akrour et al., 2012: 2011; Wilson et al., 2012). Recent studies have shown the effectiveness of training deep neural networks in complex domains with thousands of preference queries, typically following a two-phase approach: first, supervised learning to train reward model, then RL to optimize the policy, For example, (Christiano et al., 2017) uses the Bradley-Terry model for pairwise preferences and methods like A2C (Mnih, 2016) to refine the policy. Subsequent studies have expanded this framework to scenarios like preference elicitation (Mukherjee et al., 2024; Lee et al.. 2021), few-shot learning (Hejna III & Sadigh, 2023), data and preference augmentation (Ibarz et al., 2018; Zhang et al., 2023), list-wise learning (Choi et al., 2024), hindsight preference learning (Gao et al., 2024), and Transformer-
Implementing this end-to-end process within the CTDE framework is far from being trivial. It requires appropri- ate mixing networks for value factorization to preserve the convexity of the preference- based learning objective and ensure local-global consistency in policy optimality. As second contribution, we introduce a simple yet effective value factorization method and provide comprehensive theoretical analysis of the convexity and local-global con- sistency requirements. This approach enables stable and efficient policy training.
learning (Kim et al., 2023).
Finally, we conduct extensive experiments on two bench- marks, SMAC and MAMuJoCo, using preference data gen- erated by both rule-based and large language model ap- proaches. The results show that our O-MAPL consistently outperforms existing methods across various tasks.
Training reward models aligned with human preferences can be costly, requiring large volumes of preference data, especially in complex domains. This has led shift towards end-to-end frameworks that directly learn optimal policies from preference data, bypassing explicit reward models. For example, (Hejna et al., 2023) and (An et al., 2023) use contrastive learning to eliminate reward modeling, while (Kang et al., 2023) employs information matching to learn optimal policies in one step. In (Hejna & Sadigh, 2024), the IPL algorithm learns a Q-function directly from expert preferences, instead of modeling the reward function.
2. Related Work
Offline multi-agent reinforcement learning (MARL). Our work is related to offline MARL, relying solely on of- fline data to learn policies without direct interaction with the environment. Unlike standard offline MARL, we consider data only showing pairwise trajectory preferences (without rewards). Like offline MARL, it faces challenges such as distributional shift and complex interactions in large joint state and action spaces. Many existing MARL methods use the CTDE framework (Oliehoek et al., 2008; Kraemer & Banerjee, 2016), enabling efficient learning while allowing independent operation of agents. Regularization techniques are also applied to mitigate distributional shift (Yang et al., 2021; Pan et al., 2022; Shao et al., 2024; Wang et al., 2022).
While preference-based RL is well-explored in single-agent settings, research in multi-agent settings remains limited due to the complexity of agent interactions and large joint state-action spaces. Only a few studies have extended the two-phase preference- based framework to multi-agent set- tings (Kang et al., 2024: Zhang et al., 2024a). Building IPL's success in single-agent settings (Hejna & Sadigh, 2024), we leverage the reward-Q-function relationship to avoid explicit reward modeling. Adapting this to multi- agent environments is challenging, requiring careful design of mixing networks within the CTDE framework and thor- ough theoretical analysis of the preference- learning objective's 's convexity and global-local policy consistency.
For example, some works extend CQL (Kumar et al., 2020), well-known single-agent offline RL algorithm, multi- agent settings (Pan et al.. 2022: Shao et al., 2024). Others adopt the popular DICE framework, which regulates policies (OOD) issues in both competitive and cooperative settings (Mat- sunaga et al., 2023; Bui et al., 2025). Additionally, (Wang et al., 2022) explore a policy constraint framework tackle OOD problems. Some studies apply sequence modeling techniques to solve offline MARL using supervised learning
in the occupancy space to address
3. Background
Multi-agent Reinforcement Learning. We focus on co- operative MARL, modeled as a multi-agent Partially Ob- servable Markov Decision Process (POMDP) defined by M A, P,r, Z, O, n, n,,, where n is the number of
2

---

O-MAPL: Offline Multi-agent Preference Learning
agents, and A fl,...,,, } is the set of agents. The true state of the environment is denoted by E S, and the joint action space is given by A = ILE A,, where A, is the set of actions for agent i E N.. At each time step, every agent i E N selects an action 0g E A,, resulting in a joint action E A. The transition dynamics are described by P(s'|s, a) S x A S + [0, IJ, which is the probability of moving to the next state s' given the current state and joint action a. The discount factor y E [o, 1) determines the relative importance of future rewards.
where Vist is the log-sum-exp of Vist Moreover, by rear- ranging the Bellman equation, we get the so-called inverse soft Bellman-operator, formulated as follows:
(T* (s. a) = Qror a) - - P(-bs,)))))) (so)
a = (a1,
An important observation here is the one-to-one mapping between any Qtot and r a), i.e., r(s. a) - toe) a). This property has been extensively utilized in inverse RL (Garg et al., 2021; Hejna & Sadigh, 2024; Bui et al., 2024). The key idea is that, rather than explicitly recov- ering reward function, the unique mapping enables the reformulation of reward learning as a Q-learning problem. This approach improves stability and can directly recover the optimal policy from the learned Q-function using (1).
In partial observability settings, each agent receives a local observation O; E O, based on the function 2,(s) S O and the joint observation is denoted by o = (01, O2...., o,). In cooperative MARI,, agents share a global reward fune- tion r(s, a): : 8 x A R. The objective is learn joint policy Two { ................................ Tn. } that maximizes the expected dis- counted cumulative rewards (1 a)~ IE== (r,s,, ]- In offinn settings, a dataset D is pre-collected by sampling from behavior policy Prot- = {,I,... J. Policy learn- ing is then carried out using this dataset D only.
Note that, in POMDP scenarios, the global state s is not directly accessible during training and is instead represented by the joint observations from the agents. For notational convenience, we use the global state s in our formulation; however, in practice, it corresponds to the joint observa- tion (s). Specifically, terms like too a) and Qior a) actually refer to Ftot (o, a) and Qtor (o, a), where o = (s).
MaxEnt Reinforcement Learning. Standard RL opti- mizes policy that maximizes the expected discounted cu- mulative rewards Ext. [>:= 2"r(s, a:))", where (s,, at) are sampled at each time step from the trajectory distri- bution induced by the joint policy Teot- In a generalized MaxEnt RL, the standard reward objective is augmented with a KL-divergence term between the joint policy and a behavior Moot that generates the offline dataset, as follows:
4. Multi-agent Preference-based RL (PbRL)
4.1. Preference-based Inverse Q-learning
Following prior works (Christiano et al., 2017; Lee et al., 2021; Kang et al., 2024), we assume access to pairwise preference data. The data, collected from humans (or ex- perts), consists of pairs of trajectories 02), where σ1 is preferred over d Each trajectory o is sequence of joint (state, action) pairs: a = {(s), ai), ai).... (SK, ax)}; }. Let P denote the preference dataset, comprising several pairwise comparisons (01, 02). The goal of PbRL is to recover the underlying reward function and expert policies from P.
E ETTo Σn ((((5)) a+) Blog
Tent (ar. (x))
Meor (a (s:)
where 3 is the regularization parameter. Setting (oot to the uniform distribution reduces this to the standard MaxEnt RL objective. The regularization term enforces a conservative KL constraint, keeping the learned policy close to the be- havior policy and addressing offline RL's -ito---dittribution challenges (Haamojj et al., 2018; Neu et al., 2017).
A common approach in PbRL is model the expert's pref- erences using the simple and intuitive Bradley-Tere model (Bradley & Terry, 1952), which computes the probability of the expert preferring σn over 02 (denoted as 0i > 02) as:
In the above MaxEnt framework, the soft-Bellman opera- tor B= SxA + S is defined as (B, (s, a) = r(ss a) BPP Viae (s'), where Qtor is the soft-global- Q function and Veot is the optimal soft-global-valee function computed as a log-sum-exp of Qror. as follows:
P( > +0)) (itae, *E(saa) - Diaaee (2(a.)
where r a) is the reward function of the expert. Using this model, direct approach to recovering the expert re- ward function rg involves maximizing the likelihood of the preference data P. which can be formulated as follows:
Qtor (s.a)) a)
Vioe(s) = 33 log [E [Ea-paa (st)) Meoo (a)s) exp
B
The Bellman equation (B; Qear) Qiot will yield a unique optimal global Q-function Qia, and the corresponding opti- mal policy is given by (Haarnoja et al., 2018):
max,, L(relP) max Eio In P(o1 > 02)
Qioc (s. a) Vi+ (s)
(S).
fo [a|s] = Moot (als) exp
(1)
Once the expert rewards T'g are recovered, a policy can be learned by training a MARL algorithm. This method is referred to as a two-phase approach, where the reward learning and policy optimization are performed separately.
B
We adapt the formulas from single-agent MaxEnt RL the multi-agent setting, ensuring consistency in notation.
3

---

O-MAPL: Offline Multi-agent Preference Learning
The MaxEnt RL framework discussed above provides an alternative approach to integrate reward and policy recovery into single learning process. This is achieved by leveraging the unique mapping between a reward function and Q- function. Multi-agent PbRI. is thereby transformed into the Q-space, where the preference probability over a trajectory pair (01, 02) can be computed as follows:
The mixing function Mi can be either linear combination (single-layer) nonlinear combination (e.g., two-layer network with ReLU activation). Our work uses the simple linear structure, which has two key advantages over the non- linear approach. First, a two-layer structure often causes over-fitting and poor performance, especially in offline set- tings with limited data (Bui et al., 2025). Second, the linear structure ensures convexity in the learning objectives within the Q-space, leading to stable optimization and consistent bhiningg-beellee not present under two-layer mixing network structure.
Eo, (T (8,)))
P(01 > 02lQuat) = e_,,(T (ToQ)))((ss)) + e (T-Qco)(Ssa)
After solving the maximum likelihood problem, the derived Quo and Voo can be used directly to recover a policy via the soft policy formula (I), eliminating the need for an additional MARL algorithm. This unified, single-phase approach integrates reward and policy learning, streamlining the process. It enhances training stability and consistency by reducing discrepancies that arise from separate reward and policy learning. This approach also mitigates issues like error propagation and misalignment between the reward function and policy optimization. Training in the Q-space has been shown to outperform training in the reward space.
Overall, the training objective function, under the described mixing architecture, can be now expressed as follows:
C(g. V. w) = E Einnoneer Eisaien Rw lg v[(s, a)
log (eCo, Rala))((aa) + Coz R (s.a))(saa)
+ o Raw (.. (s).a)
where (--) is a concave regularization function used to pre- vent unbounded reward functions. In our experiments choose a x x'regularizer of the form x(x) = _ + I, which is also a commonly used regularizer in prior works.
4.2. Value Factorization
The training objective in the Q-space can be formulated as:
It is important to note that Que and Via must satisfy the Bellman operator, meaning that Via needs to be the log-sum- exp of Qun- To achieve this, we train M,. [v(s) (or Viot) to approximate the log-sum-exp formulation:
max LiQuol (P) = max Qui
Dinienne In P(a1 > 02l Qtoc)
Σ
Qui
While this objective works in singlc-agent settings, applying it to multi-agent scenarios is challenging due to the large state and action spaces. To address this, we apply value factorization in the CTDE framework. However, solving PbRL under CTDE is complex, as the objective involves several components tied to Qior and Vio. Thus, a carefully designed value factorization method is needed to ensure consistency between global and local policies.
Veo((s) = :logg E.cA Heor (a (e (tonssa)))))
However, this can become computationally impractical in certain scenarios, such as environments with continuous action spaces. To address this, Extreme Q-Learning (XQL) (Garg et al., 2023) provides an efficient method to update the V V-funccion. Specifically, we define the extreme-V loss objective under our mixing framework as follows:
To address these challenges, we propose a value factoriza- tion method, specifically designed to ensure scalability in multi-agent environments while preserving the alignment between global and local objectives, thereby enabling sta- ble and effective learning. Our approach involves factoriz- ing the global value functions Q and Vo, into local func- tions using a mixing network architecture. Specifically, let q(s.a) = (i ($1, a1)..... gg((n. } be set of local Q- functions, and v(s) = (si), Un ( represent a set of local V-functions. To enable centralized learning, we in- troduce mixing network Mw. parameterized by learnable weights w, which combines the local functions q and to construct the global value functions Quer and Vine as follows:
M
J(V) ((
Mi ((s, a)] MM(v(s)
E
B
Minimizing J(v) over v ensures that M. [v()) converges to the log-sum-exp value (Garg et al., 2023):
Mw[[v(s] 3 log (Eaea Mon Maltsa)/))))
Following this approach, training the local functions q and v can proceed through the following alternating updates:
Quni a) = Mar [q(s, a)]; Vsn(s) = (s))
Update q, w: Maximize L(q, v, w). the likelihood objective for preference learning.
For notational simplicity, let us define:
R (q. a)( = Qtor a) - -PE (s')
Update v: Minimize the extreme- loss J(v) to en- force consistency with the log-sum-exp equation.
MM (ss 715 / (sa)) Mu

---

O-MAPL: Offline Multi-agent Preference Learning
The following proposition shows that the learning objective functions under our mixing architectures possess appealing properties, which contribute to stable and robust training.
the sum of (4 (s)) over all a, might not equal one. To ensure feasibility, normalization is required, but it disrupts the GLC principle, breaking consisteney between global and local policies. Also, the local functions U, and q: may not satisfy the local Bellman equality (i.e., f2 is not guaranteed to be the log-sum-exp of q:), causing the soft policy formula to misalign with MaxEnt RL principles at the local level.
Proposition 4.1 (Convexity). The loss [(q, V w) is con- cave in q and w (the parameters of the mixing networks), while the extreme-V loss function J(v) is convex in V.
Given that the objective is to maximize the likelihood func- tion C(q. v. w) and minimize the extreme-V V function J (v), the concavity of L in q and ve, and the convexity of J in v, guarantees unique convergence (theoretically) within the q and v spaces, ensures stable training process in practice.
Our weighted behavior cloning approach. We propose alternative approach that offers several advantages over the previous method. Our policy extraction is based on BC, technique commonly used in offline RL. algorithms (Garg et al., 2023; Bui et al., 2025). This approach preserves the GLC property and ensures that the extracted local policies are valid, even with nonlinear mixing structures.
It is important to note that convexity is guaranteed only under single-layer mixing structures, where M is linear in its inputs. This result is formalized below:
In general, the global policy can be extracted by solving the following weighted behavior cloning (WBC) problem:
Proposition 4.2 convextyy under two-layer mixing net- works). the mixing networks Ma [a] and M [v] are two- layer (or multi-layer) feed- based loss function L(q, V. w) is no longer concave in q or w, and the extreme-V loss function [VV) is not convex in v.
networks, the preference-
max Eega-ppu e The Elup
Q (a())) 3
log Two (als) }. (3)
where 1hot represents the feasible set of global poli- cies. Here, we assume that Thg contains decompos- able global policies, i.c., Ihot = {To
While two-layer feed-forward mixing networks have been employed in several prior online MARL works, single-layer mixing networks (i.e., linear combinations) have been fa- vored in recent offline MARL works (Wang al., 2022; Bui et al., 2025). It was demonstrated that using two- layer network can lead to over-fitting issues, resulting in worse performance compared to their single-layer counter- parts (Bui et al., 2025). The results in Prop. 4.2 further suggest that, in offline preference- learning, a single- layer setup is more efficient and better suited to achieve robust and stable performance.
_ā, Vi N,, such that Tma = ITix (a;]s:))] In other words, Hht consists of global policies that can be expressed as a product of local policies. This decomposability is highly use- ful for decentralized learning and has been widely adopted in multi-agent reinforcement learning (MARL) (Wang et al., 2022; Bui et al., 2024; Zhang et al., 2021).
While solving (3) can explicitly recover an optimal global policy and is practical via sampling (s,, a) from the data, it does not support the learning of local policies, which is essential under the CTDE principle. To address this, we propose solving the following local WBC problem:
4.3. Local Policy Extraction
Simple local-value-baseed extraction approach. Glob- ally optimal policies can be extracted from Que and ViN based on (I). For decentralized execution, local policies can be derived from local values similarly (Wang et al., 2022):
max (Eaa-pal
Que Quale Vania)
log o. (a: (a:lss))) (4)
The local WBC approach has several key advantages. First, the weighting term e Quals.a)- Var(s) directly influences local policy optimization and is computed from global observa- tions and actions. This ensures local policies are optimized with global information, maintaining consistency in cooper- ative multi-agent systems. Furthermore, as shown in Theo- rem 4.3, optimizing local policies via WBC always results in valid policies that align with the global WBC objective, pre- serving global- local consistency (GLC). Importantly, these benefits hold regardless of the mixing structure (e.g.. 1-layer 2- 2-aye networks). offering significant advantages over the caal-value-extrattin method.
क. (a.ls.) = ps(as s.) exp (w qf a:) w u(()))))
B
where wf and wo are the weights of the mixing function Mi [q] and Mw (v), and (.)()) are the local behavior poli- cies. Assuming the behavior policy is decomposable into local components, i.e., (als) = II. (a(aa) si), this pol- icy extraction method guarantees global-local consistency (GLC)- ensuring alignment between the optimal global and local policies- such that Ti (a|s) II, (s:ls:).
This approach has been used in prior work but has notable limitations. First, GLC holds only with a linear mixing struc- ture; a two-layer feed-forward network breaks this property. Second, policies recovered from (2) may not be feasible, as
Theorem 4.3 (Global-Local Consistency (GLC)). Let पद be the optimal solution to the local WBC problem in (4). Then, the global policy The defined as T a) = II, x (a.)s;). is also optimal for the global WBC problem in (3).
5

---

O-MAPL: Offline Multi-agent Preference Learning
We next formally express the relationship between recovered local policies and value functions. We assume that the behav- ior policy is decomposable, i.e., Mooo a) = 11: (a:]ss)) and the mixing structures are defined as Mu (s(, a)] =
Algorithm 1 O-MAPL
1: Input: Parameters 8, ggg te w4. Offline data P.
2: Output: Local optimized polices TT-
3: for a certain number of training steps do
E, wº ge(8;; az) + by and [v)) = E; w. (x)))) + bo-
4: Update by and 0 to maximize [(w,, Ve 0)
This relationship is formalized in the following theorem:
5: Update te to minimize the Extreme-V J (1ze
Theorem 4.4. Let oi be optimal to the local WBC, then the
Update wi to maximize the local WBC loss VP (ws)
following equality holds for all 8i E St, 0g E As:
end for
(7(s))
8: Return T:(a;) (0gloo ,,...,? 7
(a:lss) = (ss;) ps (s(0s/8s)e
(5)
where (7s))/ (ss)) are correction terms."
where Mal- is linear combination (or one-layer mixing network) of its inputs with non-negative weights:
Theorem 4.4 highlights key aspects of our approach. First, as seen in (2), directly computing local policies from the lo- cal value functions 9: and iú alone may yield invalid policies that don't 't form proper probability distributions. The term m(())/ (8:) in (9) acts as a correction factor, normalizing the policies to ensure E., (a.(s,) = 1. Furthermore, the proof of Theorem 4.4 shows that both n/ss)/ (s;) and the local policy T. (a,1st) depend on the value functions of other agents. This dependency supports the principle of credit assignment in cooperative MARL, ensuring each agent's policy accounts for the actions and rewards of others.
[b)] = vols))) T Wg + bg
(6)
Molq(o, [[y,)]] = q(o, (11g) W + bg
(7)
Here, W8. bg. W , are the weights of the mixing net- works, modeled as hyper-networks that take the global ob- servation o, joint action and the learnable parameters i as inputs. In this setup, we employ the same mixing network Mo to combine both the local V and Q functions, ensuring consistency and scalability in the aggregation process.
The practical training objective function for the local Q functions can be calculated as:
Additionally, while ViN is the log-sum-exp of Qwa. this might not be the case for the local v; and qi functions. The
following proposition demonstrates that & can indeed be expressed as a log-sum-exp of qt- along with an additional term that depends on the local functions of other agents.
L(vm Go, 0)
E Eion,osEr Emnaoyon R(o, o.)
- log (Ro., + C R..,(')))))) ))+ + d (R((,
Proposition 4.5. Each local value v, can be expressed as a (modified) log-sum-exp of the local Q-function qs:
where R(o, = Molq(o, al (b)) - MMg over, the extreme- V can be practically estimated as:
More-
3 W
2
3
(18))
(s))s)
log
log
w!
((s;)
Mulginals , 11 Ma (rcol
(2
I [1bb) = E ()o,a))
Molg(o, al al - Mal (Volsa))]
Prop. 4.5 indicates that (s))) is also determined by a log- sum-exp of qi(s;, az) with additional term log ( [ifa.))
- -((oaa)~
B
(3(s))
For the policy extraction, let IH, (r(aa,o;;,), (2) be a local policy network for each agent i, where w are learnable parameters. We update the local policies using the following local WBC:
5. Practical Algorithm
In the context of POMDPs, we do not have direct access to the global states. To better reflect the practical aspects, we change the notation of global states used previously to global observations. For example, the local value function is now defined as a function of local observations, U; (o:).
V (wi) = -o.mpp
Me M MMp(veeii))
log Tee (o:
The outline of our O-MAPL is shown in Algorithm 1.
We construct a local Q-value network qi(o,, a. (1)) and a local value network U, (,(o)/e))), where u and the are learn- able parameters. The global Q and V functions are then aggregated using two mixing networks with shared set of learnable parameters 0, formulated follows:
6. Experiments
We evaluate the performance of our O-MAPL in dif- ferent complex MARL environments, including: multi- agent StarCraft II (i.c., SMACv1 (Samvelyan et al., 2019), SMACv2 (Ellis et al., 2022)) and multi-agent Mu- joco (de Witt et al., 2020a) benchmarks. Detailed descrip- tions of these benchmarks are in the appendix.
V (o) Me[v w.)l: (u a) Mola(o, a[v,)),
2 Detailed formulations of these terms are in Appendix A.4.
6

---

O-MAPL: Offline Multi-agent Preference Learning
Rule-based
LLM-based
Tasks
O-MAPL
O-MAPL
BC
IPPL
IPL-VDN
SL-MARL
BC
IIPL
IPL-VDN
SL-MARL
(ours)
(ours)
2c_vs_64zg
59.6+25.0
60.4+24.7
71.1=22.0
63.5+24.0
74,4+24.7
65.6#24.6
60.2=25.9
77.0=21.3
65.2+21.2
19.5+19.6
5m_vs_6m
16.8+18.0
14.3=17.0
16.8=18.0
16.0=18.9
19.3:19.6
18.2+18.4
15.0=17.5
18.0=19.2
17.4
20.7+20.5
6h_vs_8z
0.6+3.8
0.2+2.2
2.5=7.6
1.6+6.8
4.5+11.0
0.8+4.3
0,4+3.1
3.5=9.2
3.7+8.9
6.1=11.2
corridor
89.3+15.5
89.8=15.4
93.9=11.6
49.0+22.8
93.2+13.5
89.6
90.6=13.6
14.5=12.5
57.6+22.2
94.5+11.2
5 vs. 5
38.1+24.2
31,4+25.2
54,5+25.9
49.0+28.2
54.3+24.2
48.4425.9
41,0#24.2
58.8+24.5
54.3+24.0
61,5+24,8
Protoss
10 vs 10
38.7+24.2
28.5+2188
47.9=27.2
40.6+23.2
53.7+23.6
46.3+24.0
41.0=24.4
57.0=23.4
52.5+22.1
61.1+24.8
vs
12.7=17.4
12.5=16.5
22.3+21.0
18.6=18.8
30.7+19.8
22.7+22.2
15.6=15.9
27.3+24.7
20.9+20.9
34,4+24.8
20 vs 20
39.8+24.9
35.4+21.5
57.0=24.8
38.7+23.1
59.8+23.
48, 48.4225.3
43.6423.6
61.5422.1
51.825.00
64.5+23.5
20 vs 23
15.2+18.5
9.0=14.2
22.7=21.7
11.1=14.6
23,4+19.2
18.0=17.4
9,4414.7
23,4=21.4
12.1=15.9
26.4+20.8
5.vs.5
27.5+24.0
16.2+19.5
36,3+24.8
34.2+23.4
39.5+24.7
31,1+22.9
34.8+23.0
41.0+23.7
36.7+24.8
43.0+23.0
Teran
10_vs_10
23.8+20.5
21.1+20.8
25.8+19.7
23.2+19.6
28.3+20.6
25.8
24.2421.6
32.0+24.4
28.9+24.7
33,2+23,4
10
10.2+15.4
7.2413.3
18.2+19.4
11.3+15.3
18.2+18.7
11.7+17.4
10.4
17.8
16.4
21.3+20.3
20.vs
13.1417.1
11.9218.2
21,5:20.4
8.8.13.5
23.0+22.4
14.5
13.7+17.4
21.1+20.4
17.2116.8
24.4+23.1
20.vs 23
3,9410.6
4.1+10.3
5.71114
2.347.3
7.2+12.9
6.4+12.2
3,549.2
7.2412.6
1.7+10.2
8.6
5.
23.4+21.1
23.6+210
31.1+20.4
33.0+22.5
35.2+25.7
31.1+22.3
26.0+22.2
34,8+23.6
35.0+23.2
40.8+21.6
10. vs. 10
25.8421.6
25.8=22.5
32.2424.6
30.7+24.0
34.8+22.1
31.4+21.9
31.1+24,8
35.5+23.9
33.0+25.0
37.9+24.0
Zegg
10 vs 11
19.3420.1
12.9=17.4
22.5+20.5
19.3=18.0
23,4+2..1
20.1
18.6420.6
22.7=18.3
23.0421.1
26.0+23.0
20 vs 20
19.9+21.0
11.1=16.2
22.5+21.4
5.7=10.9
24.8+20.8
22,9+21.7
16.0=17.3
27.3422.0
31.1+24.6 16.4+18.1
20 vs 23
13.1
7.8+12.8
12.5=15.3
7.6=13.1
18.8:18.5
15.8=18.5
10.4=15.2
16.4=19.9
13.7+17.4
16.0=19.4
Table I: Win rate comparison (in percentage) for SMACvl (first 4 tasks) & SMACv2.
Methods
Hopper-v2
Ant-v2
HalfChectah-v2
Specifically, we used offline datasets of varying quality from OMIGA (Wang et al., 2022) and ComaDICE (Bui et al., sampling one thousand pairs for MaMujoco tasks and two thousand pairs for SMAC tasks. For MaMujoco, we se- lected "mediun--rppay"" "medium", and "expert" instances, while for SMACv1, we chose "poor", "medium", and "good" instances. Note that ComaDICE only provides a "medium" dataset for SMACv2, therefore, we generated new "poor" and "expert" datasets for SMACv2. Additionally, LLM- based prompts require detailed information from trajectory states (e.g., SMAC: remaining health points, shields, relative positions, cooldown time, agent types, action meanings), which we cannot extract from MaMujoco states. Therefore, we have no L-L-- dataset for MaMujoco tasks.
BC
808.1 + 39.1
1303.9 # 122.0
+ +
IIPL
782.0 : 81.5
1312.0 = 155.6 4028.8 + 430.0
IPL-VDD
846.6 # 65.4
1376.1 # 142.0
4287.5 # 273.1
SL-MARL
890.0 # 88.7
1334.1 # 150.9
4233.9 # 303.1
O-MAPL
1114.4 # 154.1
1406.4 # 163.7 4382.0 + 189.7
Table 2: Return comparisons on MaMujoco tasks
BC
IIPL
IPL-VON
SL-MARL
O-MAPL (ours)
Hopper-v2
Ant-v2
HalfCheetah-v2
1k-
1.4k-
4.5k
4k-
500
1.2k-
3.5k
3k
0-
Baselines. We consider the following baselines for our evaluations: (i) Behavioral Cloning (BC) trains policy by directly imitating all prefered trajectories in the dataset P: (ii) Independent IPL (IIPL) is straightforward extension of the IPL approach (Hejna & Sadigh, 2024) to multi-agent learning, where the single-agent IPL. algorithm is applied independently to each agent; (iii) Supervised Learning MARL (SL-MARL) is a two-phase approach where we first learn the reward function and then use train policy with OMIGA (Wang et al., 2022), a state-of-ther -frt MARL a algorithm, serving as the offline counterpart of the two- phase approach in (Kang et al., 2024); and (iv) IPL-VDN. which is similar to our algorithm but without the mixing networks, instead employing the standard VDN approach (Sunehag et al., 2017) to aggregate local Q and V functions
100
100
50
200
Figure 2: Evaluation curves (in returns) on MaMujoco tasks
Dataset. There are no human-labeled preference datasets for MARL, so we create datasets for each task using two methods: (i) Rule-based method followed by IPL (Hejna & Sadigh, 2024), we sample trajectory pairs from offline datasets of varying quality (e.g.. poor, medium, expert) and assign binary preference labels based on dataset quality; and (ii) LLM-based method- - followed by DPM (Kang et al., 2024), we sample pairs from offline datasets and use GPT- 4o to annotate labels with prompts constructed from the global state of each trajectory (details in the appendix).
7

---

O-MAPL: Offline Multi-agent Preference Learning
BC
IPL
IPL-VDN
SL-MARL
O-MAPL (ours)
Protoss
60%
30%
0%
100
0
50
100
0
50
100
50
100
100
Terran
40%
20%
0%
100
100
50
50
100
100
Zerg
40%
20%
50
100
0
50
100
0
50
100
50
100
0
50
100
5_vs_5
10
10
10
10_vs_11 11
20 20_vs 20_vs_20
20
20_vs
20
23_
Figure I: Evaluation curves (in win rates) of our O-MAPL for SMACv2 with rule-based preference data.
via a simple linear combination with unit weights.
and detailed returns and win rates for all tasks can be found in the appendix.
Results. Overall, our experimental results demonstrate the effectiveness of O-MAPL in both continuous and discrete the
7. Conclusion
multi-agent reinforcement learning environments. following, we highlight some of our main results. Due limited space, all remaining results are in appendix.
Summary. We explored preference-based learning in multi-agent environments, proposing a novel end-to-end method based on the MaxEnt RL framework that elimi- nates the need for explicit reward modeling. To facilitate efficient training, we developed a new value factorization ap- proach that learns the global preference-basee loss function by updating local value functions. Key properties, includ- ing global-local consistency and convexity, were thoroughly examined. Extensive experiments on both rule-based and LLM-based datasets show that our algorithm outperforms existing methods across multiple benchmark tasks in the MAMuJoCo and SMAC environments.
Table 1 and provide a detailed comparison of win rates for SMACvl and SMACv2 tasks and of returns for MaMujoco. O-MAPL achieves the highest win rates/returns across most tasks, outperforming all baseline methods. For example, in the 2c.vs_642g task, O-MAPL achieves a win rate of 74.4%, significantly surpassing other methods. In corridor, O-MAPL achieves win rate of 93.2%, showcasing its ability to handle structured navigation tasks effectively.
Furthermore, Table demonstrates that our algorithm, O- MAPL, achieves higher win rates in most SMAC tasks when using LLLM-generade data than when using the ruled-based generated data. This finding highlights the potential of leveraging L.MSs for rich and cost-effective data genera- tion, substantially improving environment understanding and policy learning in complex multi-agent tasks.
Limitations and Future Work. The strong performance of LLM-based preference data suggests that leveraging LLMs, coupled with systematic value factorization ap- proach, can be highly effective for training policies in com- plex multi-agent environments. This opens promising av- enues for using LLMs to enhance both environment under- standing and policy learning. However, our work has some limitations that need further exploration. For example, we primarily focus on cooperative learning, while more chal- lenging mixed cooperative-compettivv environments would require different methodologies. Additionally, our method still depends on large number of preference-basee demon- strations for optimal policy learning. Although LLMs can quickly generate extensive demonstrations, improving sam- ple efficiency remains a key challenge, particularly when data must be collected from real human feedback.
Finally, we present evaluation curves for both SMACv2 (Fig- ure 1) and MaMujoco tasks (Figure 2). The results show that O-MAPL consistently and significantly outperforms other baselines throughout the training process. Our algorithm converges faster, achieving high win rates and returns at earlier training stages across most tasks. This demonstrates the effectiveness of our multi-agent end-to-end preference learning approach, supported by systematic and carefully designed value decomposition.
Additional details on dataset generation, hype-parametery
8

---

O-MAPL: Offline Multi-agent Preference Learning
Acknowledgment
Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in neural information pro- cessing systems, 30, 2017.
This work supported by the Lee Kong Chian Fellowship awarded Tien Mai.
Impact Statement
de Witt, C. S., Peng. B., Kamienny, P.-A., Torr, P., Böhmer, W., and Whiteson, S. Deep multi-agent reinforcement learning for decentralized continuous cooperative control. arXiv preprint ar wxxiv2003.06609, 19, 2020a.
This paper presents work whose goal is to advance the field of Machine Learning, in particular multi-agent reinforce- ment learning. There are many potential societal conse- quences of our work, none which we feel must specifi- cally highlighted here.
de Witt, C. S., Peng. B., Kamienny, P.-A., Torr, P., Böhmer, W., and Whiteson, S. Deep multi-agent reinforcement learning for decentralized continuous cooperative control. arXiv preprint arXiv: 2rXin:0003.007000 2020b. URL https: //arxiv. http://www.//abs//..000070
References
Akrour, R., Schoenauer, M., and Sebag, M. Preference- based policy learning. In Machine Learning and Knowl- edge Discovery in Databases: European Conference, ECML PKDD 2011, Athens, Greece, September 5-9, 2011. Proceedings, Part I 11, pp. 12-27. Springer, 2011.
Ellis, B., Moalla, S., Samvelyan, M., Sun, M., Mahajan, A., Foerster, J. N., and Whiteson, S. Smacv2: An im- proved benchmark for cooperative multi-agent reinforce- ment learning. arXiv preprint arXiv:2212.074883 2022.
Fu, J., Luo, K., and Levine, S. Learning robust rewards with adversarial inverse reinforcement learning. arXiv preprint arXiv: 1710. /248, 2017.
Akrour, R., Schoenauer, M., and Sebag, M. April: Active preference lcarning-based reinforcement learning. In Ma- chine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2012, Bristol, UK, September 24-28, 2012. Proceedings, Part II 23, pp. 116 131. Springer, 2012.
Fürnkranz, J., Hüllermeier, E., Cheng, W., and Park, S.- H. Preference-basee reinforcement learning: a formal framework and a policy iteration algorithm. Machine learning, 89: 123-156. 2012.
An, G., Lee, J., Zuo, X., Kosaka, N., Kim, K.-M,, and Song, H. O. Direct preference-based policy optimization with- out reward modeling. Advances in Neural Information Processing Systems, 36:70247 2023.
Gao, C.-X,, Fang, S., Xiao, C., Yu, Y., and Zhang, Z. Hind- sight preference learning for offline preference- re- inforcement learning. arXiv preprint 200%v2240..04451, 2024.
Arjona-Medina, J. A., Gillhofer, M., Widrich, M., Un- terthiner, T., Brandstetter, J., and Hochreiter, S. Rudder: Return decomposition for delayed rewards. Advances in Neural Information Processing Systems, 32, 2019.
Garg, D., Chakraborty, S., Cundy, C., Song. J., and Ermon, S. Iq-learn: Inverse soft-q learning for imitation. Ad- vances in Neural Information Processing Systems, 34: 4028-4039, 2021.
Bradley, R. A. and Terry, M. E. Rank analysis of incom- plete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-344 345, 1952.
Garg, D., Hejna, J., Geist, M., and Ermon, S. Extreme q-learning: Maxent rl without entropy. In International Conference on Learning Representations (ICLR). 2023, URL wwt.tttpp///// arwaaavvv...... attpaaxivvv.r.... orrg/ abs 2231 02.228
Bui, T. V., Mai, T., and Nguyen, Inverse factorized qeearnnig for cooperative multi-agent imitation learning. Advances in Neural Information Processing Systems, 38, 2024.
Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., al. Soft actor-critic algorithms and applications. arXiv preprint 1812 2018.
Bui, T. V., Nguyen, T. H., and Mai, T. ComaDICE: Offline cooperative multi-agent reinforcement learning with sta- tionary distribution shift regularization. In Proceedings of the International Conference on Learning Representa- tions (ICLR), 2025.
Hadfield-Menell, D., Milli, S., Abbeel, P., Russell, S. J., and Dragan, A. Inverse reward design. Advances in neural information processing systems, 30, 2017.
Haydari, A. and Yılmaz, Y. Deep reinforcement learning for intelligent transportation systems: A survey. IEEE Transactions on Intelligent Transportation Systems, 23 (1):11-32, 2020.
Choi, H., Jung, S., Ahn, H., and Moon, T. Listwise re- ward estimation for offline preference-beet reinforce- ment learning. arXiv preprint arXiv:2408.0 02888.04/990 04/90, 2024.

---

O-MAPL: Offline Multi-agent Preference Learning
Hejna, J. and Sadigh, D. Inverse preference learning: Preference-based rl without reward function. Advances in Neural Information Processing Systems, 36, 2024.
Mai, T. A., Nguyen, T. H., al. Inverse factorized soft q-learning for cooperative multi-agent imitation learn ing. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Hejna, J., Rafailov, R., Sikchi, H., Finn, C., Niekum, S., Knox, W. B., and Sadigh, D. Contrastive prefence learn- ing: Learning from human feedback without rl. arXiv preprint arXiv:2310.13666 2023.
Matsunaga, D. E., Lee, J., Yoon, J., Leonardos, S., Abbeel, P., and Kim. K.-E. Alberdice: addressing out-of- distribution joint actions in offline multi-agent rl via alter- nating stationary distribution correction estimation. Ad- vances in Neural Information Processing Systems, 36: 72648-72678, 2023.
Hejna III, J. and Sadigh, D. Few-shot preference learn- ing for human-in-thee -n-the-ooop rl. In Conference Robot Learning, pp. 2014-2025. PMLR, 2023.
Meng, L.. Wen, M., Le, C., Li, X., Xing, D., Zhang, W., Wen, Y., Zhang, H., Wang, J., Yang, Y., et al. Offline pre-trained multi-agent decision transformer. Machine Intelligence Research, 20(2):233-248, 2023.
Ho, J. and Ermon, S. Generative adversarial imitation learn- ing. Advances in neural information processing systems, 29, 2016.
Ibarz, B., Leike, J., Pohlen, T., Irving, G., Legg, S., and Amodei, D. Reward learning from human preferences and demonstrations atari. Advances in neural information processing systems, 31, 2018.
Mnih, V. Asynchronous methods for deep reinforcement learning. arXiv preprint arXiv:1602.01783 2016.
Mukherjee, S., Lalitha, A., Kalantari, K., Deshmukh, A.. Liu, G., Ma, Y., and Kveton, B. Optimal design for human preference elicitation. 2024.
Kalashnikov, D., Irpan, A., Pastor, P., Ibarz, J., Herzog, A., Jang, E., Quillen, D., Holly, E., Kalakrishnan, M., Van- houcke, V., et al. Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on robot learning, pp. 651-673. PMLR, 2018.
Neu, G., Jonsson, A., and Gómez, V. A unified view of entropy-regularizze markov decision processes. arXiv preprint arXiv: / 1705.07798, 2017.
Kang, S., Lee, Y., and Yun, S.-Y. Dpm: Dual preferences- based multi-agent reinforcement learning. In ICML 2024 Workshop on Models of Human Feedback for Al Align- ment, 2024.
Oliehoek, A., Spaan, M. T., and Vlassis, N. Optimal and
approximate q-value functions for decentralized pomdps. Journal of Artificial Intelligence Research, 32:289-3533 2008.
Kang, Y., Shi, D., Liu, J., He. L., and Wang. D. Beyond reward: Offline preference-guiee guided policy optimization. arXiv preprint arXiv:2305.16211 2023.
OpenAL Gpt-4o. 2024. https: /ttts::/cpenai.... com/ index/hello-gpt-44
OpenAL. Openai api pricing, 2025, URL https: ppenal.com//piipppiciin http///pppppiing////// Accessed: 2025-01- 31.
Kim, C., Park, Shin, J., Lee, H., Abbeel, P., and Lee, K. Preference transformer: Modeling human preferences ing transformers for rl. arXiv preprini arXi::2303.00957, 2023.
Ouyang, L.., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730 2022.
Kraemer, L. and Banerjee, B. Multi-agent reinforcement learning as a rehearsal for decentralized planning. Newro- computing, 190:82-94, 2016.
Kumar, A., Zhou, A., Tucker, G., and Levine, S. Con- servative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33: 1179-1191, 2020.
Pan, L.. Huang, L., Ma. T., and Xu, H. Plan better amid conservatism: Offline multi- aueti- reinforcement learning with actor rectification. In International conference on machine learning, pp. 12..172 17221-17233 PMLR, 2022.
Lee, K., Smith, L., and Abbeel, P. Pebble: Feedback- efficient interactive reinforcement learning via relabeling experience and unsupervised me-training. arXiv preprint wwXiv:2/06.0509,, 2021.
Peng, X. B., Coumans, F., Zhang, T., Lee, T.-W,, Tan, J., and Levine, Learning agile robotic locomotion skills by imitating animals. arXiv preprint 20X1v:2004.00784, 2020.
Levine, S., Finn, C., Darrell, T., and Abbeel, P. End-to-end training of deep visuomotor policies. Journal of Machine Learning Research, 17(39):1-44) 2016.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Er- mon, S., and Finn, C. Direct preference optimization:
10

---

O-MAPL: Offline Multi-agent Preference Learning
Your language model is secretly reward model. Ad- vances in Neural Information Processing Systems, 36, 2024.
Wilson, A., Fern, A., and Tadepalli, P. A bayesian approach for policy learning from trajectory preference queries. Advances in neural information processing systems, 25, 2012.
Ren, Z., Guo, R., Zhou, Y., and Peng. J. Learning long-term reward redistribution via randomized return decomposi- tion. arXiv preprint arXiv:2111.13455 2021.
Yahya, A., Li, A., Kalakrishnan, M., Chebotar, Y., and
Levine, S. Collective robot reinforcement learning with distributed asynchronous guided policy search. In 2017 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 79-86. IEEE, 2017.
Samvelyan, M., Rashid, T., De Witt, C. S., Farquhar, G., Nardelli, N., Rudner, T. G., Hung, C.-M., Torr, P. H., Foerster, J., and Whiteson, S. The starcraft multi-agent challenge. arXiv preprint 192iv:190..040433 2019.
Yang. Y., Ma, X., Li, C.,, Zheng, Z.., Zhang, Q., Huang, G., Yang, J., and Zhao, Q. Believe what you see: Implicit constraint approach for offline multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 34:10299-10311 2021.
Schenck, C. and Fox, D. Visual closed-loop control for pour- ing liquids. In 2017 IEEE International Conference on Robolics and Automation (ICRA), pp. 2629-2636. IEEE, 2017.
Yu, C., Velu, A., Vinitsky, E.,, Gao, J., Wang, Y., Bayen, A., and Wu, Y. The surprising effectiveness of ppo in cooper- ative multi-agent games. Advances in Neural Information Processing Systems, 35:24611 15224411-24624,
Shao, J., Qu, Y., Chen, C., Zhang, H., and Ji, X. Coun- terfactual conservative learning for offline multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
24624, 2022.
Yu, T. Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., and Levine, S. Meta-world: A benchmark and evalua- tion for multi-task and meta reinforcement learning. In Conference on robot learning, pp. 1094-1100 PMLR, 2020.
Shin, D., Dragan, A. D., and Brown, D. S. Benchmarks and algorithms for offline preference-basee based reward learning. arXiv preprint arXiv:2301.01339. 2023.
Silver, D., Schritwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al. Mastering the game of go without human knowledge. nature, 350(7676):354 359, 2017.
Zhang, N., Wang, X., Cui, Q. Zhou, R., Kakade, S. M., and Du, S. S. Multi-agent reinforcement learning from hu- man feedback: Data coverage and algorithmic techniques. arXiv preprint arXiv:2409.00717. 2024a.
Stiennon, N., Ouyang, L.., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. F. Learning summarize with human feedback. Ad- vances in Neural Information Processing Systems, 33: 3008-3021, 2020.
Zhang, T., Li, Y., Wang, C., Xie, G., and Lu, Z. Fop: Fac- torizing optimal joint policy of maximum maximum-entropy entropy multi- agent reinforcement learning. In International conference on machine learning, pp. 22491-12500 PMLR, 2021.
Zhang, Y., Du, Y., Huang, B., Wang, Z., Wang. J., Fang, M., and Pechenizkiy, M. Interpretable reward redistribution in reinforcement learning: a causal approach. Advances in Neural Information Processing Systems, 36, 2024b.
Sunehag, P., Lever, G., Gruslys, A., Czamecki, W. M., Zam- baldi, V., Jaderberg, M., Lanctot, M., Sonnerat, N., Leibo, Z., Tuyls, K., al. delle-decomposttion networks for cooperative multi-agent learning. arXiv preprint ImXiv:1706.05266, 2017.
Zhang, Z., Sun, Y., Ye, J., Liu, T.-S., Zhang, J., and Yu, Y. Flow to better: Offline preference srrerence-based reinforce- ment learning via preferred trajectory generation. In The Twelfth International Conference on Learning Represen- tations, 2023.
Tseng. W.-C., Wang, T.-H. J., Lin, Y.-C., and Isola, P. Of- fline multi-agent reinforcement learning with knowledge distillation, Advances in Neural Information Processing Systems, 35:226-237, 2022.
Zhu, H., Yu, J., Gupta, A., Shah, D., Hartikainen, K., Singh, A., Kumar, V., and Levine, S. The ingredients of real- world robotic reinforcement learning. arXiv preprint trXiv:2004.12570, 2020.
Turner, A., Ratzlaff, N., and Tadepalli, Avoiding side effects in complex environments. Advances in Neural Information Processing Systems, 33:21406-21411 2020.
Wang, X., Xu, H., Zheng, Y., and Zhan, x. Offline multi- agent reinforcement learning with implicit global-to-loce value regularization. Advances in Neural Information Processing Systems, 36, 2022.

---

O-MAPL: Offline Multi-agent Preference Learning
A. Missing Proofs
We provide proofs that are omitted in the main paper.
A.1. Proof of Proposition 4.1
The preference-based loss function Lig, v, w) is concave in and w (the parameters of the mixing networks), while the extreme-V loss function J (v) convex in V.
Proposition 4.1:
Proof. We first recall that the preference- loss function has the following form:
L(q,v, = Σ Σ
Rw q. vj a) log (Z Re (9.](((8).)) + Doz Rew (nll))))))))) + @B Ru (q, vs,
(MI. (s.a)E
We observe that under the assumption that the mixing networks are linear in their inputs, the function R (q. vl a) linear in qs,, a) and 0. This implies that for any [0, and for any vectors of local values q², q², have:
a /q v](s, a) + (1 - R. pq": v) a) la + (1 a)q²; v).
Now consider the term @ R. q. a)) Since @ concave, we have the following inequality for any a E (0, 1) and two vectors q¹.q² q²:
(R (a. a)) + (1 (R. (9" v) a)) s (aaRnl v)(s + a
( v)(s,
S o (Rn
)) v(ssaa)
(8)
which implies the concavity of Hi (q, v) a)) in
(e_,, Rxlax)((sa)) + co R Rull))((saa)
For the term log
we note the following. First:
Σ Ro aa' a), (1-a) aa)
Σ
(" a))
Σ Ro loq + ajq" v)(s,
for any trajectory a. Moreover, since the log-sum-exp function log(e"t + e²2) is convex in (41,12), we also have the following inequalities for any E 1) and two vectors ,²²
a log Eo R (g' v](s.a) + c Rwla' M(sa)) + (1 - aog (etea Rwl Rull),)(ss)) + c Ru 1a x((saa))
S log <In Ru la' xl(sa) +(1 a1- E., /n (((sa) + e Eog Bull's (lsa)))) Eoy nnd vl(sa))
= =log
( (1-0)) + Rulon' (10)))+
which implies that log (eEo, Ru Rulan((s)) + Con Rull))(s.a)) convex q.
Putting all the above together, we that L(q, v, w) is concave in q.
Finally, since the mixing networks are linear in q and similar argument shows that L(g, v, w) is also concave in
For the convexity of the extreme-V V function [Vv), we rewrite the function as:
J(v) (a) [)) -M
MM a)] -Milv(s))
Ri(s,a))ppi
B
Since the mixing network Mn, [v] is linear in v, we can see that the term
Mwlq((s a)] MM (v()))
E (x
B
is also linear in v.
Moreover, the exponential function ex always convex in x. Thus, in a similar way as shown above, we can prove that MMiiii) convex in v. All these observations imply that J(v) is convex in v, as desired.
12

---

O-MAPL: Offline Multi-agent Preference Learning
A.2. Proof of Proposition 4.2
Proposition 4.1: the mixing networks Mu [g] and My [v] are two-layer (or multi-layer) feed-forward networks, the preference- basee loss function L(q. v, w) is no longer concave in q and the extreme- loss function J J(v) is not convex in
Proof. Following standard settings in value factorization, 2-layer mixing network is typically constructed with non- negative weights and convex activations (e.g., ReLUU. Under this setting, according to (Bui et al., 2024), Mw and Murlv) are convex in and v, respectively.
From this observation, we first recall the preference-beee loss function:
L(q.v,w) = Σ Σ
Rw lg, v](s, a) - log (e ((. + Eo, Rulav)(sa)) + d(Bula vl(s, a)).
(0L,02)EP (8.a)E01
can be seen that the first term of L(q, v, w) involves Rs q, v| (s. a), which can be written as:
Rg v/ss a) = MM a)] y [Mw[v(s']]]
Since Ma [q(s, a)] is convex in q, the first term L(q, v, w) convex q, which generally implies that this function not concave in q.
In similar way, since the the mixing function ( a)] is also convex in w, implying that L(q. v, w) is also not concave in
To prove the non-convexity of the Extreme-V function J (v), we recall that:
J(v) = B(s,a)/ma Mu ( Ma Mullvaal]
((s,a))-uu (M a)) Mar [v()]] - -.
We will find counterexample to show that ((v is not convex under a 2-layer mixing network. For simplicity, since Mu [q] is fixed in (v), we select Ma (s, a) 0. We then create simple example where there is only one agent (i.c., v(s) = (s1))))). and the mixing network MM [v] takes oin-dimensioonl input with a ReLU activation (a commonly used activation function in the context). Specifically, we can write Mw([v] as:
if (s > (0)( - if v1 (s1) < 0.
(81)
Mw([v()]]
Then, a given pair as, the corresponding term in J(V) associated with (s, a) can written as:
c-c" + (E -1111 1).
Here, for simplicity, we select 8 = 1, omit the notation 8 in the function 81 (81), , and only consider the case where 81 S 0. (e" - 1) is not convex for < 0 (see the plot of this function in Figure 3).
We see that the function f(t) = cl 1-
A.3. Proof of Theorem 4.3
Theorem 4.3: Let गे be the optimal solution to the local WBC problem in (4). Then, the global policy o defined as Tin (s, a) = II nद (ag)8z), also optimal for the global WBC problem in (3). In other words, the local WBC approach yields local policies that are consistent with the desired globally optimal policy.
Proof. For notational simplicity, let G(TTr) be the objective function the global WBC problem:
(TTa) Eg,arppo
Quiin.a Van(s)
log Tix ( (a|s)
13

---

O-MAPL: Offline Multi-agent Preference Learning
3
2
-3
-2
0
Figure 3: Plot of the function f(t) ct-d' + 1.
Since we are seeking a decomposable policy maor(a)s) = IlEE TT (a:(s;), we have, for any Thot Hoo such that Toot II: me.
G(Toot))
E aaruuiu
(mitaa) Vir(a)
log II TT ( (8:)
(s.(s.m) Kyn
log tú (a:1s;) }
E
Eg Eg
Vinele
log m. (a. [s:]
iEN
Suelsia) Vor(s) log The (a|s)
E
Ex
where (a) holds because each n is optimal for the corresponding local WBC problem. Thus, we have G( S G( for any Tie E Lot implying that Tix is also optimal for the global WBC. This establishes the GLC, as desired.
A.4. Proof of Theorem 4.4
Theorem 4,4: Let nē be optimal to the local WBC, then the following equality holds for all 8; E S., az A.:
n(s:) 44 (aalsss)
(ails))
(9)
((s,)
where wi and wi are parameters of the mixing networks Malg), Mu(r), respectively. In addition, n(s:)/A(s) is correction term defined as follows:
n(s:)=
II p(dgls5)e jEN
s',a' ssfa
A(8i)
Σ
(10)
a.a.:
Proof. We first note that each mixing network Mnn [q] or Man [v] can be expressed as a linear function of its inputs:
Qun(s, = M,n(qq = Σ w (8s, a4) + by
iEN
Vin(s) = Mulv[(s)
Σ
uffu;(s) + b..
ieN
14

---

O-MAPL: Offline Multi-agent Preference Learning
Thus,
(ue(s.o) Van(s)
Now, let consider the objective function of the local WBC and write:
g(
Σ sES,aEA
Moot
(ro((a/s)e
Quate aa- Vac(s)
log vT s;)
Σ sESa
[(asss;je iEN
log ग. (a:ls:).
Thus, for each agent i M and local state 8, E Sy, we extract all the components of g(m,) that involve (a)())))) as:
g (T)) = Σ
II
((d)(16)
s',a" (o] ms
JENJ
log me mr(a(18))
Σ
n(s:)
(uila(lsi)e
log (c((ss))))
of EEA:
where
17(8))
II
m(dglof)e
8' 8'
jENjAi
which is independent of any local actions a/-
The local WBC problem thus becomes the problem of finding local policies (-(ss) that maximize (T:) for any local state si. For notational simplicity, let
8(a,, = (d(l)e
We then write the local objective function g²((T() as:
g (T:) Σ
8(ai, 8:) log (a,(8)).
a.EA,
To solve the problem max,, ge1 (Ti), let us consider general version (with simplified notation):
max g(t)
Σ iEN
a, log t
t [0, 1]" Σ+=1
where 04 2 0 and g(t): [0, 1]nh + IR. By considering the Lagrangian dual of this problem, that an optimal
solution t" must satisfy the following KKT conditions:
tx E (0,1),
Σ
=
Vi,j N. +
These conditions directly imply that:
a₂
1 ΣyEr aj
15

---

O-MAPL: Offline Multi-agent Preference Learning
We return to the maximization of g (T²), which yields optimal solution:
8(a,, si)
क7 (as)s:)
Vay E A..
Eated, s()
Putting everything together, that the following solution म. is optimal for the local WBC:
T(aasss))= = ((ss))
A(s;)
where
(ssi)up(aylss)e " ofeattiie))
A(s:) E
o.E.A;
A.5. Proof of Proposition 4.5
Proposition 4.5: Each local value vi can be expressed as a (modified) log-sum-exp of the local Q-function qs:
B
B
(((s)
vz(st)
log
Σ
log
(11)
W
A (8:)
())(())
Proof. Since रं is valid probability distribution, we have Ex, T. (aslss) Substituting the closed-form formula of .. stated in Theorem 4.4, we have:
Σ (ss)) A(s)) (i(alls)ee
"Pusloswes of wlecl
Taking ( outside the summation, we get:
Σ nsss) A(si) u4 (i(aiss)e
This directly leads to the log-sum-exp formula:
vs(s)) w!
B
log
Σ
(8))) Assi)
(a(lsss)e
log
Σ
3 log w
m(()) ((ss))
w
a 8))
as desired.
B. Additional Details
B.1. Offline Preference Multi-Agent Datasets
In this section, we provide detailed description of how we constructed the dataset for preference learning tasks. Our datasets span both discrete and contimuous domains, covering the environments SMACv1, SMACv2, and MaMujoco. The datasets are designed to include varying qualities of data, sampled trajectory pairs, and their preference labels to facilitate preference learning. To create datasets suitable for preference learning, we sampled trajectory pairs from varying quality offline datasets and generated preference labels. The labeling process was performed using two approaches:
Rule-based Methods: Following IPL (Hejna & Sadigh, 2024), we sampled trajectory pairs and assigned binary preference labels based on dataset quality (e.g.. poor, medium, expert).
LLM-based Methods: Following DPM (Kang et al., 2024), we sampled trajectory pairs and annotated them using preference policies from large language models (e.g., Llama GPT-4o).
16

---

O-MAPL: Offline Multi-agent Preference Learning
For MaMujoco tasks, Ik trajectory pairs were sampled, while for SMAC tasks, 2k trajectory pairs sampled. Table summarizes the dataset details, including state dimensions, action dimensions, sample sizes, and average returns.
The datasets constructed for this study span a diverse range of environments and tasks, ensuring comprehensive evaluation of preference learning algorithms. The inclusion of varying quality levels and both rule-based and LLM-based labeling methods provides a robust foundation for preference-based multi-agent reinforcement learning research.
Tasks
State dim
Obs. dim
Act. dim
Samples
Max len.
Avg. returns
File size
Ma- Mujoco
Hopper-v2
42
14
1000
1000
1354.0=112.0
255 MB
Ant-v2
226
113
1000
1000
1514.94435.8
1003 MB
HalfCheetah-v2
138
23
1000
1000
1640.5+1175.7
1802 MB
SMACvI
2c 64zg
1350
478
70
2000
280
13.99+4.75
401 MB
5m vs 6m
780
124
12
2000
36
13.26#5.02
72 MB
6h_vs_8z
1278
172
14
2000
48
13.01#3.95
182 MB
corridor
2610
2000
394
12.69=6.30
979 MB
protoss.5.vs.5
130
92
11
2000
142
16.07=4.94
56 MB
1ootoss_1__vs..
310
182
2000
178
15.72+4.28
209 MB
protoss. 10_vs.11
327
191
17
2000
146
13.45
218 MB
protoss. 20.tos_20_.v_200
820
362
2000
200
15.6344.76
726 MB
mrotoss_20.vs..r
901
389
29
2000
200
14.44+4.73
799 MB
SMACV2
(erran.5_vs.3
120
2000
200
16.20=6.37
MB
terram_ _irram____s__0000
290
162
16
2000
200
14.86+5.78
151 MB
terran 10. vs. 11
306
170
17
2000
200
13.52+5.44
165 MB
Cerran_20_vs.2
780
322
2000
200
13.52+5.76
530 MB
terran_20 20_vs_200
858
346
29
2000
200
10.67=5.11
563 MB
zerg_5_v
120
82
2000
57
14.79+7.70
31 MB
1erg-10_vs_10
290
162
16
2000
70
14.61
99 MB
zerg-10 10_
306
2000
104
13.6745.71
MB
xcrg_20_vs _0__v___0__0
780
322
26
2000
134
12.14
303 MB
zerg 20
858
346
2000
99
10.88+4.36
313 MB
Table 3: Datasets
B.I.1. SMAC DATASET
SMACv1 (Samvelyan et al., 2019) is benchmark environment for cooperative multi-agent reinforcement learning (MARL). built on Blizzard's StarCraft II RTS game. It leverages the StarCraft II Machine Learning API and DeepMind's PySC2 to enable autonomous agent interaction with StarCraft II Unlike PySC2, SMACv1 focuses on decentralized micromanagement scenarios, where each unit is controlled by individual RI agent.
We evaluate on the following tasks: 2c_vs_64 5m_vs_6m 6m, 6h_vs_8z, and corridor. Among these, 2c_v8_6 and 5m_vs_6m are categorized as hard tasks, while 6h_vs_82 and corridor are considered super hard. The offline dataset for SMACv1 was sourced from the work of Meng al., where MAPPO was used train agents. These agents were then used to generate offline datasets for the community. The dataset quality varies across poor, medium, and good levels, ensuring comprehensive coverage of different learning stages.
w4.664 64zg
SMACv2 (Ellis et al., 2022) builds upon SMACv1, introducing enhancements to challenge contemporary MARL algorithms. incorporates randomized start positions, randomized unit types, and adjustments to unit sight and attack ranges. These changes increase the diversity of agent interactions and align the sight range with the true values in StarCraft II. Tasks in SMACv2 are grouped by factions (protoss. te terran, zerg) and instances (5_va_5, 10.vs 10.vs_11, 20_vs.20, 20 20_vs_233 23). The difficulty increases progressively from 5_vs_5 200_v... v3.
The offline dataset for SMACv2 derived from the ComaDICE, paper (Bui et al., 2025), where MAPPO (Yu al., 2022) was used to train agents over 10c6 steps, followed by random sampling of Ik trajectories. This dataset primarily represents medium-quality data, To ensure varying quality levels, we created additional datasets for poor and expert levels.
17

---

O-MAPL: Offline Multi-agent Preference Learning
B. B.1.2. MAMUJOCO DATASET
MaMujoco (de Witt et al., 2020b) is benchmark for continuous cooperative multi-agent robotic control. Derived from OpenAI Gym's MuJoCo suite, MaMujoco introduces scenarios where multiple agents within a single robot must solve tasks cooperatively. We evaluate on the tasks Hopper-v2, Ant-vž and Hal ECheetah-ve The offline dataset for MaMujoco was sourced from the work of Xiangsen et al., who used the HAPPO method to train agents. Each task includes datasets with varying quality levels: medium-replay, medium, and expert.
B.2. LLM-based Preference Annotations
To generate preference annotations for trajectory pairs, we utilized GPT-4o (OpenAI, 2024). This model was prompted with detailed trajectory state information, including key metrics such as health points, shields, relative positions, cooldown times, agent types, and action meanings. The inclusion of such detailed state information significantly improves the ability of the L.LM to evaluate trajectory pairs effectively. Following the methodology of DPM (Kang et al., 2024), we extracted critical state details such as the health points of allied and enemy agents, the number of agent deaths (both allied and enemy), and the total remaining health the final state of each trajectory. These extracted metrics were then used construct prompts for the LLM, shown in Table 5.
The OpenAl Batch API (OpenAI, 2025) was employed to submit these prompts to GPT-4o, and the associated token usage and costs are summarized in Table 4. The total cost for generating LLM-based annotations across all tasks was approximately S42, with each dataset containing 2,000 trajectory pairs. While this approach is effective, becomes costly when scaling to larger datasets or additional tasks.
It is important note that this method is particularly suited for environments like SMACvI and SMACv2, where trajectory states provide meaningful and interpretable information. However, the approach has limitations in environments such as MaMujoco, which lack detailed trajectory state information. In MaMujoco tasks, the trajectory states do not include interpretable metrics like health points or agent-specific details, making it infeasible to construct meaningful prompts for LLMs.. As result, only rule-based methods were used to generate preference labels for MaMujoco datasets.
This limitation highlights a broader challenge of the DPM approach (Kang et al., 2024): it relies on the availability of meaningful final state information, which restricts its applicability to specific environments. It is less suitable for long-horizon transitions environments with image-based observations, where extracting detailed and interpretable state information is either infeasible computationally expensive.
B.3. Implementation Details
All experiments implemented using PyTorch and executed in parallel a single NVIDIA@ H100 NVL Tensor Core GPU to ensure computational efficiency. We developed two versions of our proposed method, O-MAPL, tailored to the specific characteristies of continuous and discrete action domains:
Continuous Domain (MaMujoco): For continuous environments, we utilized a Gaussian distribution (torch. distriiutt instrsbuiions..... Normal) 1) to model the policy. Each agent's action is sampled from this distribution, which is parameterized by the mean and standard deviation outputted by the policy network.
Discrete Domains (SMACv1 & SMACv2): For discrete environments, we employed a Categorical distribution (torch. iessriiuttion Categori cal) to model the policy. The probability of each action for an agent is computed using the softmax operation over only the available actions for that agent. Actions that not available are assigned probability of zero. This ensures that the log-likelihood calculation is accurate and avoids penalizing the agent for infeasible actions.
B.4. Hyperparameters
Table 6 reports hyperparameters used consistently across all experiments:
B.5. Baseline Comparisons
We compared O-MAPL against four baseline methods evaluate its performance:
18

---

O-MAPL: Offline Multi-agent Preference Learning
Tasks
Completion Tokens
Prompt Tokens
Estimated API Cost
2c_vs_64zg
5,920
2,498,000
$3.13
5m_vs_6 1-6m
5,913
1,386,000
$1.74
6h vs 8z
5,941
1,462,000
$1.83
corridor
5,926
1,772,000
$2.22
protoss.5..m 3_vs.5
5,920
1,460,000
$1.83
protoss_IO.vs_Il
5,918
1,660,000
$2.08
protoss_10. 10_v___11
5,940
1,680,000
$2.11
protoss 20_vs_20
5,901
2,060,000
$2.58
protoss 20 vs 23
5,990
2, 2,122,000
$2.66
terran
5,990
1,442,000
$1.81
terran. 10_vv0__vv_vv
5,925
1,642,000
$2.06
terran 10 11
5,930
1,662,000
$2.08
terran_20 serran220_v_20
5,944
2,042,000
$2.56
terran Ferra.220.vs.22
5,977
2, 2,104,000
$2.64
zerg _er__5_vs.5 _
5,940
1,448,000
$1.82
zerg 10 10
5,914
1,648,000
$2.07
zerg-10.vv 10_.v00v.... vs.111
5,912
1,668,000
$2.09
zerg. 20 ys 20
5,942
2,048,000
$2.57
zerg 2rrg-20_vs.22 _v33.23
5,913
2,110,000
$2.64
Total
112,756
33,914.000
$42.53
Table 4: GPT-4o API costs
BC (Behavior Cloning): A simple BC supervised learning approach based preferred trajectories in the dataset.
IPL (Independent Inverse Preference Learning): Implements IPL. (Hejna Sadigh, 2024) independently for each agent without considering inter-agent coordination.
IPL-VDN (Inverse Preference Learning with VDN): Similar to our O-MAPL algorithm, except that the global Q and functions are aggregated by summing the local Q-values of individual agents, instead using a mixing network (Sunehag et al., 2017).
SL-MARL (Supervised Learning for MARL): A two-step approach where the reward function is first learned via supervised learning, followed by policy training through a MARL algorithm (i.e. OMIGA (Wang et al., 2022)), using the learned reward function.
B.6. Evaluation Metrics
We report two key metrics assess agent performance:
Measures the average cumulative rewards achieved by the agents across episodes (applicable to all the environments).
Mcan/Standard Deviation of Returns:
Mean/Standard Deviation of Win Rates: Applicable only competitive environments (only applicable SMACvl and SMACv2). This metric evaluates the percentage of episodes where agents achieve victory.
Each metric is computed as the average and standard deviation of the final results across all four random seeds. Additionally, we present evaluation curves for each method, depicting performance trends during the agent training process using offline datasets.
19

---

O-MAPL: Offline Multi-agent Preference Learning
Prompt
You are helpful and honest judge of geod game playing and progress in the helpfully possible, while
StarCraft Multi-Agent Challenge game. Always being truthful.
IE you don't know the answer to a question, please don't share false information. I'm looking to have you evaluate a scenario in the StarCraft Kulti-Agent Challenge. Your role will be assess how much the actions taken by multiple agents in given situation have contributed to achicving victory.
The basic information for the evaluation is as folloxs.
Scenario 5m 6m
Allied Team Agent Configuration five Marines (Marines are ranged units in StarCraft 2).
Enemy Team Agent Configuration six Marines (Marines are ranged units in StarCraft 2)
Situation Description : The situation involves the allied tear and the enemy team engaging in combat, where victory achieved by defeating all the enemies. Objective Defeat all enemy agents while ensuring as many allied agents as possible survive.
Important Notice preserved while significantly recucing the enemy's 's health. In similar situations, you should prefer shorter trajectory lengths.
should prefer the trajectory where allies' health is
will provide you with two trajectories, and you should select the better trajectory based on the outcomes of these trajectories. Regarding the trajectory, it will inform you about the final states, and you should select the better case based on these two trajectories.
[Trajestory 11
1. Final State Information
1) Allied Agents Health : 0.000, 0.000, 0.067, 0.067, 0.000
2) Enemy Agents Health 0.000, 0.000, 0. 0.000, 0.000, 0.000, o. 040
3) Number of Allied Deaths
4) Number of Enemy Deaths
5) Total Remaining Health Allies : 0.1
6) Total Remaining Health of Enemies 0.040
2. Total Number of Steps 28
[Trajectory 2]
1. Final State Information
1) Allied Agents Health : 0.000, 0. 0.000, 0.000, 0.000, 0.000
2) Enemy Agents Health 0.120, 0.000, o. 0.000, 0.000, 0.000, 200
3) Number of Allied Deaths
4) Number Enemy Deaths
5) Total Remaining Health of Allies : 0.000
6) Total Remaining Health of Enemies 0.320
2. Total Number of Steps 23
Your task to inform whish one better between [Trajectery]] and [Trajectory2] based on the information mentioned above. For example, if [Trajectory 1] seems better, output #1, and it [Trajectory 21 seems better, output #2. If it's difticult to judge they
similar, please cutput #0. Important : Generally,
is considered better when fewer allied agents are killed or injured while inflicting more damage en the enemy.
Omit detailed explanations and just previde the answer
Table 5: Sample prompt to generate preference data in SMAC environments.
20

---

O-MAPL: Offline Multi-agent Preference Learning
Hyperparameter
Value
Optimizer
Adam
Learning rate (Q-value and policy networks)
le-4
Tau (soft update target rate)
0.005
Gamma (discount factor)
0.99
Batch size
32
Agent hidden dimension
256
Mixer hidden dimension
64
Number of seeds
Number of episodes per evaluation step
32
Number of evaluation steps
100
Table 6: Hyperparameters used in all experiments.
Notes on Evaluation: For MaMujoco, win rates are not applicable as it is not competitive environment. Evaluation scores are averaged over the results of all four seeds to ensure statistical robustness. The performance trends and comparisons are visualized in detailed figures to provide insights into the training dynamics each method.
This setup ensures a fair and comprehensive comparison between O-MAPL and the baseline methods in both continuous and discrete multi-agent reinforcement learning tasks.
B.7. Recovered Rewards
Rule-based
LLM-based
Tasks
Lower
Higher
Lower
Higher
2c_vs_6Azg
-.364+0.26
9.25#0067
-12.87+0.73
14.14=0.80
5m_vs_6n
4.4940.12
4.80+0.15
4.02++..00
4.51
6h_v8_8z
4.7240.28
5.15+0.22
-5.1140.32
5.28+0.16
corridor
-12.5940.11
11.23+1.06
-12.9740.33
10.93+0.45
mrotoss_5_vs..
-..110..2
6.54+0.51
8.06+0.64
7.46+0.77
protoss 10. 10
17.73+0.1
7.92+0.32
-10.55+1.15
9.32+0.91
-7.95+0.69
8.31+0.91
11.01+0..9
10.43+1.57
protoss mrotoss_20_vs_20
20_vs_20
-..1140.35
8.1920.16
10.57400.6
9.54+0.74
protoss 20_vs_23
-8.0140.22
9.10#0.14
12.1740.72
12.09+0.80
_or_n_5__v_55
6.85+0.30
6.93+0.56
7.85++..27
7.82+0.57
terran terran_10.vv 10
-.2540.82
7.35+0.61
10..73+1.49
8.1620.56
derran_10_vs_I
-8.3++0.67
9.62+0.54
19.12...3
10.97+1.38
terran 20 vs 20
-.5944.33
8,44+0.22
1004440.96
10.79+1.00
terran 200_vs22
-8.4940.65
8.91+0.27
14.90+2.06
17.95+2.91
zerg.5.. 5_v8.5
-3.7440.14
1.6400114
-5.09+0.19
3.5160.06
verg-10_vs_I1
4.1640.116
4.27+0.16
-5.9340.43
6.1440.64
zerg 10 vs. 11
4.5420.06
4.6020.14
-7.2840.50
6.20+0.50
zerg _0__20__22 vs.20
-5.1140.08
5.25+0.20
-7.7140.54
7.24+0.23
zerg 20 vs. 23
4.78+0.12
5.08+0.15
-8.2++1.13
8.00+0.43
Table 7: Mean/std recovered rewards of the higher and lower preferred trajectories
Using recovered reward function R(o, o') MM ( a)] 7Mg ]. report the mean/std returns of the higher and lower preferred trajectories in Table 7. Across all tasks, the higher preferred trajectories consistently achieve positive rewards, while the lower preferred trajectories exhibit negative rewards. This indicates that the preference- based
21

---

O-MAPL: Offline Multi-agent Preference Learning
learning framework effectively captures and differentiates between preferred and less-preferred trajectories. The consistent separation in rewards suggests that the model successfully aligns policy learning with preference signals, reinforcing high-reward behaviors while penalizing undesirable ones. Moreover, the absolute values of both higher and lower preferred rewards tend to be more extreme in the LLM-based approach, compared to the Sule-based approach. This suggests that L.MM-assed learning amplifies both positive and negative behaviors, potentially leading to more decisive policy updates, which can be beneficial for clear preference-driven learning.
There are noticeable variations in how different task domains respond to preference- learning. The Protoss and Terran tasks exhibit larger reward variations, suggesting that these environments benefit more from preference learning, In contrast, Zerg tasks show more moderate reward differences, indicating that either the task dynamics are inherently more balanced or that preference signals have weaker impact in these settings. Additionally, the corridor task, a structured navigation environment, shows similar performance across rule-based and LLM-based approaches.
B.8. Additional Experimental Details
B.8.1. RULE-BASED RETURNS
We present experimental details, in terms of returns, for all tasks (MAMuJoCo, SMACvI, and SMACv2) using rule-based preference datasets.
Table 8 reports the returns and Figure 4 plots the evaluation curves for MaMujoco tasks with Rule-hased preference data and Figure The results demonstrate that O-MAPL consistently outperforms all baselines across Hopper-v2, Ant-v2, and HalfCheetah-v2, highlighting its effectiveness in rule-based preference learning. Notably, in Hopper-v2, O-MAPL achieves 25.2% higher return than the next-best method, SL-MARL, suggesting superior preference alignment. While SL-MARL performs well in simpler environments, its advantage diminishes in Ant-12 and HalfCheetah-v2, where IPL-VDN shows stronger results. BC significantly underperforms, reinforcing the need for preference-based learning over naive imitation.
O-MAPL
Tasks
BC
IIPL
IPL-VDN
SL-MARL
(ours)
H2pper-v2 Ant-v2
808. = 39.1
782.0 81.5
846.6 : 65.4
890.0 + 88.7
1114.4 # 154.1
1303.9 + 122.0
1312.0 # 155.6
1376.1 # 142.0
1334. + 150.9
1406.4 # 163.7
HalfCheetah-v2
4119.9 = 350.7
4028.8 + 430.0
4287.5 = 273.1
4233.9 + 303.1
4382.0 = 189.7
Table 8: Returns for MAMujoco tasks with Rule-based preference data.
BC
IIPL
IPL-VDN
SL-MARL
O-MAPL (ours)
Hopper-v2
Ant-v2
HalfCheetah-v2
1k
1.4k-
4.5k-
500
3.5k
3k
0
50
100
50
100
50
100
Figure 4: Evaluation curves (returns) for MAMujoco tasks with Rule-based preference data.
Table 9 report the returns and Figure 5 plots the evaluation curves for SMACv1 tasks. The results show that O-MAPI. consistently achieves the highest returns across most SMACvl tasks. While the performance differences are relatively small in simpler tasks like 2c_v8_64zp 642g and 5m_vs. mmm, O-MAPL outperforms all baselines more complex scenarios such as 6h_vs_8z, where achieves 12.1, compared the next-best method (SL-MARL, 11.8). Notably, SL-MARL struggles the corridor task, achieving significantly lower return (14.3) than other methods, suggesting that its reliance on separate reward modeling phase may be less effective in environments requiring strong coordinated behaviors.
22

---

O-MAPL: Offline Multi-agent Preference Learning
O-MAPL (ours)
Tasks
BC
IPL
IPL-VDN
SL-MARL
2c_vs_647g
19.0 1.1
0.8
19.3 1.1
1.8
19.3 1.4 2.1
5m 6m
1.1
10.8
1.0
11.1
6h_vs_8z corridor
11.0
0.7
11.7
1..8
12.1 + 1.3
1.0
1.0
19.6
14.3 2.8
19.6 0.9
Table 9: Returns for SMACvI tasks with Rule-based preference data.
BC
IIPL
IPL-VDN
SL-MARL
O-MAPL (ours)
2c _s_g
5m_vs [i__v__6mm _s_6m
6h vs 82
corridor
20
12-
20
15
10
8-
Figure Evaluation curves (returns) for SMACvl tasks with Rule-based preference data.
Table 10 shows the returns and Figure 6 plots the evaluation curves for SMACv2 tasks (the most complicated ones). The results show that O-MAPL consistently achieves competitive performance across all SMACv2 tasks, often outperforming other baselines. In Protoss tasks, O-MAPL achieves the highest returns in most cases, particularly in protoss 10_vs. ss_ and 10.v suggesting its effectiveness in complex team-based coordination. Similarly, in Terran tasks, O-MAPL consistently outperforms SL-MARL and IPL-VDN, with a noticeable advantage in terran terran_20.we 2_0 and terran 2atrran_00_v__222 23.. In Zerg environments, O-MAPL continues to show strong results, outperforming all baselines zerg v5v zerg -eeerrii_vvve vl_ and zerg 20 vs 20.
O-MAPL
Tasks
BC
IIPL
IPL-VDN
SL-MARL
(ours)
protoss_5.0m v5.
2.4
2.5
17.1
15.8
16.8 = 2.4
protoss_1O.vv_100
1..0
1..4
17.8
16.4224
17.9 + 1.9
protoss_10_vs.11
1.3
12.7
14.7
14.2#2.
14.9 2.0
protoss 2ottos_200.vs220 20__vs200
1..7
1..9
1..0
1..
18.0 1.5
protoss 20 vs 23
1.13
1.81
1.99
1.96
14.9
terran
1..2
11.2
11.9
13.0
12.8
terran 10_vs_10
1.99
1..
1..6
1..4
11.8
terran 10 11
8.6
8.1
1.0
9.6
11.0
terran_20.00 20rran_00_v_20
10.1
2..2
2.6
11.8 10.9
Burran-20.vs
7.6
2.1
8.7
7.7
9.4
zerg_5.cm 2erg_5_vs.5
1..2
1.8
2.71
12.7
13.1
zerg 10 10
2.4
1.74
2.6
13.2
14.0 + 2.6
zerg_10_vs_I1
11.1
10.7
2..1
11.8
zerg. 20 vs 20
13.0
1.22
13.
1..2
13.9 1.8
7erg.20_vs_23
12.1 + 2.3
1..3
1..1
1.6
12.7 + 2.0
Table 10: Returns for SMACv2 tasks with Rule-based preference data.
23

---

O-MAPL: Offline Multi-agent Preference Learning
BC
IPL
IPL-VDN
SL-MARL
O-MAPL (ours)
Protoss
12-
100
100
Terran
50
Zerg
12
50
100
50
100
50
100
0
50
100
50
100
5
$_v
10_ 1__vs__
10
v__v1_11
11
30_vs_20
20
20
$0_vs_2_
Figure Evaluation curves (returns) for SMACv2 tasks with Rule-based preference data.
B.8.2. RULE-BASED WINRATES
For SMACvI and SMACv2, win rates provide a more meaningful comparison of algorithm performance. The following tables and figures present win rates for SMAC tasks using rule-based preference data.
Table 11 shows the winrates and Figure plots the evaluation curves (in terms of winrates) SMACv1 tasks. The results indicate that O-MAPL consistently achieves the highest winrates across most tasks. In 2c_vs_642g, O-MAPL outperforms all baselines, achieving win rate of 74.4, surpassing IPL-VDN and SL-MARL. Similarly, in 5m 6m and 6h. 8z, O-MAPL achieves the highest winrates, though the performance gap is less pronounced. Notably, in the corridor task, IPL.VDN slightly outperforms O-MAPL, while SL-MARL strugglle significantly, indicating that its two-phase approach may be less effective in highly structured navigation tasks. These results suggest that O-MAPL is well-suited for complex coordination tasks, offering robust winrates across diverse SMACv1 environments.
O-MAPL (ours)
Tasks
BC
IIPL
IPL-VDD
SL-MARL
2c_xs_64zg 5m_vs_6m
59.6 + 25.0
60.4 24.7
71.1 + 22.0
24.7 19.3 + 19.6 4.5 11.0
16.8 + 18.0
16.8 # 18.0
16.0
6h vs 8z corridor
0.6 3.8
2.2
2.5
1.6
89.3 + 15.5
89.8 + 15.4
93.9 + 11.6
49.0 22.8
93.2 # 13.5
Table 11: Winrates for SMACv1 tasks with Rule-based preference data.
BC
IIPL
IPL-VDN
SL-MARL
O-MAPL (ours)
2c s0vvv_6444 642g
5m vs_ 6m
6h vs 8z
corridor
80%
30%
6%
80%
40%
3%
40%
0
50
100
0
50
100
50
100
0
50
100
Figure 7: Evaluation curves (in winrates) for SMACvl tasks with Rule-based preference data.
Table 12 shows the winrates and Figure plots the evaluation curves (in terms winrates) for SMACv2 tasks. The results, again, show that O-MAPI. consistently achieves the highest winrates across most SMACv2 tasks. In the Protoss tasks,
24

---

O-MAPL: Offline Multi-agent Preference Learning
O-MAPL outperforms all baselines, particularly in protoss_1O_vs_1 and protoss _0__v_20... where it shows significant improvement the other methods. In Terran tasks, O-MAPL. also achieves the best performance, with notable advantages in terran 22.00vvss.0..1e 20 and terran_20.ve 23, where other methods struggle to achieve high winrates. Similarly, in Zerg tasks, O-MAPL consistently achieves the best results, particularly in zerg vs. 20 and zerg 20.
O-MAPL (ours)
Tasks
BC
IIL
IPL-VDN
SL-MARL
protoss_55 1_rtoss___vs...
38.1 + 24.2
54.5 3
49.0 28.2
54.3 + 24.2
1ottoss_1O.vs_100
=8.7
28.8
23.6 272
protoss 10 vs 11
1
16.5
21.00
18.6
30.7 + 19.8
protoss 20_2200vvv.. 20
39.8
21.5
24.8
23.7
protoss 20 vs 23
1 18.5
9..2
22.7
11.1
23.4 19.2
terran 5 vs 5
24.0
26.2 + 19.5
34.2 23.4
39.5 + 24.7
terran_100 10_vss_10
20.5
21.1 =
25.8 19.7
23.2 19.6
28.3 + 20.6
terran_ terran_10.vs_1. 101
13.3
18.2 19.4
15.3
18.2 18.7
terran_20 2erran20_vs_20
13.1
2 18.2
8.8 13.5
terran 20 100vvss22 23
3.9 10.6
4.1
11.4
2.3
7.2 12.9
zerg-5_vs.5
21.1
31.1 23.6
20.4
33.0 22.5
35.2 #: 25.7
zerg. 10 vs 10
25.8 21.6
32.2 22.5
34.8 + 22.1
12.9 17.4
22.5
2 19.3
zerg_20. zerg_20_vs_22 2s
21.0
11.1 = 16.2
22.5
5.7
24.8 + 20.8
zerg_20_vs.22 _33
13.1 + 17.7
7.8 # 12.8
12.5
7.6
18.8 + 18.5
Table 12: Winrates for SMACv2 tasks with Rule-based preference data.
BC
IIPL
IPL-VDN
SL-MARL
O-MAPL (ours)
Protoss
60%-
30%
50
Terran
40%
0%-
50
100
0
50
100
0
50
100
50
100
50
100
Zerg
40%-
20%
0%-
50
100
0
50
100
0
50
100
50
100
100
5_vs_5
10 vs
10
10. 10_vs_10 11
20_vs_ 20
20
2o_vs_23 23
Figure 8: Evaluation curves (in winrates) for SMACv2 tasks with Rule-based preference data.
B.8.3. LLM-BASED RETURNS
We present comparisons in terms of returns using LLM- based preference datasets. As noted earlier, only SMACv1 and SMACv2 are suitable for obtaining meaningful preference-based data from LLMss. Therefore, we report comparisons exclusively for SMAC tasks.
Table 13 shows the returns and Figure 9 plots the evaluation curves (in terms of returns) for SMACvI tasks. The results in Table 13 indicate that O-MAPL consistently achieves the highest or near highest returns across all SMACv1 tasks using
25

---

O-MAPL: Offline Multi-agent Preference Learning
LLM-based preference data. In 6h_vs 8z, O-MAPL shows clear advantage, reaching 12.2, outperforming all baselines. Similarly, in corridor, it achieves the highest return (19.7), alongside IPI.-VIN, while SL-MARL struggles significantly in this task. Across 2c_vs_6f ___zzz and mn_vs_óm, performance differences are minimal, but O-MAPL remains competitive. These results highlight the effectiveness of O-MAPL in leveraging LLM-based preferences, particularly in more complex multi-agent coordination scenarios.
O-MAPL (ours)
Tasks
BC
IILL
IPL-VDN
SL-MARL
2c_vs_64zg
1.9
0.9
19.6 1.0
0.7
19.6 1.1
5m vs 6m
1.1
1..
1124
11.2
11.5 2.3
6h_vs_8z corridor
1..1
1.9 1.7
11.8
12.2 2 1.3
1.0
1.0
1.97
15.1 2.4
19.7 0.8
Table 13: Return comparison for SMACvl tasks with LLM-based preference data.
BC
IIPL
IPL-VDN
SL-MARL
O-MAPL (ours)
2c_vs_ [0_vs_66422 64zg
5m 5m_vs vs_6m
vs
corridor
20
18
16
8-
50
100
0
50
100
0
50
100
0
50
100
Figure 9; Evaluation curves (in returns) for SMACvl tasks with LLM-based preference data.
Table 14 shows the returns and Figure 10 plots the evaluation curves (in terms of returns) for SMACv I tasks. The results demonstrate that O-MAPL consistently achieves the highest returns across most SMACv2 tasks. In Protoss tasks, O- MAPL outperforms other methods, particularly in protoss_ LOooosss_Ovv_llllllll and prntoss_20_vs.oo indicating its effectiveness learning structured team-based strategies. In Terran tasks, O-MAPL generally achieves the best performance, with notable improvements in terran_20_vv 20_vs_20, suggesting its strength in complex coordination settings. In Zerg tasks, O-MAPL maintains strong performance, particularly in zerg 20 20, where it achieves the highest return.
26

---

O-MAPL: Offline Multi-agent Preference Learning
O-MAPL
Tasks
BC
IIPL
IPL-VDN
SL-MARL
(ours)
protoss_5
16.7
2.5
2.5
2..9
17.9 # 2.5
protoss 10 10
2.0
16.6
17.9
1.9+1. 1.8
18.0 1..
protoss_10.vs.11
2..
14.5
2.44
14.0
16.5
2ottoss_20.vs.200
1.7
1.3 1.7
1.9
18.9 + 1.5
protoss_20.vs_2
2.0
1.84
1.81
1.83
15.8 + 1.9
Berran_5_vss
1.15 1..
2.9
1.16
12.6 2.6
terran
[erran_10_vs_10 10_vs_10
21.3
11.7
11.6
12.1
1
terran 10 vs 11
9.2
9.3
10.1
9.9
10.7 2.5
terran_200 Serran_20_vs.ee 20_vv_20
11.
20.
1.44
13.0 11.6
Ferram_20_vs.o
2.4
2.2
8.9
8.7
9.1 +
zerg 5 vs 5
2.74
11.6
12.8
11.8
12.9 2.6
verg_10_vs_10
2.65
2.74
13.7 2.5
13.7
14.5 + 2.6
zerg- zerg-10.vvs 10.1s.11
2.3
1.79
11.7 2.0
12.8 2.4
1
zerg 20 vs 20
13.9 +
1.94
14.6 2.0
13.8 # 2.0
15.2 + 2.4
zerg.20.vs.23.
12.6 2.0
12.1 1 1.9
12.4 2..422.
12.6 1.9
12.4 + 2.2
Table 14: Return comparison for SMACv2 tasks with LLM-based preference data.
BC
IPL-VDN
SL-MARL
O-MAPL (ours)
20
Protoss
16-
12-
50
100
50
100
50
100
0
50
100
50
100
Terran
12
8-
50
100
50
100
100
50
100
Zerg
12-
8-
100
5_
10_vs_10 vs_ 10
10_ vs
20_vs_20
20_ vs_
Figure 10: Evaluation curves (in returns) for SMACv2 tasks with LLM-based preference data.
B.8.4. LLM-BASED WINRATES
We present a comparison of winrates for SMAC tasks using LLMMbasee preference datasets.
Table 15 shows the returns and Figure 11 plots the evaluation curves (in terms of winrates) for SMACv1 tasks. The results in Table 15 indicate that O-MAPL. achieves the highest win rates across most SMACvl tasks when using LLM-based preference data. In 2c vs 642g, O-MAPL. outperforms all other methods, achieving a win rate of 79.5, slightly higher than IPL-VDN (77.0) and significantly surpassing BC and IIPL. Similarly, in 5m_vs_ón and 6h_vs_8. O-MAPL achieves the best performance, though the overall win rates in these tasks remain low, indicating the increased difficulty of these environments. In the corridor task, both O-MAPI and IPL-VDD achieve the highest win rate (94.5), demonstrating their effectiveness in structured navigation tasks, while SL-MARL struggles significantly with a win rate of only 57.6.
27

---

O-MAPL: Offline Multi-agent Preference Learning
O-MAPL
Tasks
BC
IIL
IPL-VDN
SL-MARL
(ours)
24.6 65.6 2c_vs_64zg
6
77.0 21.3
65.2
79.5 19.6
5m 6m 6h_vs_8z corridor
18.2 + 18.4
15.0 17.5
18.0 19.2
17.4 19.4
20.7 20.5
0.8 + 4.3
0.4
9.2
8.9
6.1 11.2
89.6 + 15.5
90.6 + 13.6
94.5 + 12.5
57.6 22.2
94.5 = 11.2
Table 15: Winrate comparison for SMACvI tasks with LLM-based preference data.
BC
IIPL
IPL-VDN
SL-MARL
O-MAPL (ours)
2c vs 642g
5m
6mm
6h vs 8z
corridor
30%
6%
80%
40%-
15%
3%
40%-
Figure 11: Evaluation curves (in winrates) for SMACvl tasks with LLM- preference data.
The results reported in Table 16 and Figure 12 demonstrate that O-MAPI. consistently achieves the highest win rates most SMACv2 tasks using LLM--ased preference data. In the Protoss tasks, O-MAPL outperforms all baselines, particularly in protoss Protoss_OO_ve 10_vs and protoss 20 30 20, where it achieves substantial improvements over other methods. In Terran tasks, O-MAPL also shows strong performance, especially in terran 20_vs _vs00 and vervan_20.vs_2 where other baselines struggle to achieve competitive win rates. In Zerg tasks, O-MAPL maintains an advantage, particularly in zerg_5_vs_ and zerg 20_ys 22_ suggesting that its approach generalizes well across different strategic settings. Overall, these results indicate that O-MAPI. effectively integrates LL.M-based preference data to improve decision-making and coordination complex multi-agent environments.
O-MAPL (ours)
Tasks
BC
IIPL
IPL-VDN
SL-MARL
protoss 5
48.4
24.2
5
24.3
61.5 24.8
protoss Bartoass_o_vs_____
46.3
24.0
53.4
22.5
61.1 # 24.8
protoss 1atss_10vssllll
22.2
15.6
27.3
20.9
34.4 24.8
protoss 20 vs 20
48.4 25.3
23.6
22.5
51.8
64.5 + 23.5
protoss 20tooss20..vs.23 _23
18.0 17.4
9..7
21.4
12.1 15.9
26.4 + 20.8
terran 5 __vs_5
31.1
33.8
36.7 43.7
43,0 23.0
terran 10 vs 10
20.9
24.2
24.4
24.7
3 23.4
10
11.7
15.2
17.8 17.7
16.4
21.3 4 20.3
terran 20.v..200.vs.00
17.3
13.7
20.1
17.2
24.4 23.1
terran_20_vs_23
6..212.2
3.5
12.6
1..2
8.6 14.8
zerg_5_vs_5
31.1 22.3
26.0
3
35.0 23.2
40.8 + 21.6
zerg-10 1o.110vvv... 10
21.9
31.1 24.8
23.9
33,0 25.0
37.9 + 24.0
zerg. 10_vs.lvl vs.11
+ 18.2
18.6 20.6
23.0 28.3
26.0 + 23.0
zerg-20_vv _0_ys_2000
22.9 21.7
16.0 17.3
27.3 22.0
16.4 18.1
31.1 + 24.6
zerg 20. vs 23
15.8 + 18.5
10.4 + 15.2
16.4 + 19.9
13.7
16.0 + 19.4
Table 16: Winrate comparison for SMACv2 tasks with LLsed preference data.
28

---

O-MAPL: Offline Multi-agent Preference Learning
BC
IPL
IPL-VDN
SL-MARL
O-MAPL (ours)
Protoss
80%-
30%
0%-
100
100
100
Terran
40%
20%
0%
Zerg
40%
0%-
50
100
0
50
100
0
50
100
50
100
50
100
5_vs_5
10 vs
10 10_vs__r 11
20_vs_20
20
23_3 23
Figure 12: Evaluation curves (in winrates) for SMACv2 tasks with LLM- preference data.
B.8.5. IMPACT OF DATA RETENTION
To analyze the effect of reducing the amount of preference data, experiments were conducted by retaining 25%, 50%, 75%. and 100% of the dataset. Both returns and winning rates evaluated to assess the performance degradation robustness under reduced data availability. This experiment highlights the sample efficiency of O-MAPL and its ability to generalize with limited data.
(our)
50%
75%
100%
Hopper-22
3o9.4214.4
603.6474.3
706 2204
B4683655
617.2:519
701 9.53.3
7460557.1
$900258.7
628662448
234. 7
953.1275.7
111442156.
MaMajoco
Asi-2
896.2233.3
1141 5273.1
12388
13761214420
955 023052233
1675
1281321100.0
1334 12150.9
1189 3:140
1282 15155
1204 7=167.1
1405422637
HalCheeihh-2
2676-4a615.4
3501.24336.7
3650021659
4257.3227..1
3291
6e402.1
0:2444
402122263.1
4233.9330.
3548
3943.06271.3
A16112211.0
4882/221997
2: in 64zg
13.4808
15311.1
16821.
19.3a1.1
14.7114
16.21.1
174400
10.25
15.5000
17.3212
184:10
19-51-
SMANCY]
T8-277
91:33
90-89
93409
93274
11:4410
11227711
N4423
9/63.1
10
11
956222
10663.3
11::200
comider
1321144
156109
7608
18611.
115611
12.5006
166110
176-14
0.
196109
petto::
16212
$,5.5
6.6111
80114
110052
11.9131
106:34
100.29
10312
150031
9.123
118136
12.5.29
123115
SMACY2
Russ_11
10.ss.00
744119
5.2135
76e27
87619
117e3.0
91x26
101=331
118225
30225
6.7222
9.x2.1
70e15
10262.7
746222
11,422
96c26
K362.4
9.5224
10Te2.9
92:30
11/-22
$7=2.
118:26
11/:28
3139.21
1.263.1
RL3630
1126627
107526
93311 8
67623
10641
10962.3
K261.4
M.5e1.7
11::201
11A124
1.541.7
706177
$3216
6763.1
6-4-1 8
634311
75-18
77420
7.4-166
50-22
634-35
94-30
5,5.5
1.3225
94c277
11 11965.1
1211227
100e333
103-28
1231332
12.7331
106+3.1
12.0e3.2
12:6:37
131a15
10ss,,00
9/4239
98:355
12
130576
186e3.1
11:222
1255233
25.2
9563.5
11.545.2
135-27
180+2A
201
k062.4
89-33
20262.7
12.1-75
79415
11
10.43.3
90:18
11.11.
151:2.0
138:2.1
90-08
11:51.3
12.241.5
180-16
138
0.113
98.18
11.01.77
89.15
12.211.5
10411.
102122
125.220
12:322
Table 17: Returns Comparison with Reduced Preference Rule-based Data
29

---

O-MAPL: Offline Multi-agent Preference Learning
IPL-VDN
SL-MARL
O-MAPL (ours)
25%
50%
75%
100%
25%
50%
75%
100%
25%
50%
75%
100%
vs 642g
14.400.
16,5=0.9
17.4+1.0
19.6+1.0
15.9+1.1
17.640.5
18.540.7
19.5+0.7
17.5-0.8
18.0+1.2
19.0+1.
19.6+111
Sm.vs. 6.m
8.2+2.1
9.4
10.0:2.0
11.4+2.2
8.8+199
10.242,4
11.0+2.2
11.2+2.1
9.4+166
11.4=2.0
11.8226
11.5=2.3
SMACvI
6h_ys_8r
7.8+0.7
9.4+1.0
10.3+0.9
11.9
8.7+0.6
10.1
10.8+0.7
1188+1.2
10.0=0.9
11.1+1.3
12.1+14
12.2+1.3
corridor
14.3a1.1
16.4=1.2
17.5a1.0
19.7a0.9
12.8
13.6&1.2
14.6&1.3
15.122.4
17.1=0.9
18.6200.
19.3a1.1
19.7=0.8
5
11.6222
14.8=2.7
15.9a2.3
17.642.5
13.842.1
12.8
17.042.1
16.912.4
14.9=2.3
16.3a1.9
17.842.0
17.922.5
10
12.5=1.7
15.2=2.0
15.1
17.9#18
13
14.742.1
16.0+2.0
17,5
14,8=18
16.9#199
169215
18.0=2.1
protoss
1
10.5+1.6
13.8+2.5
14.5+2.1
15.4+2.
11.9+1.8
14.042.3
13884199
14.0+2.4
13.6=2.5
8
15.9+2.0
16.5-22
20.vs.20
12.821.4
14.2+1.3
17.8+1.4
18.5+1.3
14.1+1.2
16.241,4
15.8&1.8
18.2+1,9
16.7=1.7
17.5
18.2+1.3
18.9115
20_va_23
10.922
12.9=2.6
14.5+1.
15.1+1.8
10.8+1.6
12.7+1.8
14,1+1.8
14.3+1.8
13.0=2.1
14.5+2.
15.5#1.9
15,8=19
5.vs_5
11.8+3.3
12.6=3.9
12.433.0
13,4+2.9
10.542.8
12.143.2
13.343.0
12.6+3.1
10.5:2.4
12.0=2.9
12.3+2.9
12.6=2.6
10 10
8.442.6
10.242.8
10.743.2
11.622.7
10.242.6
9.912.3
11,012.5
12.1+288
9.743.0
11.242.6
11/422.6
12.5=2.7
SMACV2
teran
10
7.5t3.2
8.5+2.9
8.4+2.3
10.1
8.0+2.3
8.0+2.6
8.5+1.7
9.942.6
8:2+2.0
9.6+2.9
10.5#2.4
10.7=2.5
20. vs.
10:242.2
11.02.5
11,4
8.6+1.7
9.842.1
9.6+1.8
11.6+2.2
9.1+1.6
10.4+20
12.8+2.6
13.0=2.8
20.vs_23
5.8+2.2
7.3+2.4
8.2+2,1
8.9=2.1
6,8+1.7
7.6+2.2
7.6+2.4
8,7+1.9
8.1+2.1
8.7+1.5
9.0+2.1
9.1+23
5_vi_5
7.6+2.4
10.4=2.3
10.6+2.3
12.8+3.3
10.2+2,9
11,6+2,5
11.0+2.1
11,8+2,9
11.7+2.8
11.5+3.4
11.8220
12.9+2.6
9.542.7
12.1=2.3
12.02200
13.7+2.5
11.1+1.9
12.042.6
12.342.4
13.743.0
12.9=18
13.4
13.242.3
14.5=2.6
zorg
10.vs.11
8.5=2.0
10.242.2
10.3:1.9
11.742.0
10.4+2.1
10.242.5
12.0+2.3
12.8+2.4
11:2:2.
11:0118
11.022.0
12.6#2.5
20 vs. 20
9.622.3
13.3422
13.521.9
14.622.0
10.4
11.141.4
11.8&1.5
13.842.0
12.4+188
12.8
15.0a1.8
15.2:24
20.vs.23
9.5
10.422.2
11.4219
12.4+2.3
9.521.7
11.2+188
11.4+1.6
12
10.3:1.
11.7
11.722.0
12.4=2.2
Table 18: Returns Comparison with Reduced Preference LLM-based Data
B.8.6. COMPARISON OF 1-LAYER VS. 2-LAYER MIXERS
This experiment evaluates the impact of using l-layer linear mixers versus 2-layer non-linear mixers in the value decomposi- tion process. The comparison is carried out on both returns and winning rates for SMACv1 and SMACv2 tasks, using both rule-based and LLM-based datasets. The goal is to validate the theoretical claim that 1- 1-layer mixers provide better convexity and stability, while examining whether 2-layer mixers offer any practical advantage.
Returns
Winning Rates
Rule-based
LLM-based
Rule-based
L.L--based
1-layer
2-layer
1-layer
2-layer
1-layer
2-layer
1-layer
2-layer
2e_v8_642g
19.3+1.4
12.9
19.6#1.1
13.7+1.6
74.4+24.7
2.3+7.3
19.5+19.6
3.9+11.0
5m_vs. 6m
1
7.1+0.8
11.5+2.3
7.5=0.5
19.3=19.6
0.8+4.3
20.7+20.5
0.0=0.0
SMACv1
6h.vs.8z 8z
12.1
9.5+0.4
12.2+1.3
9.940.6
4.5+11.0
0.0+0.0
6.1+11.2
0.0=0.0
corridor
19.640.9
4.4+14
19.7=0.8
2.7
93.2+13.5
0.0=0.0
94.5+11.2
0.0=0.0
vs 5
16.842.4
13.822.7
17.9=2.5
16.3+2.9
54.3=24.2
36.7425.8
61.5+24.8
48. 48.4=29.3
10_vs_10
17.9&1.9
15.5
18.0+2.1
16.7+1.9
53.7423.6
42.9+20.1
61. 61.18
29.7+19.2
protoss
10 vs. 11
14.9+2.0
13.7+2.1
16.5=2.2
13.3+2.1
30.7=19.8
14.1=16.2
34,4=24.8
21.1
20_vs_20
18.0&1.5
15.3+1.7
18.9&1.5
15.6+1.6
59.8+23.2
13.4
64.5+23.5
29.7+24.6
20_vs 23
14.9+2.0
12.5#1.6
15.8=1.9
13.5+1.6
23.4=19.2
9.8+66.5
26.4=20.8
11.7=16.5
__vs_5
12.853.5
10,242,9
12.6#2.6
11.242.7
39.5=24.7
26.0=20.8
43.0=23.0
22.7=20.1
10 vs 10
11.842.6
9.0+2.6
12.5+2.7
10.0+2.4
28.3+20.6
19.4=23.0
33.2+23.4
26.7+22.0
SMACV2
terran
10_vs.11
11.0+2.8
8.1+2.4
10.7+2.5
9.0=3.1
18.2=18.7
7.0+14.3
21.3+20.3
13.3=20.7
20 vs. 20
11.842.4
9.4+2.0
13.0#2.8
11.9+2.4
23.0#22.4
12.0+16.5
24.4+23.1
10.9
20.vs 23
9.4+2.0
8.0+2.0
9.1+2.3
1.3
7.2+12.9
5.5+10.3
8.6214.8
1.6#6.1
5_vs_5
13.15
10.1+3.7
12.9=2.6
9.4+3.0
35.2+25.7
30.6#30.6
40.8+21.6
25.6+24.4
10 vs 10
14.042.6
10.5+3.0
14.5=2.6
11.242,8
34.8=22.1
24.6228.5
37.9=24.0
27.5427.1
zerg
10_vs_11
12.8+2.7
11.9
12.6a2.5
11.7+1.8
23.4+21.1
15.6*16.2
26.0=23.0
10.9=13.9
20 vs 20
13.9+1.8
11.0=1.4
15.2+2.4
12.2+1.7
24.8=20.8
12.0=15.3
31.1=24.6
12.5=14.0
20_vs_23
12.742.0
1..
12.4+2.2
9.9+1.5
18.8
3.148.3
16.0=19.4
1.9
Table 19: Comparison of our O-MAPL with 1-layer vs. 2-layer Mixers: Returns and Winning Rates (in percentage)
B.8.7. COMPARISON OF GPT-40 vs. GPT-40-MINI
To evaluate the effect of preference data quality, the performance of O-MAPL is compared when using high-quality LLM-ggnerated preferences (GPT-4o) versus lower-quality preferences (GPT-4o-mini). Both returns and winning rates are analyzed to understand the sensitivity of O-MAPL to the quality of preference annotations.
30

---

O-MAPL: Offline Multi-agent Preference Learning
Returns
Winning Rates
40
4o-mini
4o
4o-mini
2c_vs_64zg
19.6+1.1
17.0#1,4
79.5=19.6
62,8+17.4
5m vs. vs.6m
11.5+2.3
9.6+2.2
20.7420.5
19.4420.2
SMACv
6h_vs_8z
12.2+1.3
10.8#1.2
6.1+11.2
2.8+8.3
corridor
19.740.8
17.940.5
94.5+11.2
88.148.1
5_vs.5
17.9+2.5
15.6+2,4
61.5+24.8
58. 1+28.8 8
10_vs_10
18.0+2.1
17.542.4
61.1+24.8
54.4428.2
protoss
10 I1
16.5+2.2
14.242.5
34.4424.8
29. 192288
20-vs.20
18.921.5
17.0#1.8
64.5+23.5
60.0+27.6
20. 23
15.8+1.9
15.242.6
26.4420.8
26.2430.9
5.vs. 5
12.642.6
14.3+3.8
43.0423.0
45.9428.8
10_vs_10
12.5+2.7
9.6+3.3
33.2+23.4
30,0+30,9
SMACv2
terran
10
10.742.5
9.842.9
21.3420.3
13. 138.5
20. vs. 20
13.0+2.8
11,742.5
24.4+23.1
20.6+26.9
20
9.142.3
7.741.9
8.6214.8
1.91116
5 vs
12.942.6
11.243.4
40.8421.6
38.3428.8
10_vs_10
14.5+2.6
12.742.4
37.9424.0
34.7+25.2
zerg
10_vs_11
12.6+2.5
11.0+2.8
26.0423.0
22.5+23.6
20.
15.242.4
12.942.2
31,1424.6
25.3425.2
20_vs_23
12.4+2.2
10.2+2.2
16.0+19.4
14.2+16.6
Table 20: Comparison of our O-MAPL with GPT-4o vs. GPT-4o-mini: Returns and Winning Rates (in percentage)
31


Reviews:
[{"review_id": "O3odrIQdFU", "reviewer": "Reviewer_xng9", "summary": "The paper introduces O-MAPL, a novel framework for multi-agent reinforcement learning that leverages human preference data to train cooperative agents without explicit reward modeling. Traditional MARL methods often require separate stages for reward learning and policy optimization, leading to instability and inefficiency. O-MAPL addresses this by directly learning the soft Q-function from pairwise trajectory preferences, using a centralized training with decentralized execution (CTDE) paradigm. The approach incorporates a value decomposition strategy to ensure global-local consistency and convexity in the learning objective, enabling stable and efficient policy training.\n\nThe authors conduct extensive experiments on two benchmarks. Results show that O-MAPL consistently outperforms existing methods across various tasks, demonstrating its effectiveness in complex multi-agent environments. The paper also highlights the potential of using LLMs for generating rich and cost-effective preference data, which significantly improves policy learning.\n\nKey contributions include:\n1. An end-to-end preference-based learning framework for MARL that avoids explicit reward modeling.\n2. A value factorization method that ensures global-local consistency and convexity in the learning objective.\n3. Extensive empirical validation showing superior performance over existing methods.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for carefully reading our paper and providing valuable feedback. Below, we address your concern in detail.\n> I have one concern about the correctness of the derivation of the implicit-reward method. In section 4.2, the authors proposed to design a training objective function to avoid explicit reward modeling. However, this training objective relies on the inverse soft Bellman-operator. In L(), I did not see how to guarantee the inverse soft Bellman-operator which requires the soft Q function to be a fixed point of a soft Bellman-operator. Therefore, in the training pipeline, if the updated Q function can not be guaranteed to be a fixed point of a soft Bellman-operator, then the algorithm can not provide a convergence guarantee to an optimal policy. In practice, this issue usually lead to instability or sub-optimal solutions of the implicit-reward methods [1, 2].\n\nWe thank the reviewer for the comment. To clarify, we can formally prove that the trained $Q_{\\text{tot}}$ resulting from our preference-based training objective is guaranteed to be a fixed-point solution of the soft Bellman equation. The proof closely follows the structure of prior work such as IQ-Learn, and we provide a version tailored to our formulation and notation in the following.\n\n**PROOF**: \nWith the definition of $\\mathcal{T}^* Q_{\\text{tot}}$ as\n\n$(\\mathcal{T}^* Q_{\\text{tot}})(s, a)$ = $Q_{\\text{tot}}(s, a)$ - $\\gamma$ $E_{s'}$ $[V_{\\text{tot}}(s')]$\n\nwe can rewrite the inverse Bellman equation as follows:\n\n$Q_{\\text{tot}}(s, a)$ = $(\\mathcal{T}^* Q_{\\text{tot}})(s, a)$ + $\\gamma$ $E_{s'}$ $\\left[ V_{\\text{tot}}(s') \\right]$,\n\nwhich implies:\n\n$Q_{\\text{tot}}(s, a)$ = $(B_r^*$ $Q_{\\text{tot}})(s, a)$,\n\nwhere the reward function is defined as \n\n$r(s, a)$ = $(\\mathcal{T}^* Q_{\\text{tot}})(s, a)$, and $(B_r^*$ $Q_{\\text{tot}})(s, a)$ is the soft Bellman backup operator defined in our paper. This result shows that $Q_{\\text{tot}}$ satisfies the soft Bellman equation.\n\nTherefore,  if we train the preference-based objective using $(\\mathcal{T}^* Q_{\\text{tot}})(s, a)$ — i.e., via the inverse soft Bellman equation — then the resulting $Q_{\\text{tot}}$ is guaranteed to be a **fixed-point solution** to the soft Bellman equation with reward $r(s, a)$ = $(\\mathcal{T}^*$ $Q_{\\text{tot}})(s, a)$.\n\n**EndProof**\n\nWe will incorporate the above discussion into the revised paper to clarify the fixed-point convergence of our method.\n\n---\n\n*We hope that the above responses satisfactorily address your questions and concerns. If you have further questions, we are happy to provide additional clarification.*", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "This paper considers a natural extension from existing implicit-reward offline RL method to MARL setting.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "I have one concern about the correctness of the derivation of the implicit-reward method. In section 4.2, the authors proposed to design a training objective function $L(q,v,\\omega)$ to avoid explicit reward modeling. However, this training objective relies on the inverse\nsoft Bellman-operator. In $L(q,v,\\omega)$, I did not see how to guarantee the inverse soft Bellman-operator which requires the soft Q function to be a fixed point of a soft Bellman-operator. Therefore, in the training pipeline, if the updated Q function can not be guaranteed to be a fixed point of a soft Bellman-operator, then the algorithm can not provide a convergence guarantee to an optimal policy. In practice, this issue usually lead to instability or sub-optimal solutions of the implicit-reward methods [1, 2].\n\n[1] Garg, Divyansh, et al. \"Iq-learn: Inverse soft-q learning for imitation.\" Advances in Neural Information Processing Systems 34 (2021): 4028-4039.\n\n[2] Rafailov, Rafael, et al. \"Direct preference optimization: Your language model is secretly a reward model.\" Advances in Neural Information Processing Systems 36 (2023): 53728-53741.", "other_comments_or_suggestions": "I think how to guarantee the stability and convergence of the implicit-reward method can be critical to the proposed method.", "questions_for_authors": "it will be helpful if the authors show that the Q function updated in their designed training pipeline can be a fixed point to one soft Bellman operator.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2Mdc5m0Y5P", "reviewer": "Reviewer_TgVH", "summary": "This paper proposes a multi-agent offline reinforcement learning algorithm named O-MAPL, addressing the problem of directly training multi-agent policies using human preference data under offline data conditions.\nThe authors highlight limitations of traditional two-stage methods (first learning a reward model, then learning a policy), including:\n1. The requirement for large preference datasets covering state and action spaces.\n2. Performance degradation due to misalignment between the two stages.\n\nTo address these issues, they propose an end-to-end framework that directly learns policies from preference data in an offline setting. Specifically, leveraging the correspondence between soft Q-functions and reward functions under the Maximum Entropy Reinforcement Learning (MaxEnt RL) framework, they introduce a method to train Q-functions (i.e., agent policies) directly using preference data.\n\nKey technical contributions include:\n1. A mixing network for value factorization to enable multi-agent coordination.\n2. A linear value decomposition network designed to align with preference learning objectives and the IGM (Individual-Global-Maximization) principle, ensuring training stability and global-local consistency.\n\nExperimental results demonstrate that O-MAPL outperforms existing methods across benchmark environments such as SMAC (StarCraft Multi-Agent Challenge) and MaMuJoCo (Multi-Agent MuJoCo).", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the positive evaluation and insightful feedback. Below, we provide detailed responses to your comments. To address your questions, we have conducted additional experiments, which are detailed in the **PDF** available via this anonymous link: https://1drv.ms/b/s!AgChHLa7t5Bza5QJpMfi7YJX6PI\n\n---\n\n> How does the method perform when preference feedback is limited?\n\nWe thank the reviewer for the insightful comment. To address your question, we conducted additional experiments in which we systematically reduced the amount of preference data to 75%, 50%, and 25% of what was used in our main experiments. The comparison results — reported in Tables 1 and 2 in the **PDF** — demonstrate how our method and several important baselines perform under reduced preference feedback. As expected, the overall performance of all methods degraded with less data; however, O-MAPL consistently outperformed the other baselines across all settings, showing robustness to limited preference feedback. \n\n> Insufficient demonstration of the value decomposition module’s specific contributions.\n\nWe thank the reviewer for the comment. To clarify, we have included several variants of our method to evaluate the specific contributions of the value decomposition module. Specifically, IPL-VDN is a variant where we fix the weights in the value decomposition module and do not learn them, while IIPL represents a variant in which each local value function is learned independently per agent, without being aggregated through our mixing architecture. \n\nIn addition, we have conducted additional experiments comparing the performance of *1-layer* and *2-layer* mixing networks in our value decomposition approach. The comparison results are reported in Table 3 of the **PDF**.\n\n> Can the approach be extended to mixed cooperative-competitive scenarios?\n\nWe thank the reviewer for the question. While the value decomposition approach has potential applicability to mixed cooperative-competitive scenarios, it would require substantial adaptations to handle such complexities effectively. We consider this extension beyond the current scope but agree it is a valuable direction for future exploration.\n\n---\n\n*We hope that the above responses satisfactorily address your questions and concerns. If you have further questions, we are happy to provide additional clarification.*", "claims_and_evidence": "The effectiveness of the proposed O-MAPL algorithm is sufficiently supported by experiments.\n\nValidity of the end-to-end learning framework: Theoretical analysis and experimental results (on SMAC and MaMuJoCo benchmarks) demonstrate that learning policies directly from preference data outperforms traditional two-stage methods (baselines include single-agent methods extended to multi-agent settings, e.g., IPPL, SL-MARL, and methods similar to this work but lacking the proposed value decomposition architecture, e.g., IPS-VDN).\n\nImportance of value decomposition: Theoretical analysis of the convexity of single-layer mixing networks and global-local consistency.", "methods_and_evaluation_criteria": "The proposed approach of directly learning multi-agent policies from preference data in an offline setting is meaningful for multi-agent preference learning. In multi-agent scenarios, obtaining accurate reward signals is challenging, and preference learning offers a promising solution. However, preference learning in MARL remains underexplored and faces issues such as training efficiency and human involvement costs. Existing two-stage methods (first learning a reward function, then training policies) require extensive preference data and suffer from misalignment between stages. Thus, directly training policies from preference data is a significant contribution to multi-agent preference learning. The use of SMAC and MaMuJoCo for evaluation is appropriate, as these environments are widely adopted for benchmarking multi-agent reinforcement learning methods. Training with both rule-based and LLM-generated preference data is reasonable, as these methods generate preference data of varying quality, enabling comprehensive algorithm evaluation.", "theoretical_claims": "The theoretical analysis in Section 4 includes:\n1. Theorem 4.1 (Convexity Analysis): The loss function under a single-layer linear mixing network is convex.\n2. Proposition 4.2 (Non-convexity in Nonlinear Networks): Two-layer mixing networks disrupt convexity.\n3. Proposition 4.3 (Global-Local Consistency): Local policy optimization aligns with global objectives.\nThese theoretical analyses are clear and free of obvious errors.", "experimental_designs_or_analyses": "1. Experiments are conducted on widely used SMAC and MaMuJoCo environments, ensuring representativeness.\n2. Training with rule-based and LLM-generated preference data covers diverse feedback quality.\n3. Results are statistically significant, with mean values and standard deviations provided.", "relation_to_broader_scientific_literature": "The proposed method is closely related to existing literature:\n1. It extends the MaxEnt RL framework to multi-agent offline preference learning, enabling end-to-end training.\n2. It aligns with preference learning research by leveraging preference data for multi-agent policy training.", "essential_references_not_discussed": "The related work section adequately covers offline RL and preference learning.\nThis work builds upon multi-agent offline RL and integrates preference data for policy training.", "other_strengths_and_weaknesses": "Strengths:\n1. Practical relevance: The offline learning paradigm avoids costly trial-and-error interactions.\n2. Novelty: A new paradigm for multi-agent preference learning that mitigates defects of two-stage methods.\n3. Rigorous theoretical analysis and experiments.\n\nWeaknesses:\n1. No evaluation under limited preference data, a common real-world constraint.\n2. Insufficient demonstration of the value decomposition module’s specific contributions.", "other_comments_or_suggestions": "Suggest adding implementation details in the appendix to facilitate reproducibility.", "questions_for_authors": "1. How does the method perform when preference feedback is limited?\n2. Can the approach be extended to mixed cooperative-competitive scenarios?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FMXooT46R9", "reviewer": "Reviewer_5DBX", "summary": "This paper introduces O-MAPL, an end-to-end preference-based reinforcement learning framework for cooperative multi-agent systems, addressing the challenge of inferring reward functions from demonstrations in complex MARL settings. Prior methods often separate reward learning and policy optimization, leading to instability. The authors propose a novel approach that bypasses explicit reward modeling by leveraging the relationship between rewards and soft Q-functions in MaxEnt RL.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the reviewer's detailed feedback and constructive questions. To address your questions, we have conducted additional experiments, which are detailed in the **PDF** available via this anonymous link: https://1drv.ms/b/s!AgChHLa7t5Bza5QJpMfi7YJX6PI\n\n----\n\nBelow, we respond point-by-point to your valuable suggestions and concerns.\n\n> There are no comparisons of training curves, convergence rates, or sensitivity analyses against baselines like IIPL or IPL-VDN …\n\nWe thank the reviewer for highlighting this important point. To clarify, the curves presented in our paper (Figure 1 and additional figures in the appendix) represent **evaluations during training progress**. Specifically, these curves illustrate the evolution of the value function during training, comparing different approaches (O-MAPL, IIPL, IPL-VDN). \n\n> The experiments use LLM-generated preference data for training O-MAPL but do not clarify whether baseline methods (e.g., IIPL, BC) were evaluated under the same data conditions.\n\nWe thank the reviewer for the insightful comment. We would like to clarify that all baselines have been evaluated using the **same preference data**. We will explicitly state this point in the revised manuscript.\n\n> Additionally, there is no ablation study comparing rule-based vs. LLM-generated preferences for O-MAPL\n\nWe appreciate the reviewer's suggestion. To clarify, we already have Table 1 and several figures in the appendix which show a clear comparative analysis between **rule-based and LLM-based preference data** across all methods considered. \n\n> Implementation details lacked critical specifics on preference dataset generation (e.g., LLM hyperparameters) and omitted pseudocode, hindering reproducibility. \n\nWe thank the reviewer for this feedback. To clarify, we have already included several details in the appendix on our data generation methods (rule-based and LLM-based), including hyperparameters and specific prompts used to obtain LLM-generated preferences. We believe this information is sufficient for reproducibility. \n\nRegarding the pseudocode, we agree that our algorithm description is currently quite brief. We will include a more detailed and structured presentation of the algorithm in the updated paper to improve clarity and reproducibility. \n\n> Additional experiments provided limited ablation studies (e.g., linear mixing tested only on SMAC, not MAMuJoCo) and did not address computational costs or robustness to preference noise\n\nWe thank the reviewer for raising these important points. To clarify, our experiments did apply linear mixing and our proposed value decomposition methods to **both SMAC and MAMuJoCo tasks**.\nRegarding computational costs, we have provided some details in section B.2. If the reviewer could specify which particular aspects of preference noise we have overlooked, we would be happy to provide further clarification and analysis.\n\n> The work focuses exclusively on cooperative settings, leaving mixed cooperative-competitive environments unexplored. …\n\nWe thank the reviewer for highlighting this important consideration. Our current work focuses specifically on cooperative settings. As noted in our conclusion, addressing mixed cooperative-competitive scenarios would indeed necessitate significantly different methodologies, which lie beyond the current scope. We agree that exploring these mixed scenarios is a valuable direction for future research.\n\n> The experiments do not include comparisons with non-linear mixing variants. …\n\nWe thank the reviewer for this insightful comment. We focused our experiments on the linear mixing setting. This decision was based on previous studies indicating that two-layer mixing networks perform significantly worse than one-layer linear mixing networks especially in offline settings with limited data [1]\n\n[1] Bui et al. ComaDICE: Offline cooperative multi-agent reinforcement learning with stationary distribution shift regularization. In ICLR 2025.\n\nTo address your question, we have conducted additional comparative experiments between linear and nonlinear mixing structures. The results, shown in **Table 3** in the **PDF**, clearly demonstrate that the linear mixing structure outperforms the two-layer structure. \n\n> The paper does not analyze the quality of LLM-generated labels.... How sensitive is O-MAPL to potential inaccuracies or biases in LLM-generated preferences?\n\nWe thank the reviewer for the insightful comment. To address this concern, we conducted additional experiments comparing the performance of our method using two versions of ChatGPT: **4o and 4o-mini** (the latter being considered weaker in terms of long-term reasoning capabilities). The comparison results, reported in **Table 4** in the **PDF**.\n\n---\n\n*We hope that the above responses satisfactorily address your questions and concerns. If you have further questions, we are happy to provide additional clarification.*", "claims_and_evidence": "While the paper argues that prior methods suffer from instability due to phased training, no empirical evidence is provided to demonstrate that O-MAPL actually achieves greater stability. For instance, there are no comparisons of training curves, convergence rates, or sensitivity analyses against baselines like IIPL or IPL-VDN that decouple reward and policy learning. Without such comparisons, the claim remains speculative.\n\nThe experiments use LLM-generated preference data for training O-MAPL but do not clarify whether baseline methods (e.g., IIPL, BC) were evaluated under the same data conditions. Performance gains could stem from the quality or scale of LLM-generated data rather than the algorithm itself. Additionally, there is no ablation study comparing rule-based vs. LLM-generated preferences for O-MAPL, making it impossible to disentangle the contribution of the framework from the data source.", "methods_and_evaluation_criteria": "The methods are well aligned with the challenges of multi-agent preference-based RL. By integrating MaxEnt RL with value decomposition under CTDE, O-MAPL addresses the instability of prior two-phase approaches while ensuring global-local policy consistency. The use of linear mixing networks and weighted BC for policy extraction is pragmatic for offline settings, balancing stability and scalability.\n\nEvaluation on SMAC (discrete) and MAMuJoCo (continuous) benchmarks covers diverse MARL scenarios, and the inclusion of LLM-generated preference data reflects real-world applicability. While human-labeled datasets are absent (a common MARL limitation), the rule-based/LLM-generated data strategy is reasonable and reproducible. Baselines are relevant, though broader comparisons (e.g., with recent offline MARL methods) could strengthen claims. Overall, the design and evaluation credibly validate the framework’s efficacy.", "theoretical_claims": "All the proofs are mathematically sound under idealized assumptions", "experimental_designs_or_analyses": "The experiments use LLM-generated preference data for training O-MAPL but do not clarify whether baseline methods (e.g., IIPL, BC) were evaluated under the same data conditions. Performance gains could stem from the quality or scale of LLM-generated data rather than the algorithm itself. Additionally, there is no ablation study comparing rule-based vs. LLM-generated preferences for O-MAPL, making it impossible to disentangle the contribution of the framework from the data source.", "relation_to_broader_scientific_literature": "The paper’s key contributions are positioned within the broader MARL literature by advancing offline preference learning and stability in decentralized value decomposition, addressing gaps in prior work while building on established frameworks.", "essential_references_not_discussed": "In my point of view, the related works that are essential to understanding the (context for) key contributions of the paper were all currently cited/discussed in the paper.", "other_strengths_and_weaknesses": "**Strengths**\n\n1. The paper introduces a novel end-to-end preference-based learning framework for multi-agent RL that bypasses explicit reward modeling by leveraging the intrinsic relationship between reward functions and soft Q-functions. This approach is original, as it creatively combines MaxEnt RL principles with a multi-agent value decomposition strategy, addressing the instability of prior two-phase methods while ensuring global-local policy consistency through theoretical guarantees.\n\n2. The authors rigorously evaluate their method across diverse benchmarks (SMAC, MAMuJoCo) using both rule-based and LLM-generated preference data. The results demonstrate consistent superiority over baselines, particularly highlighting the effectiveness of LLM-based preference data, which opens new avenues for cost-effective training in complex multi-agent systems.\n\n**Weaknesses**\n\n1. The work focuses exclusively on cooperative settings, leaving mixed cooperative-competitive environments unexplored. This restricts the framework’s applicability to real-world scenarios where adversarial interactions or heterogeneous objectives are common, and the proposed value decomposition may not generalize.\n\n2. While leveraging LLMs mitigates data generation costs, the method still requires substantial trajectory pairs for training. The paper does not address sample efficiency in low-data regimes or human-in-the-loop settings, limiting its practicality where preference queries are expensive or scarce.", "other_comments_or_suggestions": "None.", "questions_for_authors": "1. The paper emphasizes the use of linear mixing networks to preserve convexity (Prop. 4.1–4.2) but acknowledges that non-linear mixing (e.g., two-layer networks) is common in prior MARL works. However, the experiments do not include comparisons with non-linear mixing variants. Could the authors provide ablation studies or empirical evidence showing that linear mixing indeed outperforms non-linear alternatives in their framework? \n\n\n2. The LLM-based preference generation is a novel contribution, but the paper does not analyze the quality of LLM-generated labels (e.g., alignment with ground-truth human preferences or rule-based labels). How sensitive is O-MAPL to potential inaccuracies or biases in LLM-generated preferences?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XYUMnS3V5D", "reviewer": "Reviewer_XeUY", "summary": "This paper studies the problem of cooperative multi-agent reinforcement learning from preference data. The authors formulate the problem as cooperative Markov game with a global reward function where the goal of the agents is to learn optimal policies given offline pairs of trajectories with corresponding preferences. The authors make use of a one-to-one correspondence between the inverse Bellman operator of KL-regularized Q-functions and the global reward function to formulate the optimal policy objective--thereby reducing the problem to learning the optimal Q-functions. The combinatorial computational complexity due to the presence of many agents is tackled via value decomposition and mixing through linear (learnable) maps. Moreover, the problem of maintaining a local-to-global consistency while simultaneously having well-defined policies over probability simplexes is tackled through local weighted behavior cloning techniques. The authors also theoretically demonstrate the correspondence of global optimal policies with the learned local policies and the validity of the returned value functions. Extensive experimental results showcase the benefit of using their proposed method, O-MAPL.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer for carefully reading our paper, offering a positive evaluation, and providing valuable and insightful comments. To address your questions, we have conducted additional experiments, which are detailed in the **PDF** available via this anonymous link: https://1drv.ms/b/s!AgChHLa7t5Bza5QJpMfi7YJX6PI\n\n---\n\nBelow, we address each of your comments and questions:\n\n> Why do you switch from discounted MDPs to finite-horizon MDPs when you discuss preference-based learning in Section 4.1? Isn't it better to have a unified framework?\n\nWe sincerely thank the reviewer for this insightful comment. We initially adopted the discounted infinite-horizon MDP framework as it is widely utilized in RL and MARL literature due to its general applicability. However, for our objective of the preference-based learning, we chose to exclude the discount factor because the preference data commonly consists of partial or sub-trajectory information rather than complete trajectories. Therefore, our algorithm was developed around an undiscounted sum of rewards, which aligns closely with recent preference-based approaches in the single-agent setting. We will clearly articulate and emphasize this choice in the revised manuscript. \n\n> Related to my comment on the convergence, is it possible to add some convergence guarantees of O-MAPL? There is an alternating optimization problem being solved for both the optimal Q-function and value function. Due to convexity/concavity, I expect it to converge in finite time.\n\nWe sincerely thank the reviewer for this insightful comment. Our alternating optimization approach is based on reformulating the primary learning problem as an equivalent bilevel optimization problem, as shown below:\n\n $max_q L(q,v,w)$ s.t. $v= \\arg\\min  J(v)$\n\nIn this formulation, $L(q,v,w)$ is concave in $q$ and  $J(v)$ and is convex in $v$. This convex-concave structure allows the alternating optimization process — where $q$ and $v$ are updated aternatively — to converge to a stationary point. This optimization method is widely adopted and well-supported in the literature on *bilevel optimization*.\n\nWe will clarify this point and cite relevant works on bilevel optimization to support our approach. Thank you for bringing this to our attention.\n\n> However, what is not clear to me is how the presence of WBC affects convergence.\n\nWe thank the reviewer for this insightful question. To clarify, the convergence of the Q and V functions arises from solving the bilevel optimization problem using an alternating procedure. The convergence of the WBC component, on the other hand, is theoretically supported by Theorem 4.3 under the assumption that Q and V are fixed. In practice, we train the WBC component simultaneously with Q and V updates to improve computational efficiency. Empirically, one can show that as Q and V converge, the WBC also reliably converges to the optimal policy given a sufficient number of update steps. \n\nWe will better clarify this point in the revised manuscript.\n\n> How does data coverage affect the performance of O-MAPL, that is, have you tested it in datasets where not enough \"good\" trajectories are present? How does it affect the performance?\n\nWe thank the reviewer for this insightful question. To address this point, we conducted additional experiments to evaluate the effect of data quality and coverage on O-MAPL’s performance. Specifically, we varied the number of preference samples, and examined cases where LLMs provided lower-quality preference labels. The additional results are reported in **Tables 1, 2 and 4** in the **PDF**. These additional results show that O-MAPL remains robust across a range of dataset qualities, although its performance degrades gracefully as the quality of preference data decreases.\n\n We will include these additional results in the revised manuscript to highlight this robustness.\n\n> Can you comment on the presence of the additional term in Proposition 4.5? The result indicates that the value function is not \"exactly\" expressed as log-sum-exp of the Q-function, but there is an additional bias. Why is that?\n\nWe thank the reviewer for this insightful question. In Proposition 4.5, we demonstrate that the local value function *cannot be exactly expressed* as the log-sum-exp of the local function. This highlights a key limitation of prior MARL approaches that rely solely on local functions to compute local policies or values. The fundamental reason is the interdependence among agents in cooperative settings — local policies are influenced by the joint behavior of other agents. As such, expressing them purely in terms of local and functions leads to an approximation error. We will clarify this point more explicitly in the revised manuscript. \n\n---\n\n*We hope the above responses satisfactorily address your questions and concerns. If you have any further questions or comments, we would be happy to provide additional clarification.*", "claims_and_evidence": "The convexity of the parameter space over the value functions is theoretically proven. Moreover, the consistency between optimal global policies and optimal local policies, in the WBC sense, is also theoretically shown. The validity of O-MAPL algorithm is demonstrated by extensive experiments on several MARL benchmarks against several baselines.", "methods_and_evaluation_criteria": "The proposed methods and the evaluation criteria are reasonable for the problem at hand.", "theoretical_claims": "I skimmed through the proofs of the main results in the appendix, and, they seem overall fine. No blunders or unexpected claims. However, I haven't checked in detail.", "experimental_designs_or_analyses": "I have not checked the validity of the experimental results.", "relation_to_broader_scientific_literature": "There are only two papers that discuss MARL from preference data, namely, (Kang et al., 2024) and (Zhang et al., 2024), both of which go through the reward learning phase and then perform policy optimization on top. The approach of the current paper is novel, in that regularized logistic regression is used to obtain estimates of the inverse Bellman operator of the Q-function, which correspond to the logits of the optimal policy directly. Such an approach bypasses the policy optimization using estimated reward functions, which is where it diverges from previous work.", "essential_references_not_discussed": "To the best of my knowledge, there aren't any essential references missing. The related work section seems adequately comprehensive.", "other_strengths_and_weaknesses": "1. I really enjoyed reading the paper as it is well-written, clear and coherent. \n2. The proposed method has several novel components that are brought together: (i) the reduction of the reward learning problem to the Q-space; (ii) the usage of linear mixing networks for the value decomposition phase; (iii) the usage of local weighted behavior cloning to align local policies to learned global value functions. \n3. The utilized rationale behind O-MAPL is justified from theoretical results.\n4. The experimental validation of O-MAPL is extensive, which make the algorithm quite practical. \n\nI would have liked to see some simple convergence result which sheds some light into how close the output global policy is to the ground-truth global optimal policy with respect to the global reward function, in terms of problem-specific parameters such as data size, data coverage, policy class and the used mixing networks and WBC technique.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. Why do you switch from discounted MDPs to finite-horizon MDPs when you discuss preference-based learning in Section 4.1? Isn't it better to have a unified framework?\n2. Related to my comment on the convergence, is it possible to add some convergence guarantees of O-MAPL? There is an alternating optimization problem being solved for both the optimal Q-function and value function. Due to convexity/concavity, I expect it to converge in finite time. However, what is not clear to me is how the presence of WBC affects convergence. \n3. How does data coverage affect the performance of O-MAPL, that is, have you tested it in datasets where not enough \"good\" trajectories are present? How does it affect the performance?\n4. Can you comment on the presence of the additional term in Proposition 4.5? The result indicates that the value function is not \"exactly\" expressed as log-sum-exp of the Q-function, but there is an additional bias. Why is that?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}]

Author rebuttal:

Meta-review: he paper introduces O-MAPLE, an end-to-end framework for offline multi-agent reinforcement learning (MARL) from preferences, avoiding explicit reward modeling by leveraging soft Q-function estimation via inverse Bellman operators. The proposed framework integrates value decomposition with convex mixing networks and uses centralized training with decentralized execution (CTDE), addressing scalability and stability challenges in preference-based MARL. The paper includes theoretical analyses, experimental validation across standard MARL benchmarks (SMAC, MaMuJoCo), and studies the utility of LLM-generated preferences.

There is some disagreement regarding the interpretation of regularized value functions and whether the Q-function must be optimal for the inverse Bellman operator to be meaningful. The authors are encouraged to clarify the theoretical assumptions and limitations surrounding the use of the inverse soft Bellman operator when the Q-function is not yet optimal. While the derivation appears sound, a discussion of convergence conditions and practical implications would benefit readers.
Decision: Accept (poster)