MisoDICE: Multi-Agent Imitation from Mixed-Quality Demonstrations
Authors: The Viet Bui
Tien Anh Mai
Thanh Hong Nguyen
Venue: NeurIPS 2025 poster ()
Keywords: Multi-Agent Reinforcement Learning
Offline Imitation Learning
Large Language Models
Value Factorization
Primary area: reinforcement_learning

Abstract: We study offline imitation learning (IL) in cooperative multi-agent settings, where demonstrations have unlabeled mixed quality - containing both expert and suboptimal trajectories. Our proposed solution is structured in two stages: trajectory labeling and multi-agent imitation learning, designed jointly to enable effective learning from heterogeneous, unlabeled data. In the first stage, we combine advances in large language models and preference-based reinforcement learning to construct a progressive labeling pipeline that distinguishes expert-quality trajectories. In the second stage, we introduce MisoDICE, a novel multi-agent IL algorithm that leverages these labels to learn robust policies while addressing the computational complexity of large joint state-action spaces. By extending the popular single-agent DICE framework to multi-agent settings with a new value decomposition and mixing architecture, our method yields a convex policy optimization objective and ensures consistency between global and local policies. We evaluate MisoDICE on multiple standard multi-agent RL benchmarks and demonstrate superior performance, especially when expert data is scarce.

MisoDICE: Multi-Agent Imitation from Unlabeled Mixed-Quality Demonstrations
The Viet Bui
Tien Mai
Singapore Management University, Singapore theviet. bui 2023@phdcs. smu. edu.sg
Singapore Management University, Singapore atmai @smu. edu.sg
smu.d
Thanh Hong Nguyen
University of Oregon Eugene, Oregon, United States thanhhngScs. uoregon. edu
Abstract
We study offline imitation learning (IL) in cooperative multi-agent settings, where demonstrations have unlabeled mixed quality containing both expert and sub- optimal trajectories. Our proposed solution is structured in two stages: trajectory labeling and multi-agent imitation learning, designed jointly to enable effective learning from heterogeneous, unlaheled data. In the first stage, we combine ad- vances in large language models and preference-based reinforcement learning construct progressive labeling pipeline that distinguishes expert-quality trajecto- ries. In the second stage, we introduce MisoDICE, novel multi-agent IL algorithm that leverages these labels to learn robust policies while addressing the compu- tational complexity of large joint state-action spaces. By extending the popular single-agent DICE framework to multi-agent settings with a new value decompo- sition and mixing architecture, our method yields convex policy optimization objective and ensures consistency between global and local policies. We evaluate MisoDICE on multiple standard multi-agent RL benchmarks and demonstrate superior performance, especially when expert data is scarce.
Introduction
Imitation Learning (IL) is key subfield of Reinforcement Learning (RL) that focuses on learning effective policies solely from expert demonstration data, without requiring direct access reward functions [60] A broad range of IL algorithms have been developed - primarily for single-agen settings ranging from simple supervised learning methods like Behavior Cloning (BC) [41 45] to aecupancy-matching
more advanced occupancy matching techniques [16] [8 23). While these methods typically assume access to high-quality expert demonstrations, recent research has extended single-agent IL more
challenging scenarios involving limited or suboptimal data g 28 65
Motivated by the success of single-agent II., recent studies have sought extend IL. multi- agent settings characterized by complex inter-agent dynamics. For example, leading multi-agent II. algorithms have adapted well-known single-agent approaches [16
231 by incorporating game- theoretic concepts such as Nash equilibrium for competitive scenarios [51 59], or by carefully decomposing joint policies into local policies in cooperative tasks (36] key limitation of these methods is their reliance on large volumes of expert data to learn effective policies. In practice, acquiring such comprehensive datasets in complex multi-agent environments is highly impractical, due to the combinatorial explosion of joint state-action spaces as the number of agents increases.
18
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Our work tackles new highly challenging problem of offline multi-agent IL using only unlabeled demonstrations of mixed quality (i.e., containing both expert and sub-optimal trajectories). To the best of our knowledge, this is the first effort to address such a practical yer underexplored setting in multi-agent II., Our proposed solution is structured into two key stages: data labeling and multi-agent II, that work together to enable effective learning from heterogeneous unlabeled data.
The first stage, named multi-step data labeling, combines recent advances in LLMs and preference- based RL to build a progressive trajectory labeling pipeline. We begin by leveraging LLM-generater feedback to produce initial preference annotations between pairs of trajectories from the unlabeled dataset. While LLMs can offer valuable insights, their judgments are error- prone in the complex, high- nigh-dimensional dynamics of multi -agei- environments. To refine this coarse feedback, we introduce a second step using O-MAPL, leading preference-basee MARL algorithm [6. O-MAPL trains soft Q-function based on the initial L.M preferences, capturing a more grounded notion of demonstration quality. We then re- re-purpose this Q-function to extract reward signals and re-rank trajectories, enabling more accurate separation of expert and suboptimal data. This ranking lays the foundation for more effective and data-efficient multi-agent IL in our subsequent learning stage.
The second stage, the algorithmic core of our proposed framework, focuses on designing novel multi-agent IL algorithm, MisoDICE, capable of effectively leveraging the labeled demonstrations (expert vs. non-expert) to learn robust policies. MisoDICE follows an occupancy-matee matching principle, minimizing the divergence between the learned and behavior policies within a mixed -uveliiyuattyy dataset. To manage the exponential complexity in multi-agent environments, we build on the established Cen- tralized Training with Decentralized Execution (CTDE) paradigm. However, MisoDICE significantly differs from prior CTDE- based learning methods by introducing customized learning objectives and constraints that are suited to the challenges of mixed-quality demonstrations. Furthermore, MisoDICE features a unique value decomposition strategy and is supported by a theoretical analysis establishing local-global optimality consistency. Specifically, we show that under our value factorization strategy, the global learning objective is convex, which ensures both stability and efficiency during training. Moreover, we prove that once the value function is learned, each agent's 's local policy can be optimized independently and in parallel, while still guaranteeing convergence to the globally optimal joint policy. We also derive a closed-form mathematical relationship between the local policies and the value function, offering deeper insight into the structure and characteristics of the learned solution.
Finally, we conduct extensive experiments with comprehensive ablation studies on challenging multi- agent environments, i.e., SMACv1 [46). and SMACv2 [12.. The results show that our MisoDICE outperforms all baselines, highlighting the benefit of its integrated approach, combining occupancy matching, value decomposition, and effective use of the multi-step labeling process.
2 Related Work
Imitation Learning (IL). The well-known behavioral cloning (BC) frames IL as a supervised learning problem by maximizing the likelihood of expert state-action pairs under the learned policy [1 45). However, BC disregards environment dynamics, leading to compounding errors due to distributional shift. Recent methods recover the expert's reward function (explicitly implicitly) so that the learned policy can account for such dynamics 14]
16, 23
29 43 Notable examples include distribution-
matching-based GAIL and AIRL algorithms [6.
23 alternatively optimizing policy and reward
functions via an adversarial learning process, which requires access to on-policy data. In contrast, off-policy divergence objective to align state-action occupancies, enabling fully offline training. IQ-I_earn [18] bypasses reward recovery by directly learning the Q-function, leading to more stable policy learning than adversarial methods. Beyond these, other research addresses practical challenges in single-agent environments such as limited (65) or imperfect demonstrations 4 g 28 and constrained IL [47]. Additional directions include model-based and the use of diffusson models to improve policy learning from demonstrations 30 (4]
ValueDice 229 introduces
IL
12 61 [3
While many works focus on anglleagent settings. only few studies have explored IL. in multi-agent environments, whether cooperative [3] 5 BI 36 52] or competitive [3 44
51 59], primarily due
to the complex nature of multi-agent interactions. Some adapt single-agent methods like GAIL and
AIRL to multi-agent settings using game-theoretic concepts [51] 59 but they inherit instability
issues from adversarial training. recent algorithm, MIFQ (36). extends IQ-Learn [8] to the multi-
agent setting under the CTDE paradigm, which achieves outstanding performance in multi-agent IL

---

scenarios. Unlike existing multi-agent IL methods that rely on high-qualig expert data, we address a more challenging setting with only unlabeled demonstrations of various quality are available.
Preference-based Reinforcement Learning (PhRL.. PhRL is an emerging subfield of RL that can train policies only using preference data, typically in the form of pairwise trajectory comparisons. Most existing PbRL methods follow a two-phase approach: first, reward function is learned via supervised learning techniques (e.g., using the Bradley-Terry model), and then RL is applied to optimize the policy accordingly [
However, this separation introduces an additional learning component atop already complex RL algorithms, resulting high variance and instabiliity during training. To mitigate this issue, recent works proposed end-to-end methods that learn policies directly from preferences, using contrastive learning [1 20), information matching [26), or Q Q-function optimization [21, resulting in greater stability and performance. Following advances in single-agent PhRI., recent works have extended it to multi-agent settings ( 25 62], with O-
10 32
27
37 64
MAPL 6 achieving state-of-the-an art results on challenging multi-agent benchmarks. Our work uses L.LM -egneratd feedback to produce initial preference data, which is then used to train a soft Q-function via O-MAPL. We treat O-MAPL as label-refinement step by extracting reward signals from its Q-function to partition the unlabeled data into expert and non-expert trajectories.
Centralized Training with Decentralized Execution (CTDE). CTDE is well-known learning paradigm in MARL which global information are leveraged for an efficient training of joint value functions [5]
34 42 [53] while allowing agents to operate independently based on their local observations. There are various MARL algorithms exploring different value decomposition offers an advanced value decomposition method based on mixing and hyper-network concepts 19 Others such as QTRAN (50) and QPLEX (55),
strategies under CTDE. For example, QMIX [22
introduce new factorization methods which are more expressive than QMIX. Besides these value- based methods, there are policy gradient based MARL algorithms 33 588 which are extensions of the PPO algorithm 48) Building the success of CTDE in MARL, this paradigm has been extended to multi-agent IL [6 23 36), preference-bast baased RL
6
25 62) and offline MARL 38 40
49 56 57
Our multi-agent IL algorithm adopts the DICE approach to learn effective policies from mixed-quality data within the CTDE framework. Despite operating under the same CTDE framework, existing methods- including ours differ notably in learning objectives, constraints, value decomposition strategies, and theoretical guarantees, each tailored to their specific problem settings.
Preliminaries
Our work focuses on cooperative MARL, modeled as a multi-agent Partially Observable Markoy Decision Process (POMDP): M (S, A, P. r, Z, O, N) where n is the number of agents and (1,...,,) n) is the agent set. The environment has the global state space S, and the joint action
(1, == space A IIE A., where A, is the set of actions for agent i. each time step, each agent i selects an action a; E Ar, forming a joint action
(a1, a2,.... an) E A. The transition dynamics are governed by T(s' s. s. S x A x S > [0, 11, which is the probability of transitioning to global state s' from state s under action a. Finally, 7 [0, 1) is the discounting factor. We also define the occupancy measure of a joint policy T as pTTar (s, a) = (1 - 7) Er 7 P(s s. s. = a a which is the discounted visitation distribution over state-action pairs when following Tia-
In a partially observable setting, each agent receives a local observation o; E O, via an observation function (s) S - O. and the joint observation is (01, o2, On). In practice, the global state S is not directly accessible and is instead represented by the joint observation o = Zs). For notational simplicity, we use in our formulation, but it refers to o in implementation.
In cooperative MARL, all agents share a global reward function r(s, a): : A + IR. The objective is to learn joint policy Tia { T that maximizes the expected discounted return:
1==0 7'r(ss, at)
In offline MARL, learning is performed solely based on a fixed offline dataset D. collected from a behavior policy Meo = (#I..... }. In offline multi-agent IL. the reward function is not available. Instead, set of expert demonstrations is provided, and the goal is to recover both the global and local expert policies accordingly. A common objective is to minimize the divergence between the stationary distribution induced by the learned policy and that of the expert demonstrations:
π tot argmin DKL. (ptc Noc Phot)
(I)
3

---

where Piot denotes the global state-action visitation distribution under the learned policy Two and Pin denotes the corresponding distribution derived from expert demonstrations.
The DICE framework has been widely adopted to address training objectives involving KL. divergence between stationary distributions (e.g., Eq. (T). The core idea is to formulate II constrained optimization program over occupancy measures. In later section, we extend this approach by integrating DICE with value decomposition for efficient offline multi-agent IL.
4 Unlabeled Demonstrations: Multi-Step Labeling Procedure via LLMs
Addressing multi-agent IL from unlabeled demonstrations of mixed quality requires a robust pro- cedure to identify high-quality, expert-like behaviors within the dataset. We propose multi-step data labeling pipeline designed to progressively refine trajectory quality assessments and partition the unlabeled dataset unulabetea) into expert (D&) and non-expert segments (DMia). This process serves as the first crucial stage of our MisoDICE framework for subsequent efficient IL.
The initial step in this procedure involves generating preference data. A set of trajectory pairs (Np) is randomly sampled from Duntahated. An LLM (e.g., GPT-4o) is then prompted to provide preference labels for these pairs, based on high-level semantic features and game context, such as unit health, positions, and actions in SMAC environments. This yields an initial preference dataset P. While LLMs offer valuable insights, their judgments in complex multi-agent settings can be error-prone.
The next step preference seferenne--based reward recovery. The preference dataset P used to train an offline preference-basee based MARL algorithm, i.e., the leading O-MAPL [6]. O-MAPL learns a soft Q-function (and corresponding value funetion V) directly from these preferences, capturing more grounded notion of demonstration quality compared to raw LLM feedback. After training O-MAPL, an underlying reward function R(s, a) is recovered for all state-action pairs in Duntabeled- This recovery utilizes the relationship R R Q >V, adapted from MaxEnt RL principles.
Using the recovered rewards, we compute the total return G(o)) E(s.)e R(s, a) for each trajectory in D unnlabeled and rank them accordingly. The top-Kexpert trajectories are selected to form the expert dataset DE and the remaining trajectories form the suboptimal dataset DMix, These datasets are used in the subsequent multi-agent IL phase (Phase 2 of MisoDICE).
This multi-step labeling process enables MisoDICE to leverage unlabeled, mixed-quality demonstra- tions by inferring demonstration quality through preference learning and using it to guide imitation learning. The use of LLMs for initial preference generation, followed by refinement with O-MAPL, offers powerful mechanism for identifying expert behavior without prior annotations.
Multi-Agent Imitation Learning from Mixed-Quality Demonstrations
5.1 Distribution Matching Based Formulation
Our goal is to leverage two datasets (as a result of our Phase 1): (1) a limited expert dataset, DE and (2) large suboptimal dataset, DMix to learn global and local policies that closely imitate the expert behavior. We first formulate the DICE-based learning objective as a function of the global joint policy wer for multi-agent IL in similar fashion prior work (28) for single-agent IL:
min DRL. (Ptx ll ( + DKI. (AT ll (
E where Prot and Plr denote the global state-action stationary distributions induced by the expert dataset DE and the union dataset D D U MMx, respectively. The hyperparameter a 2 controls the influence of the suboptimal data in the learning objective.
We can reformulate
(2 as a constrained optimization problem over the occupancy measure Piu
Pinn (s,a) Pla (s.a) s.t. Σ, Pior a) -1 Po(s) + E T(s
Pius (s, a) + a Piot (s, a) log
Σ s.a Pur (s a) log
min (AT (sa)22)
(s.
(s Vs E S (4)
Constraint enforces the Bellman flow condition, ensuring that Por corresponds to a valid state- action visitation distribution. The objective in 33) is convex in p (s. a). Thus can incorporate the

---

flow constraints into L using Lagrangian duality, resulting in the following saddle-pon -ointt-ponntt formulation:
max min
{C(u
1 P(x)) }
where not are Lagrange multipliers corresponding the Bellman flow constraints, and the objective
Piod (s.a)
Piul (s) a)
C(v Proi) En, Piol (s, a) log
+ Piol (s, a) log
Pio (s.
Pinn (s,a) a)
(1- (0)
Es s' T(sls', a') ('ai's' E, Piol (s.
(5)
which can be simplified to function of (w wort C(v" w = 7-) 7) (s)] + ((sa))= [w'' (A (s. - (1 +1++0 at) log(w (s, a))]
follows:
Peot (5, a) + Σg T(s' alaa)) (s) and u a)
Pial a)
where A a) log (s,
Piol (s. a)
Details of this derivation, adapted to the multi-agen setting from the single-agent II. in 28 can be found in our appendix. Here, (s) can be interpreted as the value function at the global state s. The reformulation above enables to express the training objective
optimization problem over wou and :h in which the objective C(r w is linear in y and convex in wht,, Moreover, the inner problem max C(v"M, ) admits closed-fone solution, allowing the original minimax objective to be reduced to the following non-adversen adnersrrraaial minimization problem over lot;
A a) 1 +
-1)
18 Clv ) L(v't, w.) where w = exp
5.2 Value Factorization
The objective L(V'M, w[) is generally intractable in the multi-agent setting due to the high dimension- ality of the joint state and action space. To address this, we adopt a value factorization technique in which the global yor is decomposed into local value functions using mixing network. Specifically, let v(s) = {1r (s:) HeN be the set of local value functions. We introduce a mixing network Ma. parameterized by é, which aggregates the local values to form the global value function:
y ()) = Ma (s)).
To implement Ma. we adopt linear structure (e.g., single-layer network) due to: (i) it preserves convexity in the learning objective within the value function space, leading to stable and consistent training; and (ii) nonlinear mixing networks (e.g., two-layer network with ReLU activations), partic- ularly in offline settings with limited data, are prone to overfitting and have unstable performance (7]
Under the mixing architecture described above, the training objective becomes:
P, [Mo[r(s]]] + (s)))))) (w (A (s, a) (1 + a log w a))]
(6)
where the total advantage function A/ and the optimal weighting function wt are defined as:
Puil (s, a) Punl (s.
(s, a) log
+ Σ, T(s' ( M. -Molssss
A tnt (s,
-1).
w a) = exp
(8)
1
Proposition 5.1. If the mixing network Ma [s)) is a linear function of both v(s) and c. then the training objective L( is convex in both á and
linear combination of local value functions, i.e., Ma. [s() = ΣIEN (s:) + Φo. where ø = (do. $1...., Φn are learnable parameters, then the global value function is linear in both the parameters di and the local value outputs (x())) This structural property critical for stable and efficient optimization, as guarantees that the learning objective preserves convexity w.r.t both the policy parameters and the mixing network parameters.
As result, if the mixing network is constructed
Conversely, this objective convexity does not holds if we employ multi-layer neural network structures for the mixing network Ma p)] leading to unstable training dynamics and convergence to poor local minima, especially in offline learning settings where data is limited and overfitting is concern.
Proposition 5.2. If the mixing network Ma v) a multi-layer feedforward network with as input (even with convex activation functions), the objective L( v) becomes non-convex in
5

---

5.3 Occupancy Ratio Estimation
The training objective described in Section 5.2
involves the computation of the total advantage function (s, a), which in turn requires the occupancy ratio term log( (s, Pl)/Puuf5 (s, a)). However, this ratio is not directly observable from data. We now describe practical method to estimate this ratio under the CTDE framework. Following extensions from the single-agent setting, the occupancy ratio can be estimated by solving the following discriminator-basee elassification problem:
[c) A [lg c(s, a)] p [log(1 c(s,a))]
(0X.)
This objective is concave in and admits a unique optimal solution: c² a)
Pue from
Pir (
which we can recover the occupancy ratio via:
(a(aa
c'(s,al
(s.)
In the multi-agent context, directly optimizing c(s, a) is impractical due to the exponential joint state-action space. We thus leverage the CTDE framework to approximate c(s, a) in a decentralized manner. We define local discriminators c(s, a) {ca(s; (G))))))))) HEN. where each 6; corresponding to agent i. A mixing network M,, parameterized by y, is then used to aggregate these local outputs: cls, a) MM,(((s a)]. The resulting training objective for estimating the occupancy ratio becomes:
(c, Ep [log M,,c(s, a)]. Egg [log (1 M,lccs, a))[c
It can be shown that this objective is concave in if the mixing network M,, linear in its inputs, which contributes to a stable training process. We state this formally below:
Proposition 5.3. If the mixing network Mnlc(s, a)] is linear in c(s,a). then the training objective [c, n) for occupancy ratio estimation is concave in both and n,.
5.4 Policy Extraction
We now discuss how to extract the global and local policies from the output of the training objective in Section 5.2 Once the value function v² has been optimized, we can compute the corresponding optimal occupancy ratio wge, which reflects the ratio between the visitation distributions of the optimal policy x and the mixture distribution Pyr.
Abt (s, a)
-1)
Poo (s, a)
w (s. a) = eex
1 +
Pion (s,
The global optimal policy can be then recovered by solving weighted BC objective:
max Σ (Sa)~
whth (s. a) log Tior (s, a),
(10)
where denotes the feasible set of joint policies. In the multi-agent setting, especially under the CTDE framework, it is more practical and desirable to recover decentralized local policies { kee. Therefore, we adopt a local weighted behavior cloning approach to recover each agent's policy:
maxx Eisra wit (s. log T. (a; ss),
(11)
where all agents share the same global weight wht a). This enables each local policy to be optimized using globally-ifformed signals, ensuring alignment with the credit assignment the multi- agent system. Furthermore, as shown in Proposition 5.4] the optimization of local policies via this decentralized weighted BC is consistent with the global weighted BC formulation in Equation (TO).
Proposition 5.4 (Global-Local Consistency). Assume the global policy space IT is factorizable. ie.. = y ([[Thhee such that T awr(s a) = IIEN Ta si)}. Let {.; lie and x be the solutions to the local and global weighted BCs, respectively. Then for any (s, a), it holds that:
π Tor (s, a)
II कं (a; st).
The local weighted BC objective also reveals an interesting connection between the optimal local policies and the local value functions obtained from optimizing the objective Section 5.2 Recall that the total ad advantage function can be written as: Atol (s, a) r(s. a) yLE for (s') for (s).

---

Algorithm 1: -MuDiiCE-Mut-iAggeeeee Imitation Policy Learning
Initialize local networks 02.00.,7), and mixing networks (1)
for a number of discriminator training steps do
Train (0e: 11) from Eq with c(s, a) is computed in Eq. 13
Compute occupancy ratio: Sa(sa) = I-c(s.a)
<(s.a) based on updated (fc, r):
P.n(s.))
for a number of value function training steps do
L Train (0z, ②) from Eq. 66 Gusing the occupancy ratio estimated above;
7 Compute the weight wy- (s. a) using Eq. 8 w.r. updated (0g. <);
for a number of policy training steps do
L Update 0x based on weighted BC in Eq.
11 and the weight wht (s, a);
return local policies n, for I.....
where (lo)((s) (M)((((()) and the term r(s, a) log (s.a) (s. can be interpreted as global reward
function. We can now define the global Q-function as follows: qlt a) rss, a) + 7 ( (s)).
Next, let's assume that the optimal linear mixing network takes the form: Ma- (s)
EieN 0, (s;) + of. where 6* and v^ are the solutions obtained from optimizing the training objec- tive in Eq. 6 and the reward function admits linear decomposition: r(s, a) = EiEN r, a.)
oū noting that such decompositions are often feasible or can be approximated
practice. Substituting
the decomposition into the ad advantage function yields; A (s, a) = EieN of (g (st, a.) 10 (s:)
T where the local Q-function is defined as: (8; az) = (s ai) + -BBggls,,, (s](] This reformulation allows the occupancy ratio be written in decomposed, local form:
1 w (s. a) = = exxp exp ( a (Haa(
of (s, a.) v (s:)) + 70)))
(12)
This expression enables derive the following result, which shows how the optimal local policy can be explicitly expressed as a function of the local Q-function and value function: Proposition 5.5 (Local Policy as Softmax over Local Functions). Assume that the joint behavior
policy of the union dataset jd (als) is decomposable into local behavior policies p (as | 8i). Let xx be the optimal solution to the local weighted BC objective. Then the optimal local policy can be expressed as:
1
of exp (1 de (8, az) log Noggy(oo (az si)).
(a 84)
A(++)
where A(s:) = Em.es exp ( 1+a di (s,, a:) + log p (a; si)). is the normalization constant, ensuring that Tī (a: si) is valid probability distribution over the local action space.
6 Practical Algorithm
In implementation, we construct three types of local networks for every agent E N: (i) local discriminator networks (c,(s,, ag; 0.)} (parameterized by 0.); (ii) local value networks (s; 0,)] (parameterized by 0z)); and (iii) local policy networks [T.(s;; 0z)] (parameterized by 0z). addition, there are two different mixing networks: M,, (with parameters and M, (with parameters which are used to aggregate local discriminators and value functions, i.e.,
c(s, a) M,[(c a4; He)
J (s)
(13)
Note that we use the same local network architectures with shared parameters (0.: 0g,0z) 0z) for all agents for efficient training. The overview of our algorithm MisoDICH is illustrated in Algorithm 1
7 Experiments
We run various experiments on challenging MARL. henchmarks, including the StarCraft Multi-Agent Challenge version I (SMACv1) [46), and its successor, SMACv2 [3]. Both offer discrete action
7

---

Table 1: Final average returns for MisoDICE and baseline methods on SMACv2 tasks.
BC
MisoDICE
OMAPL
INDD
MARL-SL
VDN
(3 = 0.0)
( 0.5)
( 1.0)
(ours)
2c_ ___v__64gg
0.1
9.7 0.3
0.3
12.2
14.6 1.0 6.7
14.0 1.6
0.6
16.4
5m_ 6m
5.0 1.1
6.7 0.0
6.1 0.1
5.2
6.1
6.2 1.4
7.3 = 0.1
6h_ vs. 8z corridor
0.0
0.0
7.1
6.2
7.5
7.1
8.2
8.7 = 0.2
0.1
0.2
0.7
2.2
4.2
0.2
4.7 0.6
5.8 = 0.8
5_vs__
0.1
1..7
1..2
9.6
10.9
11,
1.2
12.4 # 0.5
Protoss
10_vs, vs_ys_ 10
10.3 0.6
01.
10.6
1..1
1..0
11.9
12.4
12.9
10_ [0_v__11
0.4
9.6
8.5
9.4
10.4
20 vs_ 20
0.2
0..5
10.5
9.4
1.44
13,1
12.1
13.5 0.5
20_ys_23
8.1 0.2
0.3
8.3
7.9
9.3
9.3
10.3
10.6 0.2
$_vs_5
0.8
0.5
7.6
0.6
0.9
8.1
8.3
9.1 + 0.3
Terran
10_vs_ 10
0.3
7.4
6.6
6,9
7.6
7.2
8.0
9.1 1.3
10_vs_11
0.2
5.3
5.2
4.2
5.7
0.7
6.0
6.4 0.5
20 vs
0.4
0.9
6.7
6.5
8.2
20_vs_25
0.3
0.4
0.3
4.4
5.1
0.6
5.6
5.6 + 0.4
5_vs_5
0.5
6.6
0.3
6.1
6.2
0.5
7.1
7.5
Zerg
10_vs_10
0.1
0.6
0.7
0.6
8.2
9.0
9.7
10.2 0.6
10_vs_
0.2
0.4
0.3
0.2
8.0
8.8
9.1
9.4 0.3
20 vs. 20
0.6
0.5
7.7
6.9
8.3
8.6
9.0
10.2 # 0.6
20_vs_23
0.1
7.9
7.2
7.1
8.8
0.5
9.5 0.2
BC (B=0.0)
BC (B=0.5)
BC (B=1.0)
INDD
VON
MARL-SL
MisoDICE (ours)
14-
14
12
14
Protoss
12
10
10-
10-
10
6
10
Terran
12-
10
10
10-
Zerg
6-
4-
50
50
5 $_v8_5 5
10
10
10_vs_10 11
20 _
20
20_ 23
Figure 1: Learning curves of the average return for MisoDICE and baseline methods on SMACv2.
spaces and focus on decentralized micromanagemen scenarios derived from StarCraft II 13 46 In these environments, each combat unit is controlled by an individual agent, and the team must learn cooperative strategies to defeat built-in AI opponents. SMACv2 introduces increased stochasticity and diversity by incorporating features like randomized start positions and unit types. In the appendix, also include experiments MAMuJoCo (II.. another standard MARL benchmark. We note that LLMs struggle to provide meaningful preference feedback in MAMuJoCo due lack of contextual understanding, In this setting, instead assume access rule- mased oracle capable of providing preference feedback.
Dataset We use the offline data generated from 7 566, providing trajectories categorized by quality (expert, medium, and poor) across various tasks in SMACvI and SMACv2. For our experiments focused on learning from unlabeled, mixed-quality demonstrations, we curated a suboptimal dataset for each task by sampling 200 expert and 1000 poor trajectories from the datasets in 7 [56). These were combined and shuffled to form unlabeled dataset, D This construction reflects realistic scenarios where demonstration quality is heterogeneous and unknown priori. The expert dataset DE, used in Phase of MisODICE, is derived from T based on labeling procedure in Section

---

14-
10
10
12-
10
50
200
400
800
1200
50
200
400
800
1200
200
400
800
1200
Protoss
Terran
Zerg
Figure 2: Box plots of final returns on SMACv2 by varying the number of top-k expert trajectories.
Baselines We compare MisoDICE performance against several key baselines. Multi-Agent BC (MA-BC) serves as a foundational baseline, adapted for mixed data via a coefficient 3 that balances learning between the expert dataset (DE) and the union dataset (DC). ). We evaluate MA-BC under different 3 values: 3 = 1.0 (expert-only). 3 = 0.5 (equal weighting), and = 0.0 (union-only). The end-to-end preference-based OMAPL trains policy directly from preference data generated by LLMs. Individual DemoDICE (INDD) applies the single-agent DemoDICE algorithm independently to each agent, using individual marginal distributions from D and DV, MARL with Supervised Reward (MARL-SL) follows a two-stage setup: a reward is learned from preference data via supervised learning, followed by MARL-based policy optimization. Finally, MisoDICE VDN is ablation using a standard Value Decomposition Network mixer (sum of local values) in place of MisoDICE's learnable mixing, isolating the contribution of the advanced value mixing architecture.
Results Table provides a detailed quantitative comparison of the final average returns achieved by MisoDICE and the baselines across various SMACv2 maps. MisoDICE consistently achieves the highest mean return in nearly every scenario tested, significantly outperforming all baseline methods. The INDD baseline consistently underperforms compared MisoDICE, highlighting the deficiency of independent learning and the necessity of coordinated value decomposition in multi-agen settings. While MARL-SL and MisoDICE-VE VDN represent stronger baselines by incorporating MARL opti- mization and value decomposition, their performance is still generally lower than MisoDICE. This suggests that MisoDICE's formulation for handling mixed-quality data, combined with its specific mixing architecture, provides a distinct advantage over learning reward via supervised methods first or using simpler additive decomposition like VDN. Furthermore, OMAPL's performance, derived directly from Phase preferences, while sometimes competitive, does not match MisoDICE's final results, emphasizing the benefit of the subsequent IL phase (Phase 2) employed by MisoDICE.
Figure plots the average return curve during training for MisoDICE and the baselines on the same set of SMACv2 tasks. These plots further reinforce the findings from Table The learning curve for MisoDICE (shown in red) consistently rises above those of the other methods, demonstrating not only higher final performance but often faster and more stable learning across different game tasks,
Ablation Studies Figure 2 presents box plots of the final returns achieved on SMACv2 maps when varying the number of top trajectories (k) selected as the expert dataset (DF). The x-axis is the value of k. ranging from 50 to 1200, while the y-axis shows the distribution of returns achieved across multiple runs. For all three races depicted (Protoss, Terran, and Zerg), the plots generally show that performance tends to peak when a moderate number of top trajectories are selected as experts (around à = 200). Using too few (k = 50) or too many (k = 800, 1200) expert trajectories seems to result in lower median returns and potentially higher variance, although performance remains relatively robust across differentt & values. This suggests that our data-labeling procedure successfully isolates high-quality trajectories, but the optimal amount of expert data required for the IL phase is neither the absolute minimum nor the maximum available, highlighting a trade-off in selecting the size of F In addition, conduct several ablation studies which include investigating the effect of the hyperparameter a controlling the influence of suboptimal data, and the performance difference when using L.MMs of varying capabilities (GPT-40 vs. GPT-4o-mini) and when employing simple rule-based method for initial preference generation. For details, we refer to our appendix.
8 Conclusion
We address offline multi-agent II. with unlabeled, mixed-quality demonstrations. Our proposed two-stage framework, comprising trajectory labeling pipeline and the MisoDICE algorithm, enables

---

robust learning from such restricted data. By leveraging LLMs and preference- based learning for effective labeling, and introducing a scalable, value-decomposed extension of DICE for multi-agent settings, our approach achieved both computational efficiency and global-local policy consistency. Empirical results on multi-agent benchmarks show that MisoDICE outperforms all baselines.
Limitations and Future Work. Our work is limited to cooperative settings, Extending this approach competitive environments remains an open challenge for future investigation. Additionally, the reliance on LLMs for trajectory labeling has limitations in domains like MaMuJoCo, where abstract task semantics are difficult for LLMs interpret accurately. Addressing these challenges opens up exciting directions for future research in scalable, generalizable imitation learning.
Acknowledgments and Disclosure of Funding
This work supported by the Lee Kong Chian Fellowship awarded to Tien Mai.
10

---

References
[1] G. J. X. Zuo, N. Kosaka, K.-M. Kim, and H. O. Song. Direct preference- bbseed policy optimization without reward modeling. Advances in Neural Information Processing Systems, 36:70247-70266, 2023.
[2] N. Baram, O. Anschel, and S. Mannor. Model-based adversarial imitation learning. arXiv preprint arXiv:1612.02177 2016
[3] S. Barrett, A. Rosenfeld, S. Kraus, and P. Stone. Making friends on the fly: Cooperating with new teammates. Artificial Intelligence, 242:132-171, 2017.
[4] M. Beliaev and R. Pedarsani. Inverse reinforcement learning by estimating expertise of demon- strators. Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 15532-15540, 2025.
[5] K. Bogert and P. Doshi. Multi-robot inverse reinforcement learning under occlusion with interactions. In Proceedings of the 2014 international conference on Autonomous agents and multi-agent systems, pages 173-180, 2014.
[6] V. Bui, T. Mai, and H. T. Nguyen. O-mapl: Offline multi-agent preference learning. arXiv preprint arXiv:2501. 18944, 2025.
[7] V. Bui, H. Nguyen, and T. Mai. Comadice: Offline cooperative multi-agent reinforcement learning with stationary distribution shift regularization. arXiv preprint arXiv:2410.01954, 2024.
[8] X. Cao, F.M.. Luo, J. Ye, T. Xu, 7. Zhang, and Y. Yu. Limited preference aided imitation learning from imperfect demonstrations. In Forty-first International Conference on Machine Learning, 2024.
[9] H. Choi, S. Jung, H. Ahn, and T. Moon. Listwise reward estimation for offline preference-basee reinforcement learning. arXiv preprint arXiv:2408. 0400 2024.
[10] Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
[11] C. S. de Witt, B. Peng, P.-A. Kamienny, Torr, W. Böhmer, and S. Whiteson. Deep multi- agent reinforcement learning for decentralized continuous cooperative control. arXiv preprint arXiv:2003.06709, 19, 2020.
[12] B. Ellis, J. Cook, S. Moalla, M. Samvelyan, M. Sun, A. Mahajan, J. Foerster, and S. Whiteson. Smacv2: An improved benchmark for cooperative multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 36:37567-37593, 2023.
[13] B. Ellis, S. Moalla, M. Samvelyan, M. Sun, Mahajan, J. Foerster, and S. Whiteson. Smacv2: An improved benchmark for cooperative multi-agent reinforcement learning. arXiv preprint arXiv:22
2022.
[14] C. Finn, P. Christiano, P. Abbeel, and S. Levine. A connection between generative adver- sarial networks, inverse reinforcement learning, and energy-based models. arXiv preprint arXiv:1611.03852 2016.
[15] J. Foerster, G. Farquhar, T. Afouras, N. Nardelli, and S. Whiteson. Counterfactual multi-agent policy gradients. Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[16] J. Fu, K. Luo, and S. Levine. Learning robust rewards with adversarial inverse reinforcement learning. arXiv preprint arXiv:/710.1 148,, 2017.
[17] C.-X. Gao, S. Fang, C. Xiao, Y. Yu, and 7.. Zhang. Hindsight preference learning for offline preference-based reinforcement learning. arXiv preprint arXiv:2407.044 04451, 2024.
11

---

[18] D. Garg, S. Chakraborty, C. Cundy, J. Song, and S. Ermon. Iq-leam: Inverse soft-q learning for
imitation. Advances Neural Information Processing Systems, 34:4028-4035 2021.
[19] D. Ha, A. M. Dai, and Q. V. Le. Hypernetwork.. International Conference on Learning Representations, 2017.
[20] Hejna, R. Rafailov, H. Sikchi, C. Finn, S. Niekum, W. B. Knox, and D. Sadigh. Contrastive prefence learning: Learning from human feedback without rl. arXiv preprint arXiv:2310.13639, 2023.
[21] Hejna and Sadigh. Inverse preference learning: Preference-based rl without reward function. Advances in Neural Information Processing Systems, 36, 2024.
[22] D. J. Hejna III and D. Sadigh. Few-shot preference learning for human-in-the-loop rl. In Conference on Robot Learning, pages 2014-2025. PMLR, 2023.
[23] J. Ho and S. Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
[24] B. Ibarz, J. Leike, T. Pohlen, G. Irving, S. Legg, and D. Amodei. Reward learning from human preferences and demonstrations in atari. Advances in neural information processing systems, 31, 2018.
[25] S. Kang, Y. Lee, and S.-Y. Yun. Dpm: Dual preferences-based multi-agent reinforcement learning. In ICML 2024 Workshop on Models of Human Feedback for Al Alignment, 2024.
[26] Y. Kang, D. Shi, J. Liu, L. He, and D. Wang. Beyond reward: Offline preference-guide policy optimization. arXiv preprint arXiv:2305.16211 2023.
[27] C. Kim, J. Park, J. Shin, H. Lee, P. Abbeel, and K. Lee. Preference transformer: Modeling human preferences using transformers for rl. arXiv preprint arXiv:2303.00957. 2023.
[28] G.-H. Kim, S. Seo, J. Lee, W. Jeon, H. Hwang, H. Yang, and K.-E. Kim. Demodice: Offline imitation learning with supplementary imperfect demonstrations. In International Conference on Learming Representations, 2021.
[29] I. Kostrikov, O. Nachum, and J. Tompson. Imitation learning via off off-policy distribution matching. In International Conference on Learning Representattons, 2019.
[30] C.-M. Lai, H.-C. Wang, P.-C. Hsieh, F. Wang, M.-H. Chen, and S.-H. Sun. Diffusion-reward adversarial imitation learning. Advances in Neural Information Processing Systems, 37:95456- 95487, 2024.
[31] H. M. Le, Y. Yue, P. Carr, and P. Lucey. Coordinated multi-agent imitation learning. In International Conference on Machine Learning, pages 1995-2003. PMLR, 2017.
[32] K. Lee, L. Smith, and P. Abbeel. Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. arXiv preprint arXiv:210..0509), 2021.
[33] X. Lin, P. A. Beling, and R. Cogill. Multi-agent inverse reinforcement learning for zero-sum games. arXiv preprint X13.:/403.633 2014.
[34] R. Lowe, I. Wu, A. Tamar, J. Harb, O. Pieter Abbeel, and Mordatch. Multi-agent actor-critic for mixed cooperative-compettive environments. Advances in neural information processing systems, 30, 2017.
[35] T. Mai, T. Nguyen, et al. Mimicking to dominate: Imitation learning strategies for success in multiagent games. Advances in Neural Information Processing Systems, 37:84669-84691 2024.
[36] T. A. Mai, T. H. Nguyen, et al. Inverse factorized soft q-learning for cooperative multi-agent imitation learning. The Tiihth- Annual Conference on Neural Information Processing Systems, 2024.
12

---

[37] S. Mukherjee, A. Lalitha, K. Kalantari, A. Deshmukh, G. Liu, Y. Ma, and B. Kveton. Optimal design for human preference elicitation. 2024.
[38] T. H. Nguyen, T. A. Mai, et al. Comadice: Offline cooperative multi-agent reinforcement learning with stationary distribution shift regularization. In The Thirteenth International Conference on Learning Representations, 2024.
[39] OpenAl. Gpt-4o. 2024, httpa////pppaaa..ccnnneeeee////e www/iidwe/heelll...ooooooooo gotellooppppo/////oo
[40] L. Pan, Huang, T. Ma, and Xu. Plan better amid conservatism: Offline multi-agent reinforcement learning with actor rectification. In International conference on machine learning. pages 17221-17237. PMLR, 2022.
[41] D. A. Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural computation, 3(1):88-97, 1991.
[42] T. Rashid, M. Samvelyan, C. S. De Witt, G. Farquhar, J. Foerster, and S. Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement learning, The Journal of Machine Learning Research, 21(1):7234-7284, 2020.
[43] S. Reddy, A. D. Dragan, and S. Levine. Sqil: Imitation learning via reinforcement learning with sparse rewards. arXiv preprint arXiv: 100, 2019.
[44] T. S. Reddy, V. Gopikrishna, G. Zaruba, and M. Huber. Inverse reinforcement learning for decentralized non-cooperative multiagent systems. In 2012 IEEE International Conference Systems, Man, and Cybernetics (SMC), pages 1930-1935. IEEE, 2012.
[45] S. Ross and D. Bagnell, Efficient reductions for imitation learning. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 661-668.
JMLR Workshop and Conference Proceedings, 2010.
[46] M. Samvelyan, T. Rashid, C. S. De Witt, G. Farquhar, N. Nardelli, T. G. Rudner, C.-M. Hung, P. H. Torr, J. Foerster, and S. Whiteson. The starcraft multi-agent challenge. arXiv preprint arXiv: 1902.0042.. 2019.
[47] A. Schlaginhaufen and M. Kamgarpour. Identifiability and generalizability in constrained inverse reinforcement learning. In International Conference on Machine Learning, pages 30224-30251. PMLR, 2023.
[48] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06344 2017.
[49] J. Shao, Y. Qu, C. Chen, H. Zhang, and X. Ji. Counterfactual conservative q learning for offline multi-agent reinforcement learning. Advances Neural Information Processing Systems, 36, 2024.
[50] K. Son, D. Kim, W. Kang, D. E. Hostallero, and Y. Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International conference on machine learning. pages 5887-5896. PMLR, 2019.
[51] J. Song, H. Ren, D. Sadigh, and S. Ermon. Multi-agent generative adversarial imitation learning. Advances in neural information processing systems, 31, 2018.
[52] A. Šošié, W. R. KhudaBukhsh, A. M. Zoubir, and H. Koeppl. Inverse reinforcement learning in swarm systems. arXiv preprint arxiiv:6002 054500 2016.
[53] P. Sunehag, G. Lever, A. Gruslys, W. M. Czarnecki, V. Zambaldi, M. Jaderberg, M. Lanctot, N. Sonnerat. J. Z. Leibo, K. Tuyls, et al. Value-decooppsittoo networks for cooperative multi-agent learning. arXiv preprint arXiv: (tll::7706059990 2017.
[54] B. Wang, G. Wu, T. Pang. Y. Zhang, and Y. Yin. Diffail: Diffusion adversarial imitation learning. Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 15447-15455, 2024.
13

---

[55] J. Wang, Z. Ren, T. Liu, Y. Yu, and C. Zhang. Qplex: Duplex dueling multi- agget q-learning. arXiv preprint arXiv:2008. 010662 2020.
[56] X. Wang, H. Xu, Y. Zheng, and X. Zhan. Offline multi-agent reinforcement learning with implicit global-to-local value regularization. Advances in Neural Information Processing Systems, 36, 2022.
[57] Y. Yang, X. Ma, C. Li, Z. Zheng, Q. Zhang, G. Huang. Yang. and Q. Zhao. Believe what you see: Implicit constraint approach for offline multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 34:10299-10312, 2021.
[58] C. Yu, A. Velu, E. Vinitsky, J. Gao, Y. Wang, A. Bayen, and Y. Wu. The surprising effectiveness of ppo in cooperative multi-agent games. Advances in Neural Information Processing Systems, 35:24611-2466
2022.
[59] I.. Yu, J. Song, and S. Ermon. Multi-agent adversarial inverse reinforcement learning. In International Conference on Machine Learning, pages 7194-7201. PMLR, 2019.
[60] M. Zare, P. M. Kebria, A. Khosravi, and S. Nahavandi. A survey imitation learning: Algorithms, recent developments, and challenges. IEEE Transactions Cybernetics, 2024.
[61] S. Zeng, Li, A. Garcia, and M. Hong. When demonstrations generative world models:
A maximum likelihood framework for offline inverse reinforcement learning. Advances in Neural Information Processing Systems, 36:65531-65565 2023.
[62] N. Zhang, X. Wang, Q. Cui, R. Zhou, S. M. Kakade, and S. S. Du. Multi-agent reinforcement learning from human feedback: Data coverage and algorithmic techniques. arXiv preprint arXiv:2409.0m 007107 2024.
[63] W. Zhang, Xu, H. Niu, P. Cheng, M. Li, H. Zhang, G. Zhou, and X. Zhan. Discriminator- guided model-based offline imitation learning, In Conference robot learning, pages 1266- 1276. PMLR, 2023.
[64] Z. Zhang, Y. Sun, J. Ye, T.-S. Liu, J. Zhang, and Yu. Flow to better: Offline preference- based reinforcement learning via preferred trajectory generation. In The Twelfth International Conference on Learning Representations, 2023.
[65] H. Zhao, X. Yu, D. M. Bossens, I. Tsang, and Q Gu. Beyond-expert performance with limited demonstrations: Efficient imitation learning with double exploration. In The Thirteenth International Conference on Learning Representations, 2025.
14

---

APPENDIX
Contents
Missing Proofs
16
A.1 Proof of Proposition 5.1
16
A.2 Proof of Proposition 5.2
16
A.3 Proof of Proposition 5.3
17
A.4 Proof of Proposition 5.4
17
A.5 Proof of Proposition 5.5
18
Additional Details
19
B.1
Unlabeled Imperfect Dataset
19
B.2
Rule-based vs. Lased Trajectory Ranking Techniques
19
3
Baselines
21
B.4
Hyperparameters
23
B.5
Algorithm
24
B.6
Experiment Setup
24
Evaluation
25
Additional Experimental Results
25
Ablation Study I: Impact of the Number of Identified Expert Trajectories (Top-k)
25
Ablation Study 2:
Impact of the Suboptimal Data Influence Hyperparameter
28
Ablation Study 3: Comparison of LLM Capabilities for Preference Labeling
29
Ablation Study 4:
Performance with Based Preference Labeling
C66
Ablation Study 5: Performance with Ground Truth Data Labels
32
Ablation Study 6: Sensitivity to Noisy Preference Labels
34
Ablation Study 7: Alignment Phase
Labeling with Ground Truth
35
Ablation Study 8:
Importance of Phase Trajectory Labeling
35
15

---

A Missing Proofs
A.1 Proof of Proposition 5.1
Proposition. If the mixing network MM [p()) is linear function of both (ss) and Ø, then the training objective L( v) convex both ø and
Proof. We write the training objective as:
L( = (1 - 7) Pa [s)] + (saa)~ol [w](s, a) (s, a) - -(1 + a.) log (s, a))]
where the total advantage function A5 and the optimal weighting function ut are defined as:
Piot (s, a) Piul (s. a)
(s, a) log
+ Eg T(s' s.a))Mll(ss)) -Molllsss))MMM
(14)
w5(ss a) -xx
A (s,a)
1 1+a a -1)
(15)
We can simplify the objective as:
Ab a) 1 a
|)]
L( (1 - Po [M] (1 + (sa) exp
We now observe that if Ma is linear in both vss) and Φ, then each term in A a) is also linear in (s)) and 6, since:
The log-ratio term log Pur
is constant with respect to both and v,
Pie
The remaining terms composed of expectations of linear functions (due to the linearity of M& (-) and thus remain linear.
As result, (s, a) is linear, and the exponential of linear function is convex. Since the outer expectation is linear operator, and the sum of convex and linear functions remains convex, entire objective [(Ø), vi) is convex in both e and v(s). This concludes the convexity analysis.
A.2 Proof of Proposition 5.2
Proposition. If the mixing network Mo(p] is multi-layer feedforward network with v as input (even with convex activation functions), the objective L(ø, v) becomes non-convex v.
Proof. Now assume that Mo(P(s) is constructed using a multi-layer feedforward neural network with convex activation functions (e.g., ReLU, softplus). This implies that Mo[v(s)] convex function of both à and (s)) but it is not linear. We aim to show that, under this assumption, the objective L( v) no longer convex in the joint parameters (ő,v).
Recall that the simplified objective can be written as:
Ala a) 1
-1) 1
[( = (1 - (- Ex-r% [Mo[u(s)] + +1 + a)) (sa)-pll
exp
where A a) is defined as:
A (s, a) log Punl (s.a) plan a)
+
Σ T(s' Msa)
The log-ratio term is constant with respect to (d) v), the dependence of L these variables arises entirely through Mg(M(-)) Now observe:
By assumption, the mapping (0) + Mc(p(s) is convex.
16

---

Consequently, the term 7 Es T(s') s.a) (s))) is convex in (0,v), as it is a non- negative linear combination of convex functions.
However, the term (M)( is concave in (ø, v). and hence (s, a) is generally the difference of convex functions, which necessarily convex.
Since the exponential function convex and monotonically increasing, applying to non-convex function (like AV" /11 a) 1) does not preserve convexity in general.
Hence, we conclude that the objective [(d, v) is not convex in general when MM)[((s) is convex but not linear.
A.3 Proof of Proposition 5.3
Proposition. the mixing network Mn,c(s, a)] is linear in c(s, a), then the training objective J(c, n) for occupancy ratio estimation concave in both and vy.
Proof. We write the objective
[e, n) = Epg [log. Mo,gM,, a)]l + E,g [log (1 - M,)[(s,)))))
where My [-] is linear in ry and c(s, a). Now observe:
The composition log M,|c(ss a)] is concave function in MM,[c(s, a)). and hence concave in both y) and c.
Similarly, log(1
is also concave in 1 MM, [c(s, a)), and thus in ry and c.
Since expectations preserve concavity, both terms in J[c, 17) are concave in (c,
Here we note that if M,, (c(s, a)] is convex (but non-linear) function of r) and/or c, such as when using a neural network with convex activations (e.g.. ReLU, softplus). In this case:
The function log M,fc(s, a)] the composition of a concave function (log) with a convex function. This composition is in general not concave.
Likewise, log(1 Ms,c(ss a))) is the composition of concave function with concave function, which again does not preserve concavity.
Therefore, J(c, 17) is no longer guaranteed to be concave in (c, when M,L is convex but not linear.
A.4 Proof of Proposition 5.4
Proposition. Assume the global policy space II is factorizable, i.e., II [Twi 3imThe such that (s, a) = I (a st)} Let { hen and The be the solu- tions to the local and global weighted BCs, respectively. Then for any (s, a), holds that:
T (s. a)
II
क, (a; ss).
ieN
Proof. For notational simplicity, let (TTT)) be the objective function of the global WBC problem:
+ (Tot)
Eg [wit (s, a) log Teoe (a)s)
17

---

Since we are seeking a decomposable policy Too ITIN Tr(aa(ss), we have, for any tot E II such that tox
11; T.
4 T rot = 1B
tos (s, log ms(as18s))
18 aas (s, (og
iEN
(a)
E Ba [wrt (s. a) log मं (a.[s:)]
E a. ( (s, a) log mm
where Tio = II, mT, and (a) holds because each of is optimal for the corresponding local WBC problem. Thus, we have (TTm)) S ((maa) ) for any The E II, implying that Torr also optimal for the global WBC. This establishes the jlobal-local-consistenyy
desired.
A.5 Proof of Proposition 5.5
Proposition. Assume that the joint behavior policy of the union dataset 1 (a|s) decomposable into local behavior policies u (a= ss). Let тH be the optimal solution to the local weighted BC objective. Then the optimal local policy can be expressed as:
(a: | s:) A(s:) exp ( of 1+ (s as) + log 1 (a: st)).
1
where (ss;) = Eaies, exp (1tt (3,, a;) + log dog (a; s;)), is the normalization constant, ensuring that nī (as 8:) is valid probability distribution over the local action space.
Proof. We first recall that the total weighting function can be written
u (s, a) = =e- exp i
of (g) (s,, a,) vz (s:)) (s.)) 708)
This allows to express the local weighted behavior eloning (BC) objective for each agent as: max Te
[ [ (s, a) log (. st]]]
For each state s, the corresponding sub-training objective is:
Σ u (a s) [wbt (s, a) log (a: st)]
a
EI (as | si)
1 1+ EIEN of (si, v1 + -06) log m:(a; s;)
exp
1 = Σ exp (1-a (Eie of ai) + a1) log p (as si) + const) log (: ss))
where the constant term (including (s;) and Φū) does not depend on a,, and thus can be ignored for optimizing ,,.
Since only terms involving a; affect the optimality of the local policy <;(a; ss), we isolate the relevant part and simplify the training objective for each agent i as:
Σ cxp [(or
( (+aa
(oi (st, a;) + (1 + a:) log uf (as (s))) log (a, (ss))
This is a weighted log-likelihood objective, where the weights are shaped by the exponentiated advantage-like term. The maximization of this expression yields the following closed-form solution:
p s:) exp
di
(s., as)
I+a
1
p (as 81) exp ( oi
25 (si, ai) ai)).
π (a; 81) Ex; ui (ak exp
exp
0z (81, 14a 14a
A
(Taa a
18

---

where
A = A(8)== =.. exp ( of I+a ai (8,, 0z) + log M (a: s;)),
as desired.
B Additional Details
B.1 Unlabeled Imperfect Dataset
A significant challenge in advancing offline multi-agent imitation learning (MAIL) from mixed- quality data the lack of standardized benchmarks. Currently, there are no publicly available datasets that specifically provide suboptimal or imperfect demonstrations tailored for multi-agent reinforcement learning (MARL) settings. To address this gap and facilitate research in this crucial area, constructed our own datasets.
Our approach leverages the offline datasets generated by O-MAPL (6. The O-MAPL framework contributes a valuable resource by providing distinct datasets categorized by quality expert, medium, and poor for a range of MARL. tasks. Specifically, for each task within the MaMujoco, SMACv1, and SMACv2 environments, O-MAPI. offers 1000 trajectories for each quality level (expert, medium, and poor).
For the experiments, we specifically curate a suboptimal dataset designed to test the algorithm's ability learn from mixed-quality demonstrations where the quality is not explicitly labeled. To form this dataset, we sample 200 expert trajectories and 1000 poor trajectories from the O-MAPL datasets for each considered task. These selected trajectories are then combined and shuffled thoroughly to create single, unlabeled suboptimal dataset. This process ensures that we operate on dataset that realistically mirrors scenarios where trajectory quality is heterogeneous and unknown beforehand, compelling the agent to discern and learn from the more effective behaviors embedded within the mixed data.
We further detail the general framework of using an expert dataset (DE) and mixed-quality dataset (DMir), with constructed dataset serving as DD in setting where expert demonstrations within this mix are not pre-annotated. The specific MaMujoco and SMAC environments used in our evaluations are detailed in Table
B.2 Rule-based vs. LLM-based Trajectory Ranking Techniques
To effectively utilize unlabeled datasets in our multi-agent imitation learning framework, a critical step involves ranking trajectories to identify and select high-quality examples, which then form the expert dataset (D This ranking achieved by first learning an O-MAPL (Offline Multi-agent Preference Learning) model recover the underlying reward function from pairwise preferences. Subsequently, these recovered rewards enable the ranking of all trajectories in the unlabeled dataset.
The training of O-MAPL necessitates preference dataset (P) for each task. Each instance in this dataset consists of pair of trajectories (Ø1, .2) and a label indicating which trajectory is preferred. Echoing the data generation methodologies in O-MAPL, datasets using two distinct approaches: rule-based method and an LLM-based method. For this purpose, randomly sample 2,000 trajectory pairs from the unlabeled dataset for each task, and then apply one the following methods to assign preference labels.
construct these crucial preference
B.2.1 Rule-based Preference Labeling
In the rule-based approach, operate under the assumption that the cumulative return for each trajectory within a selected pair known or can be accurately estimated. The preference assignment straightforward: the trajectory yielding higher return is labeled preferred. This method leverages the intuition that higher returns often correlate with more successful task completion.
However, this rule-based method is not without its limitations, particularly in cooperative multi- agent settings. The accuracy can be compromised because, in certain scenarios, individual agents might adopt strategies that maximize their individual rewards without contributing even at
19

---

the detriment of, the team's overall objective and cooperative success. For instance, in complex environments like StarCraft Multi-Agent Challenge (SMAC), which involves cooperative multi-agent gameplay to defeat enemies, agents might excessively focus on attacking enemies to boost their individual reward metrics. Such behavior can lead to loss of the game due to neglecting crucial strategic elements, such as maintaining adequate health points, armor regeneration, managing weapon cooldowns, or effective unit positioning and coordination. Consequently, trajectory with higher summed individual return might not always represent superior cooperative behavior.
B.2.2 LLM-based Preference Labeling
To introduce more nuanced and potentially more accurate preference signals, we also employ an LLM- based method. For this, leverage the capabilities of large language model, specifically GPT-4o, to act as an expert labeler. The goal is to achieve better generalization in preference assessment and thereby increase the accuracy of the subsequent preference learning phase.
Following the methodology described in studies such as O-MAPI. for SMAC environments, we construct prompts for the LLM that include detailed information extracted from the global state of each trajectory in a pair. This information typically encompasses key metrics like the remaining health points and shields of allied and enemy units, the number of allied and enemy deaths, relative positions of units, weapon cooldown times, agent types, and the semantics of their actions. The LLM is then tasked to evaluate which trajectory demonstrates superior gameplay strategy based on this comprehensive contextual information. This approach has shown potential generating rich and cost-effective preference data, which significantly enhance environment understanding and policy learning in complex multi-agent tasks. As noted the MisoDICE framework, L.MS can be instrumental in inferring expert-like segments even from entirely unlabeled trajectory datasets.
B.2.3 Reward Recovery and Trajectory Ranking
Once the preference datasets are established using either the rulc-based or LLM-basee method, they used train the O-MAPL model. After training O-MAPL, we utilize inverse preference learning technique, specifically leveraging the relationship R = Q - >V (an adaptation of the inverse soft Bellman operator, r(s, a) Qtot (s, a) y (s') as seen in MaxEnt RL frameworks), to
Table 2: Overview of our datasets.
State dim
Obs dim
Action dim
Instances
Trajectories
Samples
Agents
Hopper-v2
1.2K
459.2K
126
14
MaMujoco
Ant-v2
1.2K
1200K
452
113
HalfCheetah-v2
1.2K
1200K
23
_c_vs_ _._ ec_xs_64gg 5m_vs_6m 6m
1.2K
43.2K
1350
478
1.2K
28.3K
780
124
12
SMACv1
6h_vs_82 _z corridor
1.2K
31.8K
1278
172
14
1.2K
71.8K
2610
346
30
5_vs_5
1.2K
76.9K
130
92
11
10 10
1.2K
80.2K
10
310
182
16
protoss
vs.
1,2K
73.3K
191
20 vs 20
1,2K
78.3K
20
820
362
26
20_vs_23
1.2K
70.4K
20
901
389
29
5_
1.2K
60.6K
5
120
82
11
10 vs_vs_100 10
1.2K
61.7K
10
290
162
16
SMACv2
terran
10_vs_11
1.2K
58.2K
10
306
170
17
20_vs_20
1,2K
64.7K
780
322
20_vs_23
1.2K
57.1K
20
858
346
5_vs__
1,2K
34.3K
120
82
10_vs_10
1.2K
38.1K
290
162
16
zerg
10_vs_11
1.2K
37.3K
10
306
170
17
20 20_vs_20
1.2K
40.5K
20
780
322
26
20_vs_23
1.2K
38.5K
20
858
346
20

---

Sample Np pairs
LLM prompt ?
Inverse Preference Learning
Reward Recovery Function
(
== yv
Imperfect Dataset
Preference Dataset P
Top-k
(( 6(02)
D Dullubeled
trajectory C1 02
Expert Dataset D
(
R a)
((ww))
Figure 3: Generalized Expert Dataset Identification via Preference Learning
Expert Dataset
Union Dataset
(o1, ai)
(Onz amp
MLP Network
Cost Network
Network
a(o
co am)
r'(sa)) a)
(c(o.a)). o(0 and on co am)
Critic Network
H
We
Linoar
o (s,
b
Mixing Network
Linear
Policy Natwork
Hyper Network
w w (s.a)
css, a)
css, a)
Weighted BC
Figure 4: Multi-Agent Imitation Policy Learning
recover the reward for each state-action pair within every trajectory in the unlabeled dataset. With these recovered rewards, can then compute the total return for each trajectory and subsequently rank all trajectories.
Finally, from the ranked lists generated by each of the two preference labeling approaches (rule-based and (LMM-ased)) we select the top-ranked trajectories to form the respective expert datasets, denoted as D and DE These datasets are then used for the downstream imitation learning process in MisoDICE.
B.3 Baselines
To rigorously evaluate MisoDICE, we compare its performance against several key baselines. These selected to assess MisoDICE's efficacy in multi-agent imitation learning from mixed-quality demonstrations and to ablate the contributions of its core components, particularly its learnable mixing architecture.
B.3.1 Behavioral Cloning (BC) for Mixed-Quality Data (3 1.0, 0.0)
Multi Agent Behavioral Cloning (MA-BC) serves as a fundamental imitation learning baseline, where individual agent policies n,(a,lo)) are trained via supervised learning maximize the log-likelihood of demonstrated actions. Following the approach for handling mixed-quality data in DemoDICE, we adapt MA-BC using coefficient 3. This coefficient balances learning between a limited expert dataset (D*) and a larger union dataset (D D U Dr),,, where D contains uncurated, potentially suboptimal trajectories. The objective for the set of agent policies is:
N log n. (aslai)
N
JBC(B) = -8E (
Σ
- (1 - B)
Σ
log (aa
We evaluate three variants:
21

---

Algorithm 2: MisoDICE: Phase 1 Expert Dataset Identification via Preference Learning
Input: Unlabeled dataset D Dunlabeled; O-MAPL parameters paramtesr (wag, %/ (OM); Preference method
PrefMethod; Parameters for PrefMethod; Number of preference pairs N₃: Number of expert trajectories Kexpert
Output: Expert dataset DD
/* 1. Generate Preference Dataset
Sample Np trajectory pairs Spairs from Dunlabeled:
2 Generate preference dataset P using Spairs and the specified PrefMethod;
/* 2. Train O-MAPL for Reward Recovery
for a number of training steps do Update O-MAPL parameters (yo0, , N/M (0M) using P;
Maximize preference likelihood LOMAPI and minimize Extreme- V loss To MAPLI
/* 3. Recover Rewards and Select Expert Dataset Estimate R(s, a) for each (s, a) Dunlabeled using QOM 7VOM,
7 foreach trajectory a E D umiibwledd do Compute return G((o)= (ss,)ee R(s,a), a):
Rank all trajectories by G(o) in descending order;
10 Select top- Kespen trajectories as DE;
11 return D
BC (3 1.0): This variant imitates expert data (DE) exclusively, It learns by maximizing the log-likelihood of actions in the expert dataset.
BC (8 = 0.5): This provides an equal weighting between expert and mixed data imitation. It balances learning from the expert dataset and the larger, mixed-quality union dataset.
BC (3 0.0): This variant imitates the entire mixed dataset (DU) without differentiation.
It learns by maximizing the log-lik oggliike of actions in the combined expert and suboptimal dataset.
These variants establish performance benchmark against direct imitation, highlighting how ef- fectively MisoDICE utilizes -ualitty data compared to standard BC approaches that merely re-weight datasets.
B.3.2 Offline Multi-Agent Preference Learning (OMAPL)
This baseline refers to the O-MAPL algorithm, which is an end-to-end preference-basee learning framework for cooperative MARL. O-MAPI. directly learns policies from preference data (Phase in MisoDICE context) without explicitly modeling a reward function. It leverages the relationship be- tween reward functions and soft Q-functions within the MaxEnt RL framework. The learning process uses multi-agent value decomposition strategy under the Centralized Training with Decentralized Execution (CTDE) paradigm. For this baseline, the policy is learned directly from the preference dataset generated in Phase 1. Comparing MisoDICE to OMAPL is intended to demonstrate the necessity and benefit of MisoDICE's Phase 2, which involves reward recovery and policy learning using the DICE framework.
B.3.3 Individual DeMoDICE (INDD)
This baseline applies the single-agent DemoDICE algorithm independently to each agent in the multi-agent system. DemoDICE is an offline imitation learning algorithm designed learn from both expert and imperfect demonstrations by optimizing policy within the space of stationary distributions. achieves this by reducing a potentially unstable minimax optimization problem to direct convex optimization. In the INDD baseline, each agent i independently solves its own DeMoDICE objective. This involves learning a policy n. by focusing on its individual marginal stationary distribution derived from DE (its portion of expert trajectories) and D. (its portion of the union dataset), without explicit joint state-action modeling or coordinated value/policy learning across agents. INDD contrasts MisoDICE's centralized training with value decomposition against a naive
22

---

multi-agent extension of the single-agent DemoDICE. This comparison is crucial for demonstrating the benefits of MisoDICE's joint learning approach and coordinated credit assignment.
B.3.4 MARL with Supervised Reward (MARL-SL)
This baseline represents MisoDICE using Supervised Learning to learn reward recovery from Phase In this approach, reward function is first learned via supervised learning techniques from the preference data generated in Phase 1. Then, a MARL algorithm is applied to optimize the policy with respect to this learned reward. This can be seen as two-phase approach, similar to some existing preference-based RL methods. This baseline helps to evaluate the effectiveness of MisoDICE's s end- to-end DICE-based reward and policy learning in Phase 2 compared to a more decoupled supervised reward learning followed by MARL.
B.3.5 MisoDICE-VDN
This is an ablated version of our MisoDICE algorithm, where the proposed learnable mixer (with a hypernetwork) is replaced by a standard Value Decomposition Network (VDN) mixer. MisoDICE- VDN retains the core framework of MisoDICE, including its adaptation DICE principles for multi-agent imitation learning from mixed-quality data. However, for value decomposition, it uses a simple VDN approach where the joint value function (or its DICE-equivalent. such as the global Lagrange multipliers yaa advantage function A is computed as a direct summation of individual agent values: M({vs(o))) = E, (o.). This is simpler alternative to MisoDICE's architecture, which employs mixer using a hypernetwork to generate state-dependent mixing weights, similar in spirit to advanced mixers like QMIX or those described in O-MAPL. By comparing MisoDICE against MisoDICE- VDN, we can directly assess the performance impact and importance of MisoDICE's sophisticated hypernetwork-bante based mixing architecture.
in DICE literature like ComaDICE)
expressive learnable
B.4 Hyperparameters
Table 3: Hyperparameters used in MisoDICE experiments.
Hyperparameter
Value
Optimizer
Adam
Learning rate (actor)
3e-4 (for SMAC)
le-5 (for MaMujoco)
Learning rate (critic)
3e-4
Tau (r, soft update target rate)
0.005
Alpha (a))
0.05
Gamma (y, discount factor)
0.99
Number of minibatch
512
Agent hidden dimension
256
Mixer hidden dimension
64
Number of seeds
4
Number of episodes for each evaluation step
32
Number of epochs
100
All experiments are implemented using PyTorcl and run in parallel on single NVIDIA@ H100 NVL Tensor Core GPU. Given the large size of the offline datasets for each instance, we compress all datasets into the H5 format using the h5py library.
We develop two versions of our MisoDICE agent to accommodate different domain characteristics: one for continuous action spaces (MaMujoco) and another for discrete action spaces (SMACv1 SMACv2). The primary distinction lies in the output distributions of the policy networks. For continuous control tasks, we employ Gaussian distribution (torch. distributin butionst. Normal) to model the actions. For discrete action environments, we utilize a Categorical distribution (torch. distribute butiinsuoin.... Categoricall action of an agent is computed by applying softmax function over the set of currently available actions for that agent, rather than over the entire action space. Consequently, actions not available to
Notably, in the discrete version, the probability for each
23

---

an agent at given state are assigned a probability of zero. This approach ensures more accurate calculation of the log- rigliikli by the actor.
The hyperparameters used in our experiments are detailed in Table
B.5
Algorithm
The MisoDICE framework is implemented in two main phases, detailed in Algorithm and and Algorithm
3 respectivvel. The first phase focuses on identifying expert trajectories from an unlabeled dataset, while the second phase performs multi-agent imitation learning using these identified trajectories.
Algorithm 3: MisoDICE: Phase 2 Multi-Agent Imitation Policy Learning
Input: Expert dataset D unlabeled/mixed dataset D Punlabeled (or DMix), discriminator networks Ck. Mläse local value networks Vs, value mixing network @val, local policy networks 7)k- regularization coefficient (Misa learning rates
Output: Optimized local policies [4<(a (og 77k))
Let DV = D U U u
/* 1. Train Occupancy Ratio Estimator (Discriminator)
for number of discriminator training steps do
3 Update Ck2 flüie by optimizing discriminator loss [c, Mitis)) using samples from D and DV:
Ref: MisoDICE Sec 5.3 Eq.
Compute log occupancy ratio: r"(s,a) = log (s,0))
(s.a)
/* 2. Train MisoDICE Value Functions s for a number of value function training steps do Update Us and @ral by minimizing L( {vx});
// Ref: MisoDICE Sec 5.2 Eq.
Compute w (8, a) exp
(s.a) 14ss0
1):
// Ref: MisoDICE Sec 5.2 Eq.
/* 3. Train MisoDICE Local Policies via Weighted Behavior Cloning for a number of policy training steps do foreach agent ls E N do Update The by maximizing weighted BC loss LWBC(1) w as) using DV:
// Ref: MisoDICE Sec 5.4 / Eq. 11
11 return (a oos Th for k =,,,...... 1.....
B.6 Experiment Setup
This section details the experimental environments and evaluation metrics used to assess the perfor- mance of MIsoDICE.
B.6.1 Environments
To comprehensively evaluate MisoDICE, we utilize standard and challenging benchmarks widely adopted in cooperative multi-agent reinforcement learning (MARL) research. These environments are sourced from the Multi-Agent MuloCo (MaMujoco) suite [TI. the StarCraft Multi-Agent Challenge version 1 (SMACv1) [46]. and its successor, the StarCraft Multi-Agent Challenge version 2 (SMACv2) [13]. MaMujoco provides continuous control tasks where multiple agents, typically different parts of single robot, must learn coordinate their actions to achieve common goal dynamics. In contrast, SMACv1 and SMACv2 offer discrete action spaces and focus on decentralized micromanagement scenarios derived from the real-time strategy game StarCraft II 46). In these environments, each combat unit is controlled by an individual agent, and the team must learn cooperative strategies to defeat opponent units controlled by built-in game AI. SMACv2, in particular, introduces increased stochasticity and diversity compared to SMACv1 by incorporating features like randomized start positions and unit types, presenting a more demanding test for MARL algorithms
11 These tasks are known for their complex physics-basee
[13). The selection of these diverse

---

benchmarks allows for testing MisoDICE's capabilities across different action spaces (continuous for MaMujoco, discrete for SMAC) and varied cooperative challenges.
B.6.2 Evaluation Metrics
To rigorously evaluate the performance of MisoDICE and compare it against baseline methods, use standard evaluation metrics common in MARL research. The primary metrics reported are:
Returns: This metric measures the average cumulative rewards achieved by the agents across multiple evaluation episodes. Higher returns generally indicate better policy performance in maximizing the task-specific objectives. We report the mean and standard deviation of returns.
Win Rates: Specifically for competitive or mixed cooperative-comper like the StarCraft Multi-Agent Challenge (SMAC) tasks, this metric evaluates the percentage of episodes where the team of agents achieves win condition against opponents. We report the mean and standard deviation of win rates.
environments
Performance for each method is rypically averaged over multiple random seeds to ensure statistical robustness, and evaluation curves often depict performance trends throughout the training process. For MisoDICE, evaluations were run for 32 episodes for each evaluation step across 4 seeds.
C
Evaluation
C.1 Additional Experimental Results
In this section, we present the main experimental results evaluating the performance of MisoDICE. Our evaluation aims to demonstrate MisoDICE's effectiveness in learning robust multi-agent policies from mixed-quality demonstrations, particularly focusing its ability to handle scenarios with limited unlabeled expert data. We conduct comprehensive experiments across a range of challeng- ing cooperative multi-agent benchmarks, including tasks from the StarCraft Multi-Agent Challenge (SMACvI and SMACv2) and Multi-Agent MuJoCo (MaMujoco).
The results will show that MisoDICE consistently outperforms various strong baselines across these diverse environments. key aspect of our evaluation is the comparison of MisoDICE's performance when expert trajectories are identified using two distinct approaches: a traditional rule-based ranking method and, more notably, our proposed LLM-based technique for inferring expert-like segments from entirely unlabeled datasets. This latter approach showcases novel application of LLMs to significantly enhance multi-agent imitation learning in practical settings where explicit expert annotations are unavailable.
We compare MisODICE against several relevant baselines. These include standard Behavioral Cloning (BC) variants that imitate different compositions of the available data, an Independent DemoDICE (INDD) baseline applying single-agent imitation learning each agent, and supervised MARL approach (MARL-SL) that learns a reward function from preferences before policy optimization. Fur- thermore. we evaluate against OMAPL, representing the policy learned directly from the preference data generated in the first phase of our framework, to highlight the benefits of MisoDICE's second phase DICE-based policy learning. An ablated version, MisoDICE. VDN, which uses a simpler value decomposition network, is also included to demonstrate the efficacy of our proposed learnable mixing architecture.
The tables 4 and figures 5 provide detailed quantitative results in terms of average returns and win rates, alongside learning curves. These results will substantiate MisoDICE's superior performance, its robustness to the quality and quantity of expert data, and the significant potential of leveraging LLMs for expert data identification in the challenging domain of multi-agent imitation learning from suboptimal demonstrations.
C.2 Ablation Study 1: Impact of the Number of Identified Expert Trajectories (Top-k)
In this ablation study, we investigate the sensitivity of MisoDICE to the number of trajectories selected to form the expert dataset, D As outlined in Algorithm 2 the first phase of MisoDICE
25

---

involves ranking all trajectories in the unlabeled dataset 'D Punlabeted (or D DM using preference- based model (O-MAPL) trained on LLM-generated (or rule-basedi preferences. From this ranked
Table Comparison of final average returns for MisoDICE and baseline methods on SMACvl and SMACv2 tasks when using LLM-based preference labeling approach.
BC
MisoDICE
OMAPL
INDD
MARL-SL
VDN
(8 0.0)
(8
0.5)
(B 1.0)
(ours)
sevvs__ie rs_64zg
0.1
12.6 0.3
12.2
14.6 1.0
14.0
12.7
16.4
5m 6mm
5.0
6.7
6.1
5.2
6.7
6.1
6.2
7.3
6h_vs_ 8z
0,0
7.4
7.20011
6.6
7,2
7.8
8.2
8.7
coriidor
1.1
1.2
4.7
2.2
4.4
1820.2
4.7
5.8
5_ys__
0.2
11.7
1022
9.6
10.9
11,6
11,5
0.5
Protoss
10_vs_ 10
1.6
118
10.6
10.1
11.0
1.49
12.4
12.9
10_vs. 11
8.24044
9.64044
8.7
8.5
9.4
9.9
1.14
1..7
20 20
1.2
10.4
1..5
9.4
1.4
1.41
0.51
13.5
20 vo_ys_23
0.2
0.6
8.3
0.9
9.3
9.6
10.3
10.6
5_ys__
0.8
8.1
7.1
6.2
7.9
8.1
8.3
9.1
Teran
10_vs_
0.3
7.4044
6.7
6.9 1.1
7.4
0.2
8.0
9.1
10_vs_
4.2
5.7
5.2
0.6
5.7
5.7
6.0
6.5
20 20
6.9404
7.9
6.7
6.9
8.0
8.6
8.2
0.6
20_ 23
0.3
5.1
4.3
0.4
0.1
0.6
5.3
5.6
5_vs__
0.5
6.600.4
5.9
6.5
6.2
0.5
7.1
7.5
Zerg
10_vs_
0.3
8.7
7.4
0.6
8.2
9.0
9.7
10.2
10_ys_11
0.2
8.3
7.3
7.2 0.4
8.0
8.8 0.4
9.1
9.4
20 vs__s_20
0.6
9.0
0.2
6.9
6.9
8.3
0.6
9.0
10,2 +
20_vs_23
0.3
7.9
7.2
7.1
0.4
8.8
8.7
9,5
Table 5: Comparison of final average win rates for MisoDICE and baseline methods on SMACv1 and SMACv2 tasks when using an LLM-based preference labeling approach.
BC
OMAPI
INDD
MARL-SL
VDN
MisoDICE (ours)
0.0)
(Ø = 0.5)
(8 L.0)
64_z
0.2
0,5 0.3
8.9
3.9
11.7
10.6 6.0
2.7
13.0 9.0
6m
0.2
0.6
0.1
0.0
0,2
1.1
0.9
1.2 1.2
6h_vx_8r
0.2
0.0
0.3
0.0
0.1
1.0
1.2
1.1
corridor
0.1
0.6
0.3
0.0
0.1
0.9
0.7
1.4
5_vs__
13.8
1..5
14.2
14.1 10.2
1.14
1..6
10.8
20.7
Protoss
10__s__ ys 10
12.1
12.7
1.9
11.7
8.9 1.4
1188
9.5
14.1 2.1
10_vs
0.1
3.5
1.8
0.8
2.0
3,5 0.3
2.9
4.7 0.3
20
7.0
3.1
5.2+13
8.6
1.2
11.0
20 23
1.5
0.8
1.8
0.8 1.4
2.4
2.9
2.6
3.8 a 2.0
5_vs__5
2.3
12.8
3..7
4.62
10.6
1.9
10.4
14.2
Terran
10_vs__
7.3
7.7
7.9124
9.4
8.0
8.2 1.6
7.9
12.0 1.7
10_vs_ 11
1.9
3.114
2.5
0.8
1,8
2.6
2.4
4.2
20 vs. 20
7.4225
4.40.6
4.7
0.7
7.1
4.5
8.8 1.5
20 vevvv___3
0.6
1.2
0.7
0.0
0.4
1.7
1.2
1.8 1.4
5
6.2
5.6
5.9
6.8
6.0 1.4
6.7
7.0
7.9 1.0
Zeng
10_vs_10
4.5 1.4
6.9
5.0
3.4
4.6 1.0
5.2
0.2
8.9
10_vs_11
1.7
1.8
5.2
1.4
4.7
6.0
3.4
6.8 1.1
20_vs_
0.9
0.6
1.3
0.0
0.2
1.2
2.4 0.6 0.2
20_vs_23
1.2
0.7
1.6
1.6
1.4
1.9
1.8
2.7
(B=0.0)
(B=0.5)
(B=1.0)
MARL-SL
MisoDICE (ours)
6
8-
4-
10
2c_ vs_ vs_64zg
5m vs _ 6m
6h vs
_ 8z
corridor
Figure 5: Learning curves of the average return for MisoDICE and baseline methods on SMACvl tasks when using an LLM- preference labeling approach.
26

---

list, the top- Kexpert trajectories are selected to constitute the expert dataset D which is then used alongside the full mixed dataset D in the second phase for policy learning.
BC (B-0.0)
BC (B-0.5)
BC (B-1.0)
INDD
VDN
MARL-SL
MisoDICE (ours)
14-
14
Protoss
12
12
10
50
50
50
10-
10
10
Terran
50
50
50
50
50
12
10-
Zerg
6-
50
50
0
50
50
5_vs_ 5
10_vs_10 10
10
10_vs_11
20 20_vs vs 20
20 vs _s_
Figure 6: Learning curves of the average return for MisoDICE and baseline methods SMACv2 tasks when using an LLLM-basd preference labeling approach.
BC (8-0.0)
BC (B=0.5)
BC (8=1.0)
INDD
VDN
MARL-SL
MisoDICE (ours)
30%
4%
4%
4%
10%-
25
2%
0%
0%
0%
0%
0
50
50
50
50
2c_vs_ _s_ 2s__
5m vs_ 6m
vS
8z
corridor
Figure 7: Learning curves of the average win rates for MisoDICE and baseline methods on SMACvl tasks when using LLM-based preference labeling approach.
(B=0.0)
(B=0.5)
(B=1.0)
MARL-SL
MisoDICE (ours)
10%
20%
Protoss
20%
15%
10%
10%
10%
5%
10%
0%
50
10%
20%
Terran
15%
10%
10%
5%
15%
15%
15%
Zerg
10%
10%
10%
4%
4%
6%
5%
2%
2%
0%
5
VS_
10
vs_ 10
10
vs_
11
20_
VS
20
20_ vs 23
Figure 8: Learning curves of the average win rates for MisoDICE and baseline methods on SMACv2 tasks when using an LLM-based preference labeling approach.
27

---

The choice of Keppert (referred to as "top-k") is critical. A smaller K Keppert might yield higher purity expert dataset but could be too limited, potentially missing some useful expert behaviors. Conversely, larger K Kexpert might include more diverse set of expert behaviors but risks incorporating more sub- optimal trajectories, thereby degrading the quality of "D This study examines how varying K Kaspert affects the final performance of MisoDICE. We evaluate MisoDICE's performance (in terms of returns and win rates) across a range of K Keapert values, specifically K Kezpert E {50, 200, 400, 800, 1200). using the LLM-based trajectory ranking method. The results, detailed in Table 6 and Table 7 (and visualized in Figure 9 I1 and Figure 10] provide insights into the robustness of MisoDICE the size and composition the automatically curated expert dataset. This helps understand the trade- off's involved in selecting K Kezpert and its impact on learning effective policies from mixed-quality demonstrations.
Table 6: Ablation Study 1: Final average returns for MisoDICE SMACv1 and SMACv2 tasks, varying the number top-k expert trajectories (Kexpen) selected via LLM-based preference labeling.
topk=50
topk=200
topk=400
topk=800
topk=1200
2c vs 64zg
11.8 + 1.4
16.4 + 1.3
9.7 0.1
10.8 0.7
10.4
5m_vs_6m
6.9
7.3
6.8 1 0.2
0.1
6,4 0.2
6h_vs_8z
0.1
8.7
7,8 0.2
0.1
0.1
corridor
3.1
5.8
0.1
1.7
1.7 0.1
$_vs__
11.
0.5
1.2
1.2
10.9
Protoss
10. vs
1..
12.9
1.3
0.3
11.2
10 [0_vs_l 11
9.1
1..7
9.3
9.5
9.3
20. vs. 20
1..4
0.55
12.1
1..4
11.2
20 vs. 23
0.3
1..6
10.0
9.2
8.4
5_vs_5
0.3
9.1
0.4
7.7
0.5
Teman
10. vs. 10
0.4
1.1
8.1
7.7
0.4
10. vs. 11
0.1
6.5
6.1
5.7
5.4
20 vs 20
8.0
9.2
8.2
8.0
7.9
20_vs_23
0.1
5.6
5.1
5.2
4.8
5_vs_5
0.4
0.1
0.5
6.4
0.5
10 vs 10
0.3
0.6
0.3
8.8
0.3
Zerg
10. Vs 11
8.5
0.3
8.5
8.2
0.3
20 vs. 20
8.9
1.6
8.8
8.3
8.6
20 VS 23
0.4
9.5
8.8 0.2
8.5 0.2
8.0
14-
10
6-
8-
50
200
400
800
1200
50
200
400
800
1200
50
200
400
800
1200
protoss
terran
zerg
Figure 9: Ablation Study 1: Box plots of final average returns for MisoDICE SMACv2 tasks, showing the impact of varying the number of top-k expert trajectories (K (Kexpen) selected via LLM- based preference labeling. Results presented for Protoss, Terran, and Zerg game races.
C.3 Ablation Study 2: Impact of the Suboptimal Data Influence Hyperparameter
In this ablation study, we examine the influence of the hyperparameter a on the performance of MisoDICE. As defined in our learning objective (Equation 2), plays crucial role by controlling the relative importance of the union dataset D (composed of both expert DE and mixed-quality Miae trajectories) compared the purely expert dataset F Specifically, the term aD (p (P encourages the learned policy Txat to remain close to the overall distribution of the union dataset, while the Drr.PpT (PToellee))))) term pushes the policy mimic the identified expert behaviors.
The choice of a therefore dictates how MisoDICE balances learning from the potentially limited but higher-quality expert demonstrations against leveraging the broader, though more suboptimal,
28

---

data distribution of the union dataset. A small would emphasize strict imitation of D potentially suffering if D is very scarce or narrowly focused. Conversely. large a would regularize the policy more strongly towards the mixed-quality data in DV which could be beneficial for broader state-action coverage but risks learning suboptimal behaviors if the quality of DU is low overall.
This study investigates MisoDICE's sensitivity to different values of a E {0.00, 0.05, 0.10, 0.50, 1.00, 10.0). By evaluating the algorithm's performance (returns and win rates) across this range, as detailed in Table 8 and Table we aim to understand the optimal balance and the robustness of MisoDICE this critical hyperparameter when learning from demonstrations of varying quality, Our findings also help to validate the choice of 0.05 used in our main experiments (see Table 3).
C.4 Ablation Study 3: Comparison of LLM Capabilities for Preference Labeling
core component of MisoDICE, particularly when expert demonstrations are not pre-annotated, is the utilization of Large Language Models (LLMs) infer expert-like segments from unlabeled trajectories. This is achieved the first phase by using an LLM to generate preference labels for pairs of trajectories, which then informs the O-MAPL model rank demonstrations and identify the expert dataset DE
In this ablation study, investigate the impact of the chosen LLM's capability the overall performance of MisoDICE. Specifically, we compare the results when using two different versions of OpenAl's models for the preference labeling task: the more powerful GPT-4o and its smaller, potentially more efficient counterpart, GPT-4o-mint 39 The motivation is to assess whether the
Table 7: Ablation Study 1: Final average win rates for MisoDICE SMACv1 and SMACv2 tasks, varying the number of top-k expert trajectories (Kexpen) selected via LLM-based preference labeling.
topk=50
topk=200
topk=400
topk=800
topk=1200
2c _c_vs_ _s__4zz vs
5.6
13.0
1.6 0.7
1.7 0.8
0.6 = 0.7
5m 6m
0.0 0.0
0.5
0.0 0.0
0.0
0.0 0.0
6h_vs__8 8z corridor
0.0
1.1 0.8
0.0
0.0
0.0
0.0
0.6
0.0
0.0
0.0
5 5_vs_5
8.9
20.7
10.1
12.5 1.7
9.4 1.7
Protoss
10_vs__ 10
5.3
14.1 2.1
11.1
10.0 3.0
5.7
10 vs 11
0.4
0.7
2.2
1.0
1.6
20 vs 20
2.8
11.0
3.6
6.1
3.9
20 23
0.4
2.0
0.8
1.8 1.2
1.0
5_vs_5
1.9
9.8 3.11
10.4
5.1
Terran
10_vs 10
0.6
12.0 1.7
7.1
8.1
5.2
0.6
4.2
2.2
1.4
0.6
20 vs. 20
3.7
8.8 1.5
4.6
6.0 1.6
5.2
20 23
0.3
1.8 1.4
0.6
0.4
0.1
5_v8_5
1.3
7.9 1.0
5,3 1.1
4.7 1.6
3,8 0.6
10_vs_10 10
1.7
8.9
5.6
6.7
3.5
Zerg
10 11
1.0
6.8
4.6
4.3
2.8
vs. v0_
0.3
0.6
0.3
0.5
0.3
20 vs 23
0.6
2.7 0.4
1.4
1.1 1.0
1.2
20%
10%
15%
15%
10%
5%
5%
5%
0%
0%
0%
50
200
400
800
1200
50
200
400
800
1200
50
200
400
800
1200
protoss
terran
zerg
Figure 10: Ablation Study 1: Box plots of final average win rates for MisoDICE on SMACv2 tasks, showing the impact of varying the number of top-k expert trajectories (Kexpen) selected via LLM-based preference labeling. Results are presented for Protoss, Terran, and Zerg game races.
29

---

enhanced reasoning capabilitiees of a larger model like GPT-4o translate into significantly better pref- erence data and, consequently, improved downstream imitation learning performance for MisoDICE, or if more compact model like GP---4o-mini can provide comparable results with potentially lower computational and financial costs.
We evaluate MisoDICE's final performance, in terms of returns and win rates, on our benchmark tasks when the initial preference labeling (Phase is conducted by GPT-40 versus GPT-4o-mini The comparative results, presented in Table TO, will shed light on the trade-off's associated with LLM selection for data annotation the MisoDICE framework and provide insights into the level of LLM sophistication required for effective multi-agen imitation learning from unlabeled demonstrations.
50
200
1200
14
12
Protoss
12-
12
10-
10
8-
50
50
50
50
Terran
10
8-
6-
50
Zerg
9-
10
10
10-
10-
6
50
50
50
50
50
5 v_ 5
10_vs_ v__
10
vs_
20 Vs 20
20 vs 23_
Figure 11: Ablation Study 1: Learning curves of average returns for MisoDICE on SMACv2 tasks, illustrating the impact of varying the number of top-k expert trajectories (K exppe)) selected via LLM- based preference labeling. Each curve corresponds different Kexpert value from the set (50, 200, 400, 800, 1200).
50
200
400
800
1200
10%
20%
Protoss
20%
20%
15%-
10%
5%%
10%
10%
0%
10%
Terran
15%
10%
10%
5%.
10%
5%
15%
15%
6%
Zerg
10%
10%
10%
4%
4%
2%
0%
5 VS 5
10.
10
10 VS
20
20
20
23
Figure 12: Ablation Study 1: Learning curves of average win rates for MisoDICE on SMACv2
tasks, illustrating the impact of varying the number of top-k expert trajectories (K (Kexpen) selected via LLM-based preference labeling. Each curve corresponds to different K Kespent value from the set (50, 200, 400, 800, 1200).
30

---

C.5
Ablation Study 4: Performance with Rule-Based Preference Labeling
While a key contribution of our work involves leveraging Large Language Models (LLMs) for inferring expert-like trajectories from unlabeled demonstrations, is also important to understand MisoDICE's performance when relying on more traditional or simpler methods for generating initial preference data. This ablation study evaluates MisoDICE using rule-based approach for preference labeling in its first phase.
In this rule-based setting, preference labels for pairs of trajectories (2) are generated based predefined heuristic, typically by assuming that the cumulative return for each trajectory is known or
Table 8: Ablation Study 2: Final average returns for MisoDICE SMACv1 and SMACv2 tasks, evaluating the impact of different values for the suboptimal data influence hyperparameter (a:) when using LLM-based preference labeling. The hyperparameter controls the influence of the union dataset in the learning objective.
0.00
0.05
0.10
0.50
1.00
10.0
2c vs. _ Ss_ys_ 6m
15.9 1.0
16.4 1,3
1..3
14.9
13
10,9 0.3
6.6 1.4
0.1
6.3
6.5
6.2
6.11
6h_ th__s__
0.2
8.2
8.0
8.0
7.7
0.6
corridor
0.6
0.8
0.6
5.1
3.8
1.7
5_vs_5
1.44
1..4
11.7
11,8
11.9
11.7
Protoss
10_vs_
12.1
122
12.2
1.31
11.600.5
12.1
10_vs_
0.5
1..7
10.1
1..1
9.8
9.9
20
0.2
13.5
12.3
1..2
11.
12.1
20 23_vs_23
0.4
0.2
1..1
1.21
10.0
9.6
5 5_vs_
8.8
0.1
8.4
8.2
7.4
8.3
Terran
10_vs_10
8.9
9.1
8.5
8.6
7.8
10_vs_
5.9
0.5
5.8
5.9
5.7
0.6
20
9.0
0.6
8.5
0.6
8.2
8.4
20_vs_23
0.2
5.6
5.3
5.1 0.3
5.1
5.1 0.1
5_
0.5
7.1
7.2
0.2
6.4
10_vs_
9.7
1.6
9.2
9.2
8.7
8.9
Zarg
10
9.0
0.3
8.5
8.3
8.7
8.3
20 10__v_22 20
9.0
0.62
9.3
0.7
8.4
0.9
20 23
0.2
9.2
8.8
0.9
8.5
8.6
Table Ablation Study Final average win rates for MisoDICE on SMACvI and SMACv2 tasks, evaluating the impact of different values for the suboptimal data influence hyperparameter (a:) when using LLM-based preference labeling. The hyperparameter a controls the influence of the union dataset in the learning objective.
0.00
a = 0.05
a 0.10
a = 0.50
a 1.00
= 10.0
_c_vs_ ______6_gg
1..0
13.0 9.0
1..0
9.3 2.9
7.1
1,7
5m_vs. 6m
1.1
1,2
U.4
0.7
0.
0.7
6h_vs_8/
0.8
1.1
1.1
0.8 0.5
0,4
1.0
corridor
0.2
1.6
1.2
0.5 0.4
1.4077
0.9
5_vs__
12.1
20.7 +
2.14
12.9 2.0
17.2 14.1
Protoss
10_vs. 10
3.3
14.1
9.9
2.6
9.0
11.1
10_ [0_ys_1 11
3.3
0.3
4.3
3.5 1.3
3.5
4,3 1.2
20, vs. 20
0.4
11.0
5.5
6.4
6.7
7.8 2.4
20_ 20_vs__2
0.6
3.8
2.3
3.5
3.8
2.5 1.3
5_
2.94
1..2
11.2
12.1
12.0
12.2
Terran
10_ys_ 10
9.3
12.0 1.7
8.6
9.2
8.8
9,3 3.2
10, 10_vs_11 vs_11
0.3
4.2 1.6
3.5
2.3
2.2
3.6 1.5
20. vs. 20
4.8+16
8.
4.9
5.9
4.9
9.1
20_vs_23
0.7
1.8
0.2
1.8 1.7
1.2
1.7 1.0
5_vs__5
1.1
7.9
7.4
5.9
6.4
7,5
10_vs_I0 10
6.7
8.9
7.2
7.0
7.1
6.9 1.0
7eng
10. vs.
1.0
6.8
5.0
4.1
5.0
4,5 1.4
20 v0_
0.3
0.6
12
1.7221
1.6003
2.6
20 23
0.5
2.7
1.0
2.7
2.7
0.7
31

---

can be accurately estimated, with the trajectory yielding a higher return being labeled as preferred. Although straightforward. such rule-based methods can have limitations complex cooperative multi-agent settings, as individual rewards simple cumulative returns may not always accurately reflect true team-level cooperativeness or overall strategic success.
The primary motivation for this study is twofold: first, to assess the inherent capability of the MisoDICE algorithm (Phase 2) learn effective policies even when the expert dataset D is identified using these simpler, potentially less nuanced, preference signals. Second, it provides crucial benchmark quantify the additional benefits and performance improvements gained by employing the more sophisticated LLM-based preference labeling approach presented as our main method.
We present the performance of MisoDICE and relevant baselines in terms of returns and win rates when using rule-based preference data, detailed in Table 11 and Table [2] These results will illustrate MisoDICE's adaptability and provide a clearer understanding of the impact of different preference generation techniques final policy quality.
C.6 Ablation Study 5: Performance with Ground Truth Data Labels
this ablation study, we aim to establish an upper-bound performance benchmark for our MisoDICE framework by utilizing ground truth labels for the initial data segmentation in Phase 1. This means that instead of relying on -LM-generated preferences or rule-based heuristics to identify expert trajectories, directly the original quality labels (i.e., 'expert' and 'poor') available from the O-MAPL datasets from which our mixed-quality dataset D was constructed.
Specifically, the expert dataset DE is formed by selecting trajectories that were originally labeled as 'expert' in the O-MAPI. dataset, and the suboptimal dataset D comprises those originally labeled as 'poor'. This process effectively bypasses the entirety of our proposed multi-step labeling pipeline (Section 4, Algorithm 2), including LLM prompting, preference learning with O-MAPL for reward recovery, and subsequent trajectory ranking. By providing MisoDICE's Phase 2 with this perfectly partitioned data, we can evaluate the core imitation learning algorithm's 's effectiveness under idealized conditions.
Table 10: Ablation Study 3: Comparison MisoDICE performance (final average returns and win rates) on SMACvl and SMACv2 tasks when using GPT-4o versus GPT-4o-mini for LLMM-bsee preference labeling in the initial data annotation phase (Phase 1).
Returns MIsoDICE
Winrates MisoDICE
(gpt-4o-mini)
(gpt-4o)
(gpt-4o-mini)
(gpt-4o)
_c_vs__ vs_64gg
12.2
16.4 1.3
2.1 1.1
13.0
5m_vs. vs. 6m
0.9
7.3
1,2 0.4
1.2
6h _s__r__ 8z
8.1.0.2
8.7 0.2
0.1 0.1
1.1 1 0.8
corridor
0.8
5.8
0.6
1.4 0.6
5_vs__
0.3
1.54
12.6
20.7 4 0.9
Protoss
10_vs 10
0.5
1..9
3..5
14.1 2.11
10
1.3
1..7
4.0
4.7
20 20_ v0 20
0.5
1.55
6.0
11.0
20. vs. 23
0.1
10.6
2.3
3.8
5_vs_5
0.5
9.1
10.0 1.4
1..2
Terran
10_vs_10 10
0.6
9.1
9.2
12.0 1.7
10_vs_
6.3
0.5
2.2
4.2 1.6
20 20_ 20
0.5
9.2
6.1
8.8 1.5
20_vs vs
0.3
5.6
0.9
1.4
v_
7.4
0.5
5.3
7.9 1.0
10. vs_10
9.6
1.62
7.1
8,9
Zerg
10_ v1 11
9,2
9.4
5,0
6.8 1.1
20 vs_ 20
0.9
1..2
0.8
2.4 0.6
20 20_vs_23
9.0
9.2
1.7
2.7
32

---

The primary motivation for this study is to isolate and assess the performance of the MisoDICE multi-agent IL algorithm (Phase 2) when the complexities and potential inaccuracies of the trajectory labeling stage are removed. The results obtained from this setup will serve crucial reference point, allowing us to quantify how closely the performance achieved with our I.L.-basee labeling approaches (as explored in previous ablation studies and main experiments) approximates this scenario with ground truth data separation. This will provide insights into the efficacy the labeling
rule-based
Table 11: Ablation Study 4: Final average returns for MisoDICE and baseline methods MaMujoco, SMACv1, and SMACv2 tasks. For MisoDICE, the expert dataset (D was identified using a rule-based preference labeling approach in Phase 1.
BC
MisoDICE (ours)
INDD
VDN
(8 0.0)
(B = 0.5)
(3 = 1.0)
Hopper-v2
123.0 + 23.1
44.6 154.1 = 44.6
158.1 46.4
135.4 32.8
171.4 19.2
206.5 22.5
Ant-v2
1026.5 77.7
1631. 51.8
1910.8 70.2
1826.4 16.6
1869.7 + 9.1
2025.8 + 3.2
HalfChectah-v2
-201.5 7.9
-206.5 166566000
-73.4 34.5
2.77.9
-243.9 # 14.8
-72.1 31.8
2c, 2c_vs_ _e_vg_64zg
8.2
9.3
9.7
11,8
13.2
15.0 1.4
5m vs_
1.3
6.6
5.4044
6.2
6.7
7,2
6h_ _z
0.1
7.4
7.1
0.3
7.8
8.5
corridor
0.1
1.5
3.6
0.6
1.7
4.7 0.6
5_vs_5
9.8
1.44
1..2
1.54
1.6
0.7
Protoss
10 sivv__l
9.2
1..7
1..1
#..
11.8
12.6
vs.
7.7004
9.4
8.4055
9.1
9.5
10.6
20 2s_20
0.3
10.3
9.9
11.3
11.4
13.1
20_ vs.vs__
0.3
0.3
8.3
9.4
9.5
1..4
$_vs_S
6,5
0.7
6.6
7.4
8.0
8.8 # 0.5
Terran
10_vs 10
6.5
7.1
6.6
7.3
7.6
8.6 0.3
10 si.vs_
0.2
5.4066
5.0
5,5
0.7
6,2 = 0.5
20 vs.
0.5
0.5
6.7
7.6
8.4
9.1 = 0.4
20 vs_
0.3
5,0
4.2
5.5
5.0
5.5 0.4
5_vs_S
0.3
0,3
5.6
0.5
6.3
7 0.5
10_vs_10
0.2
8.2
7.2
8.0
8.8
10.0 0.2
Zerg
10_vs_11
6.5
8.2
6.1
7.9
8.4
9.1 0.6
20 vs. 20
0.2
9.0
7.6
8.1
8.6
10.0 0.6
233
0.2
0.9
6.9
0.2
8,8
9.5 + 0.4
Table 12: Ablation Study 4: Final average win rates for MisoDICE and baseline methods on SMACv1 and SMACv2 tasks. For MisoDICE, the expert dataset (DE. (l))) was identified using rule-based preference labeling approach in Phase
BC
MisoDICE (ours)
INDD
VDN
(B 0.0)
0.5)
(B L.0)
2c_vs_64zg
0.1
0.3
1.0 1.0
2.3 0.1
7.1 1 4.6
8 5.9
5m_vs_ _
0.2
0.7
0.1
0.1 0.1
1.1
1.3 0.5
6h_vs vs_
0.2
0.0
0.2
0.1 0.1
0.6
1.180.
corridor
0.1
0.6
0.3
0.1 0.100.1
0.9
1.6
5_vs__ _
15.8
1..8
13.7
10.1 1.6
1..3
18.4 1.3
Protoss
10_vs. 10
3.3
9.4332
8.7
1.6 1.6
9.8
12.2 1.6
10_vs_ 11
1.4
2.9
1.7
1.7 0.2
2.8
4.1 1.1
20_vs_20
7.6
0.7
5.0
3.9 0.9
3.6 1.4
8.7
20_vs_23
0.4
0.7
1.7122
2.0 0.6
1.6
3.2 0.8
5_vs_5
8.7
1..3
10.4
9.3
8.8
14.0
Teran
10_vs_
2.0
7.8
7.8
6.6
5.9
11.3
1.5
2.3
2.4
1.2
2.2
2.7 1.2
20. 20
3.5
5.6
3.8
4.3
5.5
7.7
20_vs_20
0.5 0.5
0.9
0.5
0.6 0.6
0.9
1.7
5 v_vs__
5.7 0.4
5.2
1,5
5.4 1.2
6.0221
6.6 1.0
Zeng
10_vs_10
3.7
3.7
3.714
4.5 + 1.1
4.3
8.8 1.7
3.3 1.0
4.1
32+14
3.6
1.0
20. 20
0.3
0.9
0.9
0.1 0.1
1.1004
1.700.
23
0.7 0.4
0.6
0.9
0.6
1.6
2.4
33

---

procedures themselves and the overall robustness of MisoDICE, We will present the performance of MisoDICE terms of returns and win rates in Table and 14 under these ground truth conditions.
Table 13:
Ablation Study 5: Final average returns on SMACvI & SMACv2 tasks for MisoDICE and baselines, using ground truth data labels for expertsabooptimal trajectory separation.
BC
MisoDICE (ours)
INDD
VDN
(3 0.0)
(B = 0.5)
(8 1.0)
2e_vs 2c_vs_6442 vs_64zg
11.7
1.1
13.1
1.31
15.9 2.0
16.1 + 1.8 7.1
5m 3s vs_ 6m
1.1
6.4
7.2
7.2
0.3
6h vs 8z corridor
0.4
8.4
8.3
8.2
8.1
8,9 1.1
0.1
1.9
5.7
2.0
5.8
6,1 1.6
5_vs_5
0.1
12.6
12.0
1.99
1..3
13.7
Protoss
10_ 10_vs_10
1.
12.
128
1.8
0.52
14.0
10_vs. vs_
1.
10.6
10.9
1011
1.44
12.0
vs. 20
1..4
12.2
13.0=0.
12.7
1.43
13.9
vs. 23
10.1
9.5
1..
10.1
1..4
10.5
5_vs_5
8.9
8.
8.7
8.
9.4
9.4
Terran
10_vs_10
1.0
8.2
7.2
0.7
8.3
0.0
10_vs_11
0.4
5.
5.8
5.7
1.2
7.3
20 20
8.6
8.3
7.9
8.
8.7
9.3
20
0.2
5.5
5.4
4.4
0.5
5.7
5_vs_5
7.8 1.2
7.0
7.1
0.3
0.8
8.0
Zerg
10_vs_10
8.7 1.2
8.2
9.2
9.6 1.2
9.6
9.9
10_vs_11
0.6
9.3
8.8
9.3 1.0
0,9
9.9
20_vs_20
9.2
0.6
9.4 1.1
9.2
9.5
11.1 + 1.3
20_v$_23
9.2 1.1
8.9
8.2
9.3 # 1.0
9.5
9.5 & 0.1
Table 14: Ablation Study 5: Final average win rates SMACvl & SMACv2 tasks for MisoDICE and baselines, using ground truth data labels for expert/suboptimal trajectory separation.
BC
MisoDICE (ours)
INDD
VDN
0.0)
(8 0.5)
(8 1.0)
2c 2s_ ___ss_64k 642g
1.6 1.6
0.0 0.0
2.6
9.2
10.2 = 7.1
11.7 + 8.1
5m 6m
0.8 1.4
1.6
0.8
0.8 1.4
1.6 1.6
1.6 1.6
6h_ vs. 8z corridor
0.0
0.8
0.0
0.0
1.6
2.3
0.8
0.8
0.8
0.8 1.4
2.7
2.6
5 5
1..3
17.24844
20.3
20.3 1..2
21.1 11.6
Protoss
10. vs. 10
10.243.
10,243.4
8.6
6.2
10,2
4..6
10. vs. 11
1.6
3.9
3.1
4.7
4.7
6,2
20. vs. 20
6.6
2.322.6
9.4
7.0
9.4222
10.9
20_vs_23
2.6
1.616
1.6
2.3 1.4
3.1
3.9 5.1
5_vs_5
10.9
4.1
9.4
11,7
1..4
17.2 8.4
Terran
10_vs_10
8.6
4.0
7.8
5.5
9.8
10.9
10_vs_11
1.
1.4
1.6
3.5
20_vs_20
2.6
6.2
4.7
4.7
8.6
9.4
20_v8_23
0.8
0.0
1.642.7
0.8
1.6
2.3 1.4
5_vs_5
2.2
1.4
3.9
6.8
4.6
4.7
10_vs_10
2.7
5.5
5.6
3.2
8.6
8.1
Zerg
10_vs_11
3.9 1.4
4.7
0.8 1.4
2.6
8.6
8.6 4.6
20_ys_20
1.6
1.6
0.8 1.4
0.0
2.6
2.3
20_v8_23
0.8
1.6 1.6
1.6 1.6
0.0
1.7
1.7
C.7 Ablation Study 6: Sensitivity to Noisy Preference Labels
We conduct ablation study analyze the framework's sensitivity to the quality of the preference feedback. This study investigates the impact of explicitly noisy preference labels Phase on the final performance of MisoDICE in Phase 2.
To simulate this, we first obtained the complete set preference labels from GPT-4o. Then, we introduced synthetic noise by randomly "flipping" the preference outcomes for certain percentage
34

---

of the trajectory pairs. We tested this at noise levels of 5%, 10%, 20%, and 30% of the samples. The performance of MisoDICE trained on these progressively noisier preference datasets is reported in Table 15
The results clearly show gradual degradation in MisoDICE's performance the level of noise increases from 0% (no noise) to 30%. For example, in the '2c_vs_ vs_64zg scenario, performance drops from 16.4 = 1.3 11.5 = 0.5. This trend is consistent across most scenarios, supporting the hypothesis that high-quality, consistent preference feedback in Phase is critical for effective trajectory labeling and optimal downstream policy learning Phase 2.
Table 15: Ablation Study 6: Sensitivity Noisy Preference Labels. We report the final average returns for MisoDICE after introducing noise (randomly flipping labels) at different rates (5%, 10%, 20%, 30%) to the preference data generated by GPT-4o in Phase
MisoDICE (0% Noise)
MisoDICE (5% Noise)
MisoDICE (10% Noise)
MisoDICE (20% Noise)
MisoDICE (30% Nolse)
Scenario
2c_vs_647z
16.4 + 1.3
14.7 + 1.0
13.0 + 0.6
11.6 + 0.7
11.5 + 0.5
5m_vs_ 6m 6h_vs__8 87
7.3 + 0.1
6.5 4 1.0
6.1 + 1.9
6.2 + 1.3
6.0 + 1.7
8.7 0.2
8.2
8.0 + 0.1
8.1 + 0.1
7.8 + 0.2
corridor
5.8 + 0.8
2.9
2.6 + 0.5
2.0 0.3
2.2 0.6
5_vs_5
12.4 0.5
11.810 1.9
11.5 0.8
10.9 1 0.1
10.0 0.5
Protoss
10 [0_vs_10 10
12.9 0.2
11.9
12.5 1.8
11.6
1..2
10 11
10.7 0.4
10.4
10.3 0.8
10.2 0.6
9.7
20_ 20
13.5
12.6110
12
1.3
20_vs_23
10.6 0.2
10.1 0.7
9.9
9.2
9.4
5_vs_5
0.3
1.6
6.8
1.8
8.1
Terran
10_vs_10
9.1
1.9
7.6
7.1
7.2
10
6.4
0.6
6.9
0.7
6.0
20
0.2
0.7
8.7
8.1
7.8
20_
0.4
5.4
0.6
0.5
4.5
5_vs_5
0.1
0.9
7.2
6.3
6.5 1.6
Zerg
10_vs_
1.6
0.3
1.5
8.7 1.3
8.0 8.0
10_vs_11
0.3
1.3
8.8
8.4
7.5
vs
10.2
0.3
9.2 0.7
0.0
8.5 0.9
20_v8_23
0.2
9.9
0.7
0.3
8.1 1.1
C.8 Ablation Study 7: Alignment of Phase 1 Labeling with Ground Truth
We conduct an experiment to measure the alignment of our Phase labeling pipeline with ground- truth preferences. This study compares the preferences generated by our LLM--uided O-MAPL. process against ground-truth preferences derived from environment rewards.
For this experiment, we sampled 2,000 trajectory pairs per task. The "learned" preferences were derived from the reward function recovered by O-MAPL (which was trained on LLM-generated feed- preferences were determined by comparing the cumulative environment
back). The "ground-trutte rewards for the two trajectories in each pair. We then calculated the accuracy as the percentage of pairs where the learned preference (expert vs. suboptimal) matched the ground- ttrdhtttthhhhue preference.
The results are reported in Table T6
The pipeline achieved a high overall accuracy (90.5% average across all tasks), demonstrating the strong reliability of the LLM feedback and O-MAPL. refinement process in Phase 1. This highlights the effectiveness of our approach filtering expert-like trajectories that closely align with the ground-truth reward-based preferences.
C.9 Ablation Study 8: Importance of Phase Trajectory Labeling
assess the importance of the entire Phase labeling pipeline by running an ablation study where this phase is skipped entirely. In this "no labeling" variant, all trajectories in the unlabeled union dataset DU are treated as optimal demonstrations for the Phase imitation learning algorithm (MisoDICE).
35

---

We compare this variant, "Mis (no labeling)," against our full MisoDICE framework (which includes Phase labeling) and the standard "BC (no labeling)" baseline (which corresponds to BC with 3 = 0.0 from our main experiments, i.e., standard BC on the entire mixed dataset).
The results in Table 17 show that the "MisoDICE (no labeling)" variant performs significantly worse than the full MisoDICE framework across all tasks. While it still slightly outperforms the simple BC baseline in cases, this result strongly highlights the importance of Phase I's labeling strategy. Simply applying the Phase 2 algorithm to the entire mixed-quality dataset as if it were all expert data suboptimal. The labeling pipeline is crucial for distinguishing high-quality demonstrations, which in turn enables the Phase 2 algorithm to learn effectively from noisy, unlabeled data.
Table 16: Ablation Study 7: Accuracy of Phase Preference Labeling vs. Ground Truth. This table shows the accuracy of the O-MAPL-recovered reward function (trained on LLM feedback) in correctly ranking trajectory pairs compared to ground-truth rankings based on environment rewards.
Scenario
Accuracy
Scenario
Accuracy
SMACvI
SMACv2 Terran
2c_ 2c_vs_64rg 5m vs 6m
79.2%
lerran_5 __rram_________ 5_vs_5
90.4%
87.0%
terran 10_vs_ 10
81.5%
6h dh_ corridor
89.9%
lerran 10_vs_11
78.3%
87.7%
terran 20 vs 20
90.4%
terran Serran_00_vs 23
96.8%
SMACV2 Protoss
SMACv2 Zerg
protoss_ 5
89.2%
7erg_5_ vs_5
98.8%
protoss_ 10_vs_1vv vs___1
91.1%
zerg zerg_10 10__s_0______ 1100_vs__ vs_10
85.3%
protoss 10 vs_ 11
90.5%
zerg. 10 ___vv_11
95.6%
protoss 20, vs 20
92.0%
zerg 20__200_v__220 _s__v_20
92.4%
protoss 20 v8_23
93.8%
zerg_ 2erg_20_vs_22
97.7%
Table 17: Ablation Study 8: Importance of Phase Labeling. We compare the final average returns of the full MisoDICE (with Phase labeling) against variant where Phase is skipped ("MisoDICE (no labeling)") and all data is treated as optimal. "BC (no labeling)" is the baseline BC trained on all data.
Scenario
BC (no laheling)
MisoDICE (with Phase 1)
MisoDICE (no labeling)
vs. vs. 652g 5m_vs__nn
8.5 + 0.1
16.4 1.3
10.9 0.3
5.0
7.3 + 0.1
5.2 + 0.1
6h_vs__8, 8z
7.0
8.7 0.2
7.2 + 0.0
corridor
1.5
5.8 0.8
1.8 - 0.2
5_vs__
9.1
12.4
9.9 1.9
Protoss
10_vs_10
10.3 0.6
12.9 0.2
10.5 = 0..
10_vs_11
8.2
10.7
9.3
20 20
1.2
13.5 0.5
11.7 0.9
20_
8.2
10.6 0.2
8.5
5_vs__
0.
9.1 0.3
6.8
Terran
10_vs_10 10
6.3
1.1
5.5
10_vs_I
0.2
6.4 0.5
4.2
20_vs, vs
6.4
9.2 0.6
7.0
20_vs_23
0.3
5.6
4.5
5_vs_5
0.5
7.5
1.1111
Zeng
10_vs_10
0.1
10.2
7.6 0.8
10
0.2
9.4
0.5
20 v0Lv__20
7.6
10.2 0.6
1.0
23
7.3
0.2
7.3
36

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Our abstract includes our main claims reflecting our main contributions and
finding.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
37

---

Justification:
We discuss limitations of the work in the conclusion section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specifiitiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texi system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: All the proofs of the theorems and propositions stated in the main paper provided in the appendix with clear references.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide details on the environments and hyper-parameter settings in the appendix. We also uploaded our source code for re-productivity purposes.
Guidelines:
38

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The data we used, along with our source code, has been uploaded with the main paper. We have also provided sufficient instructions for their use.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
Babllc//gides/Codeee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips.ccppullcc/guue htt.:///cc/uiddee///eeeubbbb
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
39

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
All the details are provided in the main paper and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We provides all the details in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provides all the details in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
40

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations in their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper develops a general algorithm for multi-agent RL. which we have tested only in simulated environments. As such, do not foresee any direct societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
functioning correctly, harms that could arise when the
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
41

---

Justification: [TODO]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We have provided clear citations the source code and data used in the
paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwwpssppppssseeeeeeeeeeeee thcode wwt.coo///aaaae com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our source code is submitted alongside the paper, accompanied by sufficient instructions. We will share the code publicly for
re-perdduce re-producibility or benchmarking purposes.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moddl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
42

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: We utilize LLMs in the first stage of our framework to help identify expert- quality trajectories from unlabeled data
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our L.LM policy httppp///aarrrpeecc///offerrrree for what should or should not be described.
43


Reviews:
[{"review_id": "qiyGoHaHBi", "reviewer": "Reviewer_VDx6", "summary": "This paper introduces MisoDICE, a novel framework for offline multi-agent imitation learning (MAIL) that addresses the challenge of unlabeled, mixed-quality demonstrations. The authors propose a two-stage solution: first, a multi-step labeling pipeline utilizes LLMs and preference-based reinforcement learning to identify expert trajectories. Second, MisoDICE, a multi-agent IL algorithm, leverages these labels and extends the single-agent DICE framework with a value decomposition and mixing architecture to learn robust policies. The research demonstrates MisoDICE's superior performance on standard MARL benchmarks and discusses its theoretical guarantees, including convex policy optimization and global-local policy consistency.", "questions": "Could preference noise lead to misleading labels that propagate into Phase 2? \n\nHave you tested how the performance of MisoDICE degrades when preference labels are noisy or contradictory?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strength\n+ MisoDICE tackles the highly challenging problem of offline MAIL using only unlabeled demonstrations of mixed quality. The proposed two stage solution is interesting. \n+ MisoDICE addresses the combinatorial explosion of joint state-action spaces in multi-agent environments by building upon the CTDE paradigm leveraging the value decomposition and a linear mixing architecture. \n+ The framework provides a theoretical analysis establishing local-global optimality consistency. \n\n### Weakness \n- MisoDICE performance heavily rely on the quality of the external preference model. Given different LLMs trained on different domains or context, the performance could be different. Though this point has been mentioned in the limitation, it is important as it being the basis of the success of this work. \n- The two stage pipeline requires multiple model components which introduces additional implementation complexity. \n- Though linear mixing provide convexity, the model might benefit from more expressive mixing strategies.", "was_revised": true, "final_justification": "Thank you for the detailed responses. I strongly encourage the authors to include the additional results in the final version of the paper. I find my original score reasonable and will keep it.", "rebuttal": "We thank the reviewer for the insightful comments. Below, we provide our detailed responses to each of your questions and concerns.\n> MisoDICE performance heavily rely on the quality of the external preference model...\n\nWe acknowledge that our method relies on a learned preference model guided by LLM-generated feedback to identify expert-like demonstrations. This design represents a significant departure from prior work, which typically assumes access to costly, high-quality expert-labeled trajectories. By leveraging LLMs as a scalable and cost-efficient supervision source, our framework enables IL from fully unlabeled datasets. This not only reduces reliance on human annotation but also broadens the applicability of imitation learning to more practical, real-world settings where labeled expert data is unavailable or difficult to collect.\n\nWhile we agree that performance heavily depends on the quality of feedback provided by the LLM, this is a common and reasonable phenomenon in the community. In fact, all existing IL algorithms similarly rely on the quality of expert demonstrations in the dataset.\n\n> The two stage pipeline requires multiple model components which introduces additional implementation complexity. \n\nWe thank the reviewer for the insightful comment. We would like to highlight that, as demonstrated through extensive ablation studies in the appendix, our two-stage framework works effectively in tandem to deliver robust training outcomes across a range of settings. Moreover, while the stages are designed to complement each other, each stage also holds independent value. Specifically, the first stage can be broadly applied to extract expert-like demonstrations from unlabeled data and is compatible with a variety of existing imitation learning algorithms. The second stage, in turn, can operate with any dataset that includes both expert and suboptimal demonstrations, regardless of how the data is labeled. Therefore, whether used jointly or independently, both stages of our framework offer meaningful contributions to the imitation learning literature.\n\n> Though linear mixing provide convexity, the model might benefit from more expressive mixing strategies.\n\nWe agree that more expressive mixing strategies—such as two-layer nonlinear mixing networks—can, in principle, enhance representational capacity. Our algorithm is designed to be general and can indeed accommodate such architectures. That said, we intentionally adopt a linear mixing network for two key reasons. First, as shown in Proposition 5.2, the linear form guarantees convexity of the training objective, which promotes more stable and reliable optimization. Second, our design is grounded in empirical evidence from recent work in MARL and multi-agent imitation learning, which consistently finds that nonlinear mixing networks are prone to overfitting and often degrade performance in offline regimes. Thus, our choice reflects a balance between expressiveness and stability, tailored for the offline learning setting\n\n> Could preference noise lead to misleading labels that propagate into Phase 2? Have you tested how the performance of MisoDICE degrades when preference labels are noisy or contradictory?\n\nYes, noisy preference signals can lead to inaccurate trajectory labeling, which in turn can degrade the performance of the imitation learning algorithm in Phase 2. To investigate the sensitivity of our framework to the quality of preference feedback, we  conducted an ablation study presented in **Section C.4** of the appendix. In this experiment, we compared two versions of the preference labeling pipeline: one using **GPT-4o** and the other using **GPT-4o-mini**. While both models are capable of generating high-level semantic judgments, GPT-4o-mini—being a lighter variant—is significantly more limited in its reasoning and contextual understanding, especially in the complex, high-dimensional environments we consider.\n\nOur results, reported in **Table 10**, reveal a clear performance gap between the two. Specifically, MisoDICE trained with GPT-4o-based labels consistently outperformed the version using GPT-4o-mini, both in terms of average returns and learning stability. This supports the hypothesis that high-quality, consistent preference feedback is critical for effective trajectory labeling and downstream policy learning. These findings also underscore the importance of choosing capable LLMs when deploying preference-based learning pipelines in practical applications.\n\nTo further address your question, we conducted an additional ablation study in which we explicitly introduced noise into the preference labeling process in Phase 1. Specifically, we obtained preference labels from GPT-4o and randomly flipped the preference outcomes for a certain percentage of trajectory pairs—at  5%, 10%, 20%, and 30% of the  samples. The results, reported below, clearly show a gradual degradation in MisoDICE’s performance as the level of noise increases, \n\n| Scenario           | MisoDICE (0%)   | MisoDICE (5%)   | MisoDICE (10%)  | MisoDICE (20%)  | MisoDICE (30%)  |\n|--------------------|-----------------|------------------|------------------|------------------|------------------|\n| 2c_vs_64zg         | 16.4 ± 1.3      | 14.7 ± 1.0       | 13.0 ± 0.6       | 11.6 ± 0.7       | 11.5 ± 0.5       |\n| 5m_vs_6m           | 7.3 ± 0.1       | 6.5 ± 1.0        | 6.1 ± 1.9        | 6.2 ± 1.3        | 6.0 ± 1.7        |\n| 6h_vs_8z           | 8.7 ± 0.2       | 8.2 ± 0.1        | 8.0 ± 0.1        | 8.1 ± 0.1        | 7.8 ± 0.2        |\n| corridor           | 5.8 ± 0.8       | 2.9 ± 0.6        | 2.6 ± 0.5        | 2.0 ± 0.3        | 2.2 ± 0.6        |\n| protoss_5_vs_5     | 12.4 ± 0.5      | 11.8 ± 1.9       | 11.5 ± 0.8       | 10.9 ± 0.1       | 10.0 ± 0.5       |\n| protoss_10_vs_10   | 12.9 ± 0.2      | 11.9 ± 0.9       | 12.5 ± 1.8       | 11.6 ± 0.3       | 11.2 ± 1.2       |\n| protoss_10_vs_11   | 10.7 ± 0.4      | 10.4 ± 0.4       | 10.3 ± 0.8       | 10.2 ± 0.6       | 9.7 ± 0.7        |\n| protoss_20_vs_20   | 13.5 ± 0.5      | 12.9 ± 0.8       | 12.6 ± 1.0       | 12.4 ± 1.0       | 12.2 ± 1.3       |\n| protoss_20_vs_23   | 10.6 ± 0.2      | 10.1 ± 0.7       | 9.9 ± 1.1        | 9.2 ± 0.9        | 9.4 ± 1.0        |\n| terran_5_vs_5      | 9.1 ± 0.3       | 8.4 ± 1.6        | 6.8 ± 0.5        | 9.0 ± 1.8        | 8.1 ± 1.4        |\n| terran_10_vs_10    | 9.1 ± 1.3       | 8.3 ± 1.9        | 7.6 ± 0.8        | 7.1 ± 0.8        | 7.2 ± 1.1        |\n| terran_10_vs_11    | 6.4 ± 0.5       | 6.6 ± 0.6        | 6.2 ± 0.9        | 6.0 ± 0.7        | 6.0 ± 1.2        |\n| terran_20_vs_20    | 9.2 ± 0.6       | 8.5 ± 0.7        | 8.4 ± 0.7        | 8.1 ± 1.1        | 7.8 ± 0.9        |\n| terran_20_vs_23    | 5.6 ± 0.4       | 5.5 ± 0.4        | 5.2 ± 0.6        | 5.0 ± 0.5        | 4.9 ± 0.5        |\n| zerg_5_vs_5        | 7.5 ± 0.1       | 7.4 ± 0.9        | 7.2 ± 1.2        | 6.3 ± 0.4        | 6.5 ± 1.6        |\n| zerg_10_vs_10      | 10.2 ± 0.6      | 9.3 ± 0.5        | 8.2 ± 0.5        | 8.7 ± 1.3        | 8.0 ± 1.4        |\n| zerg_10_vs_11      | 9.4 ± 0.3       | 9.5 ± 1.3        | 8.8 ± 0.3        | 8.4 ± 0.8        | 7.5 ± 0.5        |\n| zerg_20_vs_20      | 10.2 ± 0.6      | 9.5 ± 0.3        | 9.2 ± 0.7        | 9.0 ± 0.6        | 8.5 ± 0.9        |\n| zerg_20_vs_23      | 9.5 ± 0.2       | 9.4 ± 0.9        | 8.9 ± 0.7        | 8.5 ± 0.3        | 8.1 ± 1.1        |\n\n---\n\n*We hope the above responses satisfactorily address your concerns. We will update the paper to include the additional experiments  and discussion described above. If you have any further comments or questions, we would be happy to address them.*"}, {"review_id": "6yd5CjIXqL", "reviewer": "Reviewer_eERK", "summary": "The paper proposes multi-agent imitation learning in a cooperative setting containing mixed quality demonstrations - expert & non-expert. The authors propose a two-step approach where LLMs and preference based RL is used to label the demonstrations and then these are used likewise to train an offline imitation learning policy. The problem setup is novel and really important.", "questions": "1. What is the intuition behind R~Q-\\gamma V in section 4? Is this advantage estimation that you are doing?\n2. Does the rankings of the trajectory done in Phase-1 align with ground truth (actual rankings) ? This would help to understand how good is O-MAPL in recovering an appropriate reward function from noisy preference data.\n3. How often are the 2 mixing networks trained in this framework? How often are the local networks trained? What objective function  is used to train the 2 mixing networks?\n4. Though the evaluation returns are better for MIsoDice, the training plot seems not that statistically significant. I am curious as to why MisoDICE is almost similar in training performance in smaller configurations (Fig 1; 5v5; 10Vs 10) than bigger ones?\n5. I could not find MisoDICE-VDN as mentioned in Baselines in the Experimental plots.\n6. I would have liked to see an ablation study without phase-1; i.e: getting demonstrations and using them all as optimal demonstrations along with Phase-2 to understand how important the pretraining process is in this framework.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n\n1. Problem setting is novel, learning multi-agent IL policies in settings where demonstrations are of mixed quality is an interesting and important problem; this also occurs very often due to scarcity of expert demonstrations\n2. Related work is well written; positioning the work well in the light of prior work\n3. Experimental results are well conducted on standard MARL benchmarks of increasing difficulty, showing reasonable advantage of the proposed approach over baselines.\n\nWeakness:\n\n1. Though the paper uses techniques from prior work (like O-MAPL); they are not mentioned in the Background section making it difficult to follow specific details.\n2. The first phase of this work is summarized briefly, without providing sufficient details. If LLM provides noisy preference data, I am not sure how O-MAPL learns a soft Q-function? Can it filter out the noisy preferences?\n3. The authors use local network to learn local value functions; but I am not sure how well this would scale up for large number of agents and the no. of local value functions would be large.\n4. I do not understand the intuition behind discriminator based classification problem (looks like cross-entropy loss) as in Eqn 9? What is the intuition behind posing the estimation problem as a supervised classification problem?\n5. Though the problem statement is interesting, I think it is too computationally intensive to be able to scale up to large number of agents. There are 3 different local networks per agent and 2 different mixing networks making the overall framework extremely complicated and intensive.\n6. There is a trade off between the amount of gain achieved in terms of performance versus the computational overhead of the proposed approach; thus I am skeptical about it’s utility for furthering research. IT looks like VDN also learns pretty well in this setting.", "was_revised": true, "final_justification": "I thank the authors who clearly addressed my concerns and performed additional experiments/ablation to validate some of my concerns. I would like to increase my score post the rebuttal because this is a nice idea and has sufficient experimental evaluations /ablations to support the claims.", "rebuttal": "We thank the reviewer for the insightful comments. Below, we provide our detailed responses to each of your questions and concerns.\n>Though the paper uses techniques from prior work (like O-MAPL); they are not mentioned in the Background section…\n\nWe will revise the Background section to better highlight and clarify the importance of preference-based methods (such as O-MAPL) in our framework.\n>The first phase of this work is summarized briefly, without providing sufficient details. \n\nWe thank the reviewer for the comment. We agree that the details of Phase 1 could be more thoroughly explained in the main paper. However, due to space constraints, we chose to move some of the descriptions to the appendix, in order to allocate more space in the main text to Phase 2, which involves more technical depth.\n> If LLM provides noisy preference data, I am not sure how O-MAPL learns a soft Q-function? …\n\nWhen the feedback is noisy or less accurate, the ability of O-MAPL to correctly distinguish expert-like trajectories deteriorates, which in turn negatively affects the downstream performance in Phase 2. To empirically validate this, we already conducted an ablation study in **Section C.4** comparing **GPT-4o** and **GPT-4o-mini** for preference labeling. Due to its limited reasoning ability, GPT-4o-mini produced noisier feedback, leading to lower labeling accuracy and degraded Phase 2 performance.\n\nAdditionally, we ran an additional ablation where we introduced synthetic noise by randomly flipping GPT-4o-generated preferences at varying rates. The results, shown below, demonstrate a clear decline in MisoDICE’s performance as noise increases.\n\n| Scenario| MisoDICE (0%)| MisoDICE (5%)| MisoDICE (10%)| MisoDICE (20%)| MisoDICE (30%)|\n|--------------------|-----------------|------------------|------------------|------------------|------------------|\n| 2c_vs_64zg | 16.4 ± 1.3 | 14.7 ± 1.0| 13.0 ± 0.6| 11.6 ± 0.7| 11.5 ± 0.5|\n| 5m_vs_6m| 7.3 ± 0.1| 6.5 ± 1.0| 6.1 ± 1.9| 6.2 ± 1.3| 6.0 ± 1.7|\n| 6h_vs_8z| 8.7 ± 0.2| 8.2 ± 0.1| 8.0 ± 0.1| 8.1 ± 0.1| 7.8 ± 0.2|\n| corridor| 5.8 ± 0.8| 2.9 ± 0.6| 2.6 ± 0.5| 2.0 ± 0.3| 2.2 ± 0.6|\n| protoss_5_vs_5| 12.4 ± 0.5 | 11.8 ± 1.9| 11.5 ± 0.8| 10.9 ± 0.1| 10.0 ± 0.5|\n| protoss_10_vs_10| 12.9 ± 0.2 | 11.9 ± 0.9| 12.5 ± 1.8| 11.6 ± 0.3| 11.2 ± 1.2|\n| protoss_10_vs_11| 10.7 ± 0.4 | 10.4 ± 0.4| 10.3 ± 0.8| 10.2 ± 0.6| 9.7 ± 0.7|\n| protoss_20_vs_20| 13.5 ± 0.5 | 12.9 ± 0.8| 12.6 ± 1.0| 12.4 ± 1.0| 12.2 ± 1.3|\n| protoss_20_vs_23| 10.6 ± 0.2 | 10.1 ± 0.7| 9.9 ± 1.1| 9.2 ± 0.9| 9.4 ± 1.0|\n| terran_5_vs_5 | 9.1 ± 0.3| 8.4 ± 1.6| 6.8 ± 0.5| 9.0 ± 1.8| 8.1 ± 1.4|\n| terran_10_vs_10 | 9.1 ± 1.3| 8.3 ± 1.9| 7.6 ± 0.8| 7.1 ± 0.8| 7.2 ± 1.1|\n| terran_10_vs_11 | 6.4 ± 0.5| 6.6 ± 0.6| 6.2 ± 0.9| 6.0 ± 0.7| 6.0 ± 1.2|\n| terran_20_vs_20 | 9.2 ± 0.6| 8.5 ± 0.7| 8.4 ± 0.7| 8.1 ± 1.1| 7.8 ± 0.9|\n| terran_20_vs_23 | 5.6 ± 0.4| 5.5 ± 0.4| 5.2 ± 0.6| 5.0 ± 0.5| 4.9 ± 0.5|\n| zerg_5_vs_5| 7.5 ± 0.1| 7.4 ± 0.9| 7.2 ± 1.2| 6.3 ± 0.4| 6.5 ± 1.6|\n| zerg_10_vs_10 | 10.2 ± 0.6 | 9.3 ± 0.5| 8.2 ± 0.5| 8.7 ± 1.3| 8.0 ± 1.4|\n| zerg_10_vs_11 | 9.4 ± 0.3| 9.5 ± 1.3| 8.8 ± 0.3| 8.4 ± 0.8| 7.5 ± 0.5|\n| zerg_20_vs_20 | 10.2 ± 0.6 | 9.5 ± 0.3| 9.2 ± 0.7| 9.0 ± 0.6| 8.5 ± 0.9|\n| zerg_20_vs_23 | 9.5 ± 0.2| 9.4 ± 0.9| 8.9 ± 0.7| 8.5 ± 0.3| 8.1 ± 1.1|\n\n> There are 3 different local networks per agent and 2 different mixing networks making the overall framework extremely complicated and intensive…\n\nWe thank the reviewer for the insightful comment. In fact, our use of local networks with mixing architectures under the CTDE framework is explicitly designed to enhance scalability and efficiency in multi-agent learning—a strategy also adopted by recent SOTA MARL and IL methods. Our experiments confirm that MisoDICE scales well, even on the most challenging SMAC_v2 scenarios.\n\nTo further support our argument regarding the computational efficiency, we will include the following table summarizing the total training time for each benchmark environment. The results clearly demonstrate that MisoDICE requires a reasonable amount of training time.\n\n| Scenario| Time |\n|--------------------|-----------|\n| SMACv1| 2.30h|\n| SMACv2 (protoss)| 4.19h|\n| SMACv2 (terran) | 3.50h|\n| SMACv2 (zerg) | 2.95h|\n\n> I do not understand the intuition behind discriminator based classification problem (looks like cross-entropy loss) as in Eqn 9? …\n\nWe thank the reviewer for the question. The purpose of the discriminator in Eq. 9 is to estimate the occupancy ratio, which is a critical component of our learning objective. This classifier-based approach can be shown to recover the occupancy ratio between expert and unlabeled datasets. Similar techniques have been widely adopted in prior IL algorithms, such as GAIL, where the discriminator serves a comparable role in distinguishing between expert and non-expert trajectories.\n\n> There is a trade off between the amount of gain achieved in terms of performance versus the computational overhead…  VDN also learns pretty well in this setting.\n\nWe thank the reviewer for the comment. We agree that there is an inherent trade-off between performance gains and computational efficiency, which is a common phenomenon in ML. In our context, the VDN variant serves as a simplified version of MisoDICE. As expected, its performance is slightly lower than MisoDICE, but better than other baselines. \n\n> What is the intuition behind R~Q-\\gamma V in section 4?...\n\nThis approach is adapted from the O-MAPL paper on preference-based learning, where the expression $Q−\\gamma V$ is interpreted as a reward function defined in the Q-value space. By learning this reward function implicitly through the Q and V functions, we enable a more consistent and integrated way of learning both the reward and policy. \n> Does the rankings of the trajectory done in Phase-1 align with ground truth?...\n\nWe thank the reviewer for the insightful question. In response, we conducted an additional experiment in which we sampled 2,000 trajectory pairs per task and generated preference rewards using LLM feedback and O-MAPL. These were then compared to ground-truth preferences derived from environment rewards. The results, reported below, indicate an overall accuracy of 90.5%, demonstrating the high reliability of the LLM feedback in Phase 1. This highlights the effectiveness of our approach in Phase 1 in filtering expert trajectories that closely align with the ground-truth preferences.\n\n| Scenario| Accuracy |\n|---------------------|-----------|\n| 2c_vs_64zg | 79.2% |\n| 5m_vs_6m| 87.0% |\n| 6h_vs_8z| 89.9% |\n| corridor| 87.7% |\n| protoss_5_vs_5  | 89.2% |\n| protoss_10_vs_10| 91.1% |\n| protoss_10_vs_11| 90.5% |\n| protoss_20_vs_20| 92.0% |\n| protoss_20_vs_23| 93.8% |\n| terran_5_vs_5| 90.4% |\n| terran_10_vs_10 | 81.5% |\n| terran_10_vs_11 | 78.3% |\n| terran_20_vs_20 | 90.4% |\n| terran_20_vs_23 | 96.8% |\n| zerg_5_vs_5| 98.8% |\n| zerg_10_vs_10| 85.3% |\n| zerg_10_vs_11| 95.6% |\n| zerg_20_vs_20| 92.4% |\n| zerg_20_vs_23| 97.7% |\n> How often are the 2 mixing networks trained in this framework?….\n\nWe thank the reviewer for the question. The mixing network and local networks are trained simultaneously via gradient backpropagation. The main training objective for jointly updating both components is provided in **Eq.6**. A detailed description of the training procedure can be found in **Section B.5** of the appendix, along with a visual illustration in **Figure 4**.\n\n> I am curious as to why MisoDICE is almost similar in training performance in smaller configurations (Fig 1; 5v5; 10Vs 10) than bigger ones?\n\nWe thank the reviewer for the question. On small-scale tasks, MisoDICE and other baselines show similar performance due to low task complexity. While training plots may appear noisy, closer inspection shows that MisoDICE consistently outperforms the baselines.\n\n> I could not find MisoDICE-VDN as mentioned in Baselines in the Experimental plots.\n\nMisoDICE-VDN is actually VDN reported in our main experiments. We included it as a lightweight variant to highlight the modularity of our approach. We will clarify this in the revised version of the paper.\n\n> I would have liked to see an ablation study without phase-1; …\n\nWe thank the reviewer for the suggestion. In response, we ran an ablation where all unlabeled data were treated as optimal for Phase 2. Results show that MisoDICE without labeling performs worse than the full version but slightly better than BC, highlighting the importance of Phase 1's labeling strategy for learning from noisy, unlabeled data.\n\n| Scenario| BC (no labeling)| MisoDICE (with Phase 1) | MisoDICE (no labeling) |\n|--------------------|----------------|------------------|--------------------------|\n| 2c_vs_64zg | 8.5 ± 0.1 | 16.4 ± 1.3| 10.9 ± 0.3|\n| 5m_vs_6m | 5.0 ± 1.1 | 7.3 ± 0.1| 5.2 ± 0.1 |\n| 6h_vs_8z | 7.0 ± 0.0 | 8.7 ± 0.2| 7.2 ± 0.0 |\n| corridor | 1.5 ± 0.1 | 5.8 ± 0.8| 1.8 ± 0.2 |\n| protoss_5_vs_5 | 9.2 ± 0.1 | 12.4 ± 0.5| 9.9 ± 1.9 |\n| protoss_10_vs_10 | 10.3 ± 0.6| 12.9 ± 0.2| 10.5 ± 0.7|\n| protoss_10_vs_11 | 8.2 ± 0.4 | 10.7 ± 0.4| 9.3 ± 0.3 |\n| protoss_20_vs_20 | 10.1 ± 0.2| 13.5 ± 0.5| 11.7 ± 0.9|\n| protoss_20_vs_23 | 8.1 ± 0.2 | 10.6 ± 0.2| 8.5 ± 0.5 |\n| terran_5_vs_5 | 6.5 ± 0.8 | 9.1 ± 0.3| 6.8 ± 0.8 |\n| terran_10_vs_10 | 6.6 ± 0.3 | 9.1 ± 1.3| 5.5 ± 1.3 |\n| terran_10_vs_11 | 4.7 ± 0.2 | 6.4 ± 0.5| 4.2 ± 0.6 |\n| terran_20_vs_20 | 6.9 ± 0.4 | 9.2 ± 0.6| 7.0 ± 0.4 |\n| terran_20_vs_23 | 4.0 ± 0.3 | 5.6 ± 0.4| 4.5 ± 0.7 |\n| zerg_5_vs_5 | 5.7 ± 0.5 | 7.5 ± 0.1| 5.7 ± 1.1 |\n| zerg_10_vs_10 | 7.3 ± 0.1 | 10.2 ± 0.6| 7.6 ± 0.8 |\n| zerg_10_vs_11 | 7.3 ± 0.2 | 9.4 ± 0.3| 7.7 ± 0.5 |\n| zerg_20_vs_20 | 7.4 ± 0.6 | 10.2 ± 0.6| 7.8 ± 1.0 |\n| zerg_20_vs_23 | 7.1 ± 0.3 | 9.5 ± 0.2| 7.3 ± 0.5 |\n\n---\n\n*We hope the above responses satisfactorily address your concerns. We will update the paper to include the additional experiments and discussion described above. If you have any further comments or questions, we would be happy to address.*"}, {"review_id": "UtibACFksa", "reviewer": "Reviewer_7ov1", "summary": "The authors propose a novel framework for cooperative multi-agent offline imitation learning from unlabeled mixed-quality demonstrations. It is structured in two stages: trajectory labeling and multi-agent imitation learning. By extending the popular single-agent DICE framework to multi-agent settings with a new value decomposition and mixing architecture, their method yields a convex policy optimization objective and ensures consistency between global and local policies.", "questions": "see the weakness", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths：\n1、The paper tackles the underexplored challenge of multi-agent IL with unlabeled mixed-quality demonstrations. The two-stage design is an innovative solution, effectively bridging preference learning and occupancy matching.\n2、The technical core is strong. The convexity analysis of the objective under linear mixing is a crucial insight ensuring stable optimization. The proofs of global-local policy consistency and the connection between local policies and value functions provide theoretical grounding for the CTDE approach and decentralized execution.\n\nWeakness：\n1、The labeling phase relies on domain-specific semantic features (e.g., unit health in SMAC) for LLM feedback. For domains without interpretable features (e.g., MaMuJoCo), the proposed \"rule-based oracle\" lacks details. How was the rule-based oracle designed for MaMuJoCo? Provide algorithmic details and its performance vs. LLM in SMAC. If it requires domain-specific heuristics, does this undermine the framework’s generality?\n2、How sensitive is the labeling phase to the quality and specificity of the LLM prompts? Could low-quality or overly generic LLM outputs significantly degrade O-MAPL refinement and subsequent IL performance? \n3、 The author claims to have solved the computational complexity of the large-scale joint state-action space, but lacks relevant theoretical or experimental proof.\n4、Proposition 5.5 assumes that behavioral strategies can be decomposed into independent local strategies, but the behaviors of agents in cooperative tasks are usually correlated. Please demonstrate the rationality of this assumption.\ns", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments. Below, we provide our detailed responses to each of your questions and concerns.\n> The labeling phase relies on domain-specific semantic features ... How was the rule-based oracle designed for MaMuJoCo? ...If it requires domain-specific heuristics, does this undermine the framework’s generality?\n\nWe thank the reviewer for the insightful question. First, to clarify, the \"rule-based oracle\" used for the MAMuJoCo tasks is derived from the ground-truth reward function. In this domain, we assume access to expert feedback to generate preference labels, as LLMs are not well-suited for interpreting raw low-level state representations typical of MuJoCo environments.\n\nIn our framework, Phase 2 is more general, as it can be applied to settings where expert and suboptimal demonstrations are available, regardless of how the expert data is identified—either manually or via LLMs. In contrast, Phase 1 is somewhat less general, as it relies on the availability of contextual and interpretable features that can be understood by LLMs. However, we believe Phase 1 still offers independent value: to the best of our knowledge, this is the first approach that leverages LLMs to identify expert-like demonstrations from fully unlabeled and potentially noisy datasets. This opens promising directions for future work, particularly in exploring how to adapt LLM-based preference labeling to less interpretable domains like MuJoCo through better feature engineering or learned state abstractions.\n\n> How sensitive is the labeling phase to the quality and specificity of the LLM prompts? Could low-quality or overly generic LLM outputs significantly degrade O-MAPL refinement and subsequent IL performance? \n\nWe thank the reviewer for the insightful question. As noted in the appendix, we have already conducted an ablation study to analyze the sensitivity of MisoDICE to the quality of preference feedback in Phase 1. Specifically, in **Section C.4** of the appendix, we compare two versions of the preference labeling pipeline: one using **GPT-4o** and the other using **GPT-4o-mini**. Since **GPT-4o-mini** is a smaller model with limited reasoning and contextual understanding capabilities, it tends to generate lower-quality preference signals. Our results show that this degradation in feedback quality leads to less accurate labeling and significantly reduces the overall performance of MisoDICE in Phase 2.\n\nWe have also conducted an additional ablation study to analyze the impact of noisy preference feedback in Phase 1 on the overall performance of MisoDICE. In this experiment, we started with preference data generated by GPT-4o and randomly flipped a portion of the preference labels at varying levels (5%, 10%, 20%, etc.). The results, reported in the table below, clearly illustrate that increasing the amount of noise in the preference data leads to a noticeable decline in MisoDICE's performance, highlighting the framework’s sensitivity to label quality in the preference-based learning phase.\n\n| Scenario| MisoDICE (0%)| MisoDICE (5%)| MisoDICE (10%)| MisoDICE (20%)| MisoDICE (30%)|\n|--------------------|-----------------|------------------|------------------|------------------|------------------|\n| 2c_vs_64zg | 16.4 ± 1.3 | 14.7 ± 1.0| 13.0 ± 0.6| 11.6 ± 0.7| 11.5 ± 0.5|\n| 5m_vs_6m| 7.3 ± 0.1| 6.5 ± 1.0| 6.1 ± 1.9| 6.2 ± 1.3| 6.0 ± 1.7|\n| 6h_vs_8z| 8.7 ± 0.2| 8.2 ± 0.1| 8.0 ± 0.1| 8.1 ± 0.1| 7.8 ± 0.2|\n| corridor| 5.8 ± 0.8| 2.9 ± 0.6| 2.6 ± 0.5| 2.0 ± 0.3| 2.2 ± 0.6|\n| protoss_5_vs_5| 12.4 ± 0.5 | 11.8 ± 1.9| 11.5 ± 0.8| 10.9 ± 0.1| 10.0 ± 0.5|\n| protoss_10_vs_10| 12.9 ± 0.2 | 11.9 ± 0.9| 12.5 ± 1.8| 11.6 ± 0.3| 11.2 ± 1.2|\n| protoss_10_vs_11| 10.7 ± 0.4 | 10.4 ± 0.4| 10.3 ± 0.8| 10.2 ± 0.6| 9.7 ± 0.7|\n| protoss_20_vs_20| 13.5 ± 0.5 | 12.9 ± 0.8| 12.6 ± 1.0| 12.4 ± 1.0| 12.2 ± 1.3|\n| protoss_20_vs_23| 10.6 ± 0.2 | 10.1 ± 0.7| 9.9 ± 1.1| 9.2 ± 0.9| 9.4 ± 1.0|\n| terran_5_vs_5 | 9.1 ± 0.3| 8.4 ± 1.6| 6.8 ± 0.5| 9.0 ± 1.8| 8.1 ± 1.4|\n| terran_10_vs_10 | 9.1 ± 1.3| 8.3 ± 1.9| 7.6 ± 0.8| 7.1 ± 0.8| 7.2 ± 1.1|\n| terran_10_vs_11 | 6.4 ± 0.5| 6.6 ± 0.6| 6.2 ± 0.9| 6.0 ± 0.7| 6.0 ± 1.2|\n| terran_20_vs_20 | 9.2 ± 0.6| 8.5 ± 0.7| 8.4 ± 0.7| 8.1 ± 1.1| 7.8 ± 0.9|\n| terran_20_vs_23 | 5.6 ± 0.4| 5.5 ± 0.4| 5.2 ± 0.6| 5.0 ± 0.5| 4.9 ± 0.5|\n| zerg_5_vs_5| 7.5 ± 0.1| 7.4 ± 0.9| 7.2 ± 1.2| 6.3 ± 0.4| 6.5 ± 1.6|\n| zerg_10_vs_10 | 10.2 ± 0.6 | 9.3 ± 0.5| 8.2 ± 0.5| 8.7 ± 1.3| 8.0 ± 1.4|\n| zerg_10_vs_11 | 9.4 ± 0.3| 9.5 ± 1.3| 8.8 ± 0.3| 8.4 ± 0.8| 7.5 ± 0.5|\n| zerg_20_vs_20 | 10.2 ± 0.6 | 9.5 ± 0.3| 9.2 ± 0.7| 9.0 ± 0.6| 8.5 ± 0.9|\n| zerg_20_vs_23 | 9.5 ± 0.2| 9.4 ± 0.9| 8.9 ± 0.7| 8.5 ± 0.3| 8.1 ± 1.1|\n\n> The author claims to have solved the computational complexity of the large-scale joint state-action space, but lacks relevant theoretical or experimental proof. \n\nWe thank the reviewer for the insightful question. Our design choices—including value factorization and local network construction—are specifically aimed at addressing the computational complexity inherent in multi-agent settings, particularly when dealing with large-scale joint state-action spaces. This approach has been adopted in several recent state-of-the-art MARL methods and has been shown to significantly improve computational scalability and efficiency. We will clarify this point more explicitly in the revised version of the paper.\n\n\n> Proposition 5.5 assumes that behavioral strategies can be decomposed into independent local strategies, but the behaviors of agents in cooperative tasks are usually correlated. Please demonstrate the rationality of this assumption. \n\nWe thank the reviewer for the insightful question. This is actually a common assumption in the existing MARL literature. To clarify, we do not assume that the global policy is decomposable into separate independent local policies (The local policies are not necessarily independent.). Rather, our framework relies on the assumption of consistency between global and local policies—specifically, that the global policy can be expressed as the product of local policies. This formulation captures a synergistic relationship, where coordinated local behaviors collectively give rise to the global strategy, rather than treating each agent’s policy in isolation.\n\n--- \n\n*We hope the above responses satisfactorily address your concerns. We will update the paper to include the additional experiments and discussion described above. If you have any further comments or questions, we would be happy to address them*"}, {"review_id": "oBPYM7MPAz", "reviewer": "Reviewer_YWXt", "summary": "This paper addresses the problem of imitation learning in cooperative multi-agent settings from a dataset of unlabeled, mixed-quality trajectories. The proposed method, MisoDICE, is a two-stage framework. The first stage employs a pipeline combining Large Language Models (LLMs) and preference-based reinforcement learning to identify and label expert-quality trajectories from the mixed data. The second stage introduces MisoDICE, a novel multi-agent imitation learning algorithm that leverages these labels. By extending the single-agent DICE framework with a new value decomposition architecture, the method derives a convex optimization objective and theoretically ensures consistency between the global joint policy and local agent policies. The effectiveness of the proposed method is evaluated on standard benchmarks such as the StarCraft Multi-Agent Challenge (SMAC).", "questions": "-Clarification on BC Performance: In Figure 1, the performance of Behavioral Cloning (BC) appears to increase over the course of training. Since BC is typically a supervised learning method whose performance on a fixed offline dataset should not change with training epochs, could you clarify what \"training\" entails for the BC baseline and why its performance curve shows an upward trend?\n\n-Justification of the Two-Stage Framework: The paper's main theoretical contribution appears to be the MisoDICE algorithm (Stage 2), which is elegant and well-motivated. However, it is tightly coupled with the LLM-based labeling pipeline (Stage 1), which seems less general and brittle, as evidenced by its failure on MAMuJoCo. Could the authors elaborate on the necessity of this tight coupling? Would the paper's contribution be clearer and more robust if Stage 2 (MisoDICE) were presented as the core method, with the LLM-based approach being just one possible (and domain-dependent) method for generating the required labeled data?\n\n- Positioning of Imitation Learning: The paper begins by stating, \"Imitation Learning (IL) is a key subfield of Reinforcement Learning (RL).\" This is a strong claim that may not be universally accepted, as many researchers treat IL and RL as related but distinct paradigms. It would be beneficial to elaborate on the positioning of IL relative to RL. A more nuanced discussion of their relationship and how this work fuses concepts from both would strengthen the paper's framing, especially considering existing studies that describe the theoretical fusion of RL and IL (e.g., [Kinose+ 2020] in robot learning).\n\nKinose, A., & Taniguchi, T. (2020). Integration of imitation learning using GAIL and reinforcement learning using task-achievement rewards via probabilistic graphical model. Advanced Robotics, 34(16), 1055-1067.", "limitations": "Yes, the authors have a section discussing limitations. However, the limitation regarding the LLM-based labeling stage could be addressed more deeply. The paper acknowledges that the LLM approach struggles in environments like MAMuJoCo, but it doesn't fully discuss the implications this has for the overall framework's claimed generality. The LLM labeling is presented as a key part of the contribution in Section 4, but the experiments suggest it is not universally applicable. I would encourage the authors to be more upfront about this limitation and discuss how the MisoDICE algorithm (Stage 2) can be used with other, potentially simpler or more general, methods for data labeling. This would strengthen the paper by focusing on its most robust and generalizable contribution.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "*Strengths\n- Originality and Significance: The paper tackles a practical and important problem: offline multi-agent imitation learning where demonstration quality is heterogeneous and unknown. This is a significant and relatively under-explored area, and the proposed problem setting is a valuable contribution.\n- Theoretical Quality: The extension of the DICE framework to the multi-agent setting is theoretically sound. A key contribution is the use of a linear mixing network to maintain the convexity of the learning objective. This is a crucial choice that ensures stability and efficient optimization. The use of Lagrange multipliers for decomposition in this context is also an interesting approach.\n- Empirical Results: While the validation might be considered minimal in scope, the paper demonstrates that MisoDICE outperforms baselines on standard, challenging benchmarks, providing evidence for the method's effectiveness.\n\n*Weaknesses\n- Dependence on LLMs and Generality of the Framework: The paper's core contribution feels split between the LLM-based labeling pipeline (Stage 1) and the MisoDICE algorithm (Stage 2). The first stage's reliance on LLMs for preference generation seems to lack generality. The authors themselves note that this approach fails on tasks like MAMuJoCo, where semantic context is poor, forcing them to use a rule-based oracle. This undermines the claim that LLM-based labeling is a core, generalizable part of the contribution and raises questions about the framework's applicability to common imitation learning domains like robotic manipulation. The two stages feel disconnected, and the paper might be stronger if it focused on the more robust contribution of the MisoDICE algorithm itself.\n- Clarity of Positioning and Presentation:\nThe paper's positioning of imitation learning (IL) as a subfield of reinforcement learning (RL) could be more nuanced. While related, they are often treated as distinct paradigms. A more detailed discussion of their relationship would clarify the paper's contribution.\n- The presentation of the baselines is potentially misleading. Although methods like \"VDN\" in Figure 1 look like baseline methods, thye are clearly ablation studies of the proposed method. They should be explicitly framed as such to avoid confusion and clearly delineate the contributions of each component of the proposed architecture.", "was_revised": true, "final_justification": "I appreciate the authors’ efforts to clarify and refine their contributions, but the concerns regarding the limited generality and robustness of the LLM-based component remain unresolved, so my overall evaluation is unchanged.", "rebuttal": "We thank the reviewer for the insightful comments. Below, we provide our detailed responses to each of your questions and concerns.\n\n> Dependence on LLMs and Generality of the Framework: ... This undermines the claim that LLM-based labeling is a core, generalizable part of the contribution and raises questions about the framework's applicability to common imitation learning domains like robotic manipulation. The two stages feel disconnected, and the paper might be stronger if it focused on the more robust contribution of the MisoDICE algorithm itself.\n\nWe appreciate the reviewer’s insightful observation regarding the broader applicability of our algorithmic contributions. We agree that the LLM-based component is currently more suited to domains with interpretable features. Nonetheless, we believe that both components of our framework represent meaningful advancements. Our use of LLMs to identify expert-like demonstrations from fully unlabeled datasets is, to the best of our knowledge, a novel and impactful contribution that opens promising avenues for preference-based learning. In particular, it paves the way for future work on leveraging LLMs to identify expert behavior in more challenging and less interpretable environments, such as MAMuJoCo.\n\nImportantly, our core algorithmic design in Phase 2 is general and agnostic to how expert demonstrations are obtained, making it broadly applicable across various settings. In the revised version of the paper, we will refine our contribution statement to more clearly articulate the generality and standalone value of this component.\n\n> Clarity of Positioning and Presentation: The paper's positioning of imitation learning (IL) as a subfield of reinforcement learning (RL) could be more nuanced....\n\nWe thank the reviewer for the insightful feedback. We recognize the importance of clearly distinguishing between IL and RL, especially given the nuanced differences in their problem settings, learning objectives, and assumptions. While RL focuses on learning optimal behavior through interaction with an environment and explicit reward signals, IL seeks to learn policies by mimicking expert behavior, often in the absence of reward functions. In the revised version of the paper, we will revise the presentation to explicitly highlight this distinction—both in the introduction and background sections.\n\n> ... Although methods like \"VDN\" in Figure 1 look like baseline methods, they are clearly ablation studies of the proposed method. ...\n\nWe thank the reviewer for the insightful observation. You are correct that VDN, as used in our experiments, is a simplified adaptation of our main algorithm, MisoDICE, rather than a standalone existing baseline. We will revise the manuscript to clarify this distinction and avoid any potential confusion.\n\n> Clarification on BC Performance: In Figure 1, the performance of Behavioral Cloning (BC) appears to increase over the course of training. ...\n\nWe thank the reviewer for the insightful question. The performance curves shown in our experiments reflect evaluation during the policy update process. Specifically, we use batch samples from the offline dataset to update the policy using BC, and after a certain number of epochs, we evaluate the policy’s performance. We will clarify this evaluation procedure more explicitly in the experimental section.\n\n> Justification of the Two-Stage Framework: The paper's main theoretical contribution appears to be the MisoDICE algorithm (Stage 2), which is elegant and well-motivated. However, it is tightly coupled with the LLM-based labeling pipeline (Stage 1), which seems less general and brittle, as evidenced by its failure on MAMuJoCo. Could the authors elaborate ...? Would the paper's contribution be clearer and more robust if Stage 2 (MisoDICE) were presented as the core method, with the LLM-based approach being just one possible (and domain-dependent) method for generating the required labeled data?\n\nWe thank the reviewer for the insightful comments and thoughtful question, with which we fully agree. The close coupling between Phase 1 and Phase 2 in our framework stems from the original motivation of the work—*tackling imitation learning in the challenging setting where the dataset is entirely unlabeled*. In this context, Phase 1 plays a critical role, as effective imitation learning cannot proceed without first identifying expert-like demonstrations.\n\nWhile we agree that Phase 2 is more general and may have broader standalone applicability—making it particularly relevant to some readers—we believe that Phase 1 also offers important contributions. To the best of our knowledge, it is the first approach that leverages LLMs to identify expert-like trajectories from noisy, unlabeled data. This opens promising future directions, including extending such techniques to less interpretable domains like MuJoCo, potentially leading to broader impact in preference-based and offline IL.\n\n---\n\n*We hope the above responses satisfactorily address your concerns. We will update the paper to include the discussion described above. If you have any further comments or questions, we would be happy to address them.*"}]

Author rebuttal:

Meta-review: The paper introduces a framework for multi-agent imitation learning in cooperative settings with mixed-quality demonstrations. The proposed two-step approach uses large language models and preference-based reinforcement learning to label demonstrations, which are then applied to train an offline imitation learning policy. The method, MisoDICE, is evaluated on standard MARL benchmarks, showing improved performance and providing theoretical guarantees such as convex policy optimization and global–local policy consistency.

After discussion, most reviewers concerns are addressed and all recommended acceptance.
Decision: Accept (poster)