Title: MisoDICE: Multi-Agent Imitation from Mixed-Quality Demonstrations
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multi-Agent Reinforcement Learning

Offline Imitation Learning

Large Language Models

Value Factorization
Abstract: We study offline imitation learning (IL) in cooperative multi-agent settings, where demonstrations have unlabeled mixed quality - containing both expert and suboptimal trajectories. Our proposed solution is structured in two stages: trajectory labeling and multi-agent imitation learning, designed jointly to enable effective learning from heterogeneous, unlabeled data. In the first stage, we combine advances in large language models and preference-based reinforcement learning to construct a progressive labeling pipeline that distinguishes expert-quality trajectories. In the second stage, we introduce MisoDICE, a novel multi-agent IL algorithm that leverages these labels to learn robust policies while addressing the computational complexity of large joint state-action spaces. By extending the popular single-agent DICE framework to multi-agent settings with a new value decomposition and mixing architecture, our method yields a convex policy optimization objective and ensures consistency between global and local policies. We evaluate MisoDICE on multiple standard multi-agent RL benchmarks and demonstrate superior performance, especially when expert data is scarce.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of imitation learning in cooperative multi-agent settings from a dataset of unlabeled, mixed-quality trajectories. The proposed method, MisoDICE, is a two-stage framework. The first stage employs a pipeline combining Large Language Models (LLMs) and preference-based reinforcement learning to identify and label expert-quality trajectories from the mixed data. The second stage introduces MisoDICE, a novel multi-agent imitation learning algorithm that leverages these labels. By extending the single-agent DICE framework with a new value decomposition architecture, the method derives a convex optimization objective and theoretically ensures consistency between the global joint policy and local agent policies. The effectiveness of the proposed method is evaluated on standard benchmarks such as the StarCraft Multi-Agent Challenge (SMAC).\n\nSTRENGTHS AND WEAKNESSES:\n*Strengths\n- Originality and Significance: The paper tackles a practical and important problem: offline multi-agent imitation learning where demonstration quality is heterogeneous and unknown. This is a significant and relatively under-explored area, and the proposed problem setting is a valuable contribution.\n- Theoretical Quality: The extension of the DICE framework to the multi-agent setting is theoretically sound. A key contribution is the use of a linear mixing network to maintain the convexity of the learning objective. This is a crucial choice that ensures stability and efficient optimization. The use of Lagrange multipliers for decomposition in this context is also an interesting approach.\n- Empirical Results: While the validation might be considered minimal in scope, the paper demonstrates that MisoDICE outperforms baselines on standard, challenging benchmarks, providing evidence for the method's effectiveness.\n\n*Weaknesses\n- Dependence on LLMs and Generality of the Framework: The paper's core contribution feels split between the LLM-based labeling pipeline (Stage 1) and the MisoDICE algorithm (Stage 2). The first stage's reliance on LLMs for preference generation seems to lack generality. The authors themselves note that this approach fails on tasks like MAMuJoCo, where semantic context is poor, forcing them to use a rule-based oracle. This undermines the claim that LLM-based labeling is a core, generalizable part of the contribution and raises questions about the framework's applicability to common imitation learning domains like robotic manipulation. The two stages feel disconnected, and the paper might be stronger if it focused on the more robust contribution of the MisoDICE algorithm itself.\n- Clarity of Positioning and Presentation:\nThe paper's positioning of imitation learning (IL) as a subfield of reinforcement learning (RL) could be more nuanced. While related, they are often treated as distinct paradigms. A more detailed discussion of their relationship would clarify the paper's contribution.\n- The presentation of the baselines is potentially misleading. Although methods like \"VDN\" in Figure 1 look like baseline methods, thye are clearly ablation studies of the proposed method. They should be explicitly framed as such to avoid confusion and clearly delineate the contributions of each component of the proposed architecture.\n\nQUESTIONS:\n-Clarification on BC Performance: In Figure 1, the performance of Behavioral Cloning (BC) appears to increase over the course of training. Since BC is typically a supervised learning method whose performance on a fixed offline dataset should not change with training epochs, could you clarify what \"training\" entails for the BC baseline and why its performance curve shows an upward trend?\n\n-Justification of the Two-Stage Framework: The paper's main theoretical contribution appears to be the MisoDICE algorithm (Stage 2), which is elegant and well-motivated. However, it is tightly coupled with the LLM-based labeling pipeline (Stage 1), which seems less general and brittle, as evidenced by its failure on MAMuJoCo. Could the authors elaborate on the necessity of this tight coupling? Would the paper's contribution be clearer and more robust if Stage 2 (MisoDICE) were presented as the core method, with the LLM-based approach being just one possible (and domain-dependent) method for generating the required labeled data?\n\n- Positioning of Imitation Learning: The paper begins by stating, \"Imitation Learning (IL) is a key subfield of Reinforcement Learning (RL).\" This is a strong claim that may not be universally accepted, as many researchers treat IL and RL as related but distinct paradigms. It would be beneficial to elaborate on the positioning of IL relative to RL. A more nuanced discussion of their relationship and how this work fuses concepts from both would strengthen the paper's framing, especially considering existing studies that describe the theoretical fusion of RL and IL (e.g., [Kinose+ 2020] in robot learning).\n\nKinose, A., & Taniguchi, T. (2020). Integration of imitation learning using GAIL and reinforcement learning using task-achievement rewards via probabilistic graphical model. Advanced Robotics, 34(16), 1055-1067.\n\nLIMITATIONS:\nYes, the authors have a section discussing limitations. However, the limitation regarding the LLM-based labeling stage could be addressed more deeply. The paper acknowledges that the LLM approach struggles in environments like MAMuJoCo, but it doesn't fully discuss the implications this has for the overall framework's claimed generality. The LLM labeling is presented as a key part of the contribution in Section 4, but the experiments suggest it is not universally applicable. I would encourage the authors to be more upfront about this limitation and discuss how the MisoDICE algorithm (Stage 2) can be used with other, potentially simpler or more general, methods for data labeling. This would strengthen the paper by focusing on its most robust and generalizable contribution.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a novel framework for cooperative multi-agent offline imitation learning from unlabeled mixed-quality demonstrations. It is structured in two stages: trajectory labeling and multi-agent imitation learning. By extending the popular single-agent DICE framework to multi-agent settings with a new value decomposition and mixing architecture, their method yields a convex policy optimization objective and ensures consistency between global and local policies.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths：\n1、The paper tackles the underexplored challenge of multi-agent IL with unlabeled mixed-quality demonstrations. The two-stage design is an innovative solution, effectively bridging preference learning and occupancy matching.\n2、The technical core is strong. The convexity analysis of the objective under linear mixing is a crucial insight ensuring stable optimization. The proofs of global-local policy consistency and the connection between local policies and value functions provide theoretical grounding for the CTDE approach and decentralized execution.\n\nWeakness：\n1、The labeling phase relies on domain-specific semantic features (e.g., unit health in SMAC) for LLM feedback. For domains without interpretable features (e.g., MaMuJoCo), the proposed \"rule-based oracle\" lacks details. How was the rule-based oracle designed for MaMuJoCo? Provide algorithmic details and its performance vs. LLM in SMAC. If it requires domain-specific heuristics, does this undermine the framework’s generality?\n2、How sensitive is the labeling phase to the quality and specificity of the LLM prompts? Could low-quality or overly generic LLM outputs significantly degrade O-MAPL refinement and subsequent IL performance? \n3、 The author claims to have solved the computational complexity of the large-scale joint state-action space, but lacks relevant theoretical or experimental proof.\n4、Proposition 5.5 assumes that behavioral strategies can be decomposed into independent local strategies, but the behaviors of agents in cooperative tasks are usually correlated. Please demonstrate the rationality of this assumption.\ns\n\nQUESTIONS:\nsee the weakness\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes multi-agent imitation learning in a cooperative setting containing mixed quality demonstrations - expert & non-expert. The authors propose a two-step approach where LLMs and preference based RL is used to label the demonstrations and then these are used likewise to train an offline imitation learning policy. The problem setup is novel and really important.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\n1. Problem setting is novel, learning multi-agent IL policies in settings where demonstrations are of mixed quality is an interesting and important problem; this also occurs very often due to scarcity of expert demonstrations\n2. Related work is well written; positioning the work well in the light of prior work\n3. Experimental results are well conducted on standard MARL benchmarks of increasing difficulty, showing reasonable advantage of the proposed approach over baselines.\n\nWeakness:\n\n1. Though the paper uses techniques from prior work (like O-MAPL); they are not mentioned in the Background section making it difficult to follow specific details.\n2. The first phase of this work is summarized briefly, without providing sufficient details. If LLM provides noisy preference data, I am not sure how O-MAPL learns a soft Q-function? Can it filter out the noisy preferences?\n3. The authors use local network to learn local value functions; but I am not sure how well this would scale up for large number of agents and the no. of local value functions would be large.\n4. I do not understand the intuition behind discriminator based classification problem (looks like cross-entropy loss) as in Eqn 9? What is the intuition behind posing the estimation problem as a supervised classification problem?\n5. Though the problem statement is interesting, I think it is too computationally intensive to be able to scale up to large number of agents. There are 3 different local networks per agent and 2 different mixing networks making the overall framework extremely complicated and intensive.\n6. There is a trade off between the amount of gain achieved in terms of performance versus the computational overhead of the proposed approach; thus I am skeptical about it’s utility for furthering research. IT looks like VDN also learns pretty well in this setting.\n\nQUESTIONS:\n1. What is the intuition behind R~Q-\\gamma V in section 4? Is this advantage estimation that you are doing?\n2. Does the rankings of the trajectory done in Phase-1 align with ground truth (actual rankings) ? This would help to understand how good is O-MAPL in recovering an appropriate reward function from noisy preference data.\n3. How often are the 2 mixing networks trained in this framework? How often are the local networks trained? What objective function  is used to train the 2 mixing networks?\n4. Though the evaluation returns are better for MIsoDice, the training plot seems not that statistically significant. I am curious as to why MisoDICE is almost similar in training performance in smaller configurations (Fig 1; 5v5; 10Vs 10) than bigger ones?\n5. I could not find MisoDICE-VDN as mentioned in Baselines in the Experimental plots.\n6. I would have liked to see an ablation study without phase-1; i.e: getting demonstrations and using them all as optimal demonstrations along with Phase-2 to understand how important the pretraining process is in this framework.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces MisoDICE, a novel framework for offline multi-agent imitation learning (MAIL) that addresses the challenge of unlabeled, mixed-quality demonstrations. The authors propose a two-stage solution: first, a multi-step labeling pipeline utilizes LLMs and preference-based reinforcement learning to identify expert trajectories. Second, MisoDICE, a multi-agent IL algorithm, leverages these labels and extends the single-agent DICE framework with a value decomposition and mixing architecture to learn robust policies. The research demonstrates MisoDICE's superior performance on standard MARL benchmarks and discusses its theoretical guarantees, including convex policy optimization and global-local policy consistency.\n\nSTRENGTHS AND WEAKNESSES:\n### Strength\n+ MisoDICE tackles the highly challenging problem of offline MAIL using only unlabeled demonstrations of mixed quality. The proposed two stage solution is interesting. \n+ MisoDICE addresses the combinatorial explosion of joint state-action spaces in multi-agent environments by building upon the CTDE paradigm leveraging the value decomposition and a linear mixing architecture. \n+ The framework provides a theoretical analysis establishing local-global optimality consistency. \n\n### Weakness \n- MisoDICE performance heavily rely on the quality of the external preference model. Given different LLMs trained on different domains or context, the performance could be different. Though this point has been mentioned in the limitation, it is important as it being the basis of the success of this work. \n- The two stage pipeline requires multiple model components which introduces additional implementation complexity. \n- Though linear mixing provide convexity, the model might benefit from more expressive mixing strategies.\n\nQUESTIONS:\nCould preference noise lead to misleading labels that propagate into Phase 2? \n\nHave you tested how the performance of MisoDICE degrades when preference labels are noisy or contradictory?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}