Title: Heterogeneous Swarms: Jointly Optimizing Model Roles and Weights for Multi-LLM Systems
Venue: NeurIPS 2025
Accepted: yes
Keywords: model collaboration

multi-LLM systems
Abstract: We propose Heterogeneous Swarms, an algorithm to design multi-LLM systems by jointly optimizing model roles and weights. We represent multi-LLM systems as directed acyclic graphs (DAGs) of LLMs with topological message passing for collaborative generation. Given a pool of LLM experts and a utility function, Heterogeneous Swarms employs two iterative steps: role-step and weight-step. For role-step, we interpret model roles as learning a DAG that specifies the flow of inputs and outputs between LLMs. Starting from a swarm of random continuous adjacency matrices, we decode them into discrete DAGs, call the LLMs in topological order, evaluate on the utility function (e.g. accuracy on a task), and optimize the adjacency matrices with particle swarm optimization based on the utility score. For weight-step, we assess the contribution of individual LLMs in the multi-LLM systems and optimize model weights with swarm intelligence. We propose JFK-score to quantify the individual contribution of each LLM in the best-found DAG of the role-step, then optimize model weights with particle swarm optimization based on the JFK-score. Experiments demonstrate that Heterogeneous Swarms outperforms 17 role- and/or weight-based baselines by 18.5% on average across 12 tasks. Further analysis reveals that Heterogeneous Swarms discovers multi-LLM systems with heterogeneous model roles and substantial collaborative gains, and benefits from the diversity of language models.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Heterogeneous Swarms, a method for jointly optimizing model roles and weights in multi-LLM systems. The approach represents multi-LLM systems as directed acyclic graphs (DAGs) and uses particle swarm optimization (PSO) to iteratively optimize both the graph structure (role-step) and model weights (weight-step). The method introduces G-DECODE for converting continuous adjacency matrices to discrete DAGs and JFK-score for evaluating individual model contributions. Experiments across 12 tasks show an 18.5% average improvement over 17 baselines.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n1. **Novel Joint Optimization Approach**: The paper addresses an important limitation in existing multi-LLM systems by jointly optimizing both roles (graph structure) and weights, rather than treating them as separate problems.\n2. **Comprehensive Experimental Evaluation**: The evaluation spans 12 diverse datasets across knowledge, reasoning, agent, and miscellaneous tasks, with comparisons against 17 baselines across multiple categories.\n3. **Practical Algorithmic Contributions**:\n    - G-DECODE provides a principled way to convert continuous optimization to discrete DAG structures\n    - JFK-score offers a novel metric for quantifying individual model contributions in multi-LLM systems\n4. **Thorough Analysis**: The paper includes extensive ablation studies, collaborative gain analysis, role distribution analysis, and scaling experiments that provide valuable insights.\n5. **Strong Empirical Results**: Consistent improvements across tasks with statistically significant results on most datasets.\n\n### Weaknesses\n\n1. **Limited Theoretical Foundation**: The paper lacks theoretical analysis of convergence properties, optimality guarantees, or conditions under which the approach should work well. The choice of PSO and the specific algorithmic design appear largely empirical.\n2. **Computational Overhead**:\n    - The optimization process requires O(n(N+M)) model inferences per iteration, which could be prohibitively expensive for large model pools\n    - Limited analysis of computational cost vs. performance trade-offs\n    - The speedup strategies (sparsity, dropout) help but may not address fundamental scalability issues\n3. **Architecture Constraints**: The requirement that all models share the same architecture for weight optimization significantly limits practical applicability, as real-world scenarios often involve heterogeneous model architectures.\n4. **Evaluation Limitations**:\n    - Only tested with relatively small models (7B parameters)\n    - Limited to 10 expert models - scalability to larger pools unclear\n    - Some datasets have small evaluation sets (e.g., GAIA-text with 28 examples)\n5. **Methodological Concerns**:\n    - The continuous-to-discrete conversion in G-DECODE may lose important structural information\n    - PSO hyperparameter sensitivity not thoroughly analyzed\n    - The JFK-score formulation could be sensitive to random assignment variations\n\n### Technical Issues\n\n1. **G-DECODE Algorithm**: While novel, the algorithm's reliance on top-p sampling for both end node selection and edge creation introduces stochasticity that may affect reproducibility. The paper would benefit from analysis of this sensitivity.\n2. **JFK-Score Robustness**: The individual contribution metric depends on random model assignments. More analysis on the stability and reliability of this metric across different assignment strategies would strengthen the work.\n\nQUESTIONS:\n1. **Theoretical Justification**: Can you provide theoretical analysis of when and why PSO should work well for this problem? What are the convergence guarantees?\n2. **Scalability Analysis**: How does performance scale with the number of expert models (beyond 10)? What are the practical limits of the approach?\n3. **Architecture Heterogeneity**: How could the method be extended to handle expert models with different architectures? This seems crucial for practical deployment.\n4. **Hyperparameter Robustness**: How sensitive is the method to PSO hyperparameters? Could you show performance across a range of settings rather than just the best-found configuration?\n5. **Computational Efficiency**: Beyond the proposed speedup strategies, are there more fundamental ways to reduce the computational overhead while maintaining performance?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents Heterogeneous Swarms (HS), an algorithm that optimizes multi-LLM systems by representing them as DAGs where models exchange messages. Guided by a given utility function, the method uses Particle Swarm Optimization to jointly optimize both the LLM connection structure and their individual parameters. The optimization alternates between a \"role-step\", which learns the graph topology, and a \"weight-step\", which adapts model parameters based on a novel \"JFK-score\" metric designed to quantify each LLM's individual contribution. Through extensive experiments, the proposed method is shown to outperform 17 baselines by an average of 18.5% across 12 diverse tasks, demonstrating substantial collaborative gains.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The experiment setup is comprehensive. The authors compare HS against 17 baselines that cover a wide spectrum of approaches, including static/dynamic roles and static/dynamic weights. This rigorous comparison across 12 diverse tasks provides strong evidence for the method's effectiveness.\n2. The paper is well-written, with clear explanations of its complex methodology. The analysis is thorough and insightful. For instance, the ablation study disentangles the relative importance of roles and weights for different task types, justifying the joint optimization approach.  Furthermore, the analysis of \"collaborative gains\" provides a quantitative justification for the use of multi-LLM systems by showing that the swarm can solve problems that no single constituent model can. \n3. While the concept of multi-LLM collaboration is not new, the explicit formulation of this as a joint optimization problem over graph structures and model parameters is a significant contribution. The methods and metrics introduced, such as the iterative two-step optimization process and the JFK-score for credit assignment, are well-grounded in the problem they aim to solve. This provides a flexible yet powerful framework for designing multi-LLM systems.\n\nWeaknesses:\n1. The primary limitation of HS is its computational expense. The optimization process involves numerous inferences across a swarm of graphs and models. While the authors acknowledge this and propose sparsity and dropout strategies to mitigate the cost, the optimization phase still remains expensive.\n2. The weight-step of the algorithm, in its default implementation, requires all LLMs in the pool to share the same model architecture to ensure they share the same parameter space for optimization. While the authors suggest potential workarounds, this is a notable constraint that limits the diversity of truly \"heterogeneous\" models that can be incorporated, especially proprietary models which cannot be fine-tuned.\n3. The role-step optimizes continuous adjacency matrices, which is a very high-dimensional search space ($n^2$ for $n$ models). While PSO is suited for non-differentiable problems, its effectiveness can degrade in such large search spaces. The paper could benefit from a deeper discussion on why PSO is the right choice over other black-box optimization techniques (e.g., evolutionary strategies, Bayesian optimization) and how it scales with an increasing number of LLMs.\n\nQUESTIONS:\n1. Could you provide a more detailed comparison of the resource requirements for HS versus key baselines? Specifically, a breakdown of metrics such as total optimization time, cost for optimization and inference (e.g., total tokens processed or GPU hours), would be very helpful for understanding the practical trade-offs involved.\n2. The optimization addresses a very high-dimensional search space. How does the framework ensure adequate exploration of the search space within 20 iteration limit to find effective solutions? Was any specific initialization strategy employed beyond random initialization to guide the search?\n3. In Table 1, the \"Prediction Merge\" baseline performs worse than the \"Best Single\" expert. This is somewhat counterintuitive for an ensemble method. Could the authors offer some insight into why a simple plurality vote might degrade performance?\n4. In table 2, the results for \"Ours w/o Role\" can be interpreted as a dynamic weight optimization method on a fixed graph. This approach still outperforms many dedicated dynamic weight baselines. What is the key algorithmic difference that leads to this superior performance?\n5. In appendix, the paper demonstrates that a collaboration of weaker models can outperform a stronger single model. How does the total inference cost of the \"HS(2-10)\" or \"HS(6-10)\" swarms compare to the inference cost of the single top-1 model they outperform? Is there a trade-off between collaborative performance gain and increased inference budget?\n6. Could you clarify how the utility function $f$ was defined for each of the 12 benchmarks? While accuracy is a likely metric for many, a detailed breakdown in the appendix specifying the evaluation metric for each dataset is needed to enhance the clarity.\n\nLIMITATIONS:\nYes. Limitations have been thoroughly discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to use multiple LLMs to solve a task by organizing them as the nodes of a DAG and calling them in the topological order where the output of the parents of a node becomes part of that node’s input.\nThe paper employs the technique of Particle Swarm Optimization (PSO) to optimize the DAG topology and the LLMs’ weights for the given task.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**     \nThe paper addresses a well-motivated problem that is worth more attention.\nIt is overall written with good clarity and easy to follow.\nThe empirical improvement seems significant.\nThe paper provides analysis on the learned roles. \n\n**Weaknesses**   \nW1. The G-decode algorithm is not well justified. In what sense does it use matrix $A$ as edge probabilities? What is top-p sampling and why is it used here? And there are DAG parameterization techniques that automatically ensure acyclicility:\nCharpentier, Bertrand, Simon Kibler, and Stephan Günnemann. \"Differentiable DAG Sampling.\" ICLR 2022.\n\nW2. JFK-score is inherently limiting because it averages out an individual LLM’s performance across many teams. In general, an LLM as a team member performs well only with certain teammates in a certain team. Ideally we would like to identify the best team as a whole. Such limitations are not discussed in the paper.\n\nW3. The innovation seems incremental from Model Swarms [26], which already used PSO for weight update. The paper does not clearly describe the difference from [26].\n\nW4. Using PSO for weight update is confusing.\nPSO tries to identify a single $x$, but ideally the multiple LLMs should have their own best weight. It is unmotivated to use PSO for the weight update.\nMoreover, PSO is only applicable when the LLMs have the same architecture, but in general an multi-LLM system involves LLMs of various kinds.\n\nW5. It is unclear how meaningful it is to compare with the chosen baselines.\nHow many baselines do multi-round optimization as the proposed method does? \nIf prior works have done either dynamic weight or dynamic role, can we do a baseline that does both by combining two prior works (e.g., Model Swarms + GPT-Swarm), which could serve as a stronger baseline to test the effectiveness of the proposed role-step/weight-step?\n\nQUESTIONS:\nIn addition to the questions in the Strengths And Weaknesses section:\n\nQ1. (Line 103) What's x_best? Is it the same as g (the global best)?\n\nQ2. Isn't that the DAG itself already has an assignment? If so, why do we “randomly select an LLM for each position” (Line 145)? \n\nQ3. How difficult is it to combine a prior dynamic weight technique vs a prior dynamic role?\n\nQ4. Are the optimization rounds (role-step and weight-step) performed independently for every dataset or datapoint in Table 1?\n\nQ5. Why can the proposed method alleviate prompt engineering issues? What prompts does the proposed method use? How about the baselines\n\nLIMITATIONS:\nLimitations are discussed in the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on multi-LLM systems and figures out the fixed-weight/fixed-role limitations in the existing methods. To overcome these limitations, the authors propose an algorithm named HETEROGENEOUS SWARMS to jointly optimizes the collaborative structure (a DAG) and individual model weights via Particle Swarm Optimization (PSO). The proposed method shows impressive outperformance over numerous baselines across 12 diverse tasks, with analysis confirming substantial collaborative gains. Though I concern about its generalizability and computation cost, this work presents a powerful framework for designing adaptive multi-LLM systems and marks a valuable contribution to the field.\n\nSTRENGTHS AND WEAKNESSES:\n* Strength\n1. The paper formulate multi-LLM system problem-solving as jointly optimizing model roles (DAG) and weights. This novel modeling provides a structured alternative to prior methods which often treated these components separately or relied on manual heuristics, thereby creating a more comprehensive optimization target.\n2. The proposed algorithm offers a practical solution to a complex search problem by employing PSO combined with two specific components—G-decode for graph generation and the JFK-score for credit assignment. These methods successfully navigate the high-dimensional, non-differentiable search space of possible graph structures and model parameters.\n3. The authors conduct a broad set of experiments across 12 different tasks and compare their method against 17 baselines. The consistent performance improvements demonstrated the effectiveness of this algorithm. Furthermore, the analysis and ablation study is insightful and confirms the necessity of each design modules.\n\n* Weakness\n\nIMO the topological structures of the multiple LLMs are task-dependent, i.e., differ task by task.  While the appendix presents a single experiment on generalization (Table 6), this evidence is insufficient to fully support the claim of adaptability. The experiment shows generalization from one knowledge-based task (K-Cross) to another similar one (WikiDYK). This demonstrates in-domain generalization but does not address the more challenging case of out-of-domain generalization (e.g., a system optimized for reasoning being applied to an agentic task). The high degree of task-specific optimization inherent in the method raises concerns about how brittle these specialized systems might be when faced with novel problem types.\n\nQUESTIONS:\n1. It is a little counter-intuitive for me that BEST SINGLE usually achieves better performance than PRED. MERGE as shown in Table 1. Can you briefly explain about it?\n2. Is the utility function optimized per task or not? This point is ambiguous in the paper.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Heterogeneous Swarms, an algorithm that automatically designs multi-LLM systems by jointly optimizing two key dimensions: (i) model roles, represented as a directed acyclic graph (DAG) defining inter-model communication, and (ii) model weights, quantifying each LLM’s contribution. The method alternates between a role-step (discovering the optimal DAG via particle swarm optimization over continuous adjacency matrices) and a weight-step (estimating individual model contributions via a proposed JFK-score and optimizing weights accordingly). The approach is tested on 12 diverse tasks, outperforming 17 strong baselines with an 18.5% average improvement, and demonstrating consistent collaborative gains from heterogeneous multi-LLM systems.\n\nSTRENGTHS AND WEAKNESSES:\n*Strengths*\n\n1. The paper is well-written with clear explanations of complex components (e.g., G-DECODE, JFK-score). Figures (e.g., Figure 2) are illustrative and help convey the method’s workflow.\n\n2. The framing of roles as learnable DAG structures optimized with swarm intelligence is novel.  Combining both roles and weights in a unified optimization loop is a fresh contribution compared to prior fixed-role or fixed-weight strategies.\n\n3. The paper advances the field of multi-LLM collaboration by addressing both flexible role discovery and weight adaptation—two aspects often studied separately. The empirical gains suggest substantial practical impact.\n\n*Weaknesses*\n\n1.  The method’s reliance on repeated LLM inference during optimization could make it impractical for very large models without distributed compute.\n\n2. The paper builds on particle swarm optimization in a relatively straightforward way; while well-applied, PSO itself is not a new method.\n\n3. A few sections, especially the PSO update rule and G-DECODE sampling process, could benefit from more intuitive examples or step-by-step illustrations.\n\nQUESTIONS:\n1. Could the authors clarify how G-DECODE ensures DAG validity in extreme cases where adjacency probabilities are nearly uniform? Does this lead to highly variable graph structures between runs?\n\n2. How does the method perform if the initial pool of LLMs is not diverse (e.g., identical models)? Could the authors quantify the dependency on model diversity in more detail?\n\n3. What happens when the utility function f is noisy or discontinuous? Does PSO still converge reliably?\n\n4. Could the authors discuss scalability: what are the computational and memory costs when the LLM pool size grows beyond 10 or 20 experts?\n\n5. Are there practical strategies (e.g., early stopping heuristics) to reduce the potentially high inference cost during optimization without losing significant performance?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}