Model Swarms: Collaborative Search to Adapt LLM Experts via Swarm Intelligence
Authors: Shangbin Feng
Zifeng Wang
Yike Wang
Sayna Ebrahimi
Hamid Palangi
Lesly Miculicich
Achin Kulshrestha
Nathalie Rauschmayr
Yejin Choi
Yulia Tsvetkov
Chen-Yu Lee
Tomas Pfister
Venue: ICML 2025 poster (2025)
Keywords: evolutionary algorithm
model adaptation
swarm intelligence
Primary area: deep_learning->large_language_models

Abstract: We propose Model Swarms, a collaborative search algorithm to adapt LLMs via swarm intelligence, the collective behavior guiding individual systems. Specifically, Model Swarms starts with a pool of LLM experts and a utility function. Guided by the best-found checkpoints across models, diverse LLM experts collaboratively move in the weight space and optimize a utility function representing model adaptation objectives. Compared to existing model composition approaches, Model Swarms offers tuning-free model adaptation, works in low-data regimes with as few as 200 examples, and does not require assumptions about specific experts in the swarm or how they should be composed. Extensive experiments demonstrate that Model Swarms could flexibly adapt LLM experts to a single task, multi-task domains, reward models, as well as diverse human interests, improving over 12 model composition baselines by up to 21.0% across tasks and contexts. Further analysis reveals that LLM experts discover previously unseen capabilities in initial checkpoints and that Model Swarms enable the weak-to-strong transition of experts through the collaborative search process.

Model Swarms: Collaborative Search to Adapt
LLM Experts via Swarm Intelligence
Shangbin Feng',
Zifeng Wang²
Yike Wang's ! Sayna Ebrahimi 3 Hamid Palangi? Lesly Miculicich 2 Yejin Choi Yulia Tsvetkov Chen-Yu Lee² Tomas Pfister?
Achin Kulshrestha 4
Nathalic Rauschmayr'
Abstract
et al., 2023) that shares parameters across all languages and tasks, recent work has increasingly recognized the im- portance of modularity through multi- collaboration, where diverse models interact and complement each other in various ways (Shen al., 2024c; Feng et al., 2024a; Chan et al., 2024; Du et al., 2024). For example, mixture- of-experts (MoE) relies on the routing of queries to various neural sub-components, leveraging the specialized expertise of one model (Masoudnia & Ebrahimpour, 2014; Roller et al., 2021; Pfeiffer et al., 2022; Jiang et al., 2024). Rout- ing to domain-specifec experts demonstrates great potential, while no new model/expert is produced in the MoE pro- cess. However, challenging real-world tasks often require flexible composition and adaptation to new domains and/or capabilities that go beyond the scope of an existing expert.
We propose MODEL SWARMS, a collaborative search algorithm adapt LLMs via swarm in- telligence, the collective behavior guiding indi- vidual systems. Specifically, MODEL SWARMS starts with a pool of LLM experts and util- ity function. Guided by the best-found check- points across models, diverse LLM experts col- laboratively move in the weight space and opti- mize a utility function representing model adap- tation objectives. Compared to existing model composition approaches, MODEL SWARMS of- fers tuning- free model adaptation, works in low- data regimes with as few as 200 examples, and does not require assumptions about specific ex- perts in the swarm or how they should be com- posed. Extensive experiments demonstrate that MODEL S SWARMS could flexibly adapt LLM ex- perts to a single task, multi-task domains, reward models, as well as diverse human interests, im- proving over 12 model composition baselines by up to 21.0% across tasks and contexts. Further analysis reveals that LLM experts discover pre- viously unseen capabilities in initial checkpoints and that MODEL SWARMS enable the weak-to- -oo strong transition of experts through the collabo- rative search process. Code and data available at http:/ggiibbboooooooo//nnnnnnnne swarm.
Two lines of work aim to extend multi-LLM collaboration beyond routing to compose and produce new adapted mod- els. 1) Learn-to-fuse designs trainable components to "glue" experts together into a merged model, then fine-tunes the model with supervised objectives produce compositional experts (Jiang et al., 2023b; Wang et al., 2024b; Bansal et al., 2024). These approaches often rely on large training sets to tune the learnable parts from scratch and hardly of- fer the modularity of seamlessly adding/removing experts. 2) Model arithmetic composes LLM experts by conduct- ing arithmetic operations on model weights and/or token probabilities (Ilharco et al., 2023; Yu et al., 2024; Yadav et al., 2024; Mavromatis et al., 2024; Liu et al., 2024). These approaches often come with strong assumptions about the available experts and how the desired adaptation should be decomposed (e.g., lion indoors = lion outdoors + (dog indoors dog outdoors) (Ilharco et al., 2023)). As such, a flexible approach that does not rely on excessive tuning data or strong assumptions about existing models is crucial for adapting diverse LLM experts for wide-ranging purposes.
1. Introduction
Advancing beyond efforts to train a single, universal large language model (LLM) (Brown et al., 2020; Gemini Team
'nniversity of Washington 'Googee Cloud AI Research Google DeepMind "Google *Stanford University. Correspon- dence to: Shangbin Feng <shangbin wewe@eess.ssssse washington.eddu> Zifeng Wang <zifengw@googg www.wgggoo.oco...... Chen- Yu Lee <chenyulee @ google.com>.
To this end, we propose MODEL SWARMS, where multi- ple LLM experts collaboratively search for new adapted models in the weight space. Inspired by Particle Swarm Optimization (PSO) (Kennedy & Eberhart, 1995), MODEL SWARMS views each LLM expert as a "particle" and defines LLM adaptation as the collaborative movement of particles
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

LLM Expert
trajectory
8w
Utiity Function
PPr
8 Weight Space
Output:
Initialization
Velocity Update
Weight Update
End of Search
Figure I:
We propose MODEL SWARMS, a collaborative search algorithm to adapt LLM experts via swarm intelligence. Guided by personal best P.. global best g, and global worst gw. LLM experts update its velocity v and location x to explore the weight space and optimize utility function f. The best-found expert (global best g) in the end is retained the output.
governed by utility function representing an adaptation objective. Specifically, to model the proactive search of LLMs instead of passive merging, each expert particle starts with a location (model weights) and velocity (direction in the weight space). The velocity is iteratively impacted by inertia (the tendency to keep current velocity), personal best (the best-found location of given particle), and global best/worst (the best/worst-found location among all parti- cles), while LLM particles then take step towards the up- dated velocity direction. These velocity factors enable LLM particles to chart an independent search path and explore the aessonalgllbbl best neighborhoods. Thanks to the flexi- ble search methodology, MODEL SWARMS does not need any supervised fine-tuning data or pre-existing knowledge about the LLM experts or the utility function, adapting LLM experts solely through collaborative search and movement guided by any model-to-scalar utility function.
perts is crucial, models display emerging capabilities in initial checkpoints, and surprisingly, the best ending particle often did not start as the best. MODEL SWARMS could be accelerated with dropout-like strategies and seam- lessly extended to token probability arithmetic for experts with different model architectures. We envision MODEL SWARMS versatile framework to reimagine the poten- tial of diverse open models.
2. Methodology
We propose MODEL SWARMS, a collaborative search al- gorithm adapt LLM experts via swarm intelligence. We present an overview of MODEL SWARMS in Figure 1 and Algorithm 1.
MODEL SWARMS assumes the access to various LLM ex- perts {x, ) which could be full models or LoRA adapters (Hu et al., 2022) fine-tuned on diverse tasks and domains publicly available on model-sharing platforms (Wolf et al., 2019). also requires utilityy function f x R. mapping each expert onto scalar value that should be optimized for model adaptation. Utility functions could be dataset perfor- mance, reward model scores, human preferences (Section 3).
MODEL SWARMS achieves superior performance across four distinct LLM adaptation objectives:
Single task: Optimizing over as few as 200 instances, MODEL SWARMS outperforms 12 model composition baselines by 13.3% across datasets spanning knowledge, reasoning, and safety.
Inspired by Particle Swarm Optimization (Kennedy & Eber- hart, 1995) and evolutionary algorithms general (Bäck Schwefel, 1993), MODEL SWARMS employs several termi- nologies:
Multi-task domain: Jointly optimizing multiple tasks in medical, legal, scientific, and cultural domains, MODEL. SWARMS often produces Pareto-optimal experts than op- timizing single task.
Each LLM expert, "particle" in the model swarm, has location represented by model weights;
Optimizing reward model scores of gen- eral and conflicting preferences, MODEL SWARMS offers steerable experts that outperform baselines by up to 14.6% in controllability.
Reward model:
Each particle has velocity, direction in the model weight space that should move towards next;
Human interest: On 16 topics evaluated by humans (e.g..
electric vehicles and PhD applications), Model Swarms produces experts on par or better than existing models in 85% of cases.
Personal best p;: the best-found location of x, based on utility function f in its search history;
Global best and worst g and gw: the best/worst location in all of {x 's search history.
Empirical analyses reveal that the diversity of starting ex-
2

---

MODEL SWARMS
The location and velocity of particles enable the proactive search of L.LM experts instead of passive merging, while the personal/global best checkpoints help keep track of good locations and neighborhoods in the weight space to further explore.
unique control over exploration, where lower de means more exploration (less impacted by current velocity and more by other models) and vice versa.
Walk randomness factors Te, Tp, Tgg Tw ~ (40 1) ensure that the search is not deterministic, boosting particle explo- ration and crucial in the collaborative search process (Table 5).
Step O. Initialize To expand the pool of starting ex- perts/particles {x:)_=1> MODEL. SWARMS employs pair- wise crossover with linear interpolation. Concretely, we randomly select two experts x, and x,, from {x,]=_ and
Step 2. Weight Update Based on velocity v, the weights/locations of LLM experts are updated by taking step towards V: X, f x, Av, where À is the step length hyperparameter. The updated particles are then evaluated on the utility function f to update g, g.. and {pr) 1 if necessary.
sample ~ (10 1), new starting particle is obtained by Xnew tx. + (1 t)x,. Repeat this process for N n
times to expand (x: {<]]". into {x, j Expanding the start- ing particles allows for more rial-and-erro bandwidth in the search process.
For each particle xG, we initialize its velocity as pointing to a random particle V; = random [x,]]ر=]] x,-" We initialize its personal best as its current location p; X; and determine the global best/worst g arg max fxx and g= = arg min [x)), x E [x))..
Since MODEL SWARMS explicitly encourage randomness and exploration, particles might sometimes fail to find desir- able locations and stray away. We propose to restart unde- sirable particles and give them another chance: concretely, if for particle i the personal best P. didn't change in c,- it- erations, where G. is a hyperparameter, we put the particle back to its personal-best location with x; + P, and V, <- O, essentially granting the particle another chance with a rel- atively good starting point. In this way, MODEL SWARMS strikes balance between exploration and robustness.
Step 1. Velocity Update The movement of LLM experts is mainly governed by velocity v, directions in the weight space. We posit that the weight neighborhoods of good model checkpoints might be promising to explore (Eilertsen et al., 2020), thus the velocity of particles v; is iteratively drawn by personal best Pt. global best g, and repelled by global worst g.. Concretely:
Step 3. End of Iteration If the global best g hasn't changed in iterations (patience hyperparameter) or the maximum iteration of IC is achieved, the search process ends. Otherwise the step length À is reduced by a hyperpa- rameter factor Φx, À + À x Φx, and goes back to step 1. In the end, the global best expert g is returned as the product of MODEL SWARMS.
vi + E [rodu + tpppppp xi)
+ 1g x,) 2r x;)]
where C = re pp + 0 rw is normalization term. To dissect this formula:
3. Experiment Settings
Models and Implementation We implement a prototype of MODEL SWARMS with GEMMA-7B (google/gemma-71) it) (Gemma Team et al., 2024) in the main paper, while we also employ other LLMs such as MISTRAL-7B (Jiang et al., 2023a) in Table We create a pool of 10 initial experts/particles by fine-tuning GEMMA-7B separately on the 10 SFT data domains in Tulu-v2 (Ivison et al., 2023) with LoRA (Hu al., 2022). We fine-tune for 5 epochs with starting learning rate of 2e-4 and effective batch size of 32 by default. For MODEL SWARMS searches, we employ N = 20, Φx = 0.95, p = 10, pr = 5, K. = 50, while running grid search over other hyperparameters and report the best-found expert based on utility functioon f.
The new velocity is the weighted average of four factors: vi, the particle keeps some of its current velocity (i.e. iner- tia): (p. x,) it drawn towards its personal best; (g x;), drawn towards the global best; - -gg x:), repelled from the global worst. Inertia enables each expert to chart an independent search path, personal/global best terms encourage experts to explore good weight neighborhoods, while the global worst term repels experts to stay clear of bad model checkpoints.
Hyperparameters inertia 6.. cognitive coefficient 6,. social coefficient dg. repel coefficient &w. all E [0, 1] are configurable and govern how much the search process is impacted by p.. g, and gw. In particular, inertia &o has
We compare with 12 model composition base- lines in three categories.
Baselines
"This is to avoid all particles collapsing into the global best g like a "black hole" and reduce exploration.
All particles perform velacity and location update in parallel,
Trivial composition, 1) Best Single expert, essentially arg max f(x for x E {x 2) Data Merge, where
we omit the time stamp & for brevity.
3

---

MODEL SWARMS
Algorithm 1: Model Swarms
Input:
LLM experts (x, M= utility function f + R: Hyperparameters: swarm size N, step length X, step length schedule Φ inertia (u, cognitive coefficient Φp. social coefficient (g., repel cocfficient Φw. patience restart patience Cr, max itcration K. 1/ initialize search
pairwise interpolation to populate initial experts (x, J= populate( (x; (1=x))==.... N initialize global best checkpoint g Ø, global worst checkpoint gw 2 for to N do
initialize personal best 4- x,, velocity + random((xx) if fxx) > f(g), B + x;; if f(x)) f(gw) gw
1/=11
end
// search!
for k to K
g did not change in the last e iiterations then break 1 to N parallel' do randomness factors Tu, Tg: 1))
update velocity v, < vr + (p x.) rg00 x:) -Be xx)], where normalization term
C Te + TyOp + 0209 g + rwi
update location x; 4- X; - Av;
if (x:) > f(g), g +- x;; if f(x:) < (Bw), Bu xf; ff f(x.) > f(p:), t
t-x
if (p.) didn't 't change in o, iterations, x; + p, and v, 0
end
step length scheduling À + Ax
end
return g
the 10 SFT data domains in Tulu-v2 are merged to train one single expert: 3) Prediction Merge, where the pre- dictions of {x are ensembled via plurality vote (if applicable).
(Feng et al., 2024a)). We by default randomly sample 200 and 1000 samples as the validation/test sets: the utility function f is defined as performance the validation set.
Multi-task domain: addition to optimizing for one task, models should also be adaptable to an application domain comprising of multiple tasks. We employ 4 such domains and 2 tasks in each domain, specifically medical (MedQA (Jin et al., 2021; Li et al., 2024b) and MedM- CQA (Pal et al., 2022)), legal (hearsay and citation pre- diction classification in LegalBench (Guha et al., 2024)), scientific (SciFaat (Wadden et al., 2020) and the STEM subset of MMLU-pr (Wang et al., 2024e)), and culture (the country-based and value-based subtasks of Normad (Rao al., 2024)). The utility function is defined the harmonic mean of performance the two tasks.
Static composition, where the composed expert is in- dependent of the adaptation task/utility function f. We evaluate Uniform Soup (Wortsman et al., 2022a). Slerp, Dare-Ties (Yu et al., 2024; Yadav et al., 2024), and Model Stocks (Jang et al., 2024).
Dynamic composition, where the composed expert changes based on the utility function f. We evaluate Greedy Soup (Wortsman al., 2022a), Pack of LLMs (Mavromatis et al., 2024), cBTM (Gururangan et al., 2023), EvolMerge (Akiba et al., 2024), and LoraHub (Huang al., 2023). These approaches are also guided by the utility function f like MODEL SWARMS.
Reward model: employ three reward models (RMs) to adapt to general and conflicting preferences: gen- eral RM (eteemminnnnemmmm2----reeeee
Data and Evaluation We investigate whether MODEL SWARMS could adapt LLM experts via collaborative search on four types of adaptation objectives and the corresponding utility functions.
(InternLM Team, 2023)) and we train two conflicting RMs, verbose-RM. RM and concise-RM, adapted from the general RM and each preferring longer and more comprehensive vs. shorter and straight-to-the the-thintttttteee responses, studying whether MODEL SWARMS and baselines could offer steerability in model behavior and adapt pluralistic human preferences (Sorensen et al., 2024). We sample 200 instructions from AlpacaFarm (Dubois et al., 2024) as the validation set and 550 instructions from AlpacaFarm and Koala (Geng et al., 2023) as the test set. f is defined as the RM scores on the validation set. We additionally employ PPO (Schulman
Single task: we employ 9 datasets spanning knowledge (MMLU (Hendrycks et al., 2021), MMLU-pro (Wang et al., 2024e), Hellaswag (Zellers et al., 2019)), reasoning (GSM8k (Cobbe et al., 2021), Knowledge Crosswords (Ding et al., 2024), NLGraph (Wang et al., 2024a; Zhang et al., 2024a)), and safety (TruthfulQA (Lin et al., 2022), ScalTTxxicitPProppl (Gehman al., 2020), AbstainQA

---

MODEL SWARMS
MMLU
MMLLUppr
Hellaswag
K- K-Crrssword
GSM8K
NLGraph
TrudhfuQQA
PiPrompts
AbstainQA
val
test
val
test
val
test
val
test
val
test
val
test
val
test
val
test
val
test
BEST SINGLE
555
.537
.377
.231
.005
.601
.395
.346
.220
.237
.540
.535
365
.308
.913
860
.020
.065
DATA MERGE
.435
.445
.300
.176
-555
-527
.380
.370
.080
.143
.395
.423
.160
.107
.800
848
.900
.025
PRED. MERGI
.255
.542
.414
.173
566
586
.255
.309
.075
.044
.505
-502
.325
.276
UNIFORM SOUP
.25
.530
.314
.206
.290
.295
.270
.352
.500
.500
.395
.350
.900
875
-040
.003
SLERP
.550
.559
.386
.237
-660
6144
.350
.309
.205
.256
.520
.530
.345
.313
-915
884
.070
.128
DARE-TIES
560
.567
.414
.230
.600
.22
410
.372
.230
.307
.500
.544
.300
.337
.905
867
.110
.40
MODEL STOCKS
.357
.221
.300
.350
.400
.339
.010
.012
GREEDY SOUP
.575
.554
.371
.219
.630
.966
.395
.355
.255
.330
.545
.530
410
.345
.916
860
.105
.014
PACK OF LLMS
313
.568
.371
.235
.630
-533
.375
.352
.245
.327
.400
.532
.370
.295
.11
.861
-065
.055
CRTM
.510
.506
.286
.179
-510
.555
.320
.284
.10
198
.410
.398
360
.314
.885
.82
.060
009
EVOLMERGE
.45
.548
.371
.229
5565
5544
.300
.293
.320
.354
.510
.506
.35
.340
896
.870
.050
.018
LORAHUR
.555
.554
.386
.231
-570
-573
.345
.291
.315
.354
-565
.568
.25
.359
-903
.885
.100
.064
MODEL SWARMS
.583
.443
.254
.675
.652
.428
.395
.459
.730
.672
.455
.392
.957
.956
.200
.175
Table 1: Performance on the validation and test sets of the 9 datasets. Best in bold and second-best in underline. MODEL SWARMS outperforms TRIVIAL, STATIC, and DYNAMIC baselines by 13.3% on average and works best on the middle three reasoning tasks with an improvement of 21.0%.
Medical
Lagal
Seieven
Caltan
SWARMS achieves state-of-the-an art performance on all tasks. It outperforms the second-strongest baseline by 13.3% average and up to 29.7% on the GSM8k dataset. The aver- age improvement on reasoning tasks (middle three, 21.0%) is higher than on knowledge (first three, 4.9%) and safety (last three, 14.1 1)) tasks, indicating MODEL. SWARMS' ver- satility and unique strength in adapting to diverse reasoning- intensive contexts due to stronger generalization on reason- ing problems. DYNAMIC merging baselines achieve 11 out of all 18 second-ple placd positions, with an average perfor- mance 15.6% and 2.1% higher than TRIVIAL and STATIC approaches. This indicates that how to compose models is dese-depen,nnt, while MODEL SWARMS advances the task- specific adaptation via multi-LLM collaborative search and further outperforms DYNAMIC approaches by 20.8%.
MasQa MedMCC
Heanay
Cit
Seifbat
STBM
Corntry
Valne
BESt S
.433
.432
.638
.500
.545
.171
.54
.385
DATA MiRro
.611
346
-596
.909
.570
7113
.57
UNIPORM Sou?
4033
.28
.521
.491
690
1146
A81
.044
SLERP
424
.431
810
878
729
167
514
578
DARE-TIR Tims
424
.457
.331
537
724
.171
334
546
Monia Stocks
433
540
GRIEDY SouP
.27
.439
.31
.72
580
.161
525
553
PArK L.
165
515
CBTM
709
MEXaE
LOR
.405
429
588
536
711
1.9
541
.55
MODEE SWARDS
.702
.743
.188
.003
Table 2: Test set performance on the 8 tasks across 4 do- mains in multi-task domain adaptation. Best in bold and second-best in underline. MODEL SWARMS outperforms all 12 baselines by 5.7% on average across datasets.
et al., 2017) and DPO (Rafailov et al., 2024) as baselines to see if MODEL SWARMS offers a better use of RMs with as few as 200 instructions.
Multi- Task Domain We present test set performance across tasks and 4 domains in Table 2. Although the multi- task domain adaptation setting is more challenging, MODEL S SWARMS still leads to average improvement of 5.7% over baselines. Specifically, in the legal domain, we see the most substantial performance improvement (11.3% and 10.5%). In addition, we discover that MODEL SWARMS produces Pareto- Pareto-Optimal experts (Figure 10), i.e., jointly optimiz- ing two tasks in one shared domain often outperforms only adapting to one single task.
Human interest: in addition to preferences represented by reward models, we adapt LLM experts directly to human, their personalized needs and interest domains. Specifi- cally, 13 human annotators nominated 16 interest domains (e.g., electric vehicles and Phl) applications), we then em- ploy GEMINI-PRO to synthesize 25:25 instructions in each domain as validation/test set. f is defined as LLM-as-a- judge (Zheng et al., 2023) 1-10 scores with Gemini on the validation set, while we evaluate the adaptation to human interest topics on three fronts: improvement in ʃ scores, improvement in factuality with Facts&Evidence (Boonsanong et al., 2024), and human evaluation win rate comparing pre-swarm and post-swarm responses.
Reward Model We present the reward model scores on validation and test set instructions in Table 4. MODEL SWARMS outperforms all 14 baselines by 6.7% on aver- age, including PPO and DPO, in the low-data adaptation regime with 200 instructions only. Importantly, while on par with alignment methods on general RM, MODEL SWARMS offers impressive steerability to adapt to diverse/conffictngg user preferences, instantiated here as verbose vs. concise. While most baselines could only reflect one but not the
4. Results
Single Task We present the performance of MODEL SWARMS and baselines on datasets in Table MODEL.
5

---

MODEL SWARMS
Interest Topic
LLM Judge
Factuality
Human Eval Win Rate
Interest Topic
LLM Judge
Factuality
Human Eval Win Rate
south america
6.28 + 7.32
.50 >> .55
648
sandbox games
5.84 > 6.88
.48 -
E:4
legal AI
6.36 7.60
.46 + .48
0.35
cartoons
6.40 7.48
.50 + .72
aircraft Al
6.52 ++ 7.76
.47 + .52
music instrument
6.48 - 7.52
.73 + .76
phd application
6.16 + 7.52
.39 + .45
611
olympics
5.92 + 6.92
.77 + .79
0.8
asian food
6.28 7.20
.44 + .47
138
cconomics
6.32 - 7.56
.41
finance
6.72 7.76
.42 + .53
electric vehicles
6.56 - 7.64
.40 + .42
luxury cars
6,40 - 7.60
.12 + .30
plastic
6.28 + 7.40
.44 + ,53
833
social network
6,56 + 7.60
.3 - .48
us tourism
6.12 - 7.28
.51 - .60
852
Table LLM-aa-a-judge scores with Gemini Gemini-Flash, factuality scores with FactenEvidennee (Boonsanong et al., 2024), and human eval win rates comparing pre- and pOSt-MODEI. SWARMS across 16 human interest domains. Colors indicate WIN, TIE, and LOSE. MODEL SWARMS improve both scores by 17.6% and 17.0% on average, while achieving 70.8% average win rate across 16 topics.
General RM
Verbose RM
Concise RM
SWARMS' great potential to produce domain-speciaalzed and community-specine LLM experts.
val
test
val
tost
val
test
BEST SINGLE
.559
.522
650
.642
.533
.490
DATA MERGE
.457
.455
.527
.550
.400
.396
5. Analysis
UNIFORM SOUP
625
.612
.555
.522
.550
.400
SLERP
.327
.316
.608
.611
.220
.204
DARE-TIES
415
.403
.607
.644
.335
.355
Correctness Emergence In the collaborative search pro- cess, are LLM experts simply transferring existing capabili- ties from one model to another, or are they discovering new skills and expertise for adaptation? Specifically, there four correctness levels for a question and the pool of LLM experts: ! the answers of experts are all wrong; ② less than half correct, 33) more than half correct; and (4) all correct. The correctness level for question could change between the pre- and pOst-MODEL SWARMS experts (e.g. ( 3) indicates that none of the experts answered correctly miiiially, but after MODEL SWARMS optimization more than half answered correctly.) We define two metrics, correctness surge (C-surge) and correctness emergence (C-emerge):
MODEL STOCKS
.615
.593
.562
.567
.555
.526
GREEDY SOUP
621
.609
.650
.649
.578
.553
PACK OP LLMS
614
.609
.625
.641
.547
.529
CBTM
.567
.559
.911
466
.463
EVOLMERGE
.619
.607
.575
.570
.554
.536
LORAHUB
.621
.603
.616
.619
.589
.561
PPO DPO
.688
.574
.618
.641
.536
.527
.627
.617
.681
.682
.611
.599
MODEL SWARMS
.646
.621
.780
.770
.651
.639
Table 4: Reward model scores on the validation and test instruction sets. Best in bold and second-best in underline. MODEL SWARMS flexibly adapts to both general and steer- able preference RMs, improving by 6.7% on average.
Ensc
Ejso
1 11
other (e-g. SLERP is good on verbose but bad on concise), MODEL SWARMS achieves state-of-the-art performance on both verbose and concise RMs, indicating that the flexible collaborative search methodology presents a viable solution for aligning to diverse and pluralistic human preferences (Wang et al., 2023; Sorensen et al., 2024; Feng et al., 2024b).
C-carge
C-emerge
Ei
ExeL
where C-surge indicates the percentage of questions with increased correctness level after MODEL SWARMS, and C-emerge quantifies that out of all initially type-O ques- tions, how much was correctly answered by at least one ex- pert after MODEL SWARMS. Figure 2 illustrates the changes in correctness levels: MODEL SWARMS achieves an aver- age C-surge of 48.2% across the four datasets, indicating broad expert improvements. An interesting observation is that MODEL. SWARMS achieves 36.0% 53.5% C-emerge, indicating that the collaborative search surfaced new skills and capabilities in experts that solved 36.0% to 53.5% pre- viously "impossible" problems for all initial experts.
Human Interest We present the comparison between pre- and PODE-MODEL SWARMS experts in the 16 human- nominated interest domains in Table 3. Through adaptation with MODEL. SWARMS, experts improve 17.6% in LLM-as- a-judge scores and 17.0% in factuality scores on average when discussing the 16 topics and domains. Most impor- tantly, human evaluation reveals that MODEL SWARMS fea- tures 70.8% win rate against initial experts on average, in particular, with an impressive 96% win rate in the two most successful domains while still maintaining 44%:28%:2289 on the unfamiliar and most challenging topics. This indicates that MODEL SWARMS outputs are consistently preferred by both automatic metrics and human users, indicating MODEL.
Diamond in the Rough We observe that in MODEL SWARMS searches, the experts that ended as the best didn't 't necessarily start as the best. We illustrate this phenomenon in Figure 3: for the partieles that ended with the highest util- ity function f scores in swarm, what was its ranking based
6

---

MODEL SWARMS
MMLU
MMLU-PP
HellaSwag
K- K-Crrsswords
post
all
>half
<half
all
all
half
<half
all
half
<half
all
>alff
<half
all
pre
all
0
0
0
0
0
0
0
0
0
half@
25
0
1
0
0
0
44
0
20
6
0
<half
2
84
48
4
0
17
24
2
1
82
39
6
12
55
54
6
0
4
12
21
0
3
6
16
0
5
8
15
6
17
20
C-surge: 0.510
C-surge: 0.386
C-surge: 0.480
C-surge: 0.550
C-emerge: 0.432
C-emerge: 0.360
C-emerge: 0.464
C-emerge: 0.535
Figure 2: The number of problems in each correctness level for experts before and after MODEL. SWARMS across four datasets, along with C-surge and C-emerge metrics. Cell colors indicate UP, SAME, and DOWN changes in correctness levels. MODEL SWARMS discover new capabilities and skills through collaborative search evident in the 44.8% average C-mmerge, solving 44.8% of previously "impossible" problems for all initial model checkpoints.
SwllsSwog
K.Crosworrds
3/1
24%
x 10, 2 x 5, x 2, and 10 X 1 in Figure 4, from the least diverse to the most diverse. Experiments demonstrate a con- sistent upward trend with the increase in expert diversity, while 10 x 1 outperforms 1 x 10 by 35.3% averaged across the five datasets. This indicates that diversity matters, that the success of MODEL SWARMS hinges on the collaborative search of a diverse and wide-ranging pool of initial experts.
10%
10%
HLGraph
TrunMfuloA
34%
32%
22%
10%
Different Model Architectures with Token Swarms The default MODEL SWARMS algorithm operates on model weights, i.e. the arithmetic operation of updating particle velocity and location is instantiated with model parameter values (weight swarms). What if we need to compose ex- perts fine-tuned from different base architectures? Instead of model weights, the swarm intelligence arithmetic could be seamlessly carried out on token probability distributions for token swarms.
Figure 3: The distribution of starting model rankings for experts that ended as the best. A vast majority of 89.6% best-found models did not start as the best and 56.9% even started in the bottom half.
on I before the search? Averaged across the four datasets, we found that only 10.4% of the ending-best particles also started as the best (#I), while surprisingly the bottom half of the starting experts were able to rise to the top in 56. 6.% of the MODEL SWARMS searches. This indicates that weak experts are not inherently less effective but maybe simply not fully adapted to the task/domain/contex of use: they are rightfully diamond in the rough and MODEL SWARMS enables the weak-to-strong transition that activates their implicit expertise to produce strong adapted LLM experts.
Concretely, the n experts start with a location matrix as an identity matrix L Inxn [I. .II... where the i-th row 1, denotes the location of particle i: a one-hot vector of Os and the i-th value is 1. For text generation, denoting the next- token probability distribution of expert as t,, expert i's adjusted token probability becomes t² = =E k, and decode text with t/. the beginning, t/: = tt as the expert focuses solely on its own token probabilities. After running updates of location and velocity in the n-dimensional search space (Algorithm 1), to becomes composition of t across experts to optimize f. This resembles the collaborative de- coding paradigms in existing research (Liu et al., 2024; Shen et al., 2024b), while how to compose the distributions are auto-discovered.
Diversity Matters MODEL SWARMS rely on a pool of LLM experts to run the collaborative search algorithm and produce adapted models. Amid the 922,559 models pub- liely available on Huggingface (Wolf et al., 2019), what models should select? Specifically, do we need homo- geneous model checkpoints or diverse specialized experts? To this end, we conduct a controlled experiment: we take a distinct initial experts (Section 3) and repeat each for b times to result in the starting swarm (denoted as a x b) while con- trolling a * b as a constant, then employ MODEL SWARMS to adapt them to task/dataset. We present the results for
We run prototype of token swarms with experts fine- tuned from GEMMA-7B and 4 from MISTRAL-7B. featur- ing different model architectures. We present the pre- and post-swarm performance of the 8 experts in Figure 5. All 8 experts become better regardless of model architecture and the global best increased 5.7% and 11.9% on the two datasets. We envision full-blown implementation and anal-
Accessed on Sept 8, 2024.
7

---

MODEL SWARMS
GSMBK
TruthfuIQA
MMLU
N.Graph
HellaSwag
0.6
0.4
0.3
Expert Pool 1*10
2
5
10a1
Figure 4: MODEL SWARMS with increasing levels of diversity in initial LLM experts. Results show general upward trend and 35.3% % increase on average from the least to most diverse initial experts.
MMLU
et al., 2024b), or even LLMs (Jiang et al., 2023b): these ap- proaches often need substantial supervised data and might not be modular seamlessly add/remove experts. In addi- tion, static model arithmetic approaches propose compose experts by performing arithmetic on model weights and to- ken probabilities to reconcile sign differences (Yu et al., 2024; Yadav et al., 2024), simulate tuning effects (Liu et al., 2024), and induce compositional capabilities (Ilharco et al., 2023), and more (Davari & Belilovsky, 2023; Jang al., 2024; Deep al., 2024; Zheng al., 2024). In comparison, dynamic model arithmetic proposes merge models guided by objective function, employing perplexity heuristics (Mavromatis et al., 2024), evolutionary methods (Akiba et al., 2024), and more (Wortsman et al., 2022a; Huang et al., 2023; Gururangan al., 2023). Most these model arith- metic approaches often rely on strong assumptions about the experts how they should be composed (e.g. lion indoors lion outdoors + (dog indoors dog outdoors) (Ilharco et al., 2023)). In contrast, MODEL. SWARMS presents a mod- ular, assumption-free, and flexible approach to compose and adapt diverse LLM experts guided by as few as 200 data instances.
0.6
0.5
0.4
0.3
3
4
5
best
Hellaswag
0.5
0.4
0.3
4 prev
5
best
post
Figure 5: Performance of the token probability variation, token swarms, the first and last four experts are based GEMMA and MISTRAL, respectively. Heterogeneous ex- perts improve performance by an average 29.3% across models and datasets.
ysis as well as some modifications to the token swarms
variant as important future work.
6. Related Work
Evolutionary Algorithms and LLMs MODEL SWARMS is in part inspired by particle swarm optimization (PSO) (Kennedy & Eberhart, 1995), evolutionary algorithm (EA) solving optimization problems. This echoes recent and contemporary uptake of EAs, especially genetic al- gorithms (GAs) in ML/LLMs (Zhao et al., 2023; Lange et al., 2023; Wu et al., 2024; Chao et al., 2024; Lange et al., 2024). EvolMerge (Akiba al., 2024) seeks to compose a math LLM and Japanese LLM through discovering better weight/layer and data flows guided by genetic algorithms. PromptBreeder (Fernando et al., 2024) seeks search for specialized L.IM prompts by maintaining prompt popu- lation and conducting LLM- crossover and mutation to produce better prompts, resembling GA processes. Evo- Prompt (Guo et al., 2024a) also follows similar concepts of applying GAs for prompt optimization. We see two key
Composing Diverse LLM Experts In addition to de- veloping gargantuan general-purpose LLMs, increasing research focus on the composition of multiple models. Mixture-of-experts (MoE) models (Jiang et al., 2024; Lin et al., 2024) and methods (Roller et al., 2021; Lewis et al., 2021; Kudugunta et al., 2021; Pfeiffer et al., 2022; Du et al., 2022; Gururangan al., 2022; Shen et al., 2024a) are one of the most noted paradigms in composing models, where different approaches vary parallel neural components (Zhang et al., 2022; Li et al., 2022), routing mechanisms (Zhou et al., 2022; Dai et al., 2022), and expert partition (Gururangan et al., 2023; Jang et al., 2023). More recently, fo-se approaches propose to "glue" experts together with trainable modules (Bansal al., 2024), adapters (Wang
8

---

MODEL SWARMS
differences between MODEL SWARMS and this line of ex- isting research: most methods focus on improvements in prompt/data engineering (Fernando et al., 2024; Guo et al., 2024a), while MODEL SWARMS seek to adapt LLMs by changing model weights and inducing new expert capa- bilities (Figure 2), which is more fundamental and offers greater headroom for improvement; existing EA applica- tions mostly employed genetic algorithms that necessitate much hand-crafted rules (Lambora et al., 2019) (how should two prompts/models crossover to produce new ones, how to mutate, etc.), while MODEL. SWARMS is inspired by swarm intelligence that come with little to no manual engineering in the composition and collaboration of models.
Wang et al., 2024f) over unseen documents in conjunction with MODEL SWARMS.
MODEL SWARMS by default operate on the model weight space, enabling the collaborative search and movement of L.M experts in terms of model parameters. While this paradigm is incompatible with a pool of experts with het- erogeneous model architectures, we propose token swarms and demonstrate its preliminary success in Figure 5. We highlight the trade-off between weight swarms and token swarms: weight swarms induces more fundamental change of model capabilitiiss through weight changes, but it would require all experts to share the same architecture; token swarms is much more flexible in expert architectures, but only changes the composition of token probabilities without touching on the model's parametric capabilities. We expect full implementation and adaptations to the token swarms variant as important future work.
7. Conclusion
We propose MODEL SWARMS, a collaborative search algo- rithm to flexibly adapt diverse LLM experts to wide-ranging purposes. Guided by personal and global best-found loca- tions, LLM experts explore to optimize utility functions representing various adaptation objectives. Extensive ex- periments demonstrate that MODEL SWARMS outperforms three categories of 12 model composition baselines by up to 21.0% across four types of model adaptation. Further analy- sis reveals that MODEL SWARMS help discover new skills in the collaborative search process and bring out the best and implicit expertise of weak models for weak-to-strong expert transition.
Unsuccessful MODEL SWARMS searches might be confined to a local minimum without broad exploration of the de- sirable weight space. While 1) we take several measures in Algorithm 1 to mitigate this (random starting velocity, walk randomness factors, etc.), 2) we observe strong em- pirical performance of MODEL SWARMS and consistent improvement to the global best g. and 3) we visualize the movement of particles in Figure 6 demonstrating its conver- gence quality, one way to mitigate this concern is by anneal- ing/adding noise to go beyond the local search: changing ro: Tpp r'g. I'w ~ 2( 1) to Tv. Tp: r'g. Iw ~ ((0222, 1) so that models have a small chance of moving towards the re- verse direction and potentially jump out of local minimums.
Impact Statement
MODEL SWARMS assumes access to pool of initial ex- perts for collaborative search to adapt language models. On one hand, it might be challenging to select the right pool of LLMs while we present evidence that the diversity of initial experts is crucial to MODEL SWARMS' successes (Figure 4); On another hand, MODEL SWARMS require the update of all experts at each iteration, which might be computationally challenging. We provide time/space complexity analysis in Appendix B and present a preliminary dropout-like accel- eration scheme in Figure 7. MODEL SWARMS is uniquely suited to low-data contexts where only a few hundred ex- amples are readily available to serve as the utility function f.
We would like to highlight the dual-use risk of MODEL SWARMS: thanks to its flexible adaptation strategy by us- ing a model-to-scalar utility function f. it also leads to malicious use cases by having malicious f's. Some exam- ples could include optimizing the reverse reward model scores, optimizing for lower scores on RealToxicityrooppt (Gehman et al., 2020), optimizing for certain social and political biases (Feng et al., 2023), and more. We argue for the responsible use of the MODEL SWARMS methodology as well as the responsible release of adapted experts.
Re Reproducibility Statement
We provide all details in the implementation and evaluation of MODEL SWARMS in Appendix C. Specifically, Appendix C contains dataset details and statistics (Table 10), imple- mentation details of MODEL SWARMS, hyperparameter set- tings, details of all 12 baselines in Section 3, details of all 4 evaluation settings in Section 4, specific prompt texts in Table 1l employed in the human interest objective, and spe- cific human evaluation instructions in Table 12. Upon the final version, we will include link to a publicly accessible repository with all MODEL SWARMS implementation code, prepossessed data files and resources, adapted model check-
MODEL SWARMS aims to adupt language models based on their existing expertise rather than memorizing new in- formation that was never seen in the training of these ex- perts. While theoretically by changing model weights ex- perts could pick up new information, our preliminary ex- periments with perplexity as the utility function, a proxy for memorization, indicates that MODEL SWARMS could not reliably optimize perplexity. We envision that temporal updates could be enabled by employing retrieval augmenta- tion (Chen et al., 2023b; Jiang et al., 2023c; Shi et al., 2024;

---

MODEL SWARMS
points, well as instructions reproducing our results and using MODEL. SWARMS beyond tasks included in this paper.
International Conference on Learning Representations, 2024.
Chao, W., Zhao, J., Jiao, L., Li, L., Liu, F., and Yang, S. A match made in consistency heaven: when large language models meet evolutionary algorithms. arXiv preprint 2rXiv:2401.105100 2024.
Acknowledgements
We would like to thank Hanna Mazzawi, Tong Chen, and the UW NLP group for their feedback and Jacob Morri- son for the inspiration. We would like thank Jiajie Yan, Ningnan Wang, Yuhan Liu, Xiaochuang Han, Herun Wan, Leijie Wang, and Wenxuan Ding for their help with human evaluation. Shangbin Feng would like to thank the support of the IBM PhD Fellowship and the Jane Street Fellowship.
Chaudhary. S. Code alpaca: An instmuction-following llama model for code generation. https: //g httpt//tttuuu.u.e thub.com, codealpaca, sahil sahii280111 sah122011/ccddappcc 2023.
Chen, J., Saha, S., Stengel-Eskin, E., and Bansal, M. Magdi: Structured distillation of multi-agent interaction graphs improves reasoning in smaller language models. In Forty-first International Conference on Machine Learn- ing, 2024a.
References
Abdelnabi, S., Gomaa, A., Sivaprasad, S., Schönherr, L., and Fritz, M. Lim-deliberation: Evaluating 1lms with interactive multi-agent negotiation game. ICLR 2024 Workshop on Large Language Model (LLM) Agents, 2024.
Chen, J. C.-Y,, Saha, S., and Bansal, M. Reconcile: Round-table conference improves reasoning via consen- sus among diverse llms. arXiv preprint arXiv:2309, 13007, 2023a.
Ainsworth, S., Hayase, J., and Srinivasa, S. Git re-basin: Merging models modulo permutation symmetries. In The Eleventh International Conference on Learning Repre- sentations, 2023.
Chen, J. C.-Y,, Prasad, A., Saha, S., Stengel-Fskin, E.,, and Bansal, M. Magicore: Multi-agent, iterative, coarse-to- fine refinement for reasoning. arXiv e-prints, pp. arXiv- 2409, 2024b.
Akiba, T. Shing, M., Tang, Y., Sun, Q. and Ha, D. Evo- lutionary optimization of model merging recipes. arXiv preprint 4rXiv:2403.13187, 2024.
Chen, T., Wang, H., Chen, S., Yu, W., Ma, K., Zhao, X., Yu, D., and Zhang, H. Dense x retrieval: What retrieval gran- ularity should we use? arXiv preprint arXiv:2312.0664 2023b.
Bäck, T. and Schwefel, H.-P An overview of evolution- ary algorithms for parameter optimization. Evolutionary computation, 1(1):1-23, 1993.
Chen, W., Su, Y., Zuo, J., Yang, C., Yuan, C., Chan, C.-M., Yu, H., Lu, Y., Hung, Y.-.., Qian, C., et al. Agentverse: Facilitating multi-agent collaboration and exploring emer- gent behaviors. In The Twelfth International Conference on Learning Representations, 2024c.
Bansal, R., Samanta, B., Dalmia, S., Gupta, N., Ganapathy, S., Bapna, A., Jain, P., and Talukdar, P. Llm augmented Ilms: Expanding capabilities through composition. In The Twelfth International Conference on Learning Represen- tations, 2024.
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Fedus, W., Li, Y., Wang. X., Dehghani, M., Brahma, S., et al. instruction-fnetuueed Scaling instruction finetuned language models. Journal of Machine Learning Research, 25(70):1-53, 2024.
Boonsanong, V., Balachandran, V., Han, X., Feng, S., Wang, L., and Tsvetkov, Y. Facts&evidence: An interactive tool for transparent fine-grained factual verification of machine-generated text. arXiv, 2024.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 14168...41........ 2021.
Brown, T. et al. Language models are few-shot learners. arXiv preprint 2rXiv:2005.14166 2020.
Burns, C., Izmailov, P., Kirchner, J. H., Baker, B., Gao, L., Aschenbrenner, L., Chen, Y., Ecoffet, A., Joglekar, M., Leike, J., et al. soak-to--trong generalization: Eliciting strong capabilities with weak supervision. In Forty-first International Conference Machine Learning, 2024.
Dai, D., Dong, L., Ma, S., Zheng, B., Sui, Z., Chang, B., and Wei, F. Stablemoe: Stable routing strategy for mixture of experts. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pp. 7085-7095, 2022.
Chan, C.-M., Chen, W., Su, Y., Yu, J., Xue, W., Zhang, S., Fu, J., and Liu, Z. Chateval: Towards better based evaluators through multi-agent debate. In The Twelfth
Davari, M. and Belilovsky, E. Model breadcrumbs: Scal- ing multi-task model merging with sparse masks. arXiv preprint arXiv:23/2.06777 2023.
10

---

MODEL SWARMS
Deep, P. T., Bhardwaj, R., and Poria, S. Della-merging: Re- ducing interference in model merging through magnitude- based sampling, arXiv preprint 270.::2406.11617,, 2024.
Fernando, C., Banarse, D. S., Michalewski, H., Osindero, S., and Rocktäschel, T. Promptbreeder: Self-referential self-improvement via prompt evolution. In Forty-first International Conference on Machine Learning. 2024.
Ding, W., Feng, S., Liu, Y., Tan, Z.,, Balachandran, V., He, T., and Tsvetkov, Y. Knowledge crosswords: Geometric knowledge reasoning with large language models. In Findings of the Association for Computational Linguisties ACL. 2024, 2024.
Fu, T., Cai, D., Liu, L., Shi, S., and Yan, R. Disperse- merge: Pushing the limits of instruction tuning via align- ment tax reduction. arXiv preprint arXiv:2405.1 134.2 2024.
Gchman, S., Gururangan, S., Sap, M., Choi, Y., and Smith, N. A. Realtoxicityprompts Evaluating neural toxic de- generation in language models. In Findings of the Associ- ation for Computational Linguistics: EMNLP 2020, pp. 3356-3369, 2020.
Du, N., Huang, Y., Dai, A. M., Tong, S., Lepikhin, D., Xu, Y., Krikun, M., Zhou, Y., Yu, A. W., Firat, O., et al. Glam: Efficient scaling of language models with mixture-of- experts. In International Conference Machine Learn- ing, pp. 5547-5569. PMLR, 2022.
Gemini Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312. 2. 1805, 2023.
Du, Y. and Kaelbling, L. P. Position: Compositional gener- ative modeling: A single model is not all you need. In Forty-first International Conference on Machine Learn- ing, 2024.
Gemma Team, G., Mesnard, T., Hardin, C.. Dadashi, R. Bhupatiraju, S., Pathak, S., Sifre, L.., Rivière, M., Kale, M. S., Love, J., et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXi::440..08295 2024.
Du, Y., Li, S., Torralba, A., Tenenbaum, J. B., and Mordatch, I. Improving factuality and reasoning in language models through multiagent debate. In Forty-first International Conference on Machine Learning, 2024.
Geng, X., Gudibande, A., Liu, H., Wallace, E., Abbeel, P., Levine, S., and Song, D. Koala: A dialogue model for academic research. Blog post, April 2023.
Dubois, Y., Li, C. X., Taori, R., Zhang, T., Gulrajani. I., Ba, J., Guestrin, C.,, Liang, P. S., and Hashimoto, T. B. Al- pacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36, 2024.
Goddard, C., Siriwardhana, S., Ehghaghi, M., Meyers, L.,
Karpukhin, V., Benedict, B., McQuade, M., and Solawetz, J. Arcee's mergekit: toolkit for merging large language models. arXiv preprint arXiv:2403. / 13257, 2024.
Bilertsen, G., Jönsson, D., Ropinski, T., Unger, J., and Yn- nerman, A. Classifying the classifier: dissecting the weight space of neural networks. In ECAI 2020, pp. 1119-1126. IOS Press, 2020.
Guha, N., Nyarko, J., Ho, D., Ré, C., Chilton, A., Chohlas- Wood, A., Peters, A., Waldon, B., Rockmore, D., Zam- brano, D., et al. Legalbench: A collaboratively built benchmark for measuring legal reasoning in large lan- guage models. Advances in Neural Information Process- ing Systems, 36, 2024.
Feng. S., Park, C. Y., Liu, Y., and Tsvetkov, Y. From pre- training data to language models to downstream tasks: Tracking the trails of political biases leading to unfair nip models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 11737-11762, 2023.
Guo, Q., Wang, R., Guo, J., Li, B., Song, K., Tan, X., Liu, G., Bian, J., and Yang, Y. Connecting large language mod- els with evolutionary algorithms yields powerful prompt optimizers. In The Twelfth International Conference on Learning Representations, 2024a.
Feng, S., Shi, W., Wang, Y., Ding, W., Balachandran, V., and Tsvetkov, Y. Don't hallucinate, abstain: Identifying LLM knowledge gaps via multi-LLM collaboration. In Pro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024a.
Guo, T., Chen, X., Wang, Y., Chang, R., Pei, S., Chawla, N. V., Wiest, O., and Zhang, X. Large language model based multi-agents: A survey of progress and challenges. 33rd International Joint Conference on Artificial Intel- ligence (IJCAI 2024). 2024b.
Feng. S., Sorensen, T., Liu, Y., Fisher, J., Park, C. Y., Choi, Y., and Tsvetkov, Y. Modular pluralism: Pluralistic alignment via multi-llm collaboration. arXiv preprint arXiv:2406. 1,951, 2024b.
Gururangan, S., Lewis, M., Holtzman, A., Smith, N. A., and Zettlemoyer, L. Demix layers: Disentangling domains for modular language modeling. In Proceedings of the 2022

---

MODEL SWARMS
Conference of the North American Chapter of the Associ- ation for Computational Linguisties: Human Language Technologies, pp. 5557-5576, 2022.
Jang, D.-H., Yun, S., and Han, D. Model stock: All we need is just few fine-tuned models. arXiv preprint urXiv:2403.14 19522, 2024.
Gururangan, S., Li, M., Lewis, M., Shi, W. Althoff, T., Smith, N. A., and Zettlemoyer, L. Scaling expert lan- guage models with unsupervised domain discovery. arXiv preprint ar 2rXv:23333.4117, 2023.
Jang, J., Kim, S., Ye, S., Kim, D., Logeswaran, L.,, Lee, M., Lee, K., and Seo, M. Exploring the benefits of training expert language models over instruction tuning. In Inter- national Conference on Machine Learning, pp. 14702- 14729. PMLR, 2023.
Han, S., Zhang, Q., Yao, Y., Jin, W., Xu, Z., and He, C. Lim muulii-ggen systems: Challenges and open problems. arXiv preprint arXiv:2402.03577 2024.
Jiang, A. Q.. Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. I., Bressand, F,, Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b, arXiv preprint arXiv:2310.0663 2023a.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. International Conference on Learning Representations, 2021.
Jiang, A. Q. Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. d. 1., Hanna, E. B., Bressand, F. et al. Mixtral of experts. arXiv preprint arXiv:2401.04088 2024.
Hong. S., Zhuge, M., Chen, J., Zheng, X., Cheng, Y., Wang, J. Zhang, C., Wang, Z., Yau, S. K. S., Lin, Z. et al. Metagpt: Meta programming for multi-agent collabora- tive framework. In The Tweefft International Conference on Learning Representations, 2024.
Jiang, D., Ren, X., and Lin, B. Y. LIm-blender: Ensembling large language models with pairwise ranking and genera- tive fusion. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pp. 14165-14178, 2023b.
Hu, E. J., Wallis, P., Allen-Zhu, Z., Li, Y., Wang. S., Wang, L., Chen, W., al. Lora: Low-rank adaptation of large language models. In International Conference Learn- ing Representations, 2022.
Jiang, Z., Xu, F. F., Gao, L., Sun, Z., Liu, Q., Dwivedi- Yu, J., Yang. Y., Callan, J., and Neubig, G. Active re- trieval augmented generation. In Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 7969-7992, 2023c.
Huang, C., Liu, Q., Lin, B. Y., Pang, T., Du, C., and Lin, M. Lorahub: Efficient cross cross-task generalization via dynamic lora composition. arXiv preprint arXiv:2307. 13269, 2023.
Jin, D., Pan, E.,, Oufattole, N., Weng. W.-H., Fang, H., and Szolovits, P. What disease does this patient have? large- scale open domain question answering dataset from med- ical exams. Applied Sciences, 11(14):6421, 2021.
Huang, J.-t., Zhou, J., Jin, T., Zhou, X., Chen, Z., Wang, W., Yuan, Y., Sap, M., and Lyu, M. R. On the resilience of multi-agent systems with malicious agents. arXiv preprint arXiv:2408.00988 2024.
Kannan, S. S., Venkatesh, V. L., and Min, B.-C. Smart-llm: Smart multi-agent robot task planning using large lan- guage models. arXiv preprint arXiv:2309. 10062, 2023.
Ilharco, G., Ribeiro, M. T., Wortsman, M., Schmidt, L., Hajishirzi, H., and Farhadi, A. Editing models with task arithmetic. In The Eleventh International Conference Learning Representations, 2023.
Kennedy, J. and Eberhart, R. Particle swarm optimization. Proceedings of ICNN"95-inteenatiioal conference on neural networks, volume 4. pp. 1942-1948. iece, 1995.
InternLM Team, I. Internlm: A multilingual language model with progressively enhanced capabilities, 2023.
Köpf, A., Kilcher, Y., von Rütte, D., Anagnostidis, S.,
Ishibashi, Y. and Nishimura, Y. Self-organized agents: A Ilm multi- aaget framework toward ultra large-scale code generation and optimization. arXiv preprint arXiv:2404.021883 2024.
Tam, Z. R., Stevens, K., Barhoum, A., Nguyen, D., Stan- ley, O., Nagyfi, R., al. Openassistant conversations- democratizing large language model alignment. Advances in Neural Information Processing Systems, 36, 2024.
Ivison, H., Wang, Y., Pyatkin, V., Lambert, N., Peters, M., Dasigi, P., Jang, J., Wadden, D., Smith, N. A., Beltagy, I., et al. Camels in a changing climate: Enhancing Im adaptation with tulu 2. arXiv preprint 101002231..10702 2023.
Kudugunta, S., Huang, Y., Bapna, A., Krikun, M., Lepikhin, D.,, Luong, M.-T,, and Firat, O. Beyond distillation: Task- level mixture-of-experts for efficient inference. In Find- ings of the Association for Computational Linguistics: EMNLP 2021, I, pp- 3577-3599, 2021.
12

---

MODEL SWARMS
Lambora, A., Gupta, K., and Chopra, K. Genetic algorithm- literature review. In 2019 international conference on machine learning, big data, cloud and parallel computing (COMITCon), pp. 380-384. IEEE, 2019.
Mavromatis, C., Karypis, P., and Karypis, G. Pack of Ilms: Model fusion test-time via perplexity optimization. arXiv preprint arxiv:2404.1133/ 2024.
Pal, A., Umapathi, L. K., and Sankarasubbu, M. Medmeqa: A large-scale multi-subject culti-choice dataset for medi- cal domain question answering- In Conference on health, inference, and learning, pp. 248-260, PMLR, 2022.
Lange, R., Tang. Y., and Tian, Y. Neuroevobench: Bench- marking evolutionary optimizers for deep learning ap- plications. Advances in Neural Information Processing Systems, 01:32160-3211 2023.
Pfeiffer, J., Goyal, N., Lin, X., Li, X., Cross, J., Riedel, S., and Artetxe, M. Lifting the curse of multilinguality by pre-training modular transformers. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 3479-3495, 2022.
Lange, R., Tian, Y., and Tang. Y. Large language models as evolution strategies. In Proceedings of the Genetic and Evolutionary Computation Conference Companion, pp. 579-582, 2024.
Lewis, M., Bhosale, S., Dettmers, T., Goyal, N., and Zettle- moyer, L. Base layers: Simplifying training of large, sparse models. In International Conference Machine Learning, pp. 0265-6274 PMLR, 2021.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. Er- mon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. Ad- vances in Neural Information Processing Systems, 36, 2024.
Li, M., Gunurangan, S., Dettmers, T., Lewis, M., Althoff, T.,, Smith, N. A., and Zettlemoyer, L. Branch-train-men merge: Embarrassingly parallel training of expert language mod- els. arXiv preprini arXiv: 22Xx:2208.003006, 2022.
Rame, A., Kirchmeyer, M., Rahier, T., Rakotomamonjy, A., Gallinari, P., and Cord, M. Diverse weight averaging for out-of-distribuution generalization. Advances in Neural Information Processing Systems, 35:10221-10836 2022.
Li, P., Zhang, Z., Yadav, P., Sung. Y.-L., Cheng, Y., Bansal, M.. and Chen, T. Merge, then compress: Demystify ef- ficient smoe with hints from its routing policy. In The Twelfth International Conference on Learning Represen- tations, 2024a.
Ramé, A., Ferret, J., Vieillard, N., Dadashi, R., Hussenot, L., Cedoz, P.-L., Sessa, P. G., Girgin, S., Douillard, A., and Bachem, O. Warp: On the benefits of weight aver- aged rewarded policies. arXiv preprint arXiv:2406.16768 2024.
Li, S. S., Balachandran, V., Feng, S., Ilgen, J., Pierson, E., Koh, P. W., and Tsvetkov, Y. Mediq: Question-asking lims for adaptive and reliable medical reasoning. arXiv preprint arXiv:2406.0099 2024b.
Rame, A., Vieillard, N., Hussenot, L., Dadashi, R., Cideron, G., Bachem, O., and Ferret, J. Warm: On the benefits of weight averaged reward models. In Forty-first Interna- tional Conference on Machine Learning, 2024.
Lian, W., Goodson, B., Pentland, E., et al. Openorea: An open dataset of gpt augmented flan reasoning traces, 2023.
Rao, A., Yerukola, A., Shah, V., Reinecke, K., and Sap. M. Normad: A benchmark for measuring the cultural adaptability of large language models. arXiv preprint arXiv:2404.1222 2024.
Lin, B., Tang, Z., Ye, Y., Cui, J., Zhu, B., Jin, P., Zhang, J., Ning, M., and Yuan, L. loeval Mixture of ex- perts for large lssoonllagura models, arXiv preprint arXiv:2401. 15947, 2024.
Roller, S., Sukhbaatar, S., Weston, J., al. Hash layers for large sparse models. Advances in Neural Information Processing Systems, 34: 34:17555-17566 2021.
Lin, S., Hilton, J., and Evans, O. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Compu- fational Linguistics (Volume I: Long Papers), pp. 3214- 3252, 2022.
Schulman, J., Wolski, F,, Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint (00.vv:7770006444 2017.
Liu, A., Han, X., Wang, Y., Tsvetkov, Y., Choi, Y., and Smith, N. A. Tuning language models by proxy. arXiv preprint ar XXX::2401.08563. 2024.
Shen, S., Hou, L., Zhou, Y., Du, N., Longpre, S., Wei, J., Chung, H. W., Zoph, B., Fedus, W., Chen, X., et al. Mixture-of
meets instruction tuning: A winning combination for large language models. In The Twelfth International Conference on Learning Representations, 2024a.
Masoudnia, S. and Ebrahimpour, R. Mixture of experts: a literature survey. The Artificial Intelligence Review, 42 (2)2255, 2014.
13

---

MODEL SWARMS
Shen, S. Z.,, Lang, H., Wang, B., Kim, Y., and Sontag, D. Learning to decode collaboratively with multiple lan- guage models. arXiv preprint arXin:2403.03870, 2024b.
Wang, H., Polo, F. M., Sun, Y., Kundu, S., Xing, E., and Yurochkin, M. Fusing models with complementary exper- tise. In The Twelfth International Conference Learning Representations, 2024b.
Shen, W., Li, C., Chen, H., Yan, M., Quan, X., Chen, H., Zhang, J., and Huang, F. Small llms are weak tool learn ers: A multi-llm agent. arXiv preprint arXiv:2401.07324 2024c.
Wang, K., Dimitriadis, N., Ortiz-Jimenez, G., Fleuret, F,, and Frossard, Localizing task information for improved model merging and compression. In Forty-first Interna- tional Conference Machine Learning, 2024c.
Shi, W. Min, S., Yasunaga, M., Seo, M., James, R., Lewis, M., Zettlemoyer, L., and Yih, W.-t. Replug: Retrieval- augmented black-box language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), pp- 3364-8377, 2024.
Wang, L., Vincent, N., Rukanskaitē, J., and Zhang, A. X. Pika: Empowering non-programmmrs' to author executable governance policies in online communities. arXiv preprint arXiv:2310.0433 2023.
Wang, Q. Wang, Z., Su, Y., Tong, H., and Song, Y. Re- thinking the bounds of Il reasoning: Are multi-agent discussions the key? arXiv preprint arXiv:2402.1888 18277. 2024d.
Smit, A. P., Grinsztajn. N.. Duckworth, P., Barrett, T. D., and Pretorius, A. Should we be going mad? a look at multi-agent debate strategies for llms. In Forty-first Inter- national Conference on Machine Learning, 2024.
Wang, Y., Ma, X.,, Zhang, G., Ni, Y., Chandra, A., Guo, S., Ren, W., Arulraj. A., He, X., Jiang, Z., et al. Mmlu-pro: A more robust and challenging multi-task language under- standing benchmark. arXiv preprint arXiv:2406.01574 2024e.
Sorensen, T. Moore, J., Fisher, J., Gordon, M. I.., Mireshghallah, N., Rytting, C. M., Ye, A., Jiang, L., Lu, X., Dziri, N., et al. Position: roadmap to pluralistic alignment. In Forty-first International Conference on Machine Learning, 2024.
Wang, Z., Wang. Z., Le, L., Zheng, H. S., Mishra, S., Perot, V. Zhang, Y., Mattapalli, A., Taly, A., Shang, J., et al. Speculative rag: Enhancing retrieval augmented genera- tion through drafting. arXiv preprint arXiv:2407.08223. 2024f.
Sun, C., Huang, S., and Pompili, D. Lim-based multi-agent reinforcement learning: Current and future directions. arXiv preprint arXiv:2405. 11106, 2024.
Wolf, T., Debut, L.,, Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. Huggingface's transformers: State-of-the-art natural language processing, arXiv preprint arXiv: 1910.037,, 2019.
Talebirad, Y. and Nadiri, A. Multi-agent collaboration: Har- nessing the power of intelligent Ilm agents. arXiv preprinl arXiv:2306.033/4 2023.
Tang, A., Shen, L., Luo, Y., Yin, N., Zhang, L., and Tao, D. Merging multi-task models via weight-ensembling mix- ture of experts. In Forty-first International Conference on Machine Learning, 2024.
Wortsman, M., Ilharco, G., Gadre, S. Y., Roelofs, R., Gontijo-Lopes, R., Morcos, A. S., Namkoong, H. Farhadi, A., Carmon, Y., Kornblith, S., et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International conference on machine learning. pp. 23965 23998. PMLR, 2022a.
Wadden, D., Lin, S., Lo, K., Wang, L. L., van Zuylen, M., Cohan, A., and Hajishirzi, H. Fact or fiction: Verifying scientific claims. Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp.
2020.
Wortsman, M., Ilharco, G., Kim, J. W., Li, M., Kornblith, S., Roclofs, R., Lopes, R. G., Hajishirzi, H., Farhadi, A., Namkoong, H., et al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7959-7971, 2022b.
Wan, F. Huang, X., Cai, D., Quan, X., Bi, W., and Shi, S. Knowledge fusion of large language models. In The Twelfth International Conference on Learning Represen- tations, 2024.
Wang, H., Feng S., He. T., Tan, Z., Han, X., and Tsvetkov, Y. Can language models solve graph problems in natural language? Advances in Neural Information Processing Systems, 36, 2024a.
Wu, X., Wu, S.-h., Wu, J., Feng, L., and Tan, K. C. Evolu- tionary computation in the era of large language model: Survey and roadmap. arXiv preprint 201: 2024.
14

---

MODEL SWARMS
Xu, C., Sun, Q. Zheng, K., Geng, X., Zhao, P., Feng, J., Tao, C., and Jiang, D. WizardIm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304. 12244, 2023.
Zheng, C., Wang, Z., Ji, H., Huang, M., and Peng, N. Weak-to-strong extrapolation expedites alignment. arXiv preprint arXiv:2404. 16792, 2024.
Zheng. L.,, Chiang, W.-L.,, Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., al. Judging Ilm-as-a-judge with mt-bench and chatbot arena. Ad- vances in Neural Information Processing Systems, 36: 2023.
Yadav, P., Tam, D., Choshen, L., Raffel, C. A., and Bansal, M. mers-ng Resolving interference when merging models. Advances in Neural Information Processing Systems, 36, 2024.
Yang, E., Wang. 7., Shen, I.,, Liu, S., Guo, G., Wang, X., and Tao, D. Adamerging: Adaptive model merging for multi- task learning. In The Twelfth International Conference on Learning Reprexentations, 2024.
Zhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X., Efrat, A., Yu, P., Yu, L., al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36, 2024.
Yu, L., Yu, B., Yu, H., Huang, F., and Li, Language models super mario: Absorbing abilities from homol- ogous models free lunch. In Forty-first International Conference on Machine Learning. 2024.
Zhou, Y., Lei, T., Liu. H., Du, N., Huang, Y., Zhao, V., Dai, A. M., Le, Q. V. Laudon, et al. Mixpure-ofeexperts with expert choice routing. Advances in Neural Information Processing Systems, 15:7103-7114, 2022.
Zaman, K., Choshen, L., and Srivastava, S. Fuse to forget: Bias reduction and selective memorization through model fusion. arXiv preprini arXiv:2311 1.07682, 2023.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pp. 4890, 2019.
Zeng, Y., Wu, Y., Zhang, X., Wang, H., and Wu, Q. Autode- fense: Multi-agent I1m defense against jailbreak attacks. arXiv preprint ar WxXxx:2403.47788 2024.
Zhang, X., Shen, Y., Huang, Z., Zhou, J., Rong, W., and Xiong. Z. Mixture of attention heads: Selecting attention heads per token. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 4150-4162, 2022.
Zhang, Y., Wang, H., Feng. S., Tan, Z., Han, X., He, T., and Tsvetkov, Y. Can 1lm graph reasoning generalize beyond pattern memorization? arXiv preprint arXiv:2406. 15992, 2024a.
Zhang, Y., Yang, S., Bai, C., Wu, F., Li, X., Li, X., and Wang, Z. Towards efficient I1m grounding for embodied multi- agent collaboration. arXiv preprint 1rXiv:2405.14314 2024b.
Zhao, J., Wang, Z., and Yang, F. Genetic prompt search via exploiting language model probabilities. In IJCAI, pp. 5296-5305, 2023.
Zhao, Q. Wang. J., Zhang, Y., Jin, Y., Zhu, K., Chen, H., and Xie, X. Competeai: Understanding the competi- tion dynamics of large language model-based agents. In Forty-first International Conference on Machine Learn- ing, 2024.
15

---

MODEL SWARMS
A. Discussion
Resilience to Malicious Experts There is discussion in multi-agent research about the impact of malicious agents (Huang et al., 2024). However, MODEL SWARMS is robust to malicious experts since the only time model has influ- ence on others is when it becomes the global best g, while an intentionally "bad" model has no chance of becoming g on the "good" utility function f.
Three Key Strengths of MODEL SWARMS 1) training- free: by training-free we mean that the composition of mod- els in MODEL SWARMS doesn't 't require specific training objectives, loss function, gradient descent, back propa- gation. This alleviates data dependency: by using as few as 200 examples MODEL SWARMS could produce better adapted experts, while that is only a bit over 3 batches for training-based approaches with a typical effective batch size of 64. 2) automatic discovery or assumption-free: instead of dictating the composition of models in A=B+C-D formu- las, MODEL S SWARMS automatically discover better adapted experts through swarm intelligence without making assump- tions about experts and how they should be composed. 3) any adaptation objective: the collaborative search is only guided by a particle-to- so-alar utility function f which could be any thing: dataset performance, reward model scores, human interests, and more.
MODEL SWARMS and Multi-Agent Systems The role of all "experts" in MODEL. SWARMS is homogeneous, i.e. they pursue the same goal/adapt to the same objective as represented by utility function f. In multi-agent systems (Rame et al., 2022; Zaman et al., 2023; Ainsworth al.. 2023; Chan al., 2024; Talebirad & Nadiri, 2023; Chen et al., 2023a; Zhang et al., 2024b; Abdelnabi et al., 2024; Kannan et al., 2023; Zeng et al., 2024; Guo et al., 2024b; Sun et al., 2024; Han et al., 2024; Ishibashi & Nishimura, 2024;
Wang et al., 2024d; Zhao et al., 2024; Chen et al., 2024c;
Hong et al., 2024; Smit et al., 2024; Chen et al., 2024a;b), the agents often have different roles to jointly complete a task, albeit those roles are more or less hand-crafte and especially through prompting. We envision future work on adapting MODEL SWARMS and automatically discovering heterogeneous and collaborative agents that jointly serve a purpose.
MODEL SWARMS and Optimization Research MODEL SWARMS is in part inspired by particle swarm optimization, one algorithm in the very rich literature of optimization research. We don't claim that PSO is the only and best applicable algorithm in the modern LLM world: on the con- trary, we invite follow-up works that critically examine how classic optimization techniques, especially for non-convex problems without strong guarantees, could be revived in today's 's context.
MODEL SWARMS and Model Merging MODEL SWARMS is both searching and merging (Wortsman et al., 2022b; Davari & Belilovsky, 2023; Deep et al., 2024; Yang et al., 2024; Wan et al., 2024; Rame et al., 2024; Fu et al., 2024; Ramé al., 2024; Li et al., 2024a; Tang et al., 2024; Wang et al., 2024c; Du & Kaelbling, 2024): searching in the sense that models proactively moving in the search space for better experts instead of passively being squashed to- gether, merging in the sense that each resulting model is im- plicitly an expert taking input from other models and chang- ing its weights accordingly. Contrary to the often "many-to- one" paradigm in model merging research where there is only one merged model, MODEL SWARMS is a "many-to -to- many" operation that yields multiple adapted experts, which open the door for further model merging, another search based on the result of a previous search, MoE routing, and more.
Non-Neural Reward Models In Figure 4 we demonstrate that MODEL SWARMS could adapt to preferences repre- sented by neural reward models. However, any model-to- scalar utility function f could work and non-neural RMs are definitely possible: optimizing engagement in social media posts, optimizing throuuhh rates in online ads, optimiz- ing charity donations when advertising a righteous case. We see many positive (and also negative) possibilities when em- ploying MODEL. SWARMS in conjunction with non-neural RMs in social-cconomic contexts.
Long vs. Short In Figure 4 we demonstrate that MODEL SWARMS could steerably adapt to either verbose RM or con- cise RM, offering use agency and controllability in model behavior. We discuss the distinctions with two other po- tential solutions: 1) setting max_new max_new_tokens, which might result in cutoffs in generated texts; 2) penalizing [EOS] tokens, which might tamper with token probabilities and harm generation quality. For a more on-the-fly steerability, we suggest to separately conduct MODEL SWARMS for two conflicting objectives, then employ an interpolation of the two models with user-controlled scaler from to
B. Analysis (cont.)
Visualizing Search Trajectory Since the same arithmetic is applied equally to all model parameters, could visu- alize the search trajectory of L.M experts by plotting any two parameter values. Figure 6 demonstrates that starting as diverse LLM experts, models collaboratively search in the weight space and converge to a weight area that best optimizes the objective f.
16

---

MODEL SWARMS
VELLD Tejectory
Hellaning Thanctory
SETTING
MMLU
Hellaswug
NLGruph AbstainQA
Sixt
FULL
0.583
0.652
0.672
0.175
W/O 1
0.504
0.587
0.530
0.099
2.0
100
W/O 2
0.516
0.615
0.523
0,049
W/O
0.344
0.611
0.547
0.147
-1.0
-102
W/o 1 & 2
0.561
0.601
0.611
0.091
-5.00 -002 -10 al E02
-123 -1.0 -001
0.2
w/o & 3
0.536
0.600
0.527
0,055
W/O 2 & 3
0.554
0.606
0.532
0.082
Figure 6: Visualization of the model search trajectories on two datasets, where each colored line represents the movement in weight space for one LLM expert. Diverse experts collahoratively search for composition and converge to adapted models in the weight space.
W/O 1 & 2 & 3
0.528
0.611
0.541
0,072
Table 5: Performance with randomness in 1) initial interpo- lation, 2) starting velocity, and 3) velocity update removed.
Hallowa
Ni.roph
0.88
5.70
MVIUU
Hellaswag
0.6
0.50
0.58
2.50
0.
0.52
0.04
0.00
2.00
373
Dropk-0.2
DropN-022
Dripk-35 DropNC555
Dropk-0 DiophiG.
Figure 7: Performance with Drop-K and Drop-N, speeding up MODEL SWARMS by up to 80% with only a 6.0% drop.
Figure 8: Performance variance across runs with each circle representing the best-found expert of a run: red line indicates the best baseline. Despite randomness, MODEL SWARMS finds experts better than any baseline in 73% of the runs.
Accelerating with Dropout-K/N By default, the util- ity function f is evaluated for every LLM expert at ev- ery single iteration. To speed up, we propose Drop-K da and Drop-N dæ: randomly skipping model evaluation in di. % of iterations or for d, % of experts. We evaluate (dg,d,) = {0.2, 0.5, 0.8) and present model performance in Figure 7. With an speed up of up to 80% comes with only a slight performance drop of 6.0% on average, indicating that Drop-K and Drop-N present two helpful strategies to reduce the computational costs of MODEL. SWARMS while maintaining good expert utility.
the val/test performance variance in Figure 8. Despite the randomness, MODEL SWARMS is consistently producing adapted experts better the best baseline, outperforming it in 73% of runs.
Collaboration of Weak > Strong When we don't 't have strong starting experts to begin with, would MODEL SWARMS enable the collaboration of weaker models to beat the strong? We investigate this by sorting {xr]== by utility f. withhold the top-1 model and see whether the collabora- tion of the remaining experts would surpass it, i.e., whether Swarm (x; | x1. We also evaluate the collaboration of the bottom half, Swarm (fx, )/=n/2), and present perfor- mance in Figure 9. It is demonstrated that the collaboration of weak models could beat the top-1 expert, with an average improvement of 35.4% across the four datasets. The collab- oration of the bottom half also outperforms the top-1 in out of 3 datasets, suggesting that MODEL SWARMS enables the weak-to-strone (Burns et al., 2024) transition of LLM experts through collaborative search.
We explicitly enable randomness in MODEL SWARMS, with the hope of boosting exploration and adaptation. Specifically, randomness comes in three steps:
Randomness Ablation
1. random interpolation to grow initial experts {x, J populate((x:]) (x:) N > n;
2. random starting velocity v, <- (andom((x:).)) X;;
3. random velocity update weights re, Tp: Tg: Iw ~ (0, 1);
Compositional Capability through Joint Utility Func- tions We investigate whether MODEL SWARMS could adapt to compositional tasks by jointly optimizing two dif- ferent datasets. Specifically, we investigate "QA+gruph reasoning = multi-hop QA" with MMLU, NLGraph, and Knowledge Crosswords. We compare the joint utility func- tion of harmonic mean performances against the best single expert without search or searching to optimize one task only.
We conduct an ablation study where we disable the three randomness in Table 5. We find that the three randomness factors all contribute to model performance across the four datasets, while the deterministic variant (no 1 & 2 & 3) would result in a 23.5% drop on average.
We further visualize performance variance due to these ran- domness factors. We run for up to 200 times, and present
17

---

MODEL SWARMS
top-1
Swam(2-10;
Swamm/5-10 10)
of adapting one dataset. Results in Table 7 demonstrates that MODEL. SWARMS is general and works regardless of base model.
0.55
0.50
Hyperparameter We by default run grid search over several hyperparameters: step length X, inertia @v. cognitive coefficient Φp. social coefficient Pg. and repel coefficient Pw. We dissect performance with each hyperparameter value in Table 8. It is demonstrated that the changes are minor, thus MODEL. SWARMS is largely robust to different hyperparam- eter configurations.
MMLU
Hellswwag
NLGrraph
Figure 9: Performance of the best initial expert, s2arm 10), and swarm(6-10). The collaboration of weak models outperforms the top expert through MODEL SWARMS.
SETTING
MMLU
Hellaswag
NLGraph
AbstainQA
PULL
0.672
CROSSOVER,
0.527
0.604
0.534
0.093
CIOOKOOKIR
0.587
Time and Space Complexity For MODEL SWARMS with n. particles, k iterations, the time of validation set inference as D1. the time of test set inference as D2, the time of weight arithmetic as w, the probability of global best updating as a, then the time complexity is n(D; + 2D: + k[(5 + a)w + Di)) and is (n) and (k)) indicating linear growth with the amount of particles and iterations. For space, the peak storage requirement is 3n + 1 copies of the LoRA adapters: given the tiny size of adapters, any n < 100 should be reasonable.
VELOCUEVSSES
0.518
0.613
0.542
0.031
VELOCITY:ZBRO
0.516
0.615
0,525
0.649
REPEL
0.534
0.631
0.534
0.025
SCHEINLE
0.11
0.536
NO RESTART
0.532
0.628
0,532
0.131
Table 6: Ablation study removing the five modifications to PSO.
Figure/ 11 demonstrates that MODEL SWARMS could indeed adapt to compositional tasks by utilizing a combined utility function.
The implementation of MODEL SWARMS employs multipro- cessing, essentially distributing the evaluation of particles to GPUs with m concurrent threads. We empirically ana- lyze the time complexity of employing to 10 GPUs on cluster of 16 A100 GPUs with 96 CPU cores with 10 default initial experts. Figure 12 demonstrates that the benefit of more GPUs gradually diminishes due to multiprocessing costs, with GPUs as being optimal on our machine. We show the general trade-off between GPU computation time and multiprocessing time while the exact time per iteration is not meaningful.
Pareto-Optimal In adaptation objective 2: multi-task do- mains, argue that the joint optimization of multiple tasks in single domain might be better than separately optimiz- ing just one. We investigate this by comparing the joint optimization against only optimizing only dataset or 2 in Figure 10. MODEL SWARMS produce mostly Pareto- Optimal experts that's s better than optimizing one dataset in most cases.
Qualitative Examples We present qualitative examples for objective 4: human interests, essentially (instruction, pre response, post response) tuples, where human evaluators judge MODEL SWARMS as winning, tying, or losing to initial experts in Tables 13, 14, and 15.
Modularity: Adding and Removing Experts MODEL SWARMS presents modular multi-LLM collaboration sys- tem, where experts could be added/removed from the com- position even after a search. For adding experts, since the only time a particle has an influence on others is when it becomes global best, we only start the search with the new particle when and if it were to become g. We empirically test this by withholding several experts and injecting oth- ers in 5 + 10, 8 + 10, and 9 + 10 settings Figure 13. Adding experts in this way is generally helpful, while injecting fewer experts is more effective.
Ablation Study MODEL SWARMS features five major dif- ferences from the classic swarm intelligence for LLM opti- mization: 1) crossover through interpolation and expanding initial expert pool; 2) randomize initial velocity; 3) adding repel term; 4) adding step length schedule; 5) restarting failing particles. We conduct an ablation study for these five factors in Table 6. It is demonstrated that they are all helpful for model performance, while 1) crossover is most useful.
As for removing experts, MODEL SWARMS presents a tech- nical guarantee for completely removing an expert and all its influence on other models. We first expand the velocity update term on step t:
Other LLMs To show the generality of MODEL SWARMS, we replace GEMMA-7B with MISTRAL-7B (mistralai/MistrallP7) -nsstrtcctt) and re-run evaluation
18

---

MODEL SWARMS
beST
our
OUr 142
bestarglo
ouss
cura 112
Dest stople
curs
curs 2
dux 142
beet ongle
curs
OUTS
cun 142
1dd
dede
1-dw
Zeea
Treet 2
1902
7-dm
7dd
Figure 10: Performance of best single expert, ours only optimizing task or 2. and jointly optimizing tasks I and 2. The domains of medical, legal, science, and culture are presented from left to right. MODEL SWARMS produces Pareto-Optimal expert than uni-task optimization.
MMLU
MMLU .ro
Hellaswag
K-Crosswords
GSM8k
NLGraph
TruchfulQA
RTP
AbstainQA
dev
test
dev
lest
dev
lest
dev
test
dev
test
dev
lest
dev
test
dev
test
dev
lest
best single
0.385
0.433
0.257
0,146
0,545
0,550
0.415
0.364
0.190
0.303
0.335
0.325
0,380
0.353
0.898
0.873
-0.130
0.081
ours
0.510
0.510
0.271
0.160
0.640
0.664
0.470
0.497
0.290
0.328
0.380
0.358
0.440
0.405
0.906
0.881
0.095
0.199
Table 7: Performance of single-dataset adaptation with MISTRAL-7BB
original
only MMLLU
only Graraph
joint
MMLU
NLGraph
TruthfulQA
0:50
0.45
avg
std
avg
std
avg
std
all
0.557
0.011
0.585 0.036 0.365 0.014
0.35
0.10
0.556
0.012
0.582
0.033
0.363
0.015
0.30
inertia
0.20
0.557
0.010
0.586
0.037
0.365
0.013
0.25
0.30
0.556
0.010
0.590
0.039
0.366
0.013
dev
lest
0.10
0.557
0.010
0.584
0.041
0.362
0.015
0.20
0.558
0.011
0.588
0.037
0.364
0.013
Figure/ 11: Performance on Knowledge Crosswords with
cognitive coeff.
0.30
0.556
0.009
0.590
0.041
0.367
0.014
optimizing dataset or joint task optimization.
0.40
0.556
0.011
0.587
0.033
0.365
0.014
0.50
0.557
0.012
0.578
0.028
0.365
0.014
0.20
0.558
0.012
0.600
0.040
0.365
0.012
0.30
0.558
0.011
0.593
0.037
0.365
0.012
social coeff.
0.40
0.556
0.010
0,587
0.039
0.365
0.014
0.50
0.556
0.010
0,570
0.023
0.365
0.014
0.60
0.554
0.010
0.576
0,032
0.363
0.015
0.553
0.009
0.565
0.013
0.367
0.015
repel coeff'
0.05
0.558
0.010
0.587
0.037
0.364
0.014
0.10
0.559
0.012
0.606
0.040
0.363
0.012
0.50
0.558
0.010
0.583
0.028
0.366
0.011
0.60
0.558
0.009
0.587
0.035
0.368
0.014
0.557
0.010
0.584
0.036
0.364
0.014
step length
0,556
0.012
0.593
0.043
0.367
0.014
0.90
0.556
0.011
0.589
0.040
0.363
0.013
1.00 0.555 0.012
0.578
0.034
0.361 0.015
Figure 12: Time per iteration changes with increasing num- ber of GPUs.
Table 8: Average model performance under various hyper- parameter values.
19

---

MODEL SWARMS
MMLU
NLGroph
pre-search
OLTSS
Gemini-flash
Gemini-pro
0.70
0.4
0.50
0.3
0.50
0.2
0.50
cety crty orly
Hellaswag
Figure 13: Performance when new experts are injected, from 5 to 10, from to 10, and from 9 to 10, across three datasets. MODEL. SWARMS presents the possibility of injecting ex- perts after a search.
Figure 14: Entropy of model responses indicating sensitivity to 10 prompt versions with minor differences, the lower the better.
in the process? We visualize the change of each particle as well the global best in term of utility function f in Figure 15. Experts explore the weight space, their utility scores wax and wane, leading to consistent bumps in global best scores and consequently better adapted language models.
V, roRoVe-1 tr x1-1) + rg x,-1)
Te xx-1)
= = t (rpOp + trogg -rudw/xi-1
+ -rg088-11 - Tm pm
Prompt Variation We hypothesize that by optimizing the weights, MODEL SWARMS might offer stronger robust- ness minor prompt changes. We employ GEMINI-PRO to "Please paraphrase the question into 10 versions with minor differences." ", evaluate models the 10 versions, and cal- culate the entropy of response distributions as indicators of sensitivity. Figure 14 demonstrates that MODEL SWARMS drastically reduce the sensitivity to minor prompt changes, while still being a bit shy of Gemini flmini-flash/pro levels.
The updated location at step i then:
x, = X1-1 Ave
= -x-11 + Voooov-- -rpp0 + rg0g
- - X + Tg rrowrBw..t1
Comparison with Training Instead of running MODEL SWARMS, what if directly fine-tune models on the vali- dation set with its 200 data points? We compare the perfor- mance of pre-search best initial expert, post-search global best, and SFT in Table 9. MODEL SWARMS outperforms SFT, indicating that we offer stronger solution for model adaptation in low-data regimes with as few as 200 instances while SFT might be over-fitting.
= =rrob + Arp0pPi-1 + | Xr + trogs
- -rwbw) x1-1+ Arg0zg1-1 Ar
Note that Vt-1, 1t and x are the property of the particle itself, while g:-1 and gwa-1 are the property of potentially other particles. As a result, simply remove the gi-1 and gw,r-1 terms if g:-1 and/or gw,t-1 come from the expert the removed and normalize the remaining weight terms. For example, if g:-1 andr gw,t-1 are both from the particle to be removed, then:
C. Experiment Details
Dataset Details We employ 20 datasets in total to evaluate MODEL SWARMS and baselines: 9 for objective 1: single task, 8 for objective 2: multi-task domains, 2 for objective 3: reward models, and we synthesize 16-domain instruction dataset from Gemini
x = c[(xx - Argdg + Ara On Omn gwiz-1
for objective human evaluation. We randomly sample subsets from each dataset and present the statistics in Table 10. We also ploy the z-test with the one-tailed hypothesis and present statistical significance test results on the applicable objective 1: single task datasets.
where C Arpputrrpdpt 1-M(rnppr ti) +rr
XrobctArpoe -trrrpotrad ow
is the weight normalization factor. Starting from = 1 to = K. for every x, this removes the specified expert(s) from the composition of other models.
Implementation Details For a prototype of MODEL
Search Dynamics What exactly is happening during a MODEL. SWARMS search and how did expert utility change
SWARMS, we employ GEMMA-7B (google/gemma-7bhl as the base model checkpoint, then fine-tune it on 10 different
20

---

MODEL SWARMS
MMLU
_rroU.rro
Hellaswag
GSM8k
NI.Graph
TruthfulQA
best single
0.537
0
0.601
0.237
0.535
0.308
SET
0.450
0.167
0,513
0.279
0.585
0,359
Model Swarms
0,583
0.254
0.652
0.459
0.672
0.392
Table 9: MODEL SWARMS outperforms directly training LLMs on the 200-instance validation set.
Size
rameters and report the best-found expert based on utility function f. Specifically, we search for d. E (0.1, 0.3), 0.2,
Datasct
Source
dev
MMLU
(Hendrycks al., 2021)
200
1000
Op E {0.1, 0.2, 0.3, 0.4, 0.5), Og E {0.2, 0.3, 0.4, 0.5, 0.6), @n E {0.01, 0.05, 0.1). À E (0.5, 0.6, 0.7, 0.8, 0.9, 1.0). We run up to 200 to 1000 runs by randomly choosing over these hyperparameter search settings and report the best-found expert on utility function f. Though there is randomness, Figure 8 demonstrates that MODEL SWARMS is robust to hyperparameter settings and consistently find experts better than any of the 12 baselines.
MMLU-pr
(Wang et al., 2024c)
K- S-Chossoodss"
(Ding et al., 2024)
200
1000
Hellaswag"
(Zellers
NI Grmph
(Wang et al., 2024a)
200
1000
GSM8K
[Cobhe es al., 2021)
200
1000
TruthfaOQA
(Lin al., 2022)
200
617
ReaTToxicity "ranopooiitt
(Gehmun ol., 20200
200
1000
AbstainQA"
(Teng et 2024a)
MidOA
al. 202/bb
MedMCQA
(Pal et 2022)
200
1000
Hearsay
(Guha 2024)
Citatinn Prodiction
(Goha et al.,
108
108
SeiFact
(Wadden et al., 2020)
200
532
STEM
(Wang et 2024c)
Baseline Details We employ 12 baselines in total:
Normad wicountry
(Rao et al., 20144)
500
2000
Normad w/value
(Rao 2024)
500
2000
AlpacaFarm
(Dubois et al., 2024)
200
400
Best single expert: among the 10 initial experts, the expert that performed best on utility function f is evaluated and reported.
Koola
(Geng ol., 2023)
eval
Gomini-symthesized
16*25
16"25
Table 10: Statistics of employed datasets. s, and #** indicates the improvement on this dataset is statistically significant with p < 0.1, p < 0.05, and p < 0.01 with one-tailed z-test.
Data merge: instead of separately training 10 initial ex- perts, we combine the SFT data and train expert, then evaluate and report its performance.
Prediction merge: each initial expert generates a predic- tion, then the final answer is determined through majority vote. Note this is not applicable to open-ended genera- tion tasks such as TerooxxiiiyPrompppip or tasks where the correct behavior vary across models such as AbstainQA.
supervised fine-tuning domains to obtain 10 initial experts. we specifically employ Tulu-v2 (Ivison et al., 2023), an open collection of instruction-tuning data. We specifically em- ploy the following subsets: flan (Chung et al., 2024), CoT, Open Assistant 1 (Köpf et al., 2024), ShareGPTt, Code Alpaca (Chaudhary, 2023), LIMA (Zhou et al., 2024), Wiz- ardLM Evol-Instruct V2 (Xu et al., 2023), Open-Orca (Lian et al., 2023), and Science Literature (Ivison et al., 2023). We replace the GPT4 Alpaca subset with Gemini Alpaca, distilling generations from gemini- 1.5-p-------rr-0-00 and remove the hard-coded subset. We employ LoRA fine-tuning (Hu et al., 2022) with a learning rate of 2e,4, cosine learning rate scheduling, effective batch size of 32, warm-up ratio of 0.1, and 5 default training epochs, while we only train for epoch on the large ShareGPT subset. We similarly tne-tune MISTRAL- for the experiments in Table 7. We employ greedy decoding for text generation and a maximum new token of 10, 50, 100, 512 depending on the task.
Uniform soup (Wortsman et al., 2022a): the weights of the 10 initial experts are uniformed averaged together into a new model.
Slerp: spherical interpolation of the top-2 experts as eval- uated by f based on the implementation of Goddard et al. (2024) with default hyperparameters.
Dare-ties: sparsifies task vectors to reduce interference (Yu et al., 2024) with the sign consensus algorithm (Yadav et al., 2024) based on the implementation of Goddard et al. (2024). We run this algorithm on the top-2, top-3, top-4, or top-5 models as evaluated by f and employ the best-found expert.
Model stocks: employ geometric properties of models to determine linear interpolation weights (Jang et al., 2024), We run this algorithm on the top-3, top-4, or top-5 models as evaluated by f and employ the best-found expert based on the implementation of Goddard et al. (2024).
Hy Hyperparameter Settings For MODEL SWARMS searches, we employ N 20, Φ^ 0.95, p = 10, Pr = 5, K. = 50, while running grid search over other hyperpa-
21

---

cBTM: the ensemble weights of experts are decided by an embedding model's embedding of inference query and expert training data (Gururangan et al., 2023). We employ GOBERTA-BASE as the embedding model to fuse the top- MODEL SWARMS
Greedy soup: starting from the expert with the best f scores, iteratively add the next-best expert into the soup of uniform averaging, retains the added expert if the soup becomes better and discard if not, until every expert is considered (Wortsman et al., 2022a).
while the best-found expert is evaluated the reward model scores the test set instructions.
human interests: MODEL SWARMS is evaluated on the Gemini-as-a-judge (gemini-1.5-faasee
scores on the dev set of instructions f. with the prompt format in Table 11 (We include human-authored example of good response in the domain for calibration). At infer- ence time the best-found expert on f is evaluated on 1) the Gemini scores on the held-out test instructions, 2) Facts&Evidence (Boonsanong et al., 2024) score of factuality, and 3) human evaluation win rates against pre-swarm experts. We employ total of 13 internal re- searchers annotators interested in these domains, while we ask 3 of the annotators to annotate the domain of PhD applications, resulting in Fleiss' Kappa of 0.62 indi- cating moderate to strong present the human evaluation instructions in Table 12.
Pack of LLMs:: the linear interpolation weights of models is decided by perplexity inference queries (Mavromatis et al., 2024). We run a hyperparameter search for 200 times with temperature from 0.1, 0.2, to and report the best-found expert by f-
2. top-3, top-4, or top-5 expert and report the best-found expert.
agreement. We
EvolMerge: employing genetic algorithm to combine models based on data/layer flow engineering (Akiba et al., 2024). We run for 200 times randomly crossover the lay- ers of the top-2 experts through linear interpolation to produce new models, while we keep maximum popula- tion size of 50, retain 10 best-found at every iteration, max iteration of and report the best-found expert.
LoraHub: dynamic LoRA composition by employing genetic algorithm optimize the linear interpolation weights of different LoRA modules (Huang al., 2023). We run for 200 times by employing population size of 50, 10 max iterations, a = 0 or = 0.05, and report the best-found expert.
Evaluation Details We describe the evaluation details in the 4 objectives:
a single task: MODEL. SWARMS and baselines are evalu- ated based on the performance on the validation as the utility function f. while the best-found expert is evaluated on the held-out test set.
multi-task domain: MODEL SWARMS and baselines are evaluated based on the harmonic mean of the performance the dev set of the two tasks as while the best-found expert is evaluated on the held-out test sets of the two tasks.
reward models: for the verbose and concise reward mod- els, we run the GEMMA-7B model on the validation set, record is response length distribution, calculate length score based on the percentile of the response in the distri- bution to the short or long end, average that score with the general reward model's score to obtain an overall score. MODEL SWARMS and baselines are evaluated based on the reward model scores on the dev set instructions as f,
22

---

MODEL SWARMS
For the following instruction and response, please rate the response scale of to 10, where the worst and 10 is the best. Below is an example of good (instruction, response) pair:
Instruction: How does tourism influence property values and the real estate market in major tourist destinations, and what are some of the potential benefits and drawbacks of these changes for local residents?
Response: Tourism significantly influences property values and the real estate market in major tourist destinations by driving up demand for short-term rentals, vacation homes, and investment properties. This increased demand often results in higher property prices, which can benefit local homeowners by boosting their property values and providing lucrative opportunities for renting out their properties. However, the drawbacks for local residents include affordability issues, as higher property prices can make it challenging for locals to buy rent homes, potentially leading gentrification and displacement. Additionally, overemphasis on tourism-related development may strain local infrastructure, disrupt community cohesion, and reduce the availability of amenities and services tailored residents' needs rather than tourists.
Please respond with "Rating: 2/10" first and then provide your reason.
Instruction: [instruction]
Response: [response]
Table 11: as-a--udge prompt.
Instructions
Thank you for being part the human eval! Fach example features one instruction and two responses: please help us by judging which response is "better", in the broadest sense possible, by entering (response 1 wins), 2 (response wins), or 3 (tie). Please note that:
>
These are open-ended questions without potential "gold" answer. Please evaluate whether the responses helpful, whether they fulfill the instruction, and whether you find it satisfactory, in broadest sense possible.
Don't worry if you are unsure about the factual aspects of the responses: we have separate FactScore-like methodology for that, so you only need to focus on the general quality of responses.
>
Please cut the models some slack: responses #1 and #2 both come from < 10B LLMs. Please bear this in mind and don't 't expect
ChatGPT-like quality.
Please don't penalize responses that were partially cut off.
> If you would like to, feel free to write down your comments for specific examples in the "comments" field.
Again, thank you so much and your efforts would be aeknowledged in the eventual paper!
Table 12: Human evaluation instructions.
23

---

MODEL SWARMS
lppic
insruction
resporse
post respunse
Processing (NLP) technology can siguticnntly cohance the drafting and review processes for legal documenss, ensuring precision and compliance. Here's is how: 1. Document Summarization and Anilysis axtract- ing Key Information: NLP can analyze logal documents und extract relevant information like contracts, agreements, and regulations. This allows lawyers quickly undersiand the core of complex documents, saving time and elfort. Idemiiyyng Legal Enittles and Relatioships: NLP can identity and casegorize logal extiles (e.g., people, organizations, locations) and their relationships within documens. This faciliuttes sucurae document organization and analysis Sentiment Analysis NI can determine the sentiment of legal decuments. helping lawyers understand the tone potential bus This be valuable in contrct negotiation or dispute reclution. 2. Automated Document Drafting: Standard Clauses and Contects: NIP can generate
Natural
Entareed Draflin and Review Processs Legal Decuments Through NLP Natural language pencessing P) revalutionizing the legal industry by witommtiing and improving various tasks, inchaling document drafting and re- view NIP technologies enhunce the accurncy, efficimey. and compliance relucing time and cost logal proczedings. Doc- umeat Dratting: Sentence Geseration: NLP models can generate seuttences that odhere to legal syntax and style, ralucing the need for mamual writing Contract Deafting: NIP can geserate contrntt and other lopal dacuments based specifle
How cun natural language procesing (NLP) technul- ogy enhance drafting
logall
legal Al
und review prucesses for
legal documents, ensuring precision and complianee?
conditions... Legall Briefs; NLP create legal brieds that summarize the facts of case and argue case in coacise and persussive mamnor.
ponrrete standard legal clanses and contracts bused specific industry, location, and other factors.
Regulatory framewarks for dromes in building identificatioo should be devel- oped focus
Regulatory frameworks for dranes in huilding identinication should be devel- oged with focus casuring key areas consider; Safety Security Lisbility Insurance: Establish clear liubility and insrance requirements for drume openaions in building iden- tifictiio. This protects victims in case of accidents or misuse. Training and Cenificanion: Require
security, and responsiible Here key areas consider: I. Safety Security: Lisbility and Insurance: Establish clear libility and insurance requiremens for drone operaiors in building kōen- tifcatiion This protects victims in case of accidents or misuse. Training und Cettifiation: Require drone operaturs undergo training and abizin certiiia- tion recognized aulhoritoes, This ensures compesency und adherence say standards. Flight Restriction:: Fatablish clear flight restrictions based on weather, time of day, and proximity to buildings, infrastrccure, and sensitive Data. Security and Privacy: Emplement robust data security and privacy regulations to protect sensiivve building informstion from unauthrrizdd acuess or misuse Liahilit for Dats Frros: Establish clear lisbilay for entor in data vollected by dromes, ensuring compensation for victims.
security responsible Here
How can regulisoey frame-
works developed sure the respunsible and safe use of drones building identiication?
undergo training and obtuin certifica- tion from recugnized authurides. This ensures compelency and atherencee safety standard, Flight Restictions: Fatabiich clene flight restrictions based on weather, time of day, and proximity to buildings, infrasinucture. and sensitive Data Security and Privacy: implemen nobusi data security and privacy regulations protect senvilive huikinng information from unuttorrized access or misuse, Libillyy for Data Frrees Eeablish clear lishility for errors data collected by drunes, eosuring compensation for vietims.
sincruft AT
Chinese tea pairing with meals is delicate art that harmonizes flavors, sestures, und aromas to elevate the dining experience. Here are the essencial elements und techniques: Elements: Tea: Choose less that cumplement the meal's flavors ingrediens.
Chinese tea pairing with meals a delicate unt that has been perfocted over centuries. is harmonious blend of tradition and innovation, and
key Chinose dining experience, Eseetial Elements of Chinese Pairing with Meals: Teapot: The teapot is the centerpiece of the lea puiring experience. li a beautiful and ornate vessel that steeped in history. The tespot is used to steep the tea leaves in hot water, and it also used to sarve the scscups
essential el- ements and techniques in the art of Chinese ice puir- ing with meals, and how they enhance the dining experience?
example, green puirs with delieate dishes, while black lea complements richer unes. Water Quality: Use filtred or spring water to enhance the tea's 's flavor and urmma Teapor: Select a tespoi thut maiches the type and browing method For example, Cliwan (stamming how is usod groon while Yixing pot
Islan food
guesss Teocups: They are usually made of purcelain, and they are decorated with intricate puzserns. Tei stminer: The loi strainer is used to filter the tea leaves fromm the tea. It is suully made a fine mesh, and it is placed in the scapor.
the guests.
for black Teacop. that alkows for proper brewing and allows the ten coal down appropriately Food: The food should be propaned a way that complements the lea. For example, spicy dishes pair well with cooling loss, while sweet foods pair with floral seas.
cup
utilize bass, their stylitiic differences evident in instrument's nole, playing techniques, and ovenll musical context. Jazr: Melodical Focus: The bass ofien lakes the lead, playing intricate meladies and improisational lines. Think of the hass second leud instrument, not just thythm section member. llrmmnnic Sophistcation: Jazz baslines oflen incerponte chnomaticism, dissunance, complex chveil stractures, soding hommonic depth and complexity. Dynamic Range: Jazz hass players exhibit a wide dynamic range, playsing with both power and fineser, often switching between the two within single phrase. Rhythm Section Symergy The hass a dysamii rlythus section, amphasizing inteplaay and improvisation.
Distinctive Bass Styles Jazz and Rock: While both
Sure, are some distinct stylastic differences between the use of bass jazz versus rock music. Jazz: Meladic: The hass is often used to play the melady, ruther than just the root notes. Harmanic: The boss is used to play chords. rather than root thythm patterss, including symcopation, swing, and waltz. Dynamic: The bass is used to play a variety of dynamic changes. inclading volume changes, pitch changes, and tempo changes, Rock: Rhythm: The bass is used to play a variety of thythm potterns, including striiht syncopated, and polythythmicc
What are some distinct stylistic differences tween the use jazz veruus rock music?
Rhythan The bass used play vuriety
instrument
with the drums and piano
Rio Jameiro were held in country was experiencing a severe comnomic crisis, and the games were major event that was affected by the crisis. The economic crisis was caused by number of factios, inchuling the colligse of the oil industry, the decline of the Bruzilian oconomy, and the rise the Zike virus. The Zika the ganees, The economic crisis hnd a number of negatie effects on the games. Fiind, it made difficult to attract athletes the games. Secved, made it diffieult to sponsors the games. Third, made difficult games. Fourti, made difficult to attract media to the games. econonicc prisis also had a nember of negative offects on Rio de Ianciro. Fins, it made it difficult to ottrace tourists to Rio de Janciro. Second, made diflish Jusetro. Third, it made difficult investors Rio de Janciro. Fourth, it made difficult to attract govemment officials to Rio de Jnneirn The cconomic aisis was major threst to the 2015 Olympic Games in Rio de Jaseiro. It was major threat to the athletes, the public, and the
The 2016 Olympic
Brazil's economic and political climate in 2016 signiicantly affected the orga- exccution of 2016 Olympic Games Rio Juncio. Here's impllaattions the city: Rccoomic Impact: High Inflaiio and Currency Elucosations: Brazil was facing a severe exenomic crisis in 2016, with high intlation and plummeting currency (the Bruzilian Real). This made difficult manage Olymgic bodges pacted the cost bosting the gumes. Budget Overuns: The Olympic Games were esiimated to cost S15 billion, but the final cost exceeded that by $2 billion due to inflatiom and currency Ductiations This put a strain on Brazil' is cooummy and forced guvernment make adjustments. Impact on Tourism: While plympics boosted tourism in shoet temm. Drazil's 's's ecomomic inscbility made difficult to sntnact long-tere term investment and tourism after the games.
How did Brazil's eco-
mujor was major sounce auxieyy for the athletes and the public.
a breakdown of ampact
nomic and political cli-
mate affect the organiza-
alympics
tion execution Olympic Games
to
Rio de Janeiro, and what were the implicatioss for the bost city?
businesses
Rio de Juneiro.
Table 13: Part of LLM responses where MODEL S SWARMS win over pre-swarm experts per human evaluation.
24

---

MODEL SWARMS
topic
insruction
response
post respunse
Chile's unique gengraphic location and landscape have significantly influ- enced economic wciviles, particularty mining and agriculture. Herre's ' how; Maning: Mineral Riches: Chile known as copper capital of world due its vast mineral deposits, purtculatly copper These deposits are concentrated in the Andes Mountains, leading to a sirong locus on mining. Challenges and Oppurtumiie:: The Andes mining operations, including difficuit terrain altitude sickness, and need for speciallized equipment. llowever, these challenges also contribute to the unique skills and es experiss of Chileun miners, making the industry a major empkoyer and contritutor to the comomy Favinonmentsl Impact: The minnnn a significant envinonmental focipeint, with large mining operations impacting water resurces, air quality, and land use. This hos led to environmental concems and the noed for sustainbbee practices.
How has Chile's is unique poographic locstion and landscape influenced economic activities, partie- ularly in mining and agri- caltre?
Chile's is unique geographic location and capes have significamlly induenced its econemic activiees, purtiularly in mining and agriculture Mining: Chile the world's brrgst cupper producer and the world's largst expetter of copper.
logivtical challenges for
south america
country 's copper deposils are locased the Andes Mountains, which are the world's 's largest cupper resource. - The Andes Mountins are also home to the world's largesi pold and silver mines - Chile's is mining indusry is majorr source of the country's 's GDP and employment.
indusary
The unimation style and visual design of Peppa Pig are key factors in its uppeal for preschoul audiences. Here's 's how they contribune: 1. Simplicity and Fumiliaity; Minimalist Design: The characters with few details makes easily identifibble simple, blocky águree young children. Limited Color Palette: The show uses a limited color puletie. often siickng to primary and secondary colors. This further simplifes the visual language and avods uverwbelming young viewers. Familiar Shapes and Pattems: churacters mask up of hasic shipes circles, squares, rastangles. This helps chilkren reoognize and lears shout shapes.
The unimation style and visual design of Peppa Pig contribute its for preschool mudienees in number ways. I. Simpicity and clarity: The saimo- fion style simple and clear, with focus on the main character's expressive eyes and mouth. The charaters are designed to he eailly siemifiable, with their distinct shapes and colours. The backggoondd are simple and unclurtredd
what ways does the imation styll visual design of Porpu Fig con- ribbutc to its appeal for preschool uudienccs??
less intimidating
CUrcoons
allowing the churacters stand
Tax policy reformed better support burdening the muddle and lower classes through a varetyy of mechanisus. I, Increase the marginal tax rate on the wealthy. This would raise signišccnni amount revenue. which could be used fund social programs. The wealiby less likely be affecied by a higher marginal tax rate, they more likely to be able to afford to pay more in taxes. This would not overburden the middle und lower classss, as they are les likely to be able to affund to pay tases.
programs
over-
Tax policy refoem can he designed better suppert social programss without overhurdening the middle and lower classes through combination of ap- proaches 1. Progressive Taxation. Incresse marginal tux rates for high eamens This
In what ways can tax pol- sey be reformed to better support social programs without overburdening the
policy
the wealthy contribute grenter taxes, offsatting the burdes lower camers, Impiement prngressive property tax system: This would increase taxes on larger homes and decresse taxes on smaller homes, ensuring that pruperty taxcs are more equitale..
their
in
middle and Jower classes?
de Positive Economic Consequences Reduced fuel costs: Electric vehacles (EVs) are powered by electricity, which much cheuper than gasoline dissel fuel. This can save drivers of money efficiency: EV's are much more energy dfficient than guvoline or diesel vehicles. This meuns that they use less enengy to travel the same distance, which cun also sive drivers lot of increase energy security. This because they can charged of sourves, including the electric gid, solar punels, and wind turbines. This means that EVs cun help to reduce the impact of power er outages and other diseuptions. Incressed joh creation: The EV industry creating a lot of new jobs. This is because EV's require more energy than gasoline or diesel vehicles, which means that they need to be charged more oflen. This can creute of new johs for people who are responcible for charging EVs.
Positive Fronomic Consequences Redined fuel Flectric vehicles (FVs) elimiintte need for grsoline diesel fuel, which lead to signfficatt
costs. lncreased energy
fuel cost savings for consuerrs, Lower maintenance costs: EVVs roquire less muinrenance than traditional combustion engine vehicles (CFVs) in they have rsquie changes lsereased energy efficioncy: enegy-dtfhiieut -tfcciet CEVs, converting a higher percentage of electricity into motion. Job creation: The mandicttring and installation of EVs and related infrastructure can create new jobs various sectors, Reducad pollution: EVs produce tailpipe emissions, which improve quality and reduce health probleats related air pollution.
What are some potential economic consqueeces,
Incressed energy security EV's help vuriety
EVs
both positive and negative, of a global shift towards eloctric whhicles?
moving pants and Ess much more energy
Consumer Awareness and Demand Drive Innovation and Market Growth for Aliemaiives to Plantic Products Consumer awareness and demand are powerful
drivers of innoatton and murket growth for altematives plastic products, consumers become more informed ahout the em envirunmental impuct
Consumer
and demand powerful drivers of innovation and
How can consumer twware- ness and deaand drive
plastic products, they are incressingly seeking cut sustainable altemutives. This demand for sustinnbbe alternatives is creating a growing market for products made from recycled materials, bioplastics, and other sustainable moterials, Recycled Materials The necylling of plasiic products a key driver innosition and market goowth altematives plastic products. Recycled plastic can be med to ponduce a variety of products, including botries, tnys, and films. Recycled platic is aften more expensive than virgin plastic also envirnmentally friandly.
market grouth altematives ln plastic products. Here's how: Demand Crents Ogoomnity. Conmaner Pressur:: When consmeess express their peef- ereace for alternatives through boycotts, petitions and social modia campaigus it creates pressure manufacturers respond. Market incentives: Growing demund coo-tiiendly alternatives inceativizes minufacturess to invest in research, development, and production of sustainable products.
plastic
innovation and marker growth for alternatives
plactic proditess?
ae Eqonomic Challenges Faved by Tourism Dependent Commanises During Deweturns Tourism- dependent commanities highly vulnersble econcmic fluctations, as their economies are heavily reliant on inurism revenue. During periods of downtum, such as ecoomiic recessonn or global pandemics, these communilies face number of challengo: Redacnd tourism reverue: As tourism--ppedeen
What are the economic challenges that tourism- dependest commmnilies face during periods of downturn, sach as economic recesioos or global pandenics. and what stntegies communities employ to mitigate such ingocts??
AM Economic Challengos Faced by Tourism- Tourism-Depeedeet Communities during Doentume:
and tourism spending declines, communities rely on for emplayment, income, and revenue is significamlly reduced, Incressed unmpployment: As tourism revenue declines, businesses in noprism-dppeneent communitios are forcod lay off employees, leading declines,
Creation: Direct Impact: Tourism is majec source reverue for these communitiss and decline directly impacts koeal businesses, attractions, and government budgens. Job Loss: Tourism eneates significant number of jæs,, and their disippeamnee during downtuons leads unemploy ment, reduced wages, and
Revenue
us Jourism
increased uncugloymest Increased poverty. the poverty rale imurism- depiidnddepeedentt communilies incresses. Incressed crime: As tourism revemne declines, crime rates in tourism--peende communitics incresse. Increased social unrest As toniimm reveaue declines, social unrest in depeadent communities
instubilay.
Table 14: Part of LLM responses where MODEL SWARMS tie with pre-swarm experts per human evaluation.
25

---

MODEL SWARMS
topic
respouse
post response
I longe language model, and I don't have the capseity conduct research projects engage collaboratios research projects and collaborraiion happening in various fields. though. To give you a better idea of what's is happening in your specific depertment, I neal information. Please interested Your univenity inctitutio:: you louking for research up- portinities at specific university or inctittionn Once howe this information. ess provide you with relevant research projecs and ool collabeations happening your depurtment.
Research Projects and Collaborations linsest departmentt name] is vibrant and active research community with ongoing projects in a wide mage of mess. Some of the departmen's key research areas include: [List of key research areas] Here are some of the department's curent research projects: IList
Department Department
provide you with information about
What types of reseurch psojects and collabora-- tinns are currently under- way your department. and how can prospective studeats get involved?
phd application
pmnjecss/ These projects being conducted by faculty members, possdocs, and graduate students, Many of the depurtment's faculty members are also involved in collaborative rescurch projects with other institutoon.
Your department: What depattment you
Ethical Consideration for Public Companies Public companies have greater respunsibility the public than privale companies. They subject more
When a company goes public, takes on new level of scrutiny and responsi- bility. Iere
ethical considerations cumpanies should adhere to: Transpurency und Accummallitt:: Finuncial Reporting: Accimte timely reporting of firancial information enucial. This includes: Fimuncial State- ments: Publishing transpareni and relisble financial statemenis that accurately reflect the compuny' 's financial standing SEC Filings: Filing required SEC forms (e.g. I0-K, 10-Q)
munner. Eihical Convidera- fiees for Public Company Directors Board diversity Public compony board. should reflect the diversity of the compuny's 's wurkfonce and customer base. Independent directon:: Public compuny hoards should have majority in-- dependent directors. Board meetings: Public company boards logelarly and be prepared to act the hest interests of company. Financial disclosure: Public company beards should ensure tuat the company's 's finuncial stotements are socurate and complete. Corporatc social responciiiity: Public company boars should sible manner.
scnutiny expected
What are the ethical con- siderations that companies
finance
should when they go public?
with accumte information Corporste Govemance: Establishing strong govemtace pencticss essure acoountabilitt und project shurcholder interests. This includes: Board of Dinectun:: Having diverse and independent hoard of directors with oppropriste expertise and ex- perienez. Executive Compansation: Setting exccutive compensation packages that are ressonable and linked company performance.
that the company acting sacially respan-
Ferran's brand exclusivity and desirability careflll masaged through combination of factors, including: 1. Limited Production & High Demand: High Demand: Ferrar'ss ss vehicles are highly sought after, with waiting lists often excending desinbbility Limited Production: Ferari carefullyy
How does Perrar mtanage to maintain the esclusiv. and desirubility of its beand, including through lamised prodaction models und special edition??
Ferma''s exclusivity and desirubility are maintained through a combination of faciors, including: 1. Limited-PPooe Production Models: - Ferrat's Limited- production models are designed to be exelisive and desinable. They are year of production, over standard model. limited production models are often designed to be unique and to attract aseetion.
luxury
limited
often sold
its production ben, musurng that demand für exereds supply This exclusivity fürther drives demand and keeps prices high.
This scuncity faels the perception of exclusivit and
Moderaturs on Reddit face delicate bulancing act between fostering open discussion and protecting the community from hurmful content. Here's breakdown
How moderatoes Reddit balance need for open discussion with maintaining community standards and preventing harmfull content?
Baluncing Open Discussion with Comcunity Stundurds Reddit Modern- turs Reddit delicate baluncing ast between fostering open discussion and maintaining a safe and pusitive community. Here are some key approaches they use: I. Defining Community Stnddnnse- - Redii has establishhd guide- lines content and behavice. inchading rules against hale speech, harassmeni, bullying, and misinformation. Modderatoos varions tooks like flags, usmings, and hans.
I. Defining Comuuuiity Standurd:: Reddit's Content Policy: Reddit comprehensivee content policy that oullines acceptable behavior and content. This acts as a buseline for mederation decisions. Community Guidelines: Rodlit's subreddit stracture allows communites establish their own guidelines, often specific to their or topic. These guidelines provide addisiona contest and expectations for content. Community Involvemen:: Modersices actively engige with community to undentand their expectations and concerss. This feedhack is enucial in shoping and refining community standards.
they
complex
social network
these guidelines through
Character backstunes are a key clement of the narrative structure in Genshin Impaci. These narraiives churacter relatiindiis and Lowe Character buckstories are way
Character backstories play a crucial role in eabancing the overall gaming experience player engagement Genshin Impact. These narratives provide context. depth, and emotional weight the characters, making them relatable and memorable. Fanahishhng Character identity and Mocication:
of lore lore lore but also only give players deeper undertanding
role chur- acter hackstorios Gen- shin Impact. Hlw do these nurratives enrich the over- gaming experience player engagement?
the chamacters. to build loce and lone lone. They provdde information about the charactens' lives, their motivations, and their relasionships with other characters. This be use to for players... For example, character who a member of the Knighes of Favonius will have a differeent backstury thom a characier who is member of the Order of Knights the Roudd Table.
Backstories reveal the motivutions, goals, and aspirations of characters. For example, Xiav's backstory reveals his internal conflict between his duty
sandbox games
information
immersive and engaging experience
Harhinger his desiee protect others. They plyyes understard the characters' perspectives, values, and relationships.
instance, Diluc's buckstury reveals his complex relitionship with his father and his motivations for founding the Knights of Favonizs.
Table 15: Part of LLM responses where MODEL SWARMS lose to pre-swarm experts per human evaluation.
26

---

MODEL SWARMS
particle_ 0. o_now, particle 10. now, particle 11 now, particle 12_
0.1
0.46
0.44
02
0.42
0.4
particle, O_now, particle_ now, particle now, particle now,oiaattcle,
now,
0.4
0.46
0.42
Step
Step
particle now, particle now, particle now, particlo
0.87
0.66
0.5
0.65
0.64
0.1
Step
Stop
10
15
10
15
particle. D. now, particle 10. now, particle 11. now, particle 12.
0.7
0.2
particle__ 0_now, particle_10, now, particle_ now, _2_n
0.55
0.02
0.2
0.91
Stop
Stop
60
10
particle. O. now, particle 10. new, particle 11 now, particle 12.
0.16
-3.2
0.12
0.4
6.1
-066
0.68
Step
Step
10
20
10
Figure 15: Search dynamics of per-particle change (left) and global best change (right) of utility function f. MMLU-pro K-Crosswords, Hellaswag, NLGraph, Promoos,iityyroomts,, and AbstainQA performance are illustrated from top to bottom.
27


Reviews:
[{"review_id": "vvL0DTbxrD", "reviewer": "Reviewer_m5j1", "summary": "The authors propose utilizing a population of base language models (with the same architecture but different weight initialization) and then study how to finetune them for downstream utilization. Their method consists of treating each model as a particle in the weight space, assigning random \"exploration\" velocities to each, along with an exploitation velocity corresponding to moving towards the current best particle. Subsequently, a weight update moves each particle along its own trajectory after which this process is repeated. The authors test their method on a variety of downstream utilization paths and find improvements in comparison to several recently published baselines.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their thoughtful comments and feedback.\n\n> How does the computational cost of ModelSwarms compare with baselines?\n\nThe main computational cost comes from model inference and evaluating the LM checkpoints on the utility function, most simply performance on a small validation set. Trivial and static baselines in Table 1 are less expensive since they don’t require this evaluation. Model Swarms is on par with the dynamic baselines such as EvolMerge and Lorahub, evaluating models on the validation set at each iteration.\n\nWe refer the reviewer to lines 980 in the appendix for a more detailed discussion on the computational cost. To recap, Model Swarms has linear complexity to the number of models and to the cost of one model inference, while it takes about 10-15 iterations for each run on average. Empirically, with 5 40GB GPUs you could run a Model Swarms search under an hour, with 2 GPUs you need about 3 hours. We propose further acceleration with dropout-k and dropout-n in Figure 7.", "claims_and_evidence": "The claims made are well supported by the authors - swarm optimization seems to be a general enough paradigm to combine expert LLMs in weight space.", "methods_and_evaluation_criteria": "The authors study their method's performance in single/multitask, reward modeling and human preference settings and demonstrate improvements across the board. They perform a thorough analysis of the dynamics of training and study the trajectories of eventually successful particles.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experiments are clean, and easy to understand with informative baselines selected. Analysis of trajectories and ablations on diversity of agents is informative.", "relation_to_broader_scientific_literature": "The paper is very relevant to the AI community at large. This form of gradient free optimization is useful for post training alignment and I believe that this paper is an impressive demonstration of a well known idea at scale.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "The thoroughness of experiments and general insights from this paper are very useful. I appreciate the number of baselines studied. It was a joy to review this paper.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "How does the computational cost of ModelSwarms compare with baselines?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XMUgJIIg2Y", "reviewer": "Reviewer_LQ3k", "summary": "- This paper introduces Model Swarms, a search algorithm to adapt LLM capabilities\n- The model is based on particle swarm optimization (PSO) \n- Model Swarms (MS) = multiple LLM experts collaboratively search for new \"adapted models\"\n- The purpose of these newly adapted models is to search for capabilities beyond the current model\n- MS is evaluated in four LLM adaptation objectives, in which superior performance is achieved", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their thoughtful comments and feedback.\n\n> Will the code be released publicly upon publication?\n\nYes, along with the best-found model checkpoints for all tasks.", "claims_and_evidence": "- Empirical—MS outperforms baselines in all four LLM adaptation objectives — *this claim is supported by the empirical results and analysis in Sections 4 and 5*\n- Other claims from Section 5 (e.g. \"diversity matters\") are well-supported by the results and discussion", "methods_and_evaluation_criteria": "- The MS method is a natural—though novel and interesting—solution to the problem of producing a new adapted models\n- The baselines chosen are reasonable and seem to cover comparable training schemes in prior works\n- The four adaptation objectives appear to be well-chosen, and offer a robust set of challenges to test the capabilities of MS relative to baselines. The domains themselves also appear to be natural choices.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "From Section 3 and Appendix C, the experimental setup appears to have been carefully designed. All relevant training details (beyond the code itself) are provided; results seem reproducible from details in the paper.", "relation_to_broader_scientific_literature": "- MS is related to other works attempting to compose LLMs to produce more capable models\n- Being an evolutionary approach, it is also related to other evolutionary algorithms applied to the LLM space\n- MS appears to be a novel construction in both of these areas, from the authors' contextualization in the related work section", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths: \n- Paper is well-written, well-structured, and clear\n- The method makes sense\n- The analysis is precise and insightful\n\nWeaknesses: \n- Nothing major", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Will the code be released publicly upon publication?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZaOUGow94T", "reviewer": "Reviewer_Ndt4", "summary": "This paper proposes a Particle Swarm Optimization based Large Language Model collaborative search algorithm, where LLM weights are considered as particles and PSO is applied to search for a best performing LLM on a target task. Experimental results show that the searched LLMs outperform the initial LLMs and other LLM composition methods on diverse tasks.\n\n## update after rebuttal\n\nSince my concerns are addressed, I change the score to 3.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their thoughtful comments and feedback.\n\n> Besides, the proposed method does not provide the alignment method for LLMs with different sizes, which may limit the further application of Model Swarms.\n\n> Usually the initial expert LLMs for different fields have different model architectures and model sizes, since the proposed method could only be applied to a set of LLMs with the same architecture and size, the contribution of the proposed method might be limited.\n\nWe propose token swarms on line 372, an extended version to compose models with different sizes/architectures, by conducting PSO on the token probabilities space.\n\nInstead of defining the search space as model parameters, in token swarms the PSO search determines how token probability distributions from heterogeneous models should be aggregated. We show that this works for a swarm of 4 Gemma and 4 Mistral models in Figure 5, with performance improvement regardless of the architecture.\n\nThe token probability variant takes inspiration from [1] and it provides great context about aggregation in the token probability space, reconciling tokenizer differences, etc.\n\n[1] Liu, Alisa, et al. \"Tuning Language Models by Proxy.\" First Conference on Language Modeling.\n\n> However, this demonstration doesn’t show the efficiency of the searched LLM over state-of-the-art LLMs on these tasks.\n\n> It makes sense that the searched LLM could surpass the initial models, but the absolute performance is not evaluated.\n\n> Besides, the state-of-the-art LLMs for these tested tasks and the GEMMA-7B models fine-tuned on these tested tasks are not included. Since fine-tuning GEMMA-7B for 5*10 epochs (the resource used for training the initial models) on the target tasks may obtain better performance.\n\nThe reviewer asks for two comparisons: Model Swarms against fine-tuning Gemma-7B on the target task, and Model Swarms against “state-of-the-art LLMs”.\n\nThe former is a fair comparison. In line 1065 and Table 9 we compare Model Swarms against direct fine-tuning on the task data of the utility function. We see consistent improvement. The main issue here is generalization: when the available data is small, direct fine-tuning tends to memorize with very large validation-test set gaps while Model Swarms leads to better skill learning and composition. We fine-tuned for 5 epochs in Table 9 and empirically see that fine-tuning for 5*10 epochs exacerbate memorization and harm generalization.\n\nWe are unsure if the latter is a fair ask. We don’t think taking 7B models and applying adaptation would outperform “state-of-the-art” models that are much larger and much more extensively trained. We don’t see the goal of this work as establishing a new state-of-the-art on the evaluated tasks, but rather demonstrating the adaptation and composition of diverse LMs could be achieved through collaborative search and swarm intelligence. Thus we believe our selection of 12 model composition baselines is adequate for this purpose.\n\n> However, the initial models are fine-tuned on data in Tulu-v2 which are significantly different from the tested tasks, which may lead to biased evaluation.\n\nThis difference between SFT data and evaluation tasks is actually a good thing, meaning that the initial swarm was not overfitted for the evaluation tasks and there is genuine adaptation/generalization to these tasks on-the-fly in Model Swarms searches.\n\n> There are various advanced and adaptive PSO variants in EC community, I wonder why authors use a (relatively) simple PSO method, using advanced PSO might obtain better performance?\n\nWe agree that there is broad literature on evolutionary algorithms (EAs) about PSO and beyond, many of them could be adapted for today’s LLM research. Model Swarms take one of the first stabs at this interdisciplinary research direction so we decided to keep the EA part simple and straightforward, hoping to get the ball rolling on future evolutionary designs of LLMs.", "claims_and_evidence": "The authors claim that the PSO based LLM composition method could achieve superior performance within 200 examples. Experimental results demonstrate that the searched LLMs outperform the initial LLMs and other LLM composition methods. However, this demonstration doesn’t show the efficiency of the searched LLM over state-of-the-art LLMs on these tasks.", "methods_and_evaluation_criteria": "The overall method is to utilize PSO to collaboratively optimize a set of initial LLMs and search for a best LLM on the target task. It makes sense that the searched LLM could surpass the initial models, but the absolute performance is not evaluated. Besides, the proposed method does not provide the alignment method for LLMs with different sizes, which may limit the further application of Model Swarms.", "theoretical_claims": "There is no theoretical claims in this paper.", "experimental_designs_or_analyses": "In the experiment, authors compare Model Swarms with initial models and other compisition methods. However, the initial models are fine-tuned on data in Tulu-v2 which are significantly different from the tested tasks, which may lead to biased evaluation. Besides, the state-of-the-art LLMs for these tested tasks and the GEMMA-7B models fine-tuned on these tested tasks are not included. Since fine-tuning GEMMA-7B for 5*10 epochs (the resource used for training the initial models) on the target tasks may obtain better performance.", "relation_to_broader_scientific_literature": "1) Large Language Model: The method is proposed for LLM adaptation.\n\n2) Swarm Intelligence: PSO is employed in the proposed method for searching LLMs from a set of initial models.\n\n3) Model Composition: The proposed method adapt LLMs to new tasks by composing existing LLM models.", "essential_references_not_discussed": "The discussion on related methods is sound, I didn’t see any essential references not discussed.", "other_strengths_and_weaknesses": "Weakness: Usually the initial expert LLMs for different fields have different model architectures and model sizes, since the proposed method could only be applied to a set of LLMs with the same architecture and size, the contribution of the proposed method might be limited.", "other_comments_or_suggestions": "None.", "questions_for_authors": "There are various advanced and adaptive PSO variants in EC community, I wonder why authors use a (relatively) simple PSO method, using advanced PSO might obtain better performance?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BGZzwa6kA1", "reviewer": "Reviewer_WsmQ", "summary": "This paper introduces MODEL SWARMS, a collaborative search algorithm for adapting large language models (LLMs) through principles of swarm intelligence, leveraging collective behaviors to guide individual systems. Inspired by Particle Swarm Optimization (PSO), MODEL SWARMS optimizes collaboration among diverse LLM experts by navigating their weight space towards maximizing a defined utility function. Extensive empirical evaluations demonstrate that MODEL SWARMS effectively adapts LLM expert ensembles across single-task scenarios, multi-task domains, reward modeling, and varied human preferences, consistently outperforming 12 baseline model composition methods.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their thoughtful comments and feedback.\n\n> It would be beneficial to include a convergence analysis of the proposed update mechanism to better understand the theoretical properties and stability of the optimization process.\n\nWe investigate convergence and stability empirically in several studies in the appendix.\n\nIn line 872 and Figure 6, we visualize the search trajectory of models. Starting as diverse model checkpoints, they gradually converge to an area in the weight space that optimizes utility function f.\n\nIn line 860 on the right, we identify the three randomness factors in Model Swarms: while the algorithm is not 100% stable and deterministic, these stochastic factors actually help exploration and performance in Table 5.\n\nIn line 1028 on the right and Figure 15, we demonstrate the convergence of model performance across tasks and optimization iterations. The stability of the search is task-dependent, with some per-model performance plots smooth and some turbulent. Nevertheless, the global best performance steadily increases across tasks.\n\nThere are theoretical analyses of PSO convergence in evolutionary algorithm research such as [1], while the assumptions made in the classic optimization problems are not applicable for the LLM setting. For example, the initial particles in classic PSO are often initialized randomly or in a grid, while the initial LMs in the swarm cannot be random/arbitrary (taking a random point in the 7-billion-dimensional space would almost certainly be a failed language model). Instead they are seeded with curated model checkpoints with different training and fine-tuning data mixtures. Given these differences, we decided to take an empirical route to analyze convergence and stability properties.\n\n[1] Van den Bergh, Frans, and Andries P. Engelbrecht. \"A study of particle swarm optimization particle trajectories.\" Information sciences 176.8 (2006): 937-971.\n\n> Could the authors clarify the computational cost associated with training the proposed method and provide a detailed comparison of training efficiency relative to the baseline approaches?\n\nThe main computational cost comes from model inference and evaluating the LM checkpoints on the utility function, most simply performance on a small validation set. Trivial and static baselines in Table 1 are less expensive since they don’t require this evaluation. Model Swarms is on par with the dynamic baselines such as EvolMerge and Lorahub, evaluating models on the validation set at each iteration.\n\nWe refer the reviewer to lines 980 in the appendix for a more detailed discussion on the computational cost. To recap, Model Swarms has linear complexity to the number of models and to the cost of one model inference, while it takes about 10-15 iterations for each run on average. Empirically, with 5 40GB GPUs you could run a Model Swarms search under an hour, with 2 GPUs you need about 3 hours. We propose further acceleration with dropout-k and dropout-n in Figure 7.", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The methods and evaluation criteria make sense for the problem.", "theoretical_claims": "Not applicable.", "experimental_designs_or_analyses": "The experimental designs and analysis are detailed and promising.", "relation_to_broader_scientific_literature": "--", "essential_references_not_discussed": "--", "other_strengths_and_weaknesses": "Strengths:\n\n1.\tThe proposed method is innovative and conceptually intriguing, and the manuscript is clearly structured, well-written, and accessible to readers.\n\n2.\tThe experimental evaluation is comprehensive, covering a wide range of settings and tasks, and the empirical results convincingly demonstrate the effectiveness and promise of the approach.", "other_comments_or_suggestions": "1.\tIt would be beneficial to include a convergence analysis of the proposed update mechanism to better understand the theoretical properties and stability of the optimization process.\n\n2.\tCould the authors clarify the computational cost associated with training the proposed method and provide a detailed comparison of training efficiency relative to the baseline approaches?", "questions_for_authors": "--", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: While some questions remain around theoretical analysis and comparisons to state-of-the-art models, the authors have made a convincing case that their primary goal was to demonstrate the potential for adaptation and composition through this novel approach.
Decision: Accept (poster)