Agent Skill Acquisition for Large Language Models via CycleQD
Authors: So Kuroki
Taishi Nakamura
Takuya Akiba
Yujin Tang
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Models
Skill Acquisition
Quality Diversity
Primary area: foundation or frontier models, including LLMs

Abstract: Training large language models to acquire specific skills remains a challenging endeavor. Conventional training approaches often struggle with data distribution imbalances and inadequacies in objective functions that do not align well with task-specific performance. To address these challenges, we introduce CycleQD, a novel approach that leverages the Quality Diversity framework through a cyclic adaptation of the algorithm, along with a model merging based crossover and an SVD-based mutation. In CycleQD, each task’s performance metric is alternated as the quality measure while the others serve as the behavioral characteristics. This cyclic focus on individual tasks allows for concentrated effort on one task at a time, eliminating the need for data ratio tuning and simplifying the design of the objective function. Empirical results from AgentBench indicate that applying CycleQD to LLAMA3-8B-INSTRUCT based models not only enables them to surpass traditional fine-tuning methods in coding, operating systems, and database tasks, but also achieves performance on par with GPT-3.5-TURBO, which potentially contains much more parameters, across these domains. Crucially, this enhanced performance is achieved while retaining robust language capabilities, as evidenced by its performance on widely adopted language benchmark tasks. We highlight the key design choices in CycleQD, detailing how these contribute to its effectiveness. Furthermore, our method is general and can be applied to image segmentation models, highlighting its applicability across different domains.

Published as a conference paper at ICLR 2025
AGENT SKILL ACQUISITION FOR LARGE LANGUAGE MODELS VIA CYCLEQD
So Kuroki, Taishi Nakamura, Takuya Akiba, Yujin Tang
Sakana AI, Japan
(sokuroki, taishi, takiba, yujintang] Baaaittnnnnnsaaaaaaa.aaa esakana.
ABSTRACT
Training large language models to acquire specific skills remains a challenging endeavor. Conventional training approaches often struggle with data distribution imbalances and inadequacies in objective functions that do not align well with task-specific performance.
address these challenges, we introduce CycleQD, novel approach that leverages the Quality Diversity framework through cyclic adaptation of the algorithm, along with model merging based crossover and an SVD-based mutation. In CycleQD, each task's performance metric is alternated as the quality measure while the others serve as the behavioral characteristics. This cyclic focus individual tasks allows for concentrated effort at a time, eliminating the need for data ratio tuning and simplifying the design of the objective function. Empirical results from AgentBench indicate that applying CycleQD to LLAMA3-88 MAAMA-8----. S--INSTRUCT based models not only enables them to sur- pass traditional fine-tuning methods in coding, operating systems, and database tasks, but also achieves performance on par with SPPT-355TTRRD which poten- tially contains much more parameters, across these domains. Crucially, this en- hanced performance is achieved while retaining robust language capabilities, as evidenced by its performance on widely adopted language benchmark tasks. We highlight the key design choices in CycleQD, detailing how these contribute to its effectiveness. Furthermore, our method general and can be applied to image segmentation models, highlighting its applicability across different domains
task
INTRODUCTION
Parf on Task
Archives are Initially popultted with expert
Oyolically swap Quality (C) and Behastor Chancteritiics (BCa)
BC Perf lack
Large Language Models (LLMs) have es- tablished themselves as powerful tools the machine learning and artificial intelli- gence landscape, initially gaining promi- nence through their effectiveness in con- versational tasks (Achiam et al., 2023). As the demand grows for LLMSs to per- form a broader range of cognitive tasks, the ability to integrate linguistic under- standing with actionable outputs becomes essential. This integration facilitates the creation of intelligent LLM- -bsedddd agents, making continual agentic fine-tuning a critical development in the field (Gur et al.
Gensration
Genenttion Ke
O: Porf Task
C: Perf on Task Y
agents
BC: Perf on Tack
on
RC:
Figure 1: Method Overview. CycleQD uses QD in cyclic manner to merge LLMs. Archives are initial- ized with expert LLMs, each of which has been fine- tuned to excel in specific task. Model merging is con- ducted using QD, treating task performance as quality (Q) and behavior characteristies (BC), which are cycli- cally swapped in each generation.
2023 Liu et al. 2023b Xi al.) 2023 Wang et al. 2024b). However, train- ing LLMs to acquire various agent skills presents two major challenges that compli- cate their development.
2024
https://githbb: /atom//aaaaaa///// /Cyceee

---

Published as a conference paper at ICLR 2025
A significant difficulty arises from the need to balance data ratios during training, ensure that the model learns effectively from datasets representing different skills without favoring any particular one and without neglecting others. For example, researchers have identified the significant influence of code training to a model's inferential generation and thinking capabilities (Liu et al.] 2023b). Comparing CODELLAMA and L.A.M-22, the former shows significant edge in tasks that follow a relatively static procedure after being tuned on code data, yet at the same time reveals decreased performance in tasks that require general thinking abilities.
In addition, conventional objective functions, such as next token prediction, often fail to capture the nuances required for performance across diverse tasks, typically leading to sub-optimal training outcomes. This function, although effective for general language understanding, does not foster the development of specific agent skills. An alternative, such as reinforcement learning (RL), has been shown to align LLMs with user intents more effectively: Ouyang et al. (2022) demonstrated aligning L.Ms with user intents by RI. from human feedback, where reward model is trained from a dataset of human- ranked model outputs. This method has later on been adopted by many others and become an industrial standard. Fortunately, unlike general language tasks, the "rewards" are clearly defined in most agent tasks. On the other hand, managing the balance of rewards in learning tasks and the associated costs of fine-tuning pose additional problems.
In response to these challenges, propose CycleQD, novel framework that leverages the Quality Diversity (QD) method (Pugh et al. 2016), specifically through a cyclic adaptation of the MAP- Elites algorithm (Mouret & Clune 2015). CycleQD features a dynamic cycle where each agentic skill's performance metric (e.g., pass@1 @ for coding tasks, success rate for Operation System (OS) and Database (DB) tasks) is optimized in isolation, with other skills represented as behavioral char- acteristics (BCs). Unlike conventional crossover operations that randomly swap the parameters between two genes. CycleQD employs model merging based crossover operation (Akiba et al., 2024), which is essential for transferring skills from specialized experts to a new, cohesive model. Furthermore, we also introduce singular value decomposition (SVD)-based mutation method to extrapolate model capabilities while preventing overfitting. Unlike gradient- based fine-tuning, Cy- cleQD reduces the need for intricate design decisions and hyperparameter tuning, while inherently capturing diverse skills and behaviors. See Figure D for an overview of CycleQD.
We adopt design choices specifically targeting the two major challenges in continual agentic fine- tuning. Our proposed method not only simplifies the complex data ratio management by focusing on one task at a time but also addresses the inadequacy of traditional loss functions by directly op- timizing task-specific performance metrics. Moreover, the QD framework proves ideal for model merging, particularly in L.MS where the computational pipelines are long. By allowing locally sub-optimal solutions persist, such temporarily under-performing layer, QD saves these con- figuration in its archive and provides them a chance to improve in the future.
The empirical results from our experiments illustrate that CycleQD significantly enhances the ability based LLM agent to develop and refine multiple computer science
of a CEAMAA-SB-INNTRUUE skills. Notably, the averaged performance of the final 8-billion parameter open-weight model is on par with that of GPT-3.5-TURBE on coding, OS and DB tasks, which potentially contains much more parameters. This superior performance is achieved without the typical degradation associated with shifting focus among tasks or compromising general language capabilities.
Based on these results, it is important to note that while post-training of LLMs has been dominated by gradient-based optimizations, our method demonstrates that incorporating evolutionary algo- rithms can serve as a compelling modification to the conventional fine-tuning pipeline, effectively enhancing the training of LLM agents. Our technical contributions are summarized as the following:
We introduce CycleQD, a novel approach for merging LLM agents, each specializing in a different task, to create composite LLM that outperforms baseline methods on computer science tasks from AgentBench (Liu et al. (023b) and popular coding benchmarks while m maintaining language capabilities.
We provide ablation studies and detailed explanations of the key design choices in Cy- cleQD, illustrating their pivotal roles in enhancing the method's effectiveness.
We extend the application of CycleQD to the Segment Anything Model (SAM) (Kirillov et al. 2023), demonstrating our method's broad applicability across domains.
2

---

Published as a conference paper at ICLR 2025
2 PRELIMINARIES
Evolutionary Strategies (ES) are optimization al- gorithms inspired by natural selection. They im- prove a population of candidate solutions through mutation, crossover (i.c., recombination), and selec- tion, guided by fitness function. By maintaining diversity, ES avoids premature convergence and is well-suited for QD optimization, which aims to gen- erate diverse high-performing solutions rather than single global optimum.
Sample
Quality
Croesover
Mutate
Evaluste
Child placerennt
BC:
MAP-Elites is key algorithm within the QD paradigm, designed to optimize for both perfor- mance and diversity by maintaining an archive of clite solutions across discretized space of behav- ioral characteristics (BCs). Each cell in this archive corresponds to a unique combination of BCs, storing the best solution found for that cell. Figure iillustrates an optimization step: two parents are sampled from the archive, and child solution is generated through crossover and mutation. The child's fitness and BCs are then evaluated to determine its placement in the archive, replacing the current entry if it performs better.
Figure 2: MAP-Elites flow.
METHODS
To avoid the laborious tuning of the data mixing ratio and the objective function, we introduce CycleQD. built on MAP-Elites, a recognized implementation of QD, but distinguishes itself from traditional MAP-Elites systems in three key ways (see Algorithm for the pseudo-code):
Alternating Quality and BCs: Pugh et al. (0166) emphasizes the importance of well-aligned BCs with the quality (i.e., task metric that needs to be optimized). Traditional systems typically set the design at the onset and do not alter them throughout the QD process. In contrast, CycleQD uses task metrics both quality and BCs, ensuring alignment and dynamically alternating them during the QD process (lines 6-7 Algorithm D. This step can be viewed as the archive is rotated by 90 degrees after each generation before proceeding to the optimization, as illustrated in Figure [
Model Merging as Crossover: Unlike existing systems that optimize model parameters directly, CycleQD leverages model merging algorithm
crossover operation, replacing the heuristic, hand-desie hand-designed rules often seen in practice (line 12 in Algorithm D Le., in Figure 2 our crossover operation merges the two parent models. In CycleQD, we fine-tune expert agents, each of which only needs to specialize in one task, and use them as the seed models to initialize the model archives (lines 2-4 in Algorithm D
SVD-based Mutation: Traditional mutation operations in genetic algorithms like MAP-Elites typi- cally introduce random perturbations from a pre-defined distribution to explore new regions (i.e., the mutation operation in Figure 2). In contrast, CycleQD utilizes perturbations aligned with the prin- cipal components of the models" parameters matrices, facilitating exploration outside the convex regions formed by the parent models while avoiding overfitting (line 13 in Algorithm
CycleQD repeats these three steps in each generation, and the archives, originally only occupied by the expert models, accommodate more and more capable and diverse models, is illustrated by Figure 6 in the Appendix.
3.1 ALTERNATING QUALITY AND BEHAVIOR CHARACTERISTICS
CycleQD manages K archives, each dedicated tracking the LLMs that specialize in one of the K agent skills. Each archive evaluates LLM performance using K tasks, which serve dual roles as both the quality and BCs. An archive is structured where dy is the number of bins defined by the k-th BC, and the i-th BC is excluded from this particular archive as it is treated as the quality metric. In generation t, the i-th archive is selected for QD computation where
lattice, the size of which is defined by II de.
mod K (see Figure 1 for an illustration with K = 3). This process
3

---

Published as a conference paper at ICLR 2025
Algorithm CycleQD
function TRAIN(A, EXPERTS)
A are the archives, EXPERTS are the expert LLM agents
2: for i < 1 to K do
b K is the number of tasks > Place experts properly i-th archive
3:
A[1] f- UPDATEARCHIVE(A1]] EXPERTS)
4: end for
5: for
1--11 N do 1 --- 1 mod K 7: A @DSTEP((A.)
N is the total number generations
6:
Cyclically swaps the task (archive) for the next QD step > Conduct one step the QD algorithm
8: end for
9: end function
10: function QDSTEP(AL R)
Ag is the archive corresponding to the k-th task Pi and p2 are the parents, SAMPLE() is detailed in Sec 3.1
P1.P2 < (A]:))
12:
C+ CROSSOVER(PI
D c is the child, CROSSOVER is detailed in Sec 3.2
C MUTATE(C)
> MUTATE() detailed in Sec 3.3
14:
for i < 1 to K do
A[i] +- UnDATEARCHIVE444..00
Place e properly i-th archive
16: end for end function
utilizes all available data from the tasks to evaluate and update the archives without the need for adjusting data ratios objective functions.
CycleQD shares knowledge across K archives for efficient optimization. Specifically, in generation a new model is created from the i-th archive by sampling, crossover, and mutation. This model is used to update not only the i-th archive but all K archives (lines 14-15 in Algorithm [ This allows for more effective utilization of the new model and helps facilitate optimization across K tasks.
We employ task performance metrics as our BCs, imbuing them with concrete performance implica- tions. This allows for the design of Elite sampling algorithm aimed expand the Pareto frontier. Our Elite sampling algorithm is inspired by Wang et al. (2023) which expand the Pareto frontier not only in quality but also within BCs. However, there are two major differences in CycleQD's setup: (1) Both Q and BCs are task metrics and are treated equally in a cyclic process, (2) We aggregate Q and BC values into a single metric to drive the frontier towards higher performance. Specifically, this frontier is defined by the models achieving high performance across these BCs. Concretely, when the i-th archive active, a model j inside is sampled with probability P3 ΣN N is the number of models in this archive and 75 is calculated follows:
Ta where
La min(f1sN,s) 3 II (Olow max(f1.N.) - min(f1:N.;)
(Ohigh - (loo))
Here, fj+ indicates the j-th model's performance on the i-th task. (alow, ahigh) are hyper- parameters for the purpose of normalization. Elite sampling strategically favors models that excel across the quality and various BCs, enhancing the efficiency and evolutionary potential of CycleQD.
3.2 MODEL MERGING BASED CROSSOVER
Training an LLM to specialize in a single task does not suffer from problems such as the data ratio tuning, the complex combinations of objective functions, and etc. It is therefore straightforward to initially train set of single-task experts using conventional fine-tuning methods, and then use model merging algorithms to develop more capable agents. In CycleQD, our model merging crossover operator employs the parameter space merging scheme described in Akiba et al. (2024), which creates new model by merging task vectors at the model level (Ihharco al. 2022). Specifically, for a pre-trained base LLM with parameters Opese E dgd and its fine-tuned LLM with parameters 0 E Rdd define a task vector by combining these task vectors: Ochild = Obase + (wn/(ws + (22) Tp. + (w2/(es + w2)) (e2))Tp) Tpp where Tp. and Tyz are the parents' task vectors. Here, W1 and W2 are i..d samples from N( o?). and (j,, .7) are predetermined hyper-parameters that remain fixed during the experiments. These w's do
0 0-Obase- The crossover operator then generates a model's parameters
4

---

Published as a conference paper at ICLR 2025
The parametar mabtx In
Fankk11 matriss m eyane!
Vitand
X
+
C2
+
Figure 3: SVD-based mutations. Left: task vector T contains multiple parameter matrices 77 E.g., the query projection matrix from the attention block in layer I, the key projection matrix from the attention block in layer 2, etc. Those that have a rank r > can be decomposed using SVD into components. Right: SVD-based mutation samples vector E R' and scales each component by wy, essentialll adding perturbations that are aligned with the "directions" of the components.
not need to positive numbers. By allowing negative component weights, the merged model has more freedom in optimizing its task vectors, the process of which is automatically carried out via cvolutionary search. We normalize (Tp. Tp2)'s )'s mixing coefficients to ensure the merged weights do not become outliers and cause problems for the downstream layers.
Additionally, there is potential to enhance the model merging process by training a neural network policy (m, o) + Tal {be1.- ..., bcg lp:: {be1, ..... bcx hpa) that learns the optimal distribution param- eters conditioned on the BC grid IDs (bci,..., bcx) } of the parents (k 1 in total for each parent because one serves as the quality). However, while this approach could improve the efficiency of model merging, also introduces greater initial learning burden in CycleQD.
3.3 SVD-BASED MUTATION
The formulation of our model merging based crossover bears one obvious limitation: The construc- tion of Oehia is linear combination of (Tpy Tp2): which are themselves linear combination of their parents. The entire process then reduces finding the optimal linear combination of the expert mod- els' parameters. Due this, Ochild will likely be trapped in the "convex region" in the performance space formed by the expert models, yet extrapolation that leads improved performance is desired. We introduce a mutation function dehild = ((Caaaa) after the crossover to get rid of this limitation.
In conventional methods, the mutation function h(@ama)) is often operator that samples pertur- bations from a pre-defined distribution (c.g., Gaussian with pre-determined mean and covariance matrix) and add them the gene being mutated. We find this setting introduce excess freedom and lead to overfitting in the final model. Instead, we propose to sample perturbations along the "directions" of the components from the model's parameter matrices. This mutation operation is mathematically defined as: h(Oemia) = Ohaso + concat (Uo)) M, E Rexr and Vi E Rnxr are the SVD components of U,E,V," the 1-th parameter matrix in the task vector Fehild fehild base (i.e., 77 reshaped subarray of Tehil). The perturbation vector E R" is sampled from a uniform distribution of boundaries [0, Wmna),, where Wmnxx E R+ hyper-parameter. Our SVD-based mutation becomes a pass-through operation for parameter matrices whose rank is one (e.g., those belonging to
where U/ E IR
layers).
This approach provides two key advantages. First, SVD focuses mutations on the fundamental building blocks of the task vector, which represent the essential components of agent skills. By con- straining perturbations to these meaningful directions, we avoid the excessive exploration inherent in random perturbations, reducing the risk of overfitting. Second, the ability to manipulate individual components (i.c., w) offers finer control over fchilds enabling targeted modifications at a granular level. Figure 3 iillustrates this process.
3.4 MODEL AGGREGATION
CycleQD maintains K archives during the optimization process, LLMs in each of these archives are trained to acquire one of the K agent skills. In the end, CycleQD returns hundreds or even thousands of LLM agents with various specialties and characteristics. These LLM-based agents versatile and can facilitate multi-agent research in a diverse set of directions (we discuss this topic in Section 6))
On the other hand, our goal in this work is to create a single model that achieves high performance across multiple tasks, and model aggregation method is therefore necessary. Our model aggrega-
5

---

Published as a conference paper at ICLR 2025
tion algorithm is straightforward, and can be summarized as: Page base kk=1 BK where Th is the task vector from the clite model in the k-th archive, with ties resolved by selecting the most recent model. Sg exp(fk)/ Ei=1 exp(fi) (i.e., softmax function) is the mixing coefficient calcu- lated from the elite model's task performance f& in the k-th archive. In this paper, all experimental results from CycleQD are produced with this aggregated model Aags.
4
EXPERIMENTS
4.1
EVALUATION ON COMPUTER SCIENCE TASKS
4.1.1 TASK DESCRIPTION
In this experiment, our goal is to train LLMSs to master three agent skills in computer science: coding, Operation System (OS) manipulation and Database (DB) query generation. We adopt the MBPP+ (Mostly Basic Python Programming) dataset from EvalPlus (Liu al. 2023a) for coding skill de- velopment, and utilize the OS and DB datasets from AgentBench (Liu et al. 2024) for training. For the coding task, optimize and report the pass @ 1 metric, whereas in OS and DB tasks success rate. Please refer to Appendix A.1.1 for extra and detailed task related setups.
the
4.1.2 CYCLEQD SETUPS
We employ LLAMA-3- 1A-MMA---BBIIT I--SRUUTCTT MODEL (Dubey et al. 2024 as our base model, and use supervised fine-tuning to create LLM experts in coding, OS and DB. For the OS and DB experts, use the OS and DB training datasets from Agent- Agent-FLAN Chen et al. 22224. The coding expert is fine-tuned on a combination of the MAGICODER CEAECCDDR-EVOLL---
Experts:
TOK and MAGICODER OSS-
INSTRUCT-75K 5 datasets (Wei al. 2024). See detailed configuration in Appendix A.11.2
Datasets:
We use the MBPP+ dataset, as well as the test and development datasets from OS and DB. To ensure that the experts have the same task metries across tasks, each dataset is split evenly into training and test splits. For OS, problems that could not be solved by either the expert models and GPT models are excluded beforehand reduce computation cost.
We use the performances of the three experts to determine the lower and upper bounds of the BC dimension. Specifically, the lower bound is at 85% of the performance achieved by the least proficient expert, while the upper bound is set at 1 115% of the performance achieved by the most proficient expert. All BCs are then evenly divided into 15 bins between the lower and upper bounds. We limit the number of models in each bin to one, and run CycleQD for 1200 generations. Since the quality and BCs are alternated in each generation, this equivalent to optimizing for the three skills for 400 generations each. See more detail in Appendix A.1.3
4.1.3 BASELINES
Our baselines can be categorized into fine-tuning based and merging based models.
Fine-tuning Based Models: These are models 3-6 in Table In addition to the expert LLMs men- tioned earlier, we also combine all the data from the three tasks (including the extra MAGICODER- EVOL-INSTRUCTT- 1 10K and MAGICODER-ON OSS-INSSRUSS the LLAMA3-8E 8--INITRUCT base model with supervised fine-tuning to develop an extra baseline (model 6 in Table D). In this baseline, we don't 't tune the data ratio and use cross-entropy (i.e., next token prediction) as our objective function.
datasets) and continue training
These are models 7-10 in Table [] We first introduce a naive merging method where the merged model produced by taking an average of the three experts' task vectors (model 7). Similar to this baseline, we introduce three additional learning based model merging methods where, instead of taking an average, the coefficients of a linear combination of these task vectors is learned through gradient descent policy gradients (model 8), CMA-ES, an evolutionary algorithm on the raw rewards (model 9), and NSGA-II (Deb et al. 2002), another evolutionary algorithm that optimizes multiple objectives simultaneously (model 10).
Merging Based Models:

---

Published as a conference paper at ICLR 2025
045
0:00
475
030
229
930
035
008
BC DB
9C: MBPP
BC: MBPP
Figure 4: CycleQD Archives from Computer Science Tasks. In each archive, the two axes show the BCs, and the color intensities indicate the LLM agent's 'ss quality in that grid. These archives obtained after 1200 generations of CycleQD. The red bounding boxes indicate the grids where expert models were present. See Appendix A.1.4 1.4 for archive development across generations.
4.1.4 RESULTS
Overall: We summarize results from this experiment in Table [ In addition the baselines, we also include the base model and the GPT models for references (highlighted with a lightgray background). The first thing notice is that each expert model (models 3-5), fine-tuned specifically base model (model 2), lay-
for its assigned task, performs better than the LLAMA3-88 BAA-A-3--IIINSTU ing the stepping stones for CycleQD and other model merging based methods. After evolutionary computing, our method, averaged across the three tasks, outperforms all baselines, and is approach- ing the performance of UPT-3.5-TURBO Specifically, CycleQD achieves the highest scores on the coding and the OS tasks among the baselines, and has only mild performance drop on the DB tasks compared with the expert. Figure gives the archives in CycleQD at the end of the experiment. is easy to see that CycleQD has managed to "illuminate" the three archives.
Comparison with Fine-tune Based Methods: A notable result from the table that, the model fine-tuned on all the datasets (model 6) is only partially and marginally better than the three experts, despite being trained on a larger dataset. On the other hand, the expert models can be easily trained because they do not require the data ratio object funetion adjustment. The drawback of these experts is that they specialize in only one task, prohibiting their usage in wider scenarios. This result proves the difficulty with conventional methods, underscoring the importance and technical contributions of CycleQD.
Comparison with Merging Based Methods: CycleQD also outperforms conventional model merg- ing based methods (models 7-10), and we conjecture that this is due to three reasons. First, these methods lack a mutation operation, which believe is crucial for enabling the merged model
Table Evaluation on computer science tasks. We report pass I for MBPP and success rates for the DB and OS tasks. The base model and the GPT models (in gray) are included for completeness. The MBPP results for the GPT models are extracted from Dubey et al. (2024).
#
Methods
MBPP
DB
OS
Avg
GPT-4
83.6
36.5
63.7
61.3
SPT-3.5-TURBO
82.0
41.6
38.5
53.7
Llama3-8B-Instru (base model)
67.3
5.3
25.2
32.6
Fine-tuning Based Methods
Finc-tuning (Coding expert)
70.4
21.2
20.7
37.4
Fine-tuning (DB expert)
65.8
42.4
28.5
45.6
Fine-tuning (OS expert)
66.3
0.0
30.4
32.2
6
Fine-tuning (All)
67.3
37.1
36.7
47.0
Merging Based Methods
7
Merging (w/o learning)
72.9
24.7
42.6
46.7
Merging (learning w/ GD)
69.3
41.2
29.6
46.7
Merging (learning w/ CMA-ES)
69.3
41.2
30.2
46.9
10
Merging (learning w/ NSGA-II)
75.9
42.4
36.4
51.6
CycleQD (Ours)
76.4
38.2
42.6
52.4

---

Published as a conference paper at ICLR 2025
Table 2:
Ablation studies. We add and highlight a different treatment from the previous trials.
#
Trials
MBPP
DB
OS
Avg
QD + No mutation + Random sampling
70.4
28.8
43.7
47.6
CycleQD +
No mutation Random sampling
72.9
33.5
41.9
49.4
2 CycleQD+ Gaussian mutation Random sampling
73.4
30.0
42.2
48.5
3 CycleQD + SVD mutation Random sampling
75.9
38.2
41.1
51.7
4 CycleQD SVD mutation +
Elite sampling
76.4
38.2
42.6
52,4
cape the "convex region" created by the seed models. For example, model 9 scores lower all three tasks compared to the experts. In contrast, CycleQD significantly surpasses expert performance in coding and OS tasks. Second, CycleQD's 's cyclic optimization mechanism incorporates elements re- sembling non-dominant sorting in NSGA-II (model 10). However, unlike the latter, which handles all tasks simultaneously and relies on crowding distance for diversity, CycleQD alternates between tasks and naturally encourages diversity via the QD mechanism. This allows CycleQD to be more effective. Lastly, we suspect that the absence of a mechanism to rehabilitate temporarily underper- forming models contributes to this discrepancy. Given the lengthy computation pipelines in LLMs, which include numerous layers, even a single malfunctioning layer can degrade overall performance. Unlike QD, the baselines do not have way to recover these layers, potentially promising models are prematurely discarded.
4.1.5 ABLATION STUDIES
To get insights into the design choices in CycleQD and show how they contribute its effectiveness, we conduct series of ablation studies and summarize the results in Table 2
CycleQD vs QD: We compare conventional QD with CycleQD in trials and 1. Specifically, we run QD for three runs. Within each run, one of the task's metric is treated the quality while the others are regarded as the BCs. We control each trial to have an equal computing budget as for CycleQD, and all other aggregation method to merge the best models from these three runs (see Section 3.4.. The better performance from CycleQD suggests the importance of alternating the quality and BCs.
are identical to CycleQD. We use the same model
Mutation Methods: We focus on the impact of different mutation operations with trials 1-3. Al- though mentioned earlier that mutation helps the merged model extrapolate in the performance space, naively designed mutations give excess freedom and can lead to overfitting. This is what hap- pens trial 2, the performance of which is even worse than trial that does not have any mutation operations. On the other hand, our SVD-based mutation (trial 3) successfully avoids the problem and delivers much better performance.
Finally, we demonstrate the importance of Elite sampling in CycleQD in trial 4, which has the identical settings as model 11 in Table I and gives the best score in the ablation studies. Elite sampling is effective because it prioritizes merging high-performing models, the result of which helps expand the Pareto frontier each archive more effectively.
Sampling Methods:
In summary, our ablation studies validate the importance of our design choices and shows that improvements aren't merely additive poor design choices can harm performance while our well- designed components work synergistically (e.g.. comparing rows and 2 in Table(2). The cumulative improvement from baseline QD to our final CycleQD is substantial at 4.8 percentage points (from 47.6% to 52.4%). This is a significant improvement in the context of LLMs performing complex agent tasks, particularly considering our model approaches GPT-3.5-TURBO'S performance despite having significantly fewer parameters.
4.1.6 SKILL GENERALIZATION AND LANGUAGE CAPABILITIES
In addition to agent skills, we evaluate model on out-of- coding tasks and language understanding across six categories, please see Appendix A.1.6 for detailed task descriptions. Ta- shows the results across these tasks, where normalize the scores against the LLAMA3-8B- INSTRUCT base model. On average, CycleQD outperforms the three tine-tuned experts, providing better overall generalization across tasks. Our method generalizes well on the coding tasks while
8

---

Published as a conference paper at ICLR 2025
Table Generalization performance. The scores are normalized against the base model. CycleQD generalizes well to other coding tasks unseen in the training while retaining its language capabilities.
Coding Tasks
Language Tasks
Model
Avg
HUMANEVALA
BigCodeBench
Reasoning
GSM8K
RC
CommonSense
MBPP expert
1.18
0.97
0.57
0.82
0,94
1.03
0.92
DB expert
0.80
0.84
0.84
0,87
0,98
0.98
0.89
OS expert 0,94
0.90
0.98
0.93
0,99
0.99
0,95
CycleQD 1.10
1.03
0.95
0.88
0.98
1.02
0.99
Table 4: Merging image segmentation Models. The scores are normalized against the performance of the corresponding expert models. Model similarity shows how close the pair of expert models are (the larger the similarity values, the closer the models are).
Expert
Expert
Score
Score
Avg Score
Model Similarity
0
CAM
POL
0.95
0,99
0.97
0.98
1
CAM
SKL
0,85
0,99
0,92
0.97
2
CAM
LEA
0.51
0.89
0,70
0.88
3
POL
SKL
0.98
0,95
0,96
0.99
4
POL
LEA
0.40
0.84
0.62
0.93
SKL
LEA
0.83
0.95
training performance on language tasks. In contrast, the MBPP expert, while delivering competitive scores on coding tasks, shows significantly lower performance on specific tasks in the Reason- ing category. This sharp decline in performance suggests the occurrence of catastrophic forgetting during fine-tuning, where the model loses its ability to generalize across tasks outside its specializa- tion. These findings once again underscore the efficacy of CycleQD in achieving superior and more consistent performance across diverse tasks compared to traditional fine-tuning approaches.
4.2
EVALUATION ON IMAGE SEGMENTATION TASKS
4.2.1 TASK DESCRIPTION
Besides to its applications for LLMs, CycleQD serves as a versatile method for integrating expert models across various data modalities beyond text. For example, include vision question answering (VQA) task in addition to the CS tasks and find CycleQD to be able to outperform the experts (see Section A.1. In this experiment, fusion of multiple Segment Anything Models (SAM), which are state-of-the-art computer vision models designed for image segmentation tasks. Specifically, our objective is to merge pairs of SAM models, A and B, to create models whose capabilities encompass the skill sets of both A and B.
go further beyond and extend CycleQD the
4.2.2 CYCLEQD SETUPS
Experts: We select four tasks within the image segmentation domain, each supported by extensive datasets, for training specialized models: Camouflaged Object Segmentation (CAM). Polyp Seg- mentation (POL), Skin Lesion Segmentation (SKL), and Leaf Segmentation (LEE).. CAM detects objects in cluttered environments, making segmentation more challenging than standard tasks. POL. identifies polyps in endoscopic images, essential for carly colorectal cancer detection. SKL detects various skin lesions in medical images. LEA identifies plant leaves in agricultural images, sup- porting disease control and improving crop quality. We employ SAM-ViT Huge model as our base model, which we fine-tune to develop these experts. See Appendix A.2. for details on the datasets used for tine-tuning and CycleQD setups.
Hyper-parameters: CycleQD's hyper papee-prrsettes remain the same in Section 4.1.2
4.2.3 RESULTS
Overall: Table Al shows the performance of the models merged by CycleQD, where scores normalized against the expert models. You can see visualization results in Appendix [A.22 In general, CycleQD is able to merge the experts successfully, with top models retaining than

---

Published as a conference paper at ICLR 2025
90% of the experts' performance (e.g., models 0, 1, and 3). On the other hand, models 2, 4 and 5 are less successful. This leads us to conduct further analysis and we report the findings next.
Analysis: We report the similarities between experts A and B in the last column in Table A strong correlation of 0.83 between the averaged scores and these similarities indicate both the limitations and potential areas for enhancement in CycleQD. We define the similarity between models A and B as the average cosine similarity of the singular value vectors (derived from the task vectors) across all layers in both models: s = (1/L) Et cos (diag(E,,A). diagg
Here, L is the number of weight matrices with a rank greater than 1, and Es,, denotes a diagonal matrix of singular values from the i-th weight matrix in the task vector. Although this summarizing metric does not fully encapsulate the models' characteristics, CycleQD tends to perform well when the models exhibit high similarity. This observation underpins our approach of incorporating this similarity metric as regularization technique during model fine-tuning. Notably, a similar metric has been employed as the "proximal term" in optimizing heterogeneous networks within the realm of federated learning, as evidenced by research documented in (Li et al. 2020). This precedent lends credence to our strategy, suggesting is grounded in established methodologies.
RELATED WORKS
LLM skill acquisition research has expanded beyond conversational tasks, focusing enabling autonomous problem-solving through the L.M--as-Aeet paradigm (Xi et al. 2023 Wang et al.) 2024b). Techniques like ReAct (Yao et al. 2023) integrate reasoning with action, but skill acqui- sition through training remains limited, often relying on standard fine-tuning (Zeng et al. 2024 Qin et al. 2024). which struggles with balancing multiple skills. QD (Pugh al. 2016) is an evolutionary paradigm aimed at discovering diverse, figh-performing solutions. Algorithms like MAP-Elites (Mouret& & Clune| 2015) and MOQD (Pierrot et al., (022) have refined QD for tasks with diverse behaviors and conflicting objectives. While QD has been applied to neural networks and L.M prompts (Guo et al., 2024, Xue et al.] (024)) evolving L.LM weights remain unexplored. Model merging, which combines capabilities across pre-trained models at lower training costs, has advanced through techniques like sparsification (Yadav et al. [023 Stoica et al. (224)) and optimiza- tion (Yang et al. 2024). Our work integrates QD with model merging, allowing LLM components to be preserved and optimized for diverse tasks. Please refer to Section for more related works.
6 CONCLUSIONS AND FUTURE WORKS
In this work, we introduced CycleQD, a compelling modification of the conventional LLM fine- tuning pipeline, integrating evolutionary algorithms for agent skill acquisition. CycleQD begins with single-task experts and utilizes QD to continuously generate hundreds of LLM--based agents, possessing diverse characteristies and exhibiting broader range of skills than the initial experts. Through dynamic process that cyclically swaps quality and BCs, coupled with a model merg- ing crossover operation and an SVD-based mutation operator, CycleQD has successfully enabled LLMSs to master computer science skills. Our method not only outperforms baseline approaches but also achieves on-par performance with GPT-3.5-TURBO, while generalizing to out- disttiibbttonno tasks and retaining the language capabilities. Furthermore, CycleQD's 's applicability extends across domains to image segmentation models, demonstrating its broad utility.
In terms of limitations, we acknowledge that the success of model merging hinges on the compati- bility of the source models. CycleQD may encounter challenges when the expert models originate from highly divergent settings. One way to address this is incorporating model similarity ularization term during the expert training. Furthermore, this research represents an initial foray into the integration of QD and evolutionary model merging for agent skill acquisition. There is substantial potential for improvement by leveraging advanced methodologies from both fields, such as integrating strategies from CMA-ME (Fontaine al., 2020) and PGA-MAP-Elites (Nilsson & Cully] 2021) to enhance the efficiency of the learning process. Looking ahead, a promising direc- tion for future research lies in multi-agent systems. Considering CycleQD generates an archive of diverse agents, orchestrating these agents to collaborate and compete opens up exciting possibilities for scientific exploration and practical applications.
reg-
10

---

Published as a conference paper at ICLR 2025
AUTHOR CONTRIBUTIONS
Yujin Tang and Takuya Akiba initiated the project. So Kuroki co-designed the algorithm, imple- mented CycleQD, conducted the experiments and the studies. Taishi Nakamura developed the agen- tic tasks, the expert models and the baseline methods. Takuya Akiba implemented the parallel distributed optimization and proposed the SAM experiment. Yujin Tang proposed the algorithm, designed and helped with the experiments, and managed the project. All authors contributed to the writing and approved the final version.
ACKNOWLEDGEMENTS
The authors would like thank Robert Lange for providing valuable discussions and feedback while drafting the text, and Kou Misaki and Qi Sun for providing infrastructure support. This paper is based results obtained from project, JPNP20017, subsidized by the New Energy and Industrial Technology Development Organization (NEDO).
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303.0877 08774, 2023.
Takuya Akiba, Makoto Shing, Yujin Tang, Qi Sun, and David Ha. Evolutionary optimization of model merging recipes. arXiv preprint 1331 2024.
Jorge Bernal, F Javier Sánchez, Gloria Fernández-Esparrach Debora Gil, Cristina Rodríguez, and Fernando Vilariño. Wm-dova maps for accurate polyp highlighting in colonoscopy: Validation saliency maps from physicians. Computerized medical imaging and graphics, 43:99-111, 2015.
Herbie Bradley, Andrew Dai, Hannah Benita Teufel, Jenny Zhang, Koen Oostermeijer, Marco Bellagente, Jeff Clune, Kenneth O. Stanley, Grégory Schott, and Joel Lehman. Quality- diversity through AI feedback. In The Twelfth International Conference on Learning Rep- resentations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL KKCrGYT
httts:://ppeneee eee www.eee/eeeeeeeeeeeeee
Zehui Chen, Kuikun Liu, Qiuchen Wang, Wenwei Zhang. Jiangning Liu, Dahua Lin, Kai Chen, and Feng Zhao. Agent-fland flan: Designing data and methods of effective agent tuning for large language models. arXiv preprint arXiv:2403. /2881, 2024.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv:21 10.11111 2021.
Noel CF Codella, David Gutman, M Emre Celebi, Brian Helba, Michael A Marchetti, Stephen W Dusza, Aadi Kalloo, Konstantinos Liopyris, Nabin Mishra, Harald Kittler, al. Skin lesion anal- ysis toward melanoma detection: A challenge at the 2017 international symposium on biomedical imaging (isbi), hosted by the international skin imaging collaboration (isic). In 2018 IEEE 15th international symposium on biomedical imaging (ISBI 2018). pp. 168-172. IEEE, 2018
Kalyanmoy Deb, Amrit Pratap. Sameer Agarwal, and TAMT Meyarivan. A fast and elitist mul- tiobjective genetic algorithm: Nsga-ii. IEEE transactions on evolutionary computation, 6(2): 182-197. 2002.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21 1783, 2024.
Deng-Ping Fan, Ge-Peng Ji, Guolei Sun. Ming-Ming Cheng, Jianbing Shen, and Ling Shao. Cam- ouflaged object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2777-2787, 2020a.

---

Published as a conference paper at ICLR 2025
Deng- Fan, Ge-Peng Ji. Tao Zhou, Geng Chen, Huazhu Fu, Jianbing Shen, and Ling Shao. Pranet: Parallel reverse attention network for polyp segmentation. In International conference on medical image computing and computer-assiste intervention, pp. 263 273.. Springer, 2020b.
Matthew C Fontaine, Julian Togelius, Stefanos Nikolaidis, and Amy K Hoover. Covariance matrix adaptation for the rapid illumination of behavior space. In Proceedings of the 2020 genetic and evolutionary compulation conference, pp. 94-102, 2020.
Kazuki Fujii, Taishi Nakamura, and Rio Yokota. Ilm-recipes, 2024. URL https:
com/ hann/okoe-kaz// http:/mmmreeiicpp
Qingyan Guo, Rui Wang, Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yujiu Yang. Connecting large language models with evolutionary algorithms yields pow- erful prompt optimizers. In The Twelfth International Conference on Learning Representa- tions, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https:
//openzeview.co wwwpenreview.et//frrrreeeeeee
Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. A real-world webagent with planning, long context understanding, and pro- gram synthesis. arXiv preprint arXiv:2307.12856 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Ja- cob Steinhardt. Measuring massive multitask language understanding. In International Confer- ence on Learning Representations, 2021. URL
net/ net/forummidd
177KBJmI 3CmQ.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. arXiv preprint arXiv:2212.04089, 2022.
Gabriel Ilharco, Marco Túlio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannanch Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. The Eleventh International Confer- ence on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
6t jrjj
D Jha, PH Smedsrud, MA Riegler, P Halvorsen, T De Lange, D Johansen, and HD Johansen. K Kvasir- seg: A segmented polyp dataset. multimedia modeling, 2019.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Regina Barzilay and Min- Yen Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL.), pp. 1601-1611, Vancouver, Canada, July 2017. Association for Computational Linguis- WRL.ERRLL........... htt.thttts//caccc http:///calaacccoaaoaoca wow. wtt.hhoooooo..oo/1111/.1111
tics. doi: 10. 16316/6111111711477 URL
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision, pp. 4015-4026, 2023.
Trung-Nghia Le, Tam V Nguyen, Zhongliang Nie, Minh-Triae Tran, and Akihiro Sugimoto. Anabranch network for camouflaged object segmentation. Computer vision and image under- standing, 184:45-56, 2019.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. In Inderjit S. Dhillon, Dim- itris S. Papailiopoulos, and Vivienne Sze (eds.), Proceedings of the Third Conference on Machine Leurning and Systems, MLSys 2020, Austin, TX, USA, March 2-4 2020. ml- ml layss wwtg:/ppprr_riii1cc files wiles/paperrsre sys.org, 2020. URL. haaaa//ESSSe33aa0033eeee3773333 7751777777777700000oo.o Abstracct. htmm
1020/hash// ht20/hahh/E5te83322228
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang. and LINGMING ZHANG. Is your code generated
by chatGPT really correct? rigorous evaluation of large language models for code generation.
In Shirth-eeventh Conference on Neural Information Processing Systems, 2023a. URL https:
//openrevien fww.:oorttt/tetteeoo d= 1 0Cu7
12

---

Published as a conference paper at ICLR 2025
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating Ilms as agents. arXiv preprint arXiv:2308. 03668, 2023b.
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. Agentbench: Evaluating LLMs as agents. The Twelfth International Conference on Learning Representations, 2024. BtLhctps: : http:////ppeeeeeeeeeeeeee http://wwwww.weeeeeeeeeeeeeeeee http://wwweeeteoooooooo/o//////d
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019. URL pettps://openrevicce net. wwt.:/rrttt/oooummmoe for.mrrddmmm
Bkg6RiCqY7
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? A new dataset for open book question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun iichi Tsujii (eds.), Proceedings of the 2018 Conference on Empir- ical Methods in Natural Language Processing, pp. 2381-2391, Brussels, Belgium, October- November 2018. Association for Computational Linguistics. doi: 10.18653/11/D1--D222 1260. URL htttps://acLattholoory httpr//ooooooooooo///----22
Jean-Baptiste Mouret and Jeff Clune. Illuminating search spaces by mapping elites. arXiv preprint arXiv:1504.04999, 2015.
Olle Nilsson and Antoine Cully. Policy gradient assisted map-elites. In Proceedings of the Genetic 2021.
and Evolutionary Computation Conference, pp.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35:
27730-27744, 2022.
Thomas Pierrot, Guillaume Richard, Karim Beguir, and Antoine Cully. Multi-objective quality diversity optimization. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 147, 2022.
Justin K Pugh, Lisa B Soros, and Kenneth O Stanley. Quality diversity: A new frontier for evolu- tionary computation. Frontiers Robotics and AI, 3:202845, 201228
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong. Xi- angru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li. Zhiyuan Liu, and Maosong Sun. Toolllm: Facilitating large language mod- els to master 16000+ real-world apis. In The Twelfth International Conference on Learning Representaatons, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL htttps://pperrewe
net/ www.mrrrrii.oo............. (d-rummid-dllgg20001
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable ques- tions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Compu- lational Linguistics (ACL), pp. 784-789, 2018. doi: (0.18653/V1/P18--122 URL https:
Sovit URL
Ranjan
Rath.
Leaf
discase
segmentation
dataset, 2023.
https: wwwp:://wwww.... http://www..ggle..e com/ http://wwww..aooooooooooooottttt
Mikayel Samvelyan, Sharath Chandra Raparthy, Andrei Lupu, Eric Hambro, Aram H. Markosyan, Manish Bhatt, Yuning Mao, Minqi Jiang, Jack Parker-Holder, Jakob N. Foerster, Tim Rocktäschel, and Roberta Raileanu. Rainbow teaming: Open-ended generation of diverse ad- doi: 1044550/ARXIV22422.6633 48550/ httpv..iivvv.///.. 240..22422.2. 16822
versarial prompts. CoRR, 168222 2024.
URL
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqu models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317-8326, 2019.
13

---

Published as a conference paper at ICLR 2025
Przemystaw Skurowski, Hassan Abdulameer, J Blaszczyk, Tomasz Depta, Adam Kornacki, and Koziet. Animal camouflage analysis: Chameleon database. Unpublished manuscript, 2(6):7, 2018.
George Stoica, Daniel Bolya, Jakob Bjomner, Taylor Hearn, and Judy Hoffman. Zipit! merging models from different tasks without training. In ICLR, 2024.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe Le, Ed Chi, Denny Zhou, and Jason Wei. Challenging BIG-bench tasks and whether chain-of-though can solve them. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 15003-1305L Toronto, Canada, July 2023. Association for Computational Linguis- tics. doi: 10.18663//1/2000 findings-acl.824 URL https: wwwllanhhoooog org 20233.
f Findings-an acc 824
Weihao Tan, Ziluo Ding, Wentao Zhang, Boyu Li, Bohan Zhou, Junpeng Yue, Haochong Xia, Jiechuan Jiang, Longtao Zheng, Xinrun Xu, Yifei Bi, Pengjie Gu, Xinrun Wang. Börje F. Karls- son, Bo An, and Zongqing Lu. Towards general computer control: A multimodal agent for red dead redemption 03186. URL https: /ww. www.oogg.1.... 48550/ arxiv 2403. 03186
case study. CoRR, abs/2403.03118. 2024. doi: 10.48550/AXXV.2403..
Alexey Tikhonov and Max Ryabinin. It's all in the heads: Using attention heads baseline for cross-lingual transfer in commonsense reasoning. In Findings of the Association for Computa- tional Linguistics (ACL), pp. 3534-3546, 2021. doi: 10. (8663///2222.FNNNINSS--6868
URL http:///oob.oogoo....1866/////20 findiig..nndigggs wwwpddinns-al....2 310
Koki Usui, Reiji Suzuki, and Takaya Arita. Towards open-ended evolution based cvt-map-elites
with dynamic switching between feature spaces. In ALIFE 2023: Ghost in the Machine: Pro- ceedings of the 2023 Artificial Life Conference. MIT Press, 2023.
Ke Wang, Nikolaos Dimitriadis, Guillermo Ortiz-Jiménez, François Fleuret, and Pascal Frossard. Localizing task information for improved model merging and compression. In JCML, pp. to appear, 2024a.
Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, al. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6):186344 2024b.
Ren-Jian Wang, Ke Xue, Haopu Shang, Chao Qian, Haobo Fu, and Qiang Fu. Multi-objective optimization-baser selection for quality-diversity by pon-surrounded-oominae Artificial Intellgence, IJ- CAI 2023, 19th-25th August 2023, Macao, SAR, China, pp. 4335 4343. ijcai.org, 2023. doi: 10.24963/JICAI.222 2023/488. httpttps: /tt:://dooi.. /tt///i.ogg///// 249 htt./iicaa.... icai 2023/4882
sorting. In
Proceedings of the Thirty-Second International Joint Conference
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoe V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Ad- vances in Neural Information Processing Systems, 2022. utLhttps: /ptpreeiieperrvie....m..m net/ Forum?ide Brrummidd__iooMeee VjO1Me
Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and LINGMING ZHANG. Magicoder: Em- powering code generation with OSS-instruct. In Forty-firsl International Conference on Machine Learning, 2024. URL
net/ forum? wtt.rrummmdxxxxoooooooo d= d33
Ryan Wickman, Bibek Poudel, Michael Villarreal, Xiaofei Zhang, and Weizi Li. Efficient quality- diversity optimization through diverse quality species. In Sara Silva and Luis Paquete (eds.), Companion Proceedings of the Conference on Genetic and Evolutionary Computation. GECCO 2023, Companion Volume. Lisbon, Portugal, July 15-19, 2023, pp. 699-702. ACM, 2023, doi: 3590581
10.1145535831333355555 URL httl:ttps:// //tp::///ooi.. http://doooooogg/.// /
Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith,
14

---

Published as a conference paper at ICLR 2025
and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models im- proves accuracy without increasing inference time. In Kamalika Chaudhuri, Stefanic Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML. 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of PMLR, 2022.
Proceedings of Machine Learning Research, pp.
https:
7/proceedings mlr press /vl62
Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. The rise and potential of large language model based agents: A survey. arXiv preprint arXiv:2309.07880
2023.
Zhiheng Xi, Yiwen Ding, Wenxiang Chen, Boyang Hong, Honglin Guo, Junzhe Wang, Dingwen Yang, Chenyang Liao, Xin Guo, Wei He, et al. Agentgym: Evolving large language model-based agents across diverse environments. arXiv preprint arXiv:2406.04 041551 2024.
Tianqi Xu, Linyao Chen, Dai-Jie Wu, Yanjun Chen, Zecheng Zhang, Xiang Yao, Zhiqiang Xie, Yongchao Chen, Shilong Liu, Bochen Qian, Philip Torr, Bernard Ghanem, and Guohao Li. CRAB: crossennvirommen agent benchmark for multimodal language model agents. CoRR, Abs/2400.005111 2024.
doi: 10445550/ARXI..240.00555 48550/arXi www.aarwww...........
URL htttps://dos ooto/i...
Ke Xue, Ren-Jian Wang. Pengyi Li, Dong Li, Jianye Hao, and Chao Qian. Sample-efficient quality- diversity by cooperative coevolution.
The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https://<penxevii net/ forum? (erruummdddDDudubbpppe 2DpF2
Prateek Yadav, Derek Tam, Leshem Choshen, Colin A. Raffel, and Mohit Bansal. Ties- merging: Resolving interference when merging models. In Alice Oh, Tristan Nau- mann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference
Neural Informa-
fion Processing Systemy 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http:// http:::://ppppsspppppp nn ps. ww/paper fi es pttp//apppp//22///a/////e (G44CA4f22.b79116744666222222222 ottp://aaateetteaaoaaaaaaaaa--aa Absstacsttttt.s Cotfe--oofeeeeeeeee html
Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao.
Adamerging: Adaptive model merging for multi-task learning. In ICLR, 2024.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representattions, ICLR 2023, Kigali. Rwanda, May 1-5, 2023. OpenRe- view.net, 2023. URL
wet/forummieiie Wid=WE vluYUI-X
Chenchen Ye, Ziniu Hu, Yihe Deng, Zijie Huang, Mingyu Derek Ma. Yanqiao Zhu, and Wei Wang. MIRAI: evaluating LLM agents for event forecasting. CoRR, abs/2407 012211 2024. doi: 10. AS550/ARXIV.2407.02233
http:ttt::// //dii. org/10......oooooooooooooo
(1555/0/axi..2220 2407 01231
Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yonghin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In ICMI., pp. to appear, 2024.
Rowan Zellers, Ani Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi HellaSwag: Can a machine really finish your sentence? In Anna Korhonen, David Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791-4800, Florence, Italy, July 2019. Association for Computational Linguisties. doi: 10. 18653/v 1//55//1//19--11 1472. URL https::///// http://aaaaatthooocooooo wwtp/19-111P11 1472
Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. Agent- tuning: Enabling generalized agent abilitics for 1lms. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (cds.). Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August l1 11-16, 2024, pp. 3053-307377700 Association for Computational Linguistics, 2024. URL ac www.hnnnntoogyyyyo wwwp//24..202...o ftt.:.......iii.. windings-dnniie ac1. 181
Zihan Zhong, Zhiqiang Tang, Tong He, Haoyang Fang, and Chun Yuan. Convolution meets lora: Parameter efficient finetuning for segment anything model. ICLR, 2024.
15

---

Published as a conference paper at ICLR 2025
Temry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul. et al. Bigcodebench: Bench- marking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406. 15877, 2024.
16

---

Published as a conference paper at ICLR 2025
A
APPENDIX
A.1 COMPUTER SCIENCE TASKS
A.1.1 EXTRA TASK SETUPS
The MBPP+ dataset, based on the original MBPP, uses a subset 399 hand-verified problems from MBPP-sanitize to ensure well-formed programming tasks. Each problem includes a problem statement, function signature, and test cases. EvalPlus extends this dataset with additional test cases to provide a more rigorous evaluation of code generation capabilities, For our evaluation, we use the pass@1 metric on the Base Tests of MBPP+, which reflects the model's ability generate correct code on the first attempt using the original test cases. The OS dataset evaluates LLMs in genuine interactive bash environments (Ubuntu Docker) on human questions with deterministic answers and practical operational tasks. The DB dataset assesses LLMs' abilities operate real databases via SQL, encompassing the complete pipeline of database analysis with authentic SQL interfaces and multiple tables. Both OS and DB datasets success tasks are designed to be solved within a maximum interaction turns and employ 1-shot setup. The DB tasks have similar interaction limit but are evaluated in 0-shot manner.
the primary evaluation metric. The OS
A.1.2 TRAINING CONFIGURATION FOR EXPERTS
We utilize lIm-recipes (Fujii et al. 2024 (commit 606cdfb) for fine-tuning. We adopt the Adam W optimizer (Loshchilovv Hutter 2019) with 31 = 0.9 and 83 0.95. A global batch size of 64 used across all fine-tuning processes. We employ cosine learning rate scheduling with range of [4 x 10 6, x 10 10- starting with a linear warmup for the first 10% of the total training steps. The OS and DB experts trained for epoch, while the code model is trained for 3 epochs due to its larger training data size.
A.1.3 CYCLEQD HYPER-PAREE PARAMETERS
3.54
We set (low 0.5 and Onigh 0.8 in Elite sampling, = 1.0 and o = 0.03 in model merging- crossover, and Wmax 0.3 in our SVD-based mutations. These hyperpa- rameters are determined preliminary studies, where we arbitrarily determined a set of values, ran CycleQD for small number of generations and picked the set with the best performance. We believe the performance of CycleQD can be further improved when incorporating sophisti- cated hyperparameter searching methods.
0.52
0.50
Average Score
2.46
3,42
200
400
600
300
1000
1200
To investigate the influence of the number of generations on CycleQD performance, we an- alyzed the average scores of the computer sci- ence tasks every 100 generations. Figure il- lustrates these results. The performance gradu- ally increases over the generations with no obviously significant oscillations. While still showing slight upward trend beyond this point, it stabilizes around 1000 generations.
Gemeration
Figure 5: Averaged performance on computer science tasks. Evaluated every 100 generations.
A.1.4 CYCLEQD DEVELOPMENT OF ARCHIVES ACROSS GENERATIONS.
Figure 6 shows the development of archives across generations. The archives are shown in incre- ments of 300 generations from top to bottom. The corresponding generation for each archive is displayed the left side of the figure. The red bounding boxes indicate the grids where expert policies were present in each archive. It can be observed that the frontier of the archives expands with each passing generation.
17

---

Published as a conference paper at ICLR 2025
A.1.5 ADDING A VISION QUESTION ANSWERING TASK
In addition to the computer science tasks, we add visual question answering (VQA) task Cy- cleQD to further demonstrate its generally applicability across tasks that differ drastically. The VQA task lies within the lsionnlanguage modeling (VLM) domain, which presents challenges when merging it with tasks from LLM domain.
Specifically, we use the TextVQA dataset (Singh et al. 2019) as part the CycleQD experiments along with the coding, DB, and OS tasks. The experimental setup follows the same configurations described in Section 4.222) including parameters and the 1200 generations training budget. To create optimize the -LAMA3-LLAVV-NE Tex data as training data. Additionally, we allocate 500 distinct samples, separated from the training data, for CycleQD training. These 500 samples evenly divided into optimization and test datasets.
VQA expert,
8BB model using 4000 samples from the
During the CycleQD optimization process, extract the LLAMA3-88 SB-INSTRUCT component of the VLM and perform merge and mutate operations. The results, presented in Table 5 show that CycleQD achieves the best average performance compared to both the base and the expert models. Notably, CycleQD outperforms the several expert models on VQA, coding, and OS tasks, highlight- ing its effectiveness in combining knowledge across diverse domains.
Table 5: Evaluation on computer science tasks and VQA. We add VQA task to computer science tasks and evaluate CycleQD.
# Methods
VQA
MBPP
DB
OS
Avg
0
Llama3 -B-na3-8B-lln Insttrucct (base model)
39.0
67.3
5.3
25.2
34.2
Experts models
VQA expert
51.4
4.0
0..
1.5
14.2
2
Coding expert
32,9
70.4
21.2
20.7
36.3
3
DB expert
45.4
65.8
42.4
28.5
45.5
4
expert
46.1
66.3
30.4
35.7
$ CycleQD (Ours)
54.1
72.9
32.4
39.6
49.7
A. BENCHMARK DATASETS FOR SKILL GENERALIZATION AND LANGUAGE UNDERSTANDING
In the Coding category, we employ two key benchmarks: HUMANEVAL+ (Liu et al. 2023a). eval- uated in a O-shot setting using the pass@) metric on the Base Tests, and BigCodeBench (Zhuo al.) 2024], which is designed to assess code generation with diverse function calls and complex instruc- tions. We report the average Pass @ scores for both BigCodeBench- -ooppletee and BigCodeBench- Instruct variants in the full setting, using 0-shot evaluation. For the General Knowledge and Rea- soning (Reasoning) category, we utilize two comprehensive benchmarks: MMLU (Hendrycks et al.) 2021 with 5-shot evaluation, and BBH (Big Bench Hard) (Suzgun et al. 2023) using 3-shot chain-of-thought (Wei et al. 2022) prompting approach. These benchmarks provide holistic as- sessment of model's 's ability handle wide range of knowledge-basee and reasoning tasks. To evaluate Mathematical Reasoning, we employ the GSM8K benchmark (Cobbe et al. 2021 with a 4- shot evaluation, challenging our model's ability to solve complex mathematical problems. For Read- ing Comprehension (RC), we use two established benchmarks: SQuAD2 (Rajpurkar et al., (018)) and TriviaQA (Joshi al. 2017). both evaluated using 4-shot approach. These datasets assess the model's 's capacity to understand and reason over complex textual information. In the Commonsense Reasoning (CommonSense) category, we employ three diverse benchmarks: HellaSwag (Zellers) et al.) 2019) for commonsense inference, OpenBookQA (Mihaylov et al. 2018) for elementary- level science question answering that requires both core scientific knowledge and broader common knowledge, and XWinograd (English version) (Tikhonov & Ryabinin 2021 for cross cross-lingus lingual com- monsense reasoning and coreference resolution. All these evaluations are conducted with 4-shot prompts.
18

---

Published as a conference paper at ICLR 2025
A.1.7 TRAINING TIME COMPARISON
We used NVIDIA H100 GPUs for our experiments. The gradient fine-tuning method (model #6 in Table took approximately 200 GPU hours, and our method (model #1 in Table took about 410 GPU hours (excluding the expert models training time). We wish to point out that (1) the extra time spent on CycleQD is mostly due to agentic task evaluations rather than the optimization, (2) due to GPU memory constraints, the merging process was carried out on CPU, (3) CycleQD produces an archive of models whereas fine-tuning generates only one. Furthermore, it is important to note that fine-tuning a mature and well-established approach with software implementations that are optimized for efficiency. By contrast, the proposed method was developed primarily to assess its performance, and as such, there is ample room for further efficiency improvements.
A.2 IMAGE SEGMENTATION TASKS
1.2.1 DATASETS FOR FINE-TUNING EXPERTS AND CYCLEQD TRAINING
We followed Zhong et al. (2024) to prepare the datasets. For Camouflaged Object Segmentation, we use three datasets: CODTOK (Fan et al. 2020a), CHAMELEON (Skurowski et al. 2018), and CAMO (Le al. 2. Following Fan et al. (2020a) we train on a combined dataset consisting of the 4040 training images from CODTOK and CAMO for 20 epochs, randomly splitting 10% of the images from the training set for validation. The model is then tested on the 250 COMO test images. For Polyp Segmentation, we use two datasets: Kvasir (Jha et al. 2019) and CVC-ClinieDB/CVC. 612 (Bernal et al. 2015). Following Fan et al. (2020b), we divide the images into a 9:1 ratio for training and testing, resulting in 1450 training images. We then randomly split 20 % of the training set for validation. The model is trained for 30 epochs and tested on the 101 Kvasir test images. For Skin Lesion Segmentation, we use the ISIC 2017 dataset (Codella al. 2018). We train the model on the 2000 training and 150 validation images for 30 epochs and evaluate it on the 600 test images.
For Leaf Segmentation, we use the Leaf Disease Segmentation Dataset (Rath 2023). We train the model on the 498 training image, using 80% for training and 20% for validation for 30 epochs, and evaluate it on 90 test images.
A.2.2 VISUALIZATION RESULT
The visualization results are shown in Figure SKL and POL, and Fig SKL and LEA. In both figures, "Ours" refers to a single merged model via CycleQD, identical to model #3 (for SKL and POL) and model #5 (for SKL and LEA) in Table In Figure 7 our model retains near expert- level performance for both tasks after merging. However, in Figure g while our model generally performs at an expert level for both tasks, a slight drop is observed. Notably, in the bottom images of SKL, the model seemss to detect the edges of the skin lesions instead of the lesions themselves, as for
if confusing the task with leaf discase detection. The expert model similarities are SKL and POL. and 0.95 for SKL and LEA. This corresponds with the observation that the merged model from the highly similar SKL and POL performs better than the one from SKL and LEA.
B MORE RELATED WORKS
Beyond conversational tasks, increasing attention has been given to en- abling LLMs to acquire skills to take action. This allows LLMS to function as agents capable of solving problems autonomously, and this approach is known as LLM-as-Agent (Xi al.) 2023 Wang et al. 20245). Benchmarks for LLM-aasAgent have been developed, and research has demon- strated that LLMs can successfully perform tasks such as computer operations and web browsing to a certain extent (Gur et al.] 2023] Liu et al. 2023b) Xi et al. 2024, Xu et al., 2024] (Tan et al.) 2024 al. 2024). ReAct Yao et al. (023) is the most frequently used approach for constructing LLM-aa-Agent systems, which integrates CoT reasoning with action. ReAct is typically employed through prompting and few-shot in-context learning, where models learn skills from a small number of examples. However, attempts to equip LLMs with such skills through actual training with more examples are still quite limited, with a few studies focusing only on standard fine-tuning (Zeng et al. 2024 Qin et al.] 2024). This fine-tuning approach often struggles with balancing multiple agent skills and suffers from the gap between next token prediction loss and actual task performance. Our proposed method uses QD techniques to optimize each skill independently and directly.
LLM Skill Acquisition:
19

---

Published as a conference paper at ICLR 2025
Quality Diversity: QD is emerging paradigm in evolutionary computation that aims discover a diverse collection of high-perfore performmig solutions rather than a single optimal result (Pugh et al. 2016). QD algorithmss balance two key aspects: quality, which represents solution's 's performance, and behavior characterization (BC), which describes its observable properties. These algorithms maintain an archive of diverse, high-quality solutions, continuously updated through an evolution- ary process. MAP-Elites (Mouret & Clune 2015) is a prominent QD algorithm that maintains an archive defined by the BCs, discretizes into tessellation, and stores the best-performing solutions in each cell to preserve diversity. In each generation, it samples parents from the archive, creates offspring through crossover and mutation, and evaluates their quality and BCs. Offspring replace existing elites if they perform better in their respective cells. This process simultaneously explores diverse behaviors and optimizes performance within each niche, resulting diverse solutions. Recent improvements to MAP-Elice include a new selection method for better diversity (Wang et al. 2023) and an approach that removes the need for predefined niches (Wickman et al. 2023). Additionally, to address potential stagnation in fixed BC space, [ et al. (2023) propose dynamic switching of the BC space. CycleQD goes further by aiming for MOQD, rotating both the BCs and the quality across tasks. MOQD (Pierrot al.) (022) has been proposed to address problems with multiple, potentially conflicting objectives while maintaining diversity. Instead of the
map of high-quality,
fitness, it maximizes Pareto front hyper-volume in each niche of the descriptor space. CycleQD is a simplified form of MOQD in the sense that it uses task metrics as both BCs and the quality, the product of which approximates the hyper- hoper-volume, and the coordinate ascent style of optimization allows it to avoid complex hyper-volume calculations for higher-dimensiinn
archives.
Quality Diversity for LLMs: Evolutionary computation, particularly QD, has occasionally been applied to neural networks and LLMs (Guo et al., 2024 Xue et al., 2024 Akiba et al. 2024) How- ever, there has been no prior research that evolves the main weights of LLMs using QD. Saamvelynn et al. (2024) proposed Rainbow Teaming, which applies QD to evolve diverse adversarial prompts for evaluating LLM safety. Bradley et al. (2024) introduced QDAIF, which applies QD with LLM feedback to evolve populations of generated creative texts. Our work represents the first attempt to merge LLMs through QD. We believe this approach holds great promise, as it allows individual components of the model be preserved even when they are temporarily sub-optimal, with the potential to contribute to future performance improvements. Given the cost and complexity of opti- mizing the many layers of LLMs, QD offers unique advantage by retaining useful configurations in archives that may later enhance overall model performance.
Model Merging: Model merging refers combining multiple pre-trained models into single model. This technique is gaining attention because it can enhance model performance and integrate multiple capabilities at much lower training cost. Additionally, unlike ensemble methods, model merging does not increase inference costs either. This technique is particularly effective when ap- plied to different fine-tuned versions of the same base model. The most basic method involves a linear combination of weights (Wortsman et al. [022 Ilharco et al. (023). More advanced merging techniques have been proposed, incorporating methods such as election and sparsification (Yadav) et al. 22233 Yu et al. 2024; Wang et al. 20244a] Stoica et al.) 2024). Furthermore, the applicability and performance of these methods have been significantly improved through optimization (Akiba et al. 2024| Yang al.) 2024).. Inspired by these merging strategies, CycleQD combines QD with model merging, allowing for more effective synthesis of multiple capabilities in LLMs.
20

---

Published as a conference paper at ICLR 2025
Generations
G. MBPP
C: 0.
0.08
060
475
0
6:00
2.30
6:20
620
0.3
NEFP
MSPP
C: MBPP
08
4.30
875
oas
A000
000
300
025
020
03
0.00
MIPP
FIC: MIPP
MBPP
06
0:06
0.8
6.40
079
0:25
4.20
600
0.20
2.0
4.5
6.10
BC:
BC: NEFP
NDPP
0:
0.03
030
0.00
900
425
000
0.15
0.9
MSPP
MBPP
MBPP
0:08
08
0.00
025
6030
1200
Figure 6: CycleQD Development of Archives Across Generations. In each archive, the two axes represent the BCs, and the color intensities in each grid indicate the quality of the LLM agent that grid. The archives are shown in increments of 300 generations from top to bottom. The correspond- ing generation for each archive is displayed on the left side of the figure. The red bounding boxes indicate the grids where expert policies were present in each archive. The experiment shown in this figure is the same as in Figure and the archive for 1200 generations is identical to that in Figure
21

---

Published as a conference paper at ICLR 2025
Skin Lesion Segmentation (SKL)
Polyp Segmentation (POL)
Image
Ground Iruth
Expert
Ours
Image
Ground Iruth
Expert
Ours
Figure 7: CycleQD of SAM for SKL and POL. Ours is single merged model via CycleQD, and is the same model as the one Table 4 (# 3). The similarity between the two expert models is high (0.99), and the model maintains expert-level performance across both tasks.
Skin Lesion Segmentation (SKL)
Leaf Segmentation (LEA)
Image
Ground truth
Expert
Ours
Image
Ground tuuth
Expert
Ours
Figure 8: CycleQD of SAM for SKL and LEA. Ours is single merged model via CycleQD, and it is the same model as the one in Table 4 (# 5). The similarity between the two expert models is 0.95, which is lower than #3 (0.99) from Table resulting in slight decrease in performance, as observed when comparing the bottom images in SKL.
22


Reviews:
[{"review_id": "hqepJjUeGG", "reviewer": "Reviewer_tJqK", "summary": "This paper presents CycleQD, a novel framework for making llm acquire specific skills through Quality Diversity (QD) optimization. CycleQD addresses challenges in skill training, such as data imbalance and inadequate objective functions, by cyclically alternating quality measures for different tasks.  CycleQD shows competitive results, achieving performance on par with GPT-3.5-TURBO.", "questions": "I am also curious about any real-world application tests using the proposed method across different scenarios, given that the scenarios in the experiments are quite similar.", "limitations": "", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper presents an innovative approach to LLM skill acquisition, uniquely applying the Quality Diversity paradigm to cycle through task-specific optimizations, which is quite novel. \nThe experimental results on the proposed CycleQD framework show a substantial performance gain, validating the model’s effectiveness and overall performance.\nOverall, CycleQD introduces a scalable method to merge agent skills into LLMs, addressing critical challenges in agent-based LLM design.", "weaknesses": "Although the CycleQD  is designed for the agent skill acquirment of LLMs, the experiments predominantly focus on computer science tasks, such as OS and DB. Its applicability to other fields as agents will futher strength this paper.\n\n The methodology, though effective, involves a multi-step process (crossover, mutation, cyclic quality alternation) that may be complex. The author may need to provide a clearer illustration of the methodology, such as an overview figure of the method pipeline.\n\nThe ablation study does not provide substantial evidence of the proposed method's effectiveness, as the performance gains of each component are minimal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xf2JbeHeXq", "reviewer": "Reviewer_mjB3", "summary": "This paper introduces evolutionary approaches for merging large language models (trained on individual skills), as an alternative to fine-tuning (jointly on multiple skills). This is the first work to merge models using Quality Diversity (QD) algorithms. This work introduces a cyclic adaptation for QD: the optimization is conducted for one task at a time (in a cycle). Models are merged in the crossover operation by linearly combining task vectors. Finally, the mutation function modifies the \"child\" (from crossover) by sampling perturbations from the SVD components of its task vector.\n\nThis approach is tested using coding, OS, and DB skills. This evolutionary approach for model merging outperforms both fine-tuning based methods (by +5.4 pts) and other model merging methods (by +5.5 pts). The ablations show that each component of this method (cycle adaptation, SVD-based mutation, and elite sampling) lead to gains (on average). This method also generalizes to out-of-distribution coding tasks (more than fine-tuned, single-task experts) while also retaining performance on language skills. On the other hand, the merged model produced by this approach underperforms on individual tasks compared to fine-tuned experts for the domain of image segmentation.", "questions": "Formatting: I see that a couple half-column figures are used (concatenated horizontally with text). Please make sure that this abides by the formatting rules and correct if not.\n\nIs it normal for model merging approaches to test using just 2-3 tasks?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This work appears to be very original: (1) it is different from conventional fine-tuning and even model merging methods, and (2) it alleviates certain design decisions, like data mixing ratios and different objectives, when fine-tuning on multiple tasks.\n\nI understand that previous evolutionary approaches don't update the weights directly and this method proposes to do so (using the framing of model merging). I think this is a significant contribution. I think the specific adaptations they propose are also positive contributions.\n\nFrom what I can see, this method establishes sufficient gains over the alternative methods in the computer science tasks. Since this paper introduces several components, I was looking for (and was satisfied to find) the ablations in Table 3.", "weaknesses": "I think the writing is mostly clear, but this paper assumes some extra background knowledge about evolutionary methods. I would really appreciate it if the authors (briefly) introduce such approaches and terminology from first principles, before the details of their augmentations.\n\nIt would also be great if the authors could again motivate the need for evolutionary approaches (rather than fine-tuning) in training or merging models. Is it just the avoidance of design decisions that I mentioned above? Are there more fundamental motivations (like the diversity of skills captured by this method, which was briefly mentioned)? Or is this direction of work primarily exploratory?\n\nI am not already familiar with evolutionary approaches, so no significant methodological weaknesses are obvious to me. I will wait to see what other reviewers have to say, but generally hold a positive opinion for now.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rbcMuDv4FR", "reviewer": "Reviewer_8w66", "summary": "The paper proposed CycleQD to achieve superior performance on multiple agent skills while retaining robust language capabilities.", "questions": "1. Questions on presentation issues: see above. \n2. Is it possible to combine finetuning and CycleQD, e.g., SFT after CycleQD? What would be the result.\n3. Computation and time cost comparisons between CycleQD and gradient-based fine-tuning.\n4. The number of generations is critical for selecting elites. How sensitive is CycleQD regarding this hyperparameter?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "- The paper discussed a critical problem: achieving superior performance on multiple skills while retaining language capabilities. \n- The proposed method is novel.", "weaknesses": "Overall, the major weakness of this paper is its presentation. For the audience not familiar with evolutionary computation, the paper is hard to follow.\n1. [Presentation] Problems in understanding the proposed method.\n\t- Explain the evolution of figure 4 by referring to algorithm 1. The figures are meant to give direct intuitions about how the algorithm works, but the current delivery fails to achieve this purpose. \n\t- In the elite sampling (line 217), what does the formula intuitively means? Specifically, what is the meaning of $ \\gamma$ . \n\t- In the crossover (Sec. 3.2), there are no explanations for why $w$ are sampled from gaussian? From my understanding, $w$ are used to determine component weights for combining task vectors. The component weights should be greater than $0$ . But Gaussian samples has infinite range. \n\t- In the mutation (Sec. 3.3), there are no explanations for why SVD can address the challenge of excessive exploration in parameter space or overfitting. In addition, clarify how the proposed SVD-based mutation allows finer control of $\\theta_{child}$ (line 283).\n\t- In the model aggregation (Sec. 3.4), there are no explanations for why the proposed convex combination is adopted for obtain the final model. In particular, what is the connection between this design choice and the major goal of this paper, i.e., achieving superior performance on multiple tasks without degradation on other tasks.\n2. [Contribution] CycleQD on SAM is irrelevant with the main topic of this paper, which focuses on language models.\n3. [Soundness] The current experiments include only three tasks. Including more tasks to support the generality of CycleQD is better.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Epe2w5dpvT", "reviewer": "Reviewer_kupr", "summary": "This paper introduces CycleQD, a framework for enhancing large language models' skill acquisition through cyclical quality diversity optimization. CycleQD first trains expert models for individual skills, then combines them using model merging as crossover operations and SVD-based mutations, while cycling through different task metrics as quality measures. The method alternates between optimizing different skills by maintaining separate archives, each tracking models with diverse behavioral characteristics.", "questions": "1. Why is cyclic optimization more effective than simultaneous optimization of multiple objectives?\n2. What is the theoretical basis for using SVD decomposition in mutation, how does it help explore the parameter space more effectively?\n3. Can this method scale to combining skills from vastly different domains (e.g., math, creative writing, and visual reasoning)? Would it still outperform traditional SFT?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Demonstrates strong empirical results by achieving comparable performance to GPT-3.5-TURBO on coding, OS and DB tasks using only an 8B parameter model\n2. Avoids the complex tuning of data ratios and objective functions by decomposing multi-skill training into single-skill training followed by model merging\n3. Provides full ablation studies validating the necessity of key components including cyclic adaptation, SVD-based mutation, and elite sampling", "weaknesses": "1. Elite sampling strategy lacks theoretical foundation and uses arbitrary hyperparameters (0.5-0.8 range) without clear justification.\n2. Computational costs and efficiency not analyzed or compared with baseline methods.\n3. Missing comparison with established multi-objective optimization approaches like NSGA-II or Pareto-based methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)