Agent Skill Acquisition for Large Language Models via CycleQD

## Abstract

Abstract Training large language models to acquire specific skills remains a challenging endeavor.
Conventional training approaches often struggle with data distribution imbalances and inadequacies in objective functions that do not align well with task-specific performance.
To address these challenges, we introduce CycleQD, a novel approach that leverages the Quality Diversity framework through a cyclic adaptation of the algorithm, along with a model merging based crossover and an SVD-based mutation.
In CycleQD, each task’s performance metric is alternated as the quality measure while the others serve as the behavioral characteristics.
This cyclic focus on individual tasks allows for concentrated effort on one task at a time, eliminating the need for data ratio tuning and simplifying the design of the objective function.
Empirical results from AgentBench indicate that applying CycleQD to Llama3-8B-Instruct based models not only enables them to surpass traditional fine-tuning methods in coding, operating systems, and database tasks, but also achieves performance on par with gpt-3.5-turbo , which potentially contains much more parameters, across these domains.
Crucially, this enhanced performance is achieved while retaining robust language capabilities, as evidenced by its performance on widely adopted language benchmark tasks.
We highlight the key design choices in CycleQD, detailing how these contribute to its effectiveness.
Furthermore, our method is general and can be applied to image segmentation models, highlighting its applicability across different domains 1 1 1 https://github.com/SakanaAI/CycleQD .

## 1 Introduction

Figure: Figure 1: Method Overview. CycleQD uses QD in a cyclic manner to merge LLMs. Archives are initialized with expert LLMs, each of which has been fine-tuned to excel in a specific task. Model merging is conducted using QD, treating task performance as quality (Q) and behavior characteristics (BC), which are cyclically swapped in each generation.
Refer to caption: x1.png

Large Language Models (LLMs) have established themselves as powerful tools in the machine learning and artificial intelligence landscape, initially gaining prominence through their effectiveness in conversational tasks (Achiam et al., ).
As the demand grows for LLMs to perform a broader range of cognitive tasks, the ability to integrate linguistic understanding with actionable outputs becomes essential.
This integration facilitates the creation of intelligent LLM-based agents, making continual agentic fine-tuning a critical development in the field (Gur et al., ; Liu et al., ; Xi et al., ; ; Wang et al., ).
However, training LLMs to acquire various agent skills presents two major challenges that complicate their development.

A significant difficulty arises from the need to balance data ratios during training, to ensure that the model learns effectively from datasets representing different skills without favoring any particular one and without neglecting others.
For example, researchers have identified the significant influence of code training to a model’s inferential generation and thinking capabilities (Liu et al., ).
Comparing codellama and llama-2, the former shows a significant edge in tasks that follow a relatively static procedure after being tuned on code data, yet at the same time reveals decreased performance in tasks that require general thinking abilities.

In addition, conventional objective functions, such as next token prediction, often fail to capture the nuances required for performance across diverse tasks, typically leading to sub-optimal training outcomes.
This function, although effective for general language understanding, does not foster the development of specific agent skills.
An alternative, such as reinforcement learning (RL), has been shown to align LLMs with user intents more effectively: Ouyang et al. () demonstrated aligning LLMs with user intents by RL from human feedback, where a reward model is trained from a dataset of human-ranked model outputs.
This method has later on been adopted by many others and become an industrial standard.
Fortunately, unlike general language tasks, the “rewards” are clearly defined in most agent tasks.
On the other hand, managing the balance of rewards in learning tasks and the associated costs of fine-tuning pose additional problems.

In response to these challenges, we propose CycleQD, a novel framework that leverages the Quality Diversity (QD) method (Pugh et al., ), specifically through a cyclic adaptation of the MAP-Elites algorithm (Mouret & Clune, ).
CycleQD features a dynamic cycle where each agentic skill’s performance metric (e.g., pass@1 for coding tasks, success rate for Operation System (OS) and Database (DB) tasks) is optimized in isolation, with other skills represented as behavioral characteristics (BCs).
Unlike conventional crossover operations that randomly swap the parameters between two genes, CycleQD employs a model merging based crossover operation (Akiba et al., ), which is essential for transferring skills from specialized experts to a new, cohesive model.
Furthermore, we also introduce a singular value decomposition (SVD)-based mutation method to extrapolate model capabilities while preventing overfitting.
Because CycleQD uses task performance as its descriptors, it can be viewed as a simplified version of Multi-Objective QD (MOQD) (Pierrot et al., ) that optimizes in a coordinate ascent fashion, without the need for complex hyper-volume calculations when the BC dimension increases.
See Figure 1 for an overview of CycleQD.

We adopt design choices specifically targeting the two major challenges in continual agentic fine-tuning.
Our proposed method not only simplifies the complex data ratio management by focusing on one task at a time but also addresses the inadequacy of traditional loss functions by directly optimizing task-specific performance metrics.
Moreover, the QD framework proves ideal for model merging, particularly in LLMs where the computational pipelines are long.
By allowing locally sub-optimal solutions to persist, such as a temporarily under-performing layer, QD saves these configuration in its archive and provides them a chance to improve in the future.

The empirical results from our experiments illustrate that CycleQD significantly enhances the ability of a Llama3-8B-Instruct based LLM agent to develop and refine multiple computer science skills.
Notably, the averaged performance of the final 8-billion parameter open-weight model is on par with that of gpt-3.5-turbo on coding, OS and DB tasks, which potentially contains much more parameters.
This superior performance is achieved without the typical degradation associated with shifting focus among tasks or compromising general language capabilities.

Based on these results, it is important to note that while post-training of LLMs has been dominated by gradient-based optimizations, our method demonstrates that incorporating evolutionary algorithms can serve as a compelling modification to the conventional fine-tuning pipeline, effectively enhancing the training of LLM agents.
Our technical contributions are summarized as the following:

- •
We introduce CycleQD, a novel approach for merging LLM agents, each specializing in a different task, to create a composite LLM that outperforms baseline methods on computer science tasks from AgentBench (Liu et al., ) and popular coding benchmarks while maintaining language capabilities.
- •
We provide ablation studies and detailed explanations of the key design choices in CycleQD, illustrating their pivotal roles in enhancing the method’s effectiveness.
- •
We extend the application of CycleQD to the Segment Anything Model (SAM) (Kirillov et al., ), demonstrating our method’s broad applicability across domains.

## 2 Related works

LLM Skill Acquisition: Beyond conversational tasks, increasing attention has been given to enabling LLMs to acquire *skills* to take action.
This allows LLMs to function as agents capable of solving problems autonomously, and this approach is known as *LLM-as-Agent* (Xi et al., ; Wang et al., ).
Benchmarks for LLM-as-Agent have been developed, and research has demonstrated that LLMs can successfully perform tasks such as computer operations and web browsing to a certain extent (Gur et al., ; Liu et al., ; Xi et al., ; Xu et al., ; Tan et al., ; Ye et al., ).
*ReAct* (Yao et al., ) is the most frequently used approach for constructing LLM-as-Agent systems, which integrates CoT reasoning with action.
ReAct is typically employed through prompting and few-shot in-context learning, where models learn skills from a small number of examples.
However, attempts to equip LLMs with such skills through actual training with more examples are still quite limited, with a few studies focusing only on standard fine-tuning (Zeng et al., ; Qin et al., ).
This fine-tuning approach often struggles with balancing multiple agent skills and suffers from the gap between next token prediction loss and actual task performance.
Our proposed method uses QD techniques to optimize each skill independently and directly.

Quality Diversity:
QD is an emerging paradigm in evolutionary computation that aims to discover a diverse collection of high-performing solutions rather than a single optimal result (Pugh et al., ).
QD algorithms balance two key aspects: *quality*, which represents a solution’s performance, and *behavior characterization* (BC), which describes its observable properties.
These algorithms maintain an *archive* of diverse, high-quality solutions, continuously updated through an evolutionary process.
MAP-Elites (Mouret & Clune, ) is a prominent QD algorithm that maintains an archive defined by the BCs, discretizes it into tessellation, and stores the best-performing solutions in each cell to preserve diversity.
In each generation, it samples parents from the archive, creates offspring through crossover and mutation, and evaluates their quality and BCs.
Offspring replace existing elites if they perform better in their respective cells.
This process simultaneously explores diverse behaviors and optimizes performance within each niche, resulting in a map of high-quality, diverse solutions.
Recent improvements to MAP-Elites include a new selection method for better diversity (Wang et al., ) and an approach that removes the need for predefined niches (Wickman et al., ).
Additionally, MOQD (Pierrot et al., ) has been proposed to address problems with multiple, potentially conflicting objectives while maintaining diversity.
Instead of the fitness, it maximizes a Pareto front hyper-volume in each niche of the descriptor space.
CycleQD is a simplified form of MOQD in the sense that it uses task metrics as both BCs and the quality, the product of which approximates the hyper-volume, and the coordinate-ascent style of optimization allows it to avoid complex hyper-volume calculations for higher-dimensional archives.

Quality Diversity for LLMs:
Evolutionary computation, particularly QD, has occasionally been applied to neural networks and LLMs (Guo et al., ; Xue et al., ; Akiba et al., ).
However, there has been no prior research that evolves the main weights of LLMs using QD.
Samvelyan et al. () proposed Rainbow Teaming, which applies QD to evolve diverse adversarial prompts for evaluating LLM safety.
Bradley et al. () introduced QDAIF, which applies QD with LLM feedback to evolve populations of generated creative texts.
Our work represents the first attempt to merge LLMs through QD.
We believe this approach holds great promise, as it allows individual components of the model to be preserved even when they are temporarily sub-optimal, with the potential to contribute to future performance improvements.
Given the cost and complexity of optimizing the many layers of LLMs, QD offers a unique advantage by retaining useful configurations in archives that may later enhance overall model performance.

Model Merging: Model merging refers to combining multiple pre-trained models into a single model.
This technique is gaining attention because it can enhance model performance and integrate multiple capabilities at a much lower training cost.
Additionally, unlike ensemble methods, model merging does not increase inference costs either.
This technique is particularly effective when applied to different fine-tuned versions of the same base model.
The most basic method involves a linear combination of weights (Wortsman et al., ; Ilharco et al., ).
More advanced merging techniques have been proposed, incorporating methods such as election and sparsification (Yadav et al., ; Yu et al., ; Wang et al., ; Stoica et al., ).
Furthermore, the applicability and performance of these methods have been significantly improved through optimization (Akiba et al., ; Yang et al., ).
Inspired by these merging strategies, CycleQD combines QD with model merging, allowing for more effective synthesis of multiple capabilities in LLMs.

## 3 Methods

To avoid the laborious tuning of the data mixing ratio and the objective function, we introduce CycleQD.
It is built on MAP-Elites, a recognized implementation of QD, but distinguishes itself from traditional MAP-Elites systems in three key ways (see Algorithm 1 for the pseudo-code):

- 1.
Alternating Quality and BCs: Pugh et al. () emphasizes the importance of well-aligned BCs with the quality (i.e., task metric that needs to be optimized).
Traditional systems typically set the design at the onset and do not alter them throughout the QD process.
In contrast, CycleQD uses task metrics as both quality and BCs, ensuring alignment and dynamically alternating them during the QD process (lines 6-7 in Algorithm 1).
- 2.
Model Merging as Crossover: Unlike existing systems that optimize model parameters directly, CycleQD leverages a model merging algorithm as a crossover operation, replacing the heuristic, hand-designed rules often seen in practice (line 12 in Algorithm 1).
In CycleQD, we fine-tune expert agents, each of which only needs to specialize in one task, and use them as the seed models to initialize the model archives (lines 2-4 in Algorithm 1).
- 3.
SVD-based Mutation: Traditional mutation operations in genetic algorithms like MAP-Elites typically introduce random perturbations from a pre-defined distribution to explore new regions. In contrast, CycleQD utilizes perturbations aligned with the principal components of the models’ parameters matrices, facilitating exploration outside the convex regions formed by the parent models while avoiding overfitting (line 13 in Algorithm 1).

Figure: Algorithm 1 CycleQD

### 3.1 Alternating quality and behavior characteristics

CycleQD manages $K$ archives, each dedicated to tracking the LLMs that specialize in one of the $K$ agent skills.
Each archive evaluates LLM performance using $K$ tasks, which serve dual roles as both the quality and BCs.
An archive is structured as a lattice, the size of which is defined by $\prod_{k\neq i}{d_{k}}$, where $d_{k}$ is the number of bins defined by the $k$-th BC, and the $i$-th BC is excluded from this particular archive as it is treated as the quality metric.
In generation $t$, the $i$-th archive is selected for QD computation where $i=t\mod K$ (see Figure 1 for an illustration with $K=3$).
This process utilizes all available data from the $K$ tasks to evaluate and update the archives without the need for adjusting data ratios or objective functions.

CycleQD shares knowledge across $K$ archives for efficient optimization. Specifically, in generation $t$, a new model is created from the $i$-th archive by sampling, crossover, and mutation. This model is used to update not only the $i$-th archive but all $K$ archives (lines 14-15 in Algorithm 1). This allows for more effective utilization of the new model and helps facilitate optimization across $K$ tasks.

We employ task performance metrics as our BCs, imbuing them with concrete performance implications.
This allows for the design of an Elite sampling algorithm aimed to expand the Pareto frontier.
This Elite sampling algorithm is specifically designed for the CycleQD setup, where both quality and BCs reflect task performance. In this context, it is more effective to prioritize models that push the frontier of higher performance.
This frontier is defined by the models achieving high performance across these BCs.
Concretely, when the $i$-th archive is active, a model $j$ inside it is sampled with probability $P_{j}=\frac{\gamma_{j}}{\sum_{n=1}^{N}{\gamma_{n}}}$ where $N$ is the number of models in this archive and $\gamma_{j}$ is calculated as follows:

$$ $\gamma_{j}=\prod_{i=1}^{K}\big{(}\alpha_{\mathrm{low}}+\frac{f_{j,i}-\min(f_{1 :N,i})}{\max(f_{1:N,i})-\min(f_{1:N,i})}(\alpha_{\mathrm{high}}-\alpha_{ \mathrm{low}})\big{)}$ $$

Here, $f_{j,i}$ indicates the $j$-th model’s performance on the $i$-th task.
$(\alpha_{\mathrm{low}},\alpha_{\mathrm{high}})$ are hyper-parameters for the purpose of normalization.
Elite sampling strategically favors models that excel across the quality and various BCs, enhancing the efficiency and evolutionary potential of CycleQD.

### 3.2 Model merging based crossover

Training an LLM to specialize in a single task does not suffer from problems such as the data ratio tuning, the complex combinations of objective functions, and etc.
It is therefore straightforward to initially train a set of single-task experts using conventional fine-tuning methods, and then use model merging algorithms to develop more capable agents.
In CycleQD, our model merging crossover operator employs the parameter space merging scheme described in Akiba et al. (), which creates a new model by merging task vectors at the model level (Ilharco et al., ).
Specifically, for a pre-trained base LLM with parameters $\theta_{\mathrm{base}}\in\mathbb{R}^{d}$
and its fine-tuned LLM with parameters $\theta\in\mathbb{R}^{d}$,
we define a task vector as $\tau=\theta-\theta_{\mathrm{base}}$.
The crossover operator then generates a model’s parameters by combining these task vectors: $\theta_{\mathrm{child}}=\theta_{\mathrm{base}}+\big{(}\omega_{1}/(\omega_{1}+
\omega_{2})\big{)}\tau_{p_{1}}+\big{(}\omega_{2}/(\omega_{1}+\omega_{2})\big{)
}\tau_{p_{2}}$, where $\tau_{p_{1}}$ and $\tau_{p_{2}}$ are the parents’ task vectors.
Here, $\omega_{1}$ and $\omega_{2}$ are i.i.d samples from $\mathcal{N}(\mu,\sigma^{2})$, and $(\mu,\sigma)$ are predetermined hyper-parameters that remain fixed during the experiments.
We normalize $(\tau_{p_{1}},\tau_{p_{2}})$’s mixing coefficients to ensure the merged weights do not become outliers and cause problems for the downstream layers.

Additionally, there is potential to enhance the model merging process by training a neural network policy $(\mu,\sigma)\leftarrow\pi_{\phi}\big{(}\{\text{bc}_{1},...,\text{bc}_{k}\}_{p_
{1}},\{\text{bc}_{1},...,\text{bc}_{k}\}_{p_{2}}\big{)}$ that learns the optimal distribution parameters conditioned on the BC grid IDs $\{\text{bc}_{1},...,\text{bc}_{k}\}$ of the parents ($k-1$ in total for each parent because one serves as the quality).
However, while this approach could improve the efficiency of model merging, it also introduces a greater initial learning burden in CycleQD.

### 3.3 SVD-based mutation

The formulation of our model merging based crossover bears one obvious limitation:
The construction of $\theta_{\mathrm{child}}$ is a linear combination of $(\tau_{p_{1}},\tau_{p_{2}})$, which are themselves linear combination of their parents.
The entire process then reduces to finding the optimal linear combination of the expert models’ parameters.
Due to this, $\theta_{\mathrm{child}}$ will likely be trapped in the “convex region” in the performance space formed by the expert models, yet extrapolation that leads to improved performance is desired.
We introduce a mutation function $\theta_{\mathrm{child}}=h(\theta_{\mathrm{child}})$ after the crossover to get rid of this limitation.

In conventional methods, the mutation function $h(\theta_{\mathrm{child}})$ is often an operator that samples perturbations from a pre-defined distribution (e.g., Gaussian with pre-determined mean and covariance matrix) and add them to the gene being mutated.
We find this setting introduce excess freedom and lead to overfitting in the final model.
Instead, we propose to sample perturbations along the “directions” of the components from the model’s parameter matrices.
This mutation operation is mathematically defined as: $h(\theta_{\mathrm{child}})=\theta_{\mathrm{base}}+\text{concat}\big{(}[U_{l}(
\Sigma_{l}w)V_{l}^{\intercal}]_{l=1}^{L}\big{)}$, where $U_{l}\in\mathbb{R}^{m\times r}$, $\Sigma_{l}\in\mathbb{R}^{r\times r}$ and $V_{l}\in\mathbb{R}^{n\times r}$ are the SVD components of $\tau_{l}=U_{l}\Sigma_{l}V_{l}^{\intercal}$, the $l$-th parameter matrix in the task vector $\tau_{\mathrm{child}}=\theta_{\mathrm{child}}-\theta_{\mathrm{base}}$ (i.e., $\tau_{l}$ is a reshaped subarray of $\tau_{\mathrm{child}}$).
The perturbation vector $w\in\mathbb{R}^{r}$ is sampled from a uniform distribution of boundaries $[0,w_{\mathrm{max}}]$, where $w_{\mathrm{max}}\in\mathbb{R}^{+}$ is a hyper-parameter.
Our SVD-based mutation becomes a pass-through operation for parameter matrices whose rank is one (e.g., those belonging to layer-normalization layers).

Intuitively, if task vectors are representative of the agent skills that each fine-tuned LLM has mastered, their components are the building blocks and are therefore proxies in a finer granularity.
As a result, our SVD-based mutation allows finer control of $\theta_{\mathrm{child}}$ in the fundamental skill building blocks and reduces the risk of overfitting.
See Figure 2 for an illustration.

Figure: Figure 2: SVD-based mutations. Left: A task vector $\tau$ contains multiple parameter matrices $\tau_{l}$. E.g., the query projection matrix from the attention block in layer 1, the key projection matrix from the attention block in layer 2, etc. Those that have a rank $r>1$ can be decomposed using SVD into $r$ components. Right: SVD-based mutation samples a vector $w\in\mathbb{R}^{r}$ and scales each component by $w_{i}$, essentially adding perturbations that are aligned with the “directions” of the components.
Refer to caption: x2.png

### 3.4 Model aggregation

CycleQD maintains $K$ archives during the optimization process, LLMs in each of these archives are trained to acquire one of the $K$ agent skills.
In the end, CycleQD returns hundreds or even thousands of LLM agents with various specialties and characteristics.
These LLM-based agents are versatile and can facilitate multi-agent research in a diverse set of directions (we discuss this topic in Section 5).
On the other hand, an aggregated LLM agent that possesses the top skills acquired by the best models in each archive is useful on its own and is important for the evaluation of CycleQD.

Our model aggregation algorithm is straightforward, and can be summarized as: $\theta_{\mathrm{agg}}=\theta_{\mathrm{base}}+\sum_{k=1}^{K}{\beta_{k}\tau_{k}}$, where $\tau_{k}$ is the task vector from the elite model in the $k$-th archive, with ties resolved by selecting the most recent model.
$\beta_{k}=\exp(f_{k})/\sum_{i=1}^{K}{\exp(f_{i})}$ (i.e., softmax function) is the mixing coefficient calculated from the elite model’s task performance $f_{k}$ in the $k$-th archive.
In this paper, all experimental results from CycleQD are produced with this aggregated model $\theta_{\mathrm{agg}}$.

## 4 Experiments

We evaluate CycleQD extensively and conduct analysis to answer the following questions:

- •
Does CycleQD allow LLMs to effectively acquire various agent skills? If so, how good is it when compared with baseline methods?
- •
What are the critical design choices that lead to its effectiveness?
- •
What are the benefits and limitations of CycleQD?

### 4.1 Evaluation on computer science tasks

#### 4.1.1 Task description

In this experiment, our goal is to train LLMs to master three agent skills in computer science: coding, Operation System (OS) manipulation and Database (DB) query generation.
We adopt the MBPP+ (Mostly Basic Python Programming) dataset from EvalPlus (Liu et al., ) for coding skill development, and utilize the OS and DB datasets from AgentBench (Liu et al., ) for training.
For the coding task, we optimize and report the pass@1 metric, whereas in OS and DB tasks we use the success rate.
Please refer to Appendix A.1.1 for extra and detailed task related setups.

#### 4.1.2 CycleQD setups

Experts:
We employ Llama-3-8B-Instruct model (Dubey et al., ) as our base model, and use supervised fine-tuning to create LLM experts in coding, OS and DB.
For the OS and DB experts, we use the OS and DB training datasets from Agent-FLAN Chen et al. ().
The coding expert is fine-tuned on a combination of the Magicoder-Evol-Instruct-110K and Magicoder-OSS-Instruct-75K datasets (Wei et al., ).
See detailed configuration in Appendix A.1.2

Datasets:
We use the MBPP+ dataset, as well as the test and development datasets from OS and DB.
To ensure that the experts have the same task metrics across tasks, each dataset is split evenly into training and test splits.
For OS, problems that could not be solved by either the expert models and GPT models are excluded beforehand to reduce computation cost.

Hyper-parameters:
We use the performances of the three experts to determine the lower and upper bounds of the BC dimension.
Specifically, the lower bound is set at 85% of the performance achieved by the least proficient expert, while the upper bound is set at 115% of the performance achieved by the most proficient expert.
All BCs are then evenly divided into 15 bins between the lower and upper bounds.
We limit the number of models in each bin to one, and run CycleQD for 1200 generations.
Since the quality and BCs are alternated in each generation, this is equivalent to optimizing for the three skills for 400 generations each.
See more detail in Appendix A.1.3

#### 4.1.3 Baselines

Our baselines can be categorized into fine-tuning based and merging based models.

Fine-tuning Based Models:
These are models 3-6 in Table 1.
In addition to the expert LLMs mentioned earlier, we also combine all the data from the three tasks (including the extra Magicoder-Evol-Instruct-110K and Magicoder-OSS-Instruct-75K datasets) and continue training the Llama3-8B-Instruct base model with supervised fine-tuning to develop an extra baseline (model 6 in Table 1).
In this baseline, we don’t tune the data ratio and use cross-entropy (i.e., next token prediction) as our objective function.

Merging Based Models:
These are models 7-9 in Table 1.
We first introduce a naive merging method where the merged model is produced by taking an average of the three experts’ task vectors (model 7).
Similar to this baseline, we introduce two additional learning based model merging methods where, instead of taking an average, the coefficients of a linear combination of these task vectors is learned through gradient descent on policy gradients (model 8), and CMA-ES, an evolutionary algorithm on the raw rewards (model 9) separately.

#### 4.1.4 Results

Figure: Figure 3: CycleQD Archives from Computer Science Tasks. In each archive, the two axes show the BCs, and the color intensities indicate the LLM agent’s quality in that grid. These archives are obtained after 1200 generations of CycleQD. The red bounding boxes indicate the grids where expert models were present. See Appendix A.1.4 for archive development across generations.
Refer to caption: x3.png

**Table 1: Evaluation on computer science tasks. We report pass@1 for MBPP and success rates for the DB and OS tasks. The base model and the GPT models (in gray) are included for completeness. The MBPP results for the GPT models are extracted from Dubey et al. ().**
| # | Methods | MBPP | DB | OS | Avg |
| --- | --- | --- | --- | --- | --- |
| 0 | gpt-4 | 83.6 | 36.5 | 63.7 | 61.3 |
| 1 | gpt-3.5-turbo | 82.0 | 41.6 | 38.5 | 53.7 |
| 2 | Llama3-8B-Instruct (base model) | 67.3 | 5.3 | 25.2 | 32.6 |
| Fine-tuning Based Methods |  |  |  |  |  |
| 3 | Fine-tuning (Coding expert) | 70.4 | 21.2 | 20.7 | 37.4 |
| 4 | Fine-tuning (DB expert) | 65.8 | 42.4 | 28.5 | 45.6 |
| 5 | Fine-tuning (OS expert) | 66.3 | 0.0 | 30.4 | 32.2 |
| 6 | Fine-tuning (All) | 67.3 | 37.1 | 36.7 | 47.0 |
| Merging Based Methods |  |  |  |  |  |
| 7 | Merging (w/o learning) | 72.9 | 24.7 | 42.6 | 46.7 |
| 8 | Merging (learning w/ GD) | 69.3 | 41.2 | 29.6 | 46.7 |
| 9 | Merging (learning w/ CMA-ES) | 69.3 | 41.2 | 30.2 | 46.9 |
| 10 | CycleQD (Ours) | 76.4 | 38.2 | 42.6 | 52.4 |

Overall: We summarize our results from this experiment in Table 1.
In addition to the baselines, we also include the base model and the GPT models for references (highlighted with a lightgray background).
The first thing to notice is that each expert model (models 3-5), fine-tuned specifically for its assigned task, performs better than the Llama3-8B-Instruct base model (model 2), laying the stepping stones for CycleQD and other model merging based methods.
After evolutionary computing, our method, averaged across the three tasks, outperforms all baselines, and is approaching the performance of gpt-3.5-turbo.
Specifically, CycleQD achieves the highest scores on the coding and the OS tasks among the baselines, and has only a mild performance drop on the DB tasks compared with the expert.
Figure 3 gives the archives in CycleQD at the end of the experiment.
It is easy to see that CycleQD has managed to “illuminate” the three archives.

Comparison with Fine-tune Based Methods:
A notable result from the table is that, the model fine-tuned on all the datasets (model 6) is only partially and marginally better than the three experts, despite being trained on a larger dataset.
On the other hand, the expert models can be easily trained because they do not require the data ratio or object function adjustment.
The drawback of these experts is that they specialize in only one task, prohibiting their usage in wider scenarios.
This result proves the difficulty with conventional methods, underscoring the importance and technical contributions of CycleQD.

Comparison with Merging Based Methods:
CycleQD also outperforms conventional model merging based methods (models 7-9), and we conjecture that this is due to two reasons.
First, these methods lack a mutation operation, which we believe is crucial for enabling the merged model to escape the “convex region” created by the seed models.
For example, model 9, while having the best average performance in its category, scores lower on all three tasks compared to the experts.
In contrast, CycleQD significantly surpasses expert performance in coding and OS tasks.
Second, we suspect that the absence of a mechanism to rehabilitate temporarily underperforming models contributes to this discrepancy.
Given the lengthy computation pipelines in LLMs, which include numerous layers, even a single malfunctioning layer can degrade overall performance.
Unlike QD, the baselines do not have a way to recover these layers, potentially promising models are prematurely discarded.

#### 4.1.5 Ablation studies

To get insights into the design choices in CycleQD and show how they contribute its effectiveness, we conduct a series of ablation studies and summarize the results in Table 2.

CycleQD vs QD: We compare conventional QD with CycleQD in trials 0 and 1.
Specifically, we run QD for three runs.
Within each run, one of the task’s metric is treated as the quality while the others are regarded as the BCs.
We control each trial to have an equal computing budget as for CycleQD, and all other hyper-parameters are identical to CycleQD.
We use the same model aggregation method to merge the best models from these three runs (see Section 3.4).
The better performance from CycleQD suggests the importance of alternating the quality and BCs.

Mutation Methods: We focus on the impact of different mutation operations with trials 1-3.
Although we mentioned earlier that mutation helps the merged model extrapolate in the performance space, naively designed mutations give excess freedom and can lead to overfitting.
This is what happens to trial 2, the performance of which is even worse than trial 1 that does not have any mutation operations.
On the other hand, our SVD-based mutation (trial 3) successfully avoids the problem and delivers a much better performance.

Sampling Methods: Finally, we demonstrate the importance of Elite sampling in CycleQD in trial 4, which has the identical settings as model 10 in Table 1, and gives the best score in the ablation studies.
Elite sampling is effective because it prioritizes merging high-performing models, the result of which helps expand the Pareto frontier in each archive more effectively.

**Table 2: Ablation studies. We add and highlight a different treatment from the previous trials.**
| # | Trials | MBPP | DB | OS | Avg |
| --- | --- | --- | --- | --- | --- |
| 0 | QD + No mutation + Random sampling | 70.4 | 28.8 | 43.7 | 47.6 |
| 1 | CycleQD + No mutation + Random sampling | 72.9 | 33.5 | 41.9 | 49.4 |
| 2 | CycleQD + Gaussian mutation + Random sampling | 73.4 | 30.0 | 42.2 | 48.5 |
| 3 | CycleQD + SVD mutation + Random sampling | 75.9 | 38.2 | 41.1 | 51.7 |
| 4 | CycleQD + SVD mutation + Elite sampling | 76.4 | 38.2 | 42.6 | 52.4 |

#### 4.1.6 Skill generalization and language capabilities

In addition to agent skills, we evaluate our model on out-of-distribution coding tasks and language understanding across six categories, please see Appendix A.1.5 for detailed task descriptions.
Table 3 shows the results across these tasks, where we normalize the scores against the Llama3-8B-Instruct base model.
On average, CycleQD outperforms the three fine-tuned experts, providing better overall generalization across tasks.
Our method generalizes well on the coding tasks while retraining performance on language tasks.
In contrast, the Coding expert, while delivering competitive scores on the coding tasks, shows significantly lower performance on specific tasks in the Reasoning category.
This sharp decline in performance suggests the occurrence of catastrophic forgetting during fine-tuning, where the model loses its ability to generalize across tasks outside its specialization.
These findings once again underscore the efficacy of CycleQD in achieving superior and more consistent performance across diverse tasks compared to traditional fine-tuning approaches.

**Table 3: Generalization performance. The scores are normalized against the base model. CycleQD generalizes well to other coding tasks unseen in the training while retaining its language capabilities.**
| Model | Coding Tasks | Language Tasks | Avg |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| HUMANEVAL+ | BigCodeBench | Reasoning | GSM8K | RC | CommonSense |  |  |
| Coding expert | 1.18 | 0.97 | 0.57 | 0.82 | 0.94 | 1.03 | 0.92 |
| DB expert | 0.80 | 0.84 | 0.84 | 0.87 | 0.98 | 0.98 | 0.89 |
| OS expert | 0.94 | 0.90 | 0.98 | 0.93 | 0.99 | 0.99 | 0.95 |
| CycleQD | 1.10 | 1.03 | 0.95 | 0.88 | 0.98 | 1.02 | 0.99 |

### 4.2 Evaluation on image segmentation Tasks

#### 4.2.1 Task description

In addition to its applications for LLMs, CycleQD serves as a versatile method for integrating expert models across various data modalities beyond text.
In this experiment, we extend CycleQD to the fusion of multiple Segment Anything Models (SAM), which are state-of-the-art computer vision models designed for image segmentation tasks.
Specifically, our objective is to merge pairs of SAM models, A and B, to create models whose capabilities encompass the skill sets of both A and B.

#### 4.2.2 CycleQD setups

Experts:
We select four tasks within the image segmentation domain, each supported by extensive datasets, for training specialized models: Camouflaged Object Segmentation (CAM), Polyp Segmentation (POL), Skin Lesion Segmentation (SKL), and Leaf Segmentation (LEA).
CAM detects objects in cluttered environments, making segmentation more challenging than standard tasks.
POL identifies polyps in endoscopic images, essential for early colorectal cancer detection.
SKL detects various skin lesions in medical images.
LEA identifies plant leaves in agricultural images, supporting disease control and improving crop quality.
We employ SAM-ViT Huge model as our base model, which we fine-tune to develop these experts.
See Appendix A.2.1 for details on the datasets used for fine-tuning and CycleQD setups.

Hyper-parameters:
CycleQD’s hyper-parameters remain the same as in Section 4.1.2.

#### 4.2.3 Results

Overall:
Table 4 shows the performance of the models merged by CycleQD, where scores are normalized against the expert models. You can see visualization results in Appendix A.2.2.
In general, CycleQD is able to merge the experts successfully, with top models retaining more than 90% of the experts’ performance (e.g., models 0, 1, and 3).
On the other hand, models 2, 4 and 5 are less successful.
This leads us to conduct further analysis and we report the findings next.

Analysis:
We report the similarities between experts A and B in the last column in Table 4.
A strong correlation of 0.83 between the averaged scores and these similarities indicate both the limitations and potential areas for enhancement in CycleQD.
We define the similarity between models A and B as the average cosine similarity of the singular value vectors (derived from the task vectors) across all layers in both models:
$s=(1/L)\sum_{i=1}^{L}\cos\big{(}\text{diag}(\Sigma_{i,A}),\text{diag}(\Sigma_{
i,B})\big{)}$.
Here, $L$ is the number of weight matrices with a rank greater than 1, and $\Sigma_{i,*}$ denotes a diagonal matrix of singular values from the $i$-th weight matrix in the task vector.
Although this summarizing metric does not fully encapsulate the models’ characteristics, CycleQD tends to perform well when the models exhibit high similarity.
This observation underpins our approach of incorporating this similarity metric as a regularization technique during model fine-tuning.
Notably, a similar metric has been employed as the “proximal term” in optimizing heterogeneous networks within the realm of federated learning, as evidenced by research documented in  (Li et al., ).
This precedent lends credence to our strategy, suggesting it is grounded in established methodologies.

**Table 4: Merging image segmentation Models. The scores are normalized against the performance of the corresponding expert models. Model similarity shows how close the pair of expert models are (the larger the similarity values, the closer the models are).**
| # | Expert A | Expert B | Score A | Score B | Avg Score | Model Similarity |
| --- | --- | --- | --- | --- | --- | --- |
| 0 | CAM | POL | 0.95 | 0.99 | 0.97 | 0.98 |
| 1 | CAM | SKL | 0.85 | 0.99 | 0.92 | 0.97 |
| 2 | CAM | LEA | 0.51 | 0.89 | 0.70 | 0.88 |
| 3 | POL | SKL | 0.98 | 0.95 | 0.96 | 0.99 |
| 4 | POL | LEA | 0.40 | 0.84 | 0.62 | 0.93 |
| 5 | SKL | LEA | 0.83 | 0.84 | 0.83 | 0.95 |

## 5 Conclusions and future works

In this work, we introduced CycleQD, a compelling modification of the conventional LLM fine-tuning pipeline, integrating evolutionary algorithms for agent skill acquisition.
CycleQD begins with single-task experts and utilizes QD to continuously generate hundreds of LLM-based agents, possessing diverse characteristics and exhibiting a broader range of skills than the initial experts.
Through a dynamic process that cyclically swaps quality and BCs, coupled with a model merging crossover operation and an SVD-based mutation operator, CycleQD has successfully enabled LLMs to master computer science skills.
Our method not only outperforms baseline approaches but also achieves on-par performance with gpt-3.5-turbo, while generalizing to out-of-distribution tasks and retaining the language capabilities.
Furthermore, CycleQD’s applicability extends across domains to image segmentation models, demonstrating its broad utility.

In terms of limitations, we acknowledge that the success of model merging hinges on the compatibility of the source models.
CycleQD may encounter challenges when the expert models originate from highly divergent settings.
One way to address this is incorporating model similarity as a regularization term during the expert training.
Furthermore, this research represents an initial foray into the integration of QD and evolutionary model merging for agent skill acquisition.
There is substantial potential for improvement by leveraging advanced methodologies from both fields, such as integrating strategies from CMA-ME (Fontaine et al., ) and PGA-MAP-Elites (Nilsson & Cully, ) to enhance the efficiency of the learning process.
Looking ahead, a promising direction for future research lies in multi-agent systems.
Considering CycleQD generates an archive of diverse agents, orchestrating these agents to collaborate and compete opens up exciting possibilities for scientific exploration and practical applications.

## Author contributions

Yujin Tang and Takuya Akiba initiated the project.
So Kuroki co-designed the algorithm, implemented CycleQD, conducted the experiments and the studies.
Taishi Nakamura developed the agentic tasks, the expert models and the baseline methods.
Takuya Akiba implemented the parallel distributed optimization and proposed the SAM experiment.
Yujin Tang proposed the algorithm, designed and helped with the experiments, and managed the project.
All authors contributed to the writing and approved the final version.

## Acknowledgements

The authors would like to thank Robert Lange for providing valuable discussions and feedback while drafting the text, and Kou Misaki for providing infrastructure support.
This paper is based on results obtained from a project, JPNP20017, subsidized by the New Energy and Industrial Technology Development Organization (NEDO).

## Appendix A Appendix

### A.1 Computer science tasks

#### A.1.1 Extra task setups

The MBPP+ dataset, based on the original MBPP, uses a subset of 399 hand-verified problems from MBPP-sanitized to ensure well-formed programming tasks. Each problem includes a problem statement, function signature, and test cases. EvalPlus extends this dataset with additional test cases to provide a more rigorous evaluation of code generation capabilities. For our evaluation, we use the pass@1 metric on the Base Tests of MBPP+, which reflects the model’s ability to generate correct code on the first attempt using the original test cases.
The OS dataset evaluates LLMs in genuine interactive bash environments (Ubuntu Docker) on human questions with deterministic answers and practical operational tasks.
The DB dataset assesses LLMs’ abilities to operate on real databases via SQL, encompassing the complete pipeline of database analysis with authentic SQL interfaces and multiple tables.
Both OS and DB datasets use success rate as the primary evaluation metric. The OS tasks are designed to be solved within a maximum of 5 interaction turns and employ a 1-shot setup. The DB tasks have a similar interaction limit but are evaluated in a 0-shot manner.

#### A.1.2 Training configuration for experts

We adopt the AdamW optimizer (Loshchilov & Hutter, ) with $\beta_{1}=0.9$ and $\beta_{2}=0.95$.
A global batch size of 64 is used across all fine-tuning processes. We employ cosine learning rate scheduling with a range of $[4\times 10^{-6},2\times 10^{-5}]$, starting with a linear warmup for the first 10% of the total training steps.
The OS and DB experts are trained for 1 epoch, while the code model is trained for 3 epochs due to its larger training data size.

#### A.1.3 CycleQD hyper-parameters

We set $\alpha_{\mathrm{low}}=0.5$ and $\alpha_{\mathrm{high}}=0.8$ in Elite sampling, $\mu=1.0$ and $\sigma=0.03$ in model merging-based crossover, and $w_{\text{max}}=0.3$ in our SVD-based mutations.

#### A.1.4 CycleQD development of archives across generations.

Figure 4 shows the development of archives across generations. The archives are shown in increments of 300 generations from top to bottom. The corresponding generation for each archive is displayed on the left side of the figure. The red bounding boxes indicate the grids where expert policies were present in each archive. It can be observed that the frontier of the archives expands with each passing generation.

Figure: Figure 4: CycleQD Development of Archives Across Generations. In each archive, the two axes represent the BCs, and the color intensities in each grid indicate the quality of the LLM agent in that grid. The archives are shown in increments of 300 generations from top to bottom. The corresponding generation for each archive is displayed on the left side of the figure. The red bounding boxes indicate the grids where expert policies were present in each archive. The experiment shown in this figure is the same as in Figure 3, and the archive for 1200 generations is identical to that in Figure 3.
Refer to caption: x4.png

#### A.1.5 Benchmark datasets for skill generalization and language understanding

In the Coding category, we employ two key benchmarks: HUMANEVAL+  (Liu et al., ), evaluated in a 0-shot setting using the pass@1 metric on the Base Tests, and BigCodeBench (Zhuo et al., ), which is designed to assess code generation with diverse function calls and complex instructions.
We report the average Pass@1 scores for both BigCodeBench-Complete and BigCodeBench-Instruct variants in the full setting, using 0-shot evaluation.
For the General Knowledge and Reasoning (Reasoning) category, we utilize two comprehensive benchmarks: MMLU (Hendrycks et al., ) with a 5-shot evaluation, and BBH (Big Bench Hard) (Suzgun et al., ) using a 3-shot chain-of-thought (Wei et al., ) prompting approach.
These benchmarks provide a holistic assessment of our model’s ability to handle a wide range of knowledge-based and reasoning tasks.
To evaluate Mathematical Reasoning, we employ the GSM8K benchmark (Cobbe et al., ) with a 4-shot evaluation, challenging our model’s ability to solve complex mathematical problems.
For Reading Comprehension (RC), we use two established benchmarks: SQuAD2 (Rajpurkar et al., ) and TriviaQA (Joshi et al., ), both evaluated using a 4-shot approach.
These datasets assess the model’s capacity to understand and reason over complex textual information.
In the Commonsense Reasoning (CommonSense) category, we employ three diverse benchmarks: HellaSwag (Zellers et al., ) for commonsense inference, OpenBookQA (Mihaylov et al., ) for elementary-level science question answering that requires both core scientific knowledge and broader common knowledge, and XWinograd (English version) (Tikhonov & Ryabinin, ) for cross-lingual commonsense reasoning and coreference resolution.
All these evaluations are conducted with 4-shot prompts.

### A.2 Image segmentation tasks

#### A.2.1 Datasets for fine-tuning experts and CycleQD training

We followed Zhong et al. () to prepare the datasets.
For Camouflaged Object Segmentation, we use three datasets: COD10K (Fan et al., ), CHAMELEON (Skurowski et al., ), and CAMO (Le et al., ). Following Fan et al. () we train on a combined dataset consisting of the 4040 training images from COD10K and CAMO for 20 epochs, randomly splitting 10% of the images from the training set for validation. The model is then tested on the 250 COMO test images.
For Polyp Segmentation, we use two datasets: Kvasir (Jha et al., ) and CVC-ClinicDB/CVC-612 (Bernal et al., ). Following Fan et al. (), we divide the images into a 9:1 ratio for training and testing, resulting in 1450 training images. We then randomly split 20 % of the training set for validation. The model is trained for 30 epochs and tested on the 101 Kvasir test images.
For Skin Lesion Segmentation, we use the ISIC 2017 dataset (Codella et al., ). We train the model on the 2000 training and 150 validation images for 30 epochs and evaluate it on the 600 test images.
For Leaf Segmentation, we use the Leaf Disease Segmentation Dataset (Rath, ). We train the model on the 498 training image, using 80% for training and 20% for validation for 30 epochs, and evaluate it on 90 test images.

#### A.2.2 Visualization result

The visualization results are shown in Figure 5 for SKL and POL, and Figure 6 for SKL and LEA. In both figures, ”Ours” refers to a single merged model via CycleQD, identical to model #3 (for SKL and POL) and model #5 (for SKL and LEA) in Table 4. In Figure 5, our model retains near expert-level performance for both tasks after merging. However, in Figure 6, while our model generally performs at an expert level for both tasks, a slight drop is observed. Notably, in the bottom images of SKL, the model seems to detect the edges of the skin lesions instead of the lesions themselves, as if confusing the task with leaf disease detection. The expert model similarities are high—0.99 for SKL and POL, and 0.95 for SKL and LEA. This corresponds with the observation that the merged model from the highly similar SKL and POL performs better than the one from SKL and LEA.

Figure: Figure 5: CycleQD of SAM for SKL and POL. Ours is a single merged model via CycleQD, and it is the same model as the one in Table 4 (# 3). The similarity between the two expert models is high (0.99), and the model maintains expert-level performance across both tasks.
Refer to caption: x5.png

Figure: Figure 6: CycleQD of SAM for SKL and LEA. Ours is a single merged model via CycleQD, and it is the same model as the one in Table 4 (# 5). The similarity between the two expert models is 0.95, which is lower than #3 (0.99) from Table 4, resulting in a slight decrease in performance, as observed when comparing the bottom images in SKL.
Refer to caption: x6.png

## Peer Reviews

confidence: 1
contribution: 3
initial_rating: 6
presentation: 2
questions: I am also curious about any real-world application tests using the proposed method across different scenarios, given that the scenarios in the experiments are quite similar.
soundness: 3
strengths: The paper presents an innovative approach to LLM skill acquisition, uniquely applying the Quality Diversity paradigm to cycle through task-specific optimizations, which is quite novel. 
The experimental results on the proposed CycleQD framework show a substantial performance gain, validating the model’s effectiveness and overall performance.
Overall, CycleQD introduces a scalable method to merge agent skills into LLMs, addressing critical challenges in agent-based LLM design.
summary: This paper presents CycleQD, a novel framework for making llm acquire specific skills through Quality Diversity (QD) optimization. CycleQD addresses challenges in skill training, such as data imbalance and inadequate objective functions, by cyclically alternating quality measures for different tasks.  CycleQD shows competitive results, achieving performance on par with GPT-3.5-TURBO.
weaknesses: Although the CycleQD  is designed for the agent skill acquirment of LLMs, the experiments predominantly focus on computer science tasks, such as OS and DB. Its applicability to other fields as agents will futher strength this paper.

 The methodology, though effective, involves a multi-step process (crossover, mutation, cyclic quality alternation) that may be complex. The author may need to provide a clearer illustration of the methodology, such as an overview figure of the method pipeline.

The ablation study does not provide substantial evidence of the proposed method's effectiveness, as the performance gains of each component are minimal.

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: Formatting: I see that a couple half-column figures are used (concatenated horizontally with text). Please make sure that this abides by the formatting rules and correct if not.

Is it normal for model merging approaches to test using just 2-3 tasks?
soundness: 3
strengths: This work appears to be very original: (1) it is different from conventional fine-tuning and even model merging methods, and (2) it alleviates certain design decisions, like data mixing ratios and different objectives, when fine-tuning on multiple tasks.

I understand that previous evolutionary approaches don't update the weights directly and this method proposes to do so (using the framing of model merging). I think this is a significant contribution. I think the specific adaptations they propose are also positive contributions.

From what I can see, this method establishes sufficient gains over the alternative methods in the computer science tasks. Since this paper introduces several components, I was looking for (and was satisfied to find) the ablations in Table 3.
summary: This paper introduces evolutionary approaches for merging large language models (trained on individual skills), as an alternative to fine-tuning (jointly on multiple skills). This is the first work to merge models using Quality Diversity (QD) algorithms. This work introduces a cyclic adaptation for QD: the optimization is conducted for one task at a time (in a cycle). Models are merged in the crossover operation by linearly combining task vectors. Finally, the mutation function modifies the "child" (from crossover) by sampling perturbations from the SVD components of its task vector.

This approach is tested using coding, OS, and DB skills. This evolutionary approach for model merging outperforms both fine-tuning based methods (by +5.4 pts) and other model merging methods (by +5.5 pts). The ablations show that each component of this method (cycle adaptation, SVD-based mutation, and elite sampling) lead to gains (on average). This method also generalizes to out-of-distribution coding tasks (more than fine-tuned, single-task experts) while also retaining performance on language skills. On the other hand, the merged model produced by this approach underperforms on individual tasks compared to fine-tuned experts for the domain of image segmentation.
weaknesses: I think the writing is mostly clear, but this paper assumes some extra background knowledge about evolutionary methods. I would really appreciate it if the authors (briefly) introduce such approaches and terminology from first principles, before the details of their augmentations.

It would also be great if the authors could again motivate the need for evolutionary approaches (rather than fine-tuning) in training or merging models. Is it just the avoidance of design decisions that I mentioned above? Are there more fundamental motivations (like the diversity of skills captured by this method, which was briefly mentioned)? Or is this direction of work primarily exploratory?

I am not already familiar with evolutionary approaches, so no significant methodological weaknesses are obvious to me. I will wait to see what other reviewers have to say, but generally hold a positive opinion for now.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 1
questions: 1. Questions on presentation issues: see above. 
2. Is it possible to combine finetuning and CycleQD, e.g., SFT after CycleQD? What would be the result.
3. Computation and time cost comparisons between CycleQD and gradient-based fine-tuning.
4. The number of generations is critical for selecting elites. How sensitive is CycleQD regarding this hyperparameter?
soundness: 2
strengths: - The paper discussed a critical problem: achieving superior performance on multiple skills while retaining language capabilities. 
- The proposed method is novel.
summary: The paper proposed CycleQD to achieve superior performance on multiple agent skills while retaining robust language capabilities.
weaknesses: Overall, the major weakness of this paper is its presentation. For the audience not familiar with evolutionary computation, the paper is hard to follow.
1. [Presentation] Problems in understanding the proposed method.
	- Explain the evolution of figure 4 by referring to algorithm 1. The figures are meant to give direct intuitions about how the algorithm works, but the current delivery fails to achieve this purpose. 
	- In the elite sampling (line 217), what does the formula intuitively means? Specifically, what is the meaning of $ \gamma$ . 
	- In the crossover (Sec. 3.2), there are no explanations for why $w$ are sampled from gaussian? From my understanding, $w$ are used to determine component weights for combining task vectors. The component weights should be greater than $0$ . But Gaussian samples has infinite range. 
	- In the mutation (Sec. 3.3), there are no explanations for why SVD can address the challenge of excessive exploration in parameter space or overfitting. In addition, clarify how the proposed SVD-based mutation allows finer control of $\theta_{child}$ (line 283).
	- In the model aggregation (Sec. 3.4), there are no explanations for why the proposed convex combination is adopted for obtain the final model. In particular, what is the connection between this design choice and the major goal of this paper, i.e., achieving superior performance on multiple tasks without degradation on other tasks.
2. [Contribution] CycleQD on SAM is irrelevant with the main topic of this paper, which focuses on language models.
3. [Soundness] The current experiments include only three tasks. Including more tasks to support the generality of CycleQD is better.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Why is cyclic optimization more effective than simultaneous optimization of multiple objectives?
2. What is the theoretical basis for using SVD decomposition in mutation, how does it help explore the parameter space more effectively?
3. Can this method scale to combining skills from vastly different domains (e.g., math, creative writing, and visual reasoning)? Would it still outperform traditional SFT?
soundness: 3
strengths: 1. Demonstrates strong empirical results by achieving comparable performance to GPT-3.5-TURBO on coding, OS and DB tasks using only an 8B parameter model
2. Avoids the complex tuning of data ratios and objective functions by decomposing multi-skill training into single-skill training followed by model merging
3. Provides full ablation studies validating the necessity of key components including cyclic adaptation, SVD-based mutation, and elite sampling
summary: This paper introduces CycleQD, a framework for enhancing large language models' skill acquisition through cyclical quality diversity optimization. CycleQD first trains expert models for individual skills, then combines them using model merging as crossover operations and SVD-based mutations, while cycling through different task metrics as quality measures. The method alternates between optimizing different skills by maintaining separate archives, each tracking models with diverse behavioral characteristics.
weaknesses: 1. Elite sampling strategy lacks theoretical foundation and uses arbitrary hyperparameters (0.5-0.8 range) without clear justification.
2. Computational costs and efficiency not analyzed or compared with baseline methods.
3. Missing comparison with established multi-objective optimization approaches like NSGA-II or Pareto-based methods.

Meta-review: This paper introduces CycleQD, a framework for skill acquisition in large language models (LLMs) using Quality Diversity (QD) optimization. By cyclically alternating quality measures for different tasks, CycleQD offers a scalable alternative to fine-tuning and traditional model merging methods, addressing challenges such as data imbalance and inadequate objective functions. The proposed method demonstrates strong performance, achieving gains over fine-tuning and other model merging techniques, particularly on coding, OS, and database tasks, while also generalizing to out-of-distribution tasks. Reviewers commend the originality of applying QD algorithms to LLMs and the clear performance improvement. However, the paper primarily focuses on computer science tasks, limiting its demonstrated applicability to other domains. Additionally, the methodology could benefit from more precise illustrations and more substantial ablation evidence. While the paper is well-received for its novelty and contributions, reviewers suggest further exploration of real-world applications, broader task coverage, and more accessible explanations of evolutionary methods.

Decision: Accept (Poster)