Model Fusion through Bayesian Optimization in Language Model Fine-Tuning
Authors: Chaeyun Jang
Hyungi Lee
Jungtaek Kim
Juho Lee
Venue: NeurIPS 2024 spotlight (2024)
Keywords: fine-tuning
language model
bayesian optimization
Primary area: optimization_for_deep_networks
TL;DR: This paper introduces a novel Bayesian Optimization based model fusion method, referred to as BOMF, designed specifically for the fine-tuning scenario involving Pre-trained Language Models.

Abstract: Fine-tuning pre-trained models for downstream tasks is a widely adopted technique known for its adaptability and reliability across various domains. Despite its conceptual simplicity, fine-tuning entails several troublesome engineering choices, such as selecting hyperparameters and determining checkpoints from an optimization trajectory. To tackle the difficulty of choosing the best model, one effective solution is model fusion, which combines multiple models in a parameter space. However, we observe a large discrepancy between loss and metric landscapes during the fine-tuning of pre-trained language models. Building on this observation, we introduce a novel model fusion technique that optimizes both the desired metric and loss through multi-objective Bayesian optimization. In addition, to effectively select hyperparameters, we establish a two-stage procedure by integrating Bayesian optimization processes into our framework. Experiments across various downstream tasks show considerable performance improvements using our Bayesian optimization-guided method.

Model Fusion through Bayesian Optimization in Language Model Fine-Tuning
Chaeyun Jang' KAIST j jcy99110kaist. ac.kr
Hyungi Lee" KAIST
1ha27080kaist.ac..u ac.
Jungtaek Kim University of Pittsburgh jungtaek. kim@pitt. edu
Juho Lee' KAIST
juholee@kaist. ac.
Abstract
Fine-tuning pre-trained models for downstream tasks is widely adopted tech- nique known for its adaptability and reliability across various domains. Despite its conceptual simplicity, fine-tuning entails several troublesome engineering choices, such selecting hyperparameters and determining checkpoints from opti- mization trajectory. To tackle the difficulty of choosing the best model, one ef- fective solution is model fusion, which combines multiple models in parameter space. However, we observe a large discrepancy between loss and metric land- scapes during the fine-tuning of pre-trat rre-trained language models. Building this observation, we introduce a novel model fusion technique that optimizes both the desired metric and loss through multi-objectiiv Bayesian oplimization. In addition, to effectively select hyperparameters, we establish two-stage proce- dure by integrating Bayesian optimization processes into our framework. Experi- ments across various downstream tasks show considerable performance improve- ments using our Bayesian optimization-guidee guided method. Code will be available https: git..u...... com/ haa/cchaeooooo jang/bomf bomf git.
Introduction
The field of Natural Language Processing (NLP) has significantly advanced with the pre-training of Transformer- based models on large amounts of texts without supervision. In general, these pre-trained networks are fine-tuned supervised downstream tasks solve particular tasks. The rise of Large Language Models (LLMS) such as GPT [50] and LLaMA [63] has increased demands for huge memory and computing during fine-tuning on downstream tasks. In response, low rank approximation methods such as Low-Rank Adaptation (LOR A) [22] and Quantized Low-Rank Adaptation (QLORA) [11] have been adopted recently to fine-tune the LLM. However, the fine-tuning of Pretrained Language Models (PLMs) exhibits high sensitivity to marginal variations in hyperparameters such as learning rate and batch size, often leading to training failure and the performance drop of fine-tuned model [45], while searching hyperparameters requires vast amount of resources.
An effective strategy to seek optimal model among multiple candidates is model ensembling, which yields impressive performance on generalization and robustness [33]. However, traditional ensemble methods lead to several drawbacks including the space and computational costs that linearly scale with the number of models involved. These issues are particularly pertinent for LLMS, since individual
"Co-fire first authors
"Co--Corresa CCo-corresponding authors
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

models are costly to train and test. Therefore, we can alternatively utilize model fusion to aggregate multiple models into a single proficient model on a parameter space. One of its simplest forms, known as Stochastic Weight Averaging (SWA) [25], involves taking the average of model parameters obtained during an optimization process. Despite its simplicity, SWA and its variants have proven successful across various tasks, notably in computer vision [25, 42, 6, 46]. Recent advancement in this field is the concept of Model Soups, which has been introduced by Wortsman et al. [70]. This approach weight-averages set of models, obtained from multiple fine-tuning runs with different hyperparameters to create a powerful model that outperforms both individual and ensemble models.
The effectiveness of model fusion has predominantly been explored in the visual domain. For instance, while Model Soups have shown considerable improvements in image classification. they have not demonstrated superiority in the NLP tasks [70]. The existing averaging methods like SWA make use of their ability to encourage a fused model to locate on the center of the flatter area near local optima [25, 20], as loss landscapes are analogous to metric landscapes in computer vision tasks. Unfortunately, for PLMS, loss landscapes are substantially mismatched to metric landscapes, so that the flat loss minimum found by SWA does not necessarily correspond to the flat metric minimum making a simple averaging method fail to find a superior model.
In this paper, we present a novel model fusion approach with an efficient hyperparameter selection strategy. denoted as Bayesian Optimization Model Fusion (BOMF), specifically designed to fine-tune PLMS. To motivate our method, we start by illustrating two empirical analyses. Firstly, we demonstrate that the existing model fusion techniques are not suitable for PLMS. Secondly, we highlight that the optimal hyperparameters for PI.MS exhibit consistency on varying the number of frozen layers or the rank used in the LoRA setting [22].
Based on these findings, we introduce our proposed method to build single model, aggregated through the weighted combination of individual models. Supposing that evaluation metries are non- differentiable, we employ Bayesian Optimization (BO) [5, 18], which is a black-box optimization technique, in developing our model fusion method. To the best of our knowledge, this is the first study that utilizes BO in the context of model fusion, in order to achieve the following objectives:
Utilization of Both Metries and Loss Functions in Model Fusion. Instead of running BO with an averaged target metric, we use Multi-Objectio Bayesian Optimization (MOBO) that considers both metrics and loss functions for model fusion. Despite low correlations between loss and metric values, we find that incorporating loss values still serves as useful guidance.
Two-Stage Model Fusion. We devise our model fusion process as two-stage BO procedure. One is for optimizing hyperparameters in fine-tuning and the other is dedicated to our model fusion method. The objective of the first stage is maximize gains from the second stage to find hyperparameters leading to the optimal fused model after the BO of the second stage.
We demonstrate the effectiivneess of BOMF several NLP tasks, including both Natural Language Understanding (NLU) and Natural Language Generation (NLG), with ROBERTa, Text-toTTexi Transfer Transformer (T5) and LLaMA. Through these comprehensive experiments, we assess the performance of BOMF in diverse NLP tasks and uncover the interesting properties of our approach through various ablation studies.
2 Preliminaries
Problem Setup. In this paper, we explore the process of fine-tuning PLMS using two types of datasets: downstream training dataset Dmm and validation dataset D Assuming that given a pre-trained set of weights mit and a trainable set of weights Winit for our PLM denoted as M(O, w). Win is either a subset of 0ma or LoRA weights [22]. Specifically, in the former case, Wiit is deliberately selected from m As a special case, Wini will be identical to 0 if any layers or weights not frozen. Meanwhile, if the LORA method is employed during the fine-tuning of our model, Wini will be the LORA weights.
We use K distinct metrics, denoted as f(tti (M, D) for & E [K], to evaluate our model's performance on a given task. Each metric (kk) is typically non-differentiable, while a differentiable loss function fress is employed for training. Assuming that D is similar to the true data distribution, our goal is to

---

find the optimal set of trainable weights w that minimizes the following:
K
W arg min WEW
m (Mi(f w), Dvat),
(1)
c(a) where menric is normalized version of metric for all k E [K], and W is the space of trainable weights. However, due to the non-differentiabiiiyy of the I functions, conventional approaches resort to finding approximate solutions using gradient descent or its variants, as shown below:
W arg min floss (MM mit: w), Dun)
(2)
wEW
As will be discussed in the subsequent section, misalignment between loss and metric surfaces is more prominent in PLMS compared to computer vision models. To address this challenge, we propose novel method more adequately make use of (fre) }K and Daal by considering Equation 1.
merric k=1
Model Fusion. In the recent work [25, 70, 53, 54], there has been a growing interest in the use of weight averaging model fusion across diverse tasks. This line of research is effective strategy to achieve superior performance in downstream tasks, all while managing computational costs by aggregation of multiple models. In this context, the aggregation of multiple models in- volves the identification of set of N fine-tuned trainable weights, denoted as S = /- The objective is to derive fused weight vector, w. by utilizing S, such that outperforms all other members in S. This can be expressed as Lmetrie S arg min Aieeric (w), where Twmerreew) Σk f(A) (M w); Dvat)-
Model fusion approaches can be categorized into two main types: 1) uniform averaging and 2) weighted averaging. Uniform averaging methods, c.g., SWA [25]. Greedy Soups [70], involve the straightforward process of uniformly averaging weights within a subset S C S to obtain an improved performing weight vector W, i.e.
1 Ewes w. Here, selecting suitable subset S is an important strategy for each method. On the other hand, weighted averaging approaches, c.g., Learned Soups [70] and Rewarded Soups [54], aim determine optimized weight vector by forming a convex combination of parameters from S, expressed as w Σn 6,W,, where each averaging coefficient ó, satisfies 6, 2 O, and Ei 6, = 1. While weighted averaging methods offer more flexibility compared to uniform averaging, they often require additional training to determine suitable values for the coefficient set d through gradient descent updates based on the loss function fias. However, in our proposed method, we suggest a weighted averaging technique that considers not only the loss function flsss but also the metrics (f 1K k=
[ST
Multi-Objectiin Bayesian Optimization. BO is sample-efficient black-box optimization tech- nique with probabilistic regression. Since we assume that an objective to optimize is unknown, a surrogate function, which is generally probabilistic regression model, is estimated instead. The key desired properties of the surrogate function are attained by considering how a search space is exploited and explored through its outputs. Utilizing the surrogate function, BO eventually optimizes a specific form of optimizable function, called an acquisition function; see [5, 18] for details.
On top of generic BO, MOBO is used to solve an optimization problem, involved with K different competing objectives:
arg miggmi(//a (xx, fx(()).
Supposing that we cannot directly access f2, fk, probabilistic surrogate models, which are alternatives to unknown objectives, should used to determine a next point to evaluate. To find a solution candidate of Equation 3 using MOBO, we can consider scalarization of either the realizations of surrogate models or acquisition functions corresponding to multiple objectives [48]. In contrast to the scalarization method, the maximization of Expected HyperVolume Improvement (EHVI), metric space [16] can be used:
x1 arg max EHVI(X; Y,r),
where a hypervolume defined as the size of space between the Pareto frontier of n historical evaluations Y B and reference point r. While the scalarization determines query points
3

---

W
125
WS
1.03
SWA
0.80
SWA
0.19
0.58
0.15
Wg
W2
0.35
W3g
W2
0.10
0.12
(a) Loss of ResNet-50
(b) Metric ResNet-50
17
Wi
W2
1.1
16
0.54
12
SWA
SWA
0.48
BOMF
0.43
10
0.41
9.7
W3
W3
W3
0.4
8.8
(c) Loss of ROBERTA
(d) Metric of ROBERTA
Figure Visualization of the loss landscape over parameters (Figures 1a and 1c) and the metric landscape over parameters (Figures 1b and Id) for both the vision task (Figures la and 1b) and the NLP task (Figures Ic and 1d). The metric is accuracy and FI score for the vision task and the NLP task, respectively. In the vision task, we fine-tune the ResNet-50 model [21] pre-traines with ImageNet-21k [56] on the Caltech- 101 dataset [35], while the NI.P task, fine-tuning was performed on the pre-trained ROBERTA model on the MRPC dataset. The members of the SWA for each figure are denoted as W1; W2 W3-
by aggregating outputs with particular (potentially random) coefficients, the hypervolume im- provement maximization chooses query points that widen the expected hypervolume, which is more robust to function scales without the sampling distributions of scalarization coefficients. As reported in the previous work [10, 3], compared to other existing MOBO algorithms, gNEHVI which is variant of the EHVI method that evaluates a batch of points in parallel manner. Building on the powerful MOBO algorithm, our model fusion framework is capable of determining averaging coefficients efficiently reducing the number of evaluations required to find better fused PLMS.
3 Empirical Findings
In this section, we present empirical observations motivating model fusion strategy. In $ 3.1, we initially illustrate distinct findings: unlike in computer vision tasks, in NL.P tasks, there exists a significant misalignment between the loss and metric surfaces. This misalignment poses challenge for straightforward model fusion methods when fine-tuning PLMS. In $ 3.2, we find that the optimal fine-tuned hy perparrameeeess for analogously align across different architectural configurations varying the number of frozen layers or variations in rank in the LORA setting.
3.1 On Misalignment in Loss and Metric Landscapes
The well-known success of uniform averaging, e.g., SWA and Model Soups, in image classification tasks, has been grounded on the flatness loss landscape. As one can see Figure la, the use of uniform averaging successfully explores minima on the flatter region of the loss landscape using individual weights close to the flatter region, resulting enhanced generalization loss on a test dataset. This generalization effect is similarly observed in the case of the metric landscape, as illustrated in Figure b,, owing to the similarity between the loss and metric landscapes. This similarity the consequence of the inherent similarity between the loss function and the metric [43]. However, the

---

Optimal values
0.65
92
12(al)) layers
Valid metric (%)
0.60
Freeze layers
Valid loss
90
Freeze layers
0.55
Froeze layers
Full fine tue-tuning
0.50
Optimal values
88
0.45
Full fine-tuning
layers
86
0.40
Freeze layers
Freeze layers
0.35
Freaze layers
10
10
10-5
10"
10-1
10
10
10
10-5
10
10
Learning rate
Learning rate
0.625
86.0
Optimal values
0.600
Freeze all layers
Valid metric (%)
85.5
0.575
Valid loss
Freeze 4 layers
0.550
85.0
Freeze 6 layers
fine-tuning
0.525
84.5
X
0.500
0.475
84.0
0.450
83.5
0.425
12 Batch size
10
16
Batch size
Figure Validation results on the MRPC dataset for RoBERTa: loss (shown in left panels) and Fl score (in right panels) for varying learning rates, batch sizes, and frozen layers. Optimal hyperparameters align well across different frozen layers, except when pre-trained layers are frozen.
domain of language modeling, characterized by semantic, morphosyntactic, and pragmatic intricacies, requires the evaluation of generalization performance across a diverse array of tasks and metrics [12]. It is unlikely precisely align these metrics with training loss function [74, 39], leading misalignment that often results in more complex and less flat surfaces in language tasks compared to the loss function visually demonstrated in Figures Ic and ld.
In Figures and 1d, we find that while uniform averaging can reach high generalization performance based on the loss function, it poorly performs concerning the metric function compared to the best- performing weight in S. However, Figure 1d shows that even though the uniform averaging of three weight points degrades the metric performance, better points in terms of higher metric values exist in the convex set of the three weight points. The empirical results we observe above, which are caused by the complex and misaligned surface, motivate the need to utilize weighted averaging methods and seek the optimal combination of averaging weights based on the metric. This does not agree with the previous findings in vision tasks [70] and Figure 1b which argue minimal performance difference between the weighted averaging and the uniform averaging. Refer to Appendix C.I for numerical results that show the discrepancy between the loss and metric landscapes in PLMS.
3.2 On Hyperparameter Alignment
Discovering the optimal training hyperparameters incurs significant computational costs, particularly when fine-tuning extensive foundational models [2, 45, 66]. This challenge arises since the ideal set of hyperparameteers tends to vary in tandem with changes in both tasks and model structures.
Surprisingly, our empirical findings reveal consistent alignment optimal hyperparameters when fine-tuning PLMS, regardless of variations in the number of frozen layers or the rank of LORA. As illustrated in Figure 2, the alterations in validation loss and metric resulting from changes in the learning rate or batch size exhibit similar pattern across different numbers of frozen layers, except the case when all pre-trained layers are frozen and only the classifier layer is trained. This proves that we can decrease computational cost for searching the optimal hyperparameters by tuning on smaller models with more frozen layers LoRA with smaller ranks. Refer to Appendix C.2 to see the additional results when varying the adam beta, learning rate schedule, well as the case of the LoRA.

---

Yang et al. [72] demonstrate that employing a particular model weight initialization method and learning rate scheduling method, referred to as j-parametrization, enables the transferability of certain training hyperparameters (such as learning rate and momentum) varying the width of the model. However, it is important to note that these results specifically pertain to scenarios where models are trained from scratch. This distinction is noteworthy our context involves the fine-tuning of PLMS. would be a great future research direction to theoretically analyze this phenomenon.
Bayesian Optimization Model Fusion
In this section, ROMF unfolds in three key steps. In 4,1, present the process of constructing set of fine-tuned trainable weights S, serving as components for model fusion. In $ 4.2, we introduce method identify optimal hyperparameters crucial in the construction of the set S based on the findings explained in $ 3.2. Finally, we delve into how we conduct weighted averaging in $ 4.3, following the insights discussed in 3.1.
4.1 Fusion Member Sampling
To improve the performance of our model through model fusion, it is crucial carefully create the set S by employing an appropriate weight sampling method. There are two main types of weight sampling methods: 1) sampling from multiple training trajectories [70] and 2) sampling from single training trajectory with proper learning rate scheduling [25]. However, Wortsman et al. [70] indicate that, when applying model fusion with samples from multiple training trajectories, the performance improvement becomes less significant during the fine-tuning of PLMS compared to vision tasks. This limitation in NLP tasks attributed to the misalignment loss and metric surfaces, as discussed in 3.1. Furthermore, when employing multiple training trajectories to sample fusion members, the training computation cost increases linearly in proportion to the number of fusion members. This poses significant challenge, particularly in the context of fine-tuning PLMs. For these reasons, in our approach, we collect our fusion members from a single training trajectory. Since the fine-tuning process of PLMS involves a small number of training epochs and exhibits rapid convergence [41], start gathering fusion members after 50% of the training epochs are completed. This timing is slightly quicker than the point described in the work [25], which begins collecting after 75% of the training epochs are concluded. Once we start collecting the fusion members, we proceed to uniformly sample 15 members throughout the remaining training epochs. Refer to Appendix A for more details on the process of collecting fusion members.
4.2 Hyperparameter Search via Bayesian Optimization
In the construction of a set of fusion members S from single training trajectory, the effectiveness of the training trajectory significantly impacts the ultimate metric performance of the fused model weight w. In this context, the effectiveness of a training trajectory refers the model's metric performance using the best- performing weight within that trajectory on the valida- tion dataset D The correlation Figure 3 strongly indicates that the performance of the best best-performing weight is positively correlated with the performance of the fused weight. Consequently, to achieve the best performance of the fused weight, becomes crucial to identify the set of optimal hyperparameters À that re- sults in the most effective training trajectory. However, two primary challenges arise when searching for the optimal hyperparameters X that yield the best metric
*
Best point Worst point
performance
D. n 000000
3
75 Best single performance (%)
Figure 3: Correlation between the perfor- mance of best-performing weights in a train- ing trajectory and the performance of the fused model. We fine-tune the ROBERTa model on the RTE dataset. Each point is obtained from the evaluation of single tra- jectory with varying hyperparameters.
performance: 1) the metric functions { ( M J==1 k= are non-differentiable and 2) we need to efficiently assign computational resources in finding better hyperparameters beyond naive methods such as grid search. To remedy these two issues simultancously, in BOMF, we employ BO to find the optimal set

---

of hyperparameters:
((t) (M(Oinit w(A)), Deai),
A = arg min
(5)
where w(A) represents the best-performing weight within the training trajectory associated with the hyperparameter set X. Here, we utilize Gaussian process (GP) regression [55] and Log-Expected Improvement [3] as a surrogate function and an acquisition function, respectively. We employ three randomly initialized sets of hyperparameters as the starting point for BO, conducting 10 iterations of computations to determine the optimal set A
The sequential nature of BO computations can lead to a substantial computational load. particularly in the context of fine-tuning PLMS. To address this issue and propose more computationally efficient Bо approach, we draw insights from the observations discussed in $ 3.2. The alignment of the best hyperparameters for fine-tuning between the full model and lightweight models (e.g.. frozen layers model or reduced rank LORA) allows us to utilize the lightweight model instead of the full model when seeking the optimal set X as follows:
A arg min
Σ Fletii (M(( (())), Drai),
(6)
where w is the trainable weight of the lightweight model. Refer to $ 6 to see how our computationally efficient method decreases computation time while maintaining performance.
4.3 Multi-Objective Bayesian Optimization for Model Fusion
After completing the construction the set S with N individual models, the next stage involves selecting appropriate averaging coefficients 5 E [0, 1| to ensure the enhanced metric performance of fused model. To achieve this, we can leverage metries [ obtain optimal averaging coefficients 8", similar to the optimization process for the hyperparameter set A. However, restricting the consideration to metric performance solely on Dval may result in our fused weights w overfitting Dval and exhibiting poor generalization to the true data distribution, due to the complex and sharp nature of the metric landscape which is observed in $ 3.1. To tackle IK this challenge, when optimizing 8, we propose to minimize both floss and { I Tk=1 by employing MOBO identify a Pareto frontier defined as follows:
J==1 and apply a BO procedure to
P = 5* 8" = arg min (8) ... .x (1,,(()))..,,(())))
5
where ((5) = Joss (M(0mi: w())), Dval) and (6) = f(A) (M(0 inits w())), Drai) for & E [K]. Note that w() denotes fused set of weights with averaging coefficient vector Ó, i... w(6) EN J,W, where w, E S for E [N] and N the number of models fuse.
Here, we utilize the EHVI strategy, which is described in the work by Emmerich et al. [16]. The hypervolume, in this context, is defined the reference points as a zero vector. To enhance the optimization of the hypervolume improvement objective, employ the logarithmic form of NEHHVI algorithm [10, 3]. which is implemented with the BoTTorch framework [4]. As highlighted multi-objective optimization scenarios. This makes it well-suited to handle the complex and sharp nature of our metric landscape, enabling it to successfully identify the optimal 8*. We run MOBO for a total of 5 = 75 iterations to find the optimal coefficients 8".
volume size between P and a reference point We set
2, this algorithm has proven effective in practical
In our case, additional constraints are in place for executing MOBO, specifically 1) equality constraints
and 2) inequality constraints for 8. To address the inequality constraints (i.e., 5, 0), follow the work by Gardner et al. [17] to incorporate constraints into the acquisition function. To deal with the equality constraints Σm & = 1, we simply normalize the output of the acquisition function. Refer to Algorithm in Appendix B for the summary of BOMF.
5 Related Work
Model Fusion for Pre-Trained Language Models. Due to the increasing number of model parameters in recent PLMS, there has been significant increase in both memory requirements and
7

---

Table I: Results Medium-Sized Language Models. We conduct the text classification task using RoBERTa-base on subset of the GLUE benchmark datasets, and the question-answert task using 15-base the SQuAD2.0 dataset. ACC, FI, and EM denote accuracy, F1 score, and Exact Match, respectively.
ROBERTA-BASG
15-BASE
(AcC)
(FI)
SST-2 (AcC)
(ACC)
QUP (FI)
(ACC)
SQUAD2.0
GRID FINE-TUNE
77.78
92.39
94.87
92.62
88.16
87.41
78.18/72.85
HPBO (FULL)
78.57
92.78
95.11
93.01
88.58
87.46
78. 28/78/33 29
SWA
78.62
92.24
95.42
92.81
88.49
87.41
80.31/74.88
OTFUSION
77.08
92.82
94.27
92.22
88.34
87.43
80.75/74. 99
GREEDY SWA
80.70
92.83
95.54
93.16
87.45
20.63//55.4
LEARNED SWA
81.40
92.81
95.31
92.94
88.38
87.41
80.65/74.23
TWA
81.23
91.58
95.54
93.00
87.85
87.42
80.29/74.79
BOMF! (OURS)
81.75
93.37
95.65
94.83
88.66
87.51
80.82/75.79
BOMF (OURS)
81.40
93.90
95.54
93.50
88.68
87.86
81.82/76.21
computational costs [73, 8, 63]. Consequently, there is growing attention on a research direction aimed at enhancing the performance of PLMS while simultaneously managing computational costs and memory requirements through the exploration of model fusion methods [54, 71, 9]. However, most of these studies have focused on fusing the models fine-tuned different tasks, aiming to develop a single multi-task learner. In the context of a single-task fine-tuning scenario within PL.M, it has been observed that the previous simple weight-averaging approaches often yield marginal improvements [70, 27]; nevertheless, the exploration into the underlying rationale of this consequence remains limited. As mentioned in $ 3, we find that uniform weight averaging does not always align generalization on the loss surface with the optimal point on the metric surface, primarily due to the discrepancy between loss and metric landscapes. To address this issue, we develope single- task model fusion method based on MOBO, finding the optimal weight combination coefficients by considering both metrics and loss functions.
Bayesian Optimization.
[5. 18] is promising strategy optimize black-box function. In particular, if target objective is costly in terms of function evaluations, Specifically, Bo sequentially seeks solution candidates by modeling surrogate function and maximizing an acquisition function.
In the BO community, a GP [55] is often employed as a surrogate function but diverse regression models such as Bayesian neural networks [61, 38] and tree-based models [23, 30] can be used. As a choice of acquisition function, expected improvement [26] and GP upper confidence bound [62] are often considered. Importantly, BO more effective than other existing optimization strategies such grid search and genetic algorithms [64]. Its efficacy has been demonstrated in wide variety of applications such as hyperparameter optimization [58], nanostructured device design [31], and chemical reaction optimization [57]. Moreover, in the deep learning context, the necessity for efficient hyperparameter tuning via BO has risen following the increasing number of hyperparameters and parameters in models [59]. Consequently, BO is applied for hyperparameter optimization in various deep learning tasks, such as image classification [28, 34] and NLP tasks [44, 7].
Experiments
In this section, we present empirical results demonstrating the effectiveness of BOMF various NLP tasks. We compare our method five basic algorithms aimed finding a nigh-ppeffrming solution. Grid Fine- Tune is a simple fine-tuning method that selects the best-performing checkpoint using grid search. HPBO utilizes optimal hyperparameters obtained by $ 4.2 for fine-tuning the baselines. SWA is an optimization technique that averages model parameters obtained during training. Greedy SWA is modified version of SWA inspired by Greedy Soups [70], sorting weights based on metric performance on D and including them in S only if they improve w performance. Learned SWA, inspired by Learned Soup [70], learns the coefficients For medium-sized language models, we tested a variant of Transformer OTfusion [24], aligning pre-trained weights before averaging. Additionally, we experimented with TWA [36], recent SWA variant that reconstructs S by finding weight space basis vectors and learns 8 based on the loss.
based on the loss after fine-tuning.

---

Table 2: Results on Large Language Models. We compare the results of BOMF and baseline methods using the SAMSum and KorMedMCQA datasets for summarization and medical multiple choice question-answering tasks with I.LMMA-7B and LLMMA3-8B. RI, R2, and RI. denote Rouge-1, Rouge- 2, and Rouge-I. scores for summarization. Doctor, Nurse, and Pharm denote evaluation results for medical question answering each respective field, using accuracy as the metric.
(a) Summarization (SAMSum)
(b) Korean Medical Question Answering
METHOD
R1
R2
RL
AVG.
METHOD
DOCTOR
NURSE
PHARM
AVG.
НPBO (RANK 64)
52,66
28.22
44.33
41.73
37.89
50.15
SWA
51.81
27.61
43.55
HPBO (RANK 64)
43.62
54.64
51.49
49.92
GREEDY SWA
53.40
28.06
43.31
41.49
SWA
43.96
54.64
51.97
50.19
LEARNED SWA
52,93
28.97
44.04
41.98
GREADY SWA
43.97
54.64
$1.98
50.20
LEARNED SWA
44.06
54.94
52.28
50.43
BOME (OURS)
53.40
28.78
44.38
42.19
BOMP (OURS)
53.07
28.61
44.40
42.03
BOMF (OURS)
45.00
55.70
52.97
51.22
BOMF (OURS)
45.31
55.32
52.80
51.16
In all tables, the best performance is indicated with boldfaced underline. while the second-best value is represented with underline in each column. The final column 'Avg.' provides a summary of overall results for each method across various datasets or metrics. The terms 'Full" and "Freeze' in Table specify the exploration of optimal hyperparameters using either the entire model or a model with half of its weights frozen, as discussed in $ 4.2. Similarly, the terms 'Rank 64' and 'Rank 4' Table 2a denote that we use the Rank 64 or the lightweight Rank 4 version of the LORA model for the hyperparameter search, respectively. See Appendix A for the details of downstream datasets and hyperparameter selection.
6.1 Empirical Analysis on Medium-Sizee Language Models
We begin by evaluating the effectiveness of BOMF on medium-sized language models using RoBERTa- base [40] and b5-base [51]. For RoBERTa- base, we performed text classification tasks using the GLUE benchmark datasets [65]. For T5- we carried out the question aiooon--nwwiinggg task with the SQuAD2.0 [52] dataset. For both models, we fine-tuned the weights directly on the downstream datasets.
Table shows that BOMF consistently outperforms other baselines across all model structure and datasets. 2 Notably, the performance of HPBO, which uses hyperparameters obtained from 4.2 with the full model, surpasses Grid Fine-Tune for most datasets. These results demonstrate that our BO-based hyperparameter search framework effectively discovers optimal hyperparameters compared grid search. Refer Appendix C.4 for the performance of freeze HPBO, which uses lightweight model for hyperparameter optimization. Freeze HPBO also clearly outperforms Grid Fine-Ture which proves the effectiveness of our BO-based hyperparameter search. Also, it is evident that model fusion methods, except BOMF, lead to performance declines compared to HPBO, as discussed in 3.1, in certain datasets. On the contrary, BOMF consistently betters the performance compared HPBO, yielding that our method with MORO effectively finds optimal 8" even in complex and sharp metric landscapes. Refer to Table 16 for the complete results.
6.2 Empirical Analysis Large Language Models
We further validated the effectiveness of our proposed method by fine-tuning larger models using LORA. Specifically, experimented with LLaMA2-7B and L.LMAA-8B on tasks such as summa- rization using the SAMsum [19] dataset, Korean multi-choice medical question answering using the KorMedMCQA [32] dataset, and dialogue generation using the E2E [47] dataset. In the sum- marization task, while Learned SWA exhibited the best performance in terms of Rouge-2, BOMF surpassed Learned SWA in average performance across all metrics, as illustrated in Table 2a. Notably, for Rouge-L, only BOMF improved over HPBO, highlighting the effectiveness of the multi-objective approach in BOMF. Furthermore, as shown in Table 2b, our model not only outperforms other base- lines but also demonstrates that fine-tuning remains essential for specific tasks despite the rise of
'A+ i symbol indicates results from the trajectory found using full HPBO, while results without the symbol indicate trajectories found using freeze HPBO.

---

in-context learning (ICL) [14]. This highlights the necessity of BOMF, which efficiently identifies hyperparameters and provides an effective fine-tuning solution through model fusion. The results for F2E can be found in Appendix C.4.
6.3 Ablation Study
Number of Frozen Layers. To analyze the ef- ficiency of memory and compute when using a lightweight model in the BO procedure to find X" we conduct a study using RoBERTa-base on the RTE and MRPC datasets. As presented in Table 3, the use of lightweight model successfully iden- tifies favorable pyprpaaraettess that yield good performance while reducing the number of param- eters by up to 25% and the computation time by up to 66%. This efficiency is achieved by caching outputs from the frozen layers. By systematically freezing layers from the tail of the model, we can cache the outputs from these frozen layers and reuse them during the training process.
Table Results on the Varying Number of Frozen Layers. Comparison of the number of parameters and relative training wall-clock time per epoch when optimizing hyperparameters across different numbers of frozen layers, us- ing RoBERTa-bar fine-tuned on the RTE and MRPC datasets.
TASK
PARAMS REI RELATIVE TIME
RTE
MRPC
GRID TUNE
125M
77.78
92.39
78.57
FREEZE 2
110M
X0.53
78.50
92.39
78.34
92.36
FREEZE 6
82M
X0.34
78.49
92.72
Multiple Objectives. To validate the efficacy of using multiple objectives when determining opti- mal 8, we compare BOMF with single-objective baselines using b5-basc on the SQuAD2.0 dataset. In this task, we consider two metrics: F1 score and Exact Match. Table shows that relying only one specific metric slightly increases the objec- tive metric but results in significant performance drop for the other metric. This outcome suggests that using single-objective BO is appropriate when aiming find model optimized for specific metric, while the use of MOBO is more suitable for discovering an optimal fused model that achieves high performance across various metrics. Refer Appendix C.3 for further ablation studies.
Table 4: Comparison of Using Multi Multi-Objective and Single-Ohjective Approaches. Results of BOMF and single-objective Bо baselines with T5- ba-e fine-tuned on the SQuAD2.0 datasct.
METRIC
F1
EM
AVG.
81.01
75.09
78.05
EM ONLY
80.40
75.87
78.13
BOMF
75.79
78.31
7 Conclusion
In this paper, we empirically remarked two intriguing findings on loss and metric landscapes and hyperparameter alignment. Then, motivated by the observations mentioned above, proposed novel BO- bo-based BOMF algorithm for model fusion. Our method utilizes the BO and MOBO frameworks to seek optimal fine-tuning hyperparameters and averaging coefficients, respectively. We validated that our proposed method exhibits improved performance on both NLU and NLG tasks on middle- and large-scale PL.MS.
Limitations and Future Work. As discussed in & 3.2, compelling future research involves the the- oretical analysis of the hyperparameter aligmment phenomenon. Morcover, we empirically observed that when utilizing quantization- based low-rank approximation methods [11, 37], traditional uniform averaging methods and weighted averaging methods face challenges in effectively aggregating mod- els. These challenges arise from the quantized weight values in the models that behave differently with averaging weights. Another research direction is the development of averaging methods for the quantization-bu based low-rank approximation methods.
Acknowledgments and Disclosure of Funding
This work was partly supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (NRF-2022R) AAA708390812) and Institute of Information & communications Technology Planning Evaluation (ITTP) grant funded by the Korea government (MSIT) (No.RS 1.o.RR-2011 2019-11100075, Artificial Intelligence Graduate School Program (KAIST), 2o.2022-
10

---

0-00184, Development and Study of AI Technologies to Inexpensively Conform to Evolving Policy on Ethics, No.2022-0-00713, Meta-learning Applicable to keal-world Problems).
References
[1] A. Agarwal and A. Lavie. Meteor, M-BLEU and M-TER: Evaluation metrics for high-correlation with human rankings of machine translation output. In Proceedings of the Third Workshop on Statistical Machine Translation, June 2008. 26
[2] A. Aghajanyan, A. Shrivastava, A. Gupta, N. Goyal, L. Zettlemoye, and S. Gupta. Better fine-tuning by reducing representational collapse. arXiv preprint 2000::2008,03/56 2020. 5
[3] S. Ament, S. Daulton, D. Eriksson, M. Balandat, and E. Bakshy, Unexpected improvements to expected improvement for Bayesian optimization. In Advances in Neural Information Processing Systems (NeurIPS), 2023. 4, 7
[4] M. Balandat, B. Karrer, D. Jiang. S. Daulton, B. Letham, A. G. Wilson, and E. Bakshy, BoTorch: framework for efficient Monte-Carla Bayesian optimization. In Advances in Neural Informa- fion Processing Systems (NeurIPS), 2020. 16
[5] E. Brochu, V. M. Cora, and N. de Freitas. A tutorial on Bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. arXiv preprint arXiv: 10/2.2000 2010. 2, 3,8
[6] Cha, S. Chun, K. Lee, H.-C. Cho, Park, Y. Lee, and S. Park. SWAD: domain generalization by seeking flat minima. In Advances in Neural Information Processing Systems (NeurIPS), 2021. 2
[7] L. Chen, J. Chen, T. Goldstein, H. Huang, and T. Zhou. Instructzero: Efficient instruction optimization for black-box large language models. arXiv preprint arXiv:2306.03083 2023. 8
[8] A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113, 2023. 8
[9] A. Chronopoulou, M. E. Peters, A. Fraser, and J. Dodge. Adaptersoup: Weight averaging to improve generalization of pretrained language models. arXiv preprint arXiv:2302.07027, 2023.
8
[10] S. Daulton, M. Balandat, and E. Bakshy. Parallel Bayesian optimization of multiple noisy objectives with expected hypervolume improvement. In Advances in Neural Information Processing Systems (NeurIPS), 2021. 4,
[11] T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305. 2023. 1, 10
[12] J. Dodge, S. Gururangan, D. Card, R. Schwartz, and N. A. Smith. Show your work: Improved reporting of experimental results. In Proceedings of the 2019 Conference on Empirical Meth- ods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing EEMNL--ICCNLL 2019. 5
[13] B. Dolan and C. Brockett, Automatically constructing a corpus of sentential paraphrases. In Third International Workshop on Paraphrasing (/WP2005), 2005. 16
[14] Q Dong, L. Li, Dai, Zheng, Z. Wu, B. Chang, X. Sun, Xu, L. Li, and Sui. A survey on in-context learning. arXiv preprint arXiv:2301.0023:1 2023. 10
[15] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In Proceedings of the International Conference Learning Representations (ICLR), 2021. 21

---

[16] M. T. M. Emmerich, K. C. Giannakoglou, and B. Naujoks. Single-and multiobjective evolu- tionary optimization assisted by Gaussian random field metamodels. IEEE Transactions on Evolutionary Computation, 10(4):421 -4442,1199,,,, 2006. 3,7
[17] J. R. Gardner, M. J. Kusner, Z. E. Xu, K. Q. Weinberger, and J. P. Cunningham, Bayesian optimization with inequality constraints. In Proceedings of the International Conference on Machine Learning (ICML), 2014. 7
[18] R. Garnett. Bayesian Optimization. Cambridge University Press, 2023. 2, 2,3,
[19] B. Gliwa, I. Mochol, M. Biesek, and A. Wawer. Samsum corpus: A human-annotated dialogue dataset for abstractive summarization. arXiv preprint arXiv: /911 2019. 9, 17
[20] H. He, G. Huang, and Y. Yuan. Asymmetric valleys: Beyond sharp and flat local minima. In Advances in Neural Information Processing Systems 32 (NeurIPS 2019), 2019. 2
[21] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE International Conference Computer Vision and Pattern Recognition (CVPR), 2016.
[22] E. J. Hu, P. Wallis, Allen-Zhu, Li, S. Wang, L. Wang, W. Chen, et al. Lora: Low-rank adaptation of large language models. In Proceedings of the International Conference Learning Representations (ICLR), 2021. 1,2
[23] Hutter, H. Hoos, and K. Leyton-Browd Brown. Sequential model-based optimization for general algorithm configuration. In Proceedings of the International Conference Learning and Intelligent Optimizution (LJON), 2011.
[24] M. Imfeld, J. Graldi, M. Giordano, T. Hofmann, S. Anagnostidis, and S. Singh. Trans- former fusion with optimal transport. In The Twelfth International Conference on Learning Representations, 2023.
[25] Izmailov, Podoprikhin, Garipov, D. Vetrov, and A. G. Wilson. Averaging weights leads to wider optima and better generalization. In Proceedings of the Conference on Uncertainty in Artificial Intelligence ( (UAI), 2018. 2,
[26] D. Jones, M. Schonlau, and J. J. Welch. Efficient global optimization of expensive black-box functions. Journal of Global Optimization, 13:455-4925 1998. 8
[27] J. Kaddour, L. Liu, R. Silva, and M. J. Kusner. When do flat minima optimizers work? Advances in Neural Information Processing Systems (NeurIPS), 2022.
(28) K. Kandasamy, W. Neiswanger, J. Schneider, B. Póezos, and E. P. Xing. Neural architecture search with Bayesian optimisation and optimal transport. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
[29] M. Kilickaya, A. Erdem, N. Mäzler-Cinniss and E. Erdem. Re-evaluating automatic metrics for image captioning. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguisties: Volume I, Long Papers, 2017. 26
[30] J. Kim and S. Choi. On uncertainty estimation by tree-based surrogate models in sequen- tial model-based optimization. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), 2022. 8
[31] J. Kim, M. Li, Y. Li, A. Gómez, O. Hinder, and P. W. Leu. Multi-BOWS: Multi-fidelity multi- objective Bayesian optimization with warm starts for nanophotonic structure design. Digital Discovery, 3(2):381-39) 2024. 8
[32] S. Kweon, B. Choi, M. Kim, R. W. Park. and E. Choi. Kormedmeqa: Multi-choice question answering benchmark for korean healthcare professional licensing examinations. arXiv preprint arXiv:2403.01469. 2024. 9, 17
[33] B. Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
12

---

[34] B. Letham, R. Calandra, A. Rai, and E. Bakshy. Re-examining linear embeddings for high- dimensional Bayesian optimization. In Advances in Neural Information Processing Systems (NeurIPS), 2020. 8
[35] F.F.. Li, M. Andreeto, M. Ranzato, and P. Perona. Caltech 101, Apr 2022.
[36] T. Li, Z. Huang, Q. Tao, Y. Wu, and X. Huang. Trainable weight averaging: Efficient training by optimizing historical solutions. In Proceedings of the International Conference on Learning Representations (ICLR), 2023.
[37] Y. Li, Y. Yu, C. Liang, P. He, N. Karampatziakis, W. Chen, and T. Zhao. Loftq; Lora-fine- tuning-
tuning-aware aware quantization for large language models. arXiv preprint arXiv:2310.08659, 2023. 10
[38] Y. L. Li, T. G. Rudner, and A. G. Wilson. A study of Bayesian neural network surrogates for Bayesian optimization. arXiv preprint Xxxv:2305.20028. 2023. 8
[39] G. Liu, Z. Yang, T. Tao, X. Liang. J. Bao, Z. Li, X. He. S. Cui, and Z. Hu. Don't take it literally: An edit-invariant sequence loss for text generation. In Proceedings the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2055-2078, 2022. 5
the 2022 Conference of the
[40] Y. Liu, M. Out, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint
2019. 9
[41] P. Lu, I. Kobyzev, M. Rezagholizadeh, A. Rashid, A. Ghodsi, and P. Langlais. Improving generalization of pre-trained language models via stochastic weight averaging. arXiv preprint arXiv:22/2.05956, 2022. 6
[42] W. Maddox, T. Garipov, P. Izmailov, Vetrov, and A. G. Wilson. simple baseline for Bayesian uncertainty in deep learning. In Advances in Neural Information Processing Systems (NeurIPS), 2019. 2
[43] A. Mao, M. Mohri, and Y. Zhong, Cross-entropy loss functions: Theoretical analysis and applications. In Proceedings of the International Conference on Machine Learning (ICML), 2023.
[44] G. Melis, C. Dyer, and P. Blunsom. On the state of the art of evaluation in neural language models. In Proceedings of the International Conference on Learning Representations (ICLR),
2018. 8
[45] M. Mosbach, M. Andriushchenko, and D. Klakow. On the stability of fine-tuning bert: Mis- conceptions, explanations, and strong baselines. arXiv preprint arXiv:2006.0488 04884, 2020. 1,
[46] G. Nam, S. Jang, and Lee. Decoupled training for long-tailed classification with stochastic representations. In Proceedings of the International Conference on Learning Representations (ICLR), 2023. 2
[47] J. Novikova, O. Dušck, and V. Rieser. The e2c dataset: New challenges for end-to-end generation. arXiv preprint 1XXiv1100.0992 2017. 9, 17
[48] B. Paria, K. Kandasamy, and B. Póczos. A flexible framework for multi-objectivy Bayesian optimization using random scalarizations. Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI), 2019. 3
[49] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L.. Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems (NeurIPS), 2019. 16
[50] A. Radford, K. Narasimhan, T. Salimans, and Sutskever. Improving language understanding by generative pre-training, arXiv preprint 161.00180,. 2018.
13

---

[51] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(1):5485-555 2020.
[52] Rajpurkar, R. Jia, and P. Liang. Know what you don't know: Unanswerable questions for squad. arXiv preprint arXiv: / 036803 2018. 9, 16, 17
[53] A. Ramé, K. Ahuja, Zhang, M. Cord, L. Bottou, and Lopez-Paz. Model ratatouille: Recy- cling diverse models for out-of-distribution generalization. In Proceedings of the International Conference on Machine Learning (ICML.) 2023. 3
[54] A. Ramé, G. Couairon, M. Shukor, C. Dancette, J.-B. Gaya, L. Soulier, and M. Cord. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. In Advances in Neural Information Processing Systems (NeurIPS), 2023. 3,8
[55] C. E. Rasmussen and K. Williams. Gaussian Processes for Machine Learning. MIT Press. 2006. 7,8
[56] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211 1-222, 2015. 4
[57] B. J. Shields. J. Stevens. J. Li, M. Parasram, F. Damani, J. I. M. Alvarado, J. M. Janey, R. P. Adams, and A. G. Doyle. Bayesian reaction optimization as a tool for chemical synthesis. Nature, 590(7844):89 99, 2021. 8
[58] Snoek, H. Larochelle, and R. P. Adams. Practical Bayesian optimization of machine learning algorithms. In Advances in Neural Information Processing Systems (NeurIPS), 2012. 8
[59] Snock, O. Rippel, K. Swersky, R. Kiros, N. Satish, N. Sundaram, M. Patwary, M. Prabhat, and R. Adams. Scalable bayesian optimization using deep neural networks. In Proceedings of the International Conference on Machine Learning (ICML), 2015. 8
[60] R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Y. Ng. and C. Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631-1642, 2013. 16
[61] J. T. Springenberg, A. Klein, S. Falkner, and F. Hutter. Bayesian optimization with robust Bayesian neural networks. In Advances in Neural Information Processing Systems (NeurIPS), 2016. 8
[62] N. Srinivas, A. Krause, S. Kakade, and M. Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. In Proceedings of the International Conference on Machine Learning (ICML), 2010.
[63] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, Bhargava, S. Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint ar a0Xiv2230..09288, 2023. 1,8
[64] R. Turner, D. Eriksson, M. McCourt, J. Kiili, E. Laaksonen, Z. Xu, and Guyon. Bayesian optimization is superior to random search for machine learning hyperparameter tuning: Analysis of the black-box optimizaaton challenge 2020. Proceedings of the NeurIPS Competition and Demonstration Track, 2020. 8
[65] A. Wang, A. Singh, J. Michael, Hill, O. Levy, and S. R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the International Conference on Learning Representations (ICIR), 2018. 9, 16
[66] L. Wang. Y. Li, T. Miller, S. Bethard, and G. Savova. Two-stage fine-tuning for improved bias and variance for large pretrained language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), July 2023. 5
14

---

[67] Z. Wang, W. Hamza, and R. Florian. Bilateral multi purti-perspeettve matching for natural language sentences. arXiv preprint arXiv:1702.038/1 2017. 16
[68] A. Williams, N. Nangia, and S. R. Bowman. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv: 17Xtv:/700.05426, 2017. 16
[69] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, et al. Huggingface's s transformers: State-of-the-arr natural language processing.
arXiv preprint arXiv: 191.:3791, 2019, 16
[70] M. Wortsman, G. Ilharco. S. Y. Gadre, Roclofs, R. Gontijo-Lopes, A. S. Morcos, H. Namkoong, A. Farhadi, Y. Carmon, S. Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In Proceedings of the International Conference on Machine Learning (ICML), 2022. 2, 2,
[71] P. Yadav, D. Tam, L. Choshen, C. Raffel, and M. Bansal. Resolving interference when merging models. arXiv preprint 130.::23006./7000 2023. 8
[72] G. Yang. E. J. Hu, I. Babuschkin, S. Sidor, X. Liu, D. Farhi, N. Ryder, J. Pachocki, W. Chen, and J. Gao. Tensor programs Tuning large neural networks via zero-shot hyperparameter transfer. arXiv preprint arXiv:2203.0344 2022.
[73] S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, al. Opt: Open pre-trained transformer language models. arXiv preprini arXiv:2205. 01068, 2022.
[74] V. Zhukov, E. Golikov, and M. Kretov. Differentiable lower bound for expected bleu score. arXiv preprint arXiv:1712.04770 2017. 5
15

---

A Details of Experiments
Our implementation leverages key libraries, including PyTorch 2.0.1 [49], Huggingface Transform- ers [69], and Bo'Torch [4], to construct robust framework for our experiments. These experiments are rigorously conducted on high-perffrmance computing hardware, specifically NVIDIA RTX 3090 and NVIDIA RTX A6000 GPUs, to ensure the efficiency and scalability of our models. To further bolster the reproducibility of our results, we meticulously set and documented all experiment seeds, enabling precise replication of our experimental conditions and findings.
A.1 Medium-Sized Language Models
Table 5: Detailed ROBERTA experimental setup.
CATEGORY
DETAILS
MODEL SPECIFICATIONS
ARCHITECTURE
TRANSFORMER
TRAINING
ROBERTA BASE
OPTIMIZER
ADAMW
SCHEDULER
LINEAR SCHEDULER WITH WARMUP
WARMUP RATIO
0.2 IF RTE OR MRPC BLSE 0.1
LEARNING RATE
[IE-06, 1E-04]
BATCH SIZE
[8, 16] IF RTE OR MRPC ELSE [32, 64]
EPOCHS
20 IF RTE ELSE 10
TASK SPECIFICATIONS
TASK NAME
CLASSIFICATION
DATASET
SUBSET OF GLUE BENCHMARKS.
For the ROBERTa model, we evaluated the performance for classification and utilized a subset of the GLUE benchmark [65]. This benchmark serves as comprehensive evaluation of a language model's overall NLU capabilities. The Recognizing Textual Entailment (RTE) task, which employs neutral and contradiction instances to assign a not-entailment label, a binary classification task comprising 2,490 training instances and 277 validation instances. The Microsoft Research Paraphrase Corpus (MRPC) [13] consists of sentence pairs and corresponding labels. This task involves binary classification determine whether pair of sentences score as the metric due to label imbalance. This dataset contains a total of 3,668 training and 408 validation instances. The Stanford Sentiment Treebank (SST-2) [60] includes movie reviews with associated positive/negative labels. The task is binary classification to discern the sentiment of a given sentence as positive or negative, with 67, 47,349 training and 872 validation instances. The Stanford Question Answering Dataset (QNLI) [52] is question-answering task composed of paragraph- question pairs, where one sentence in the paragraph contains the answer the numan-generatee question. This dataset comprises (04,743 training and 5,463 validation instances. The Quora Question Pairs dataset (QQP) [67] involves determining whether two questions are semantically equivalent, again using the F1 score as the metric due to label imbalance, with 363,846 training and 40,430 test instances. Lastly, The Multi-Genre Natural Language Inference Corpus (MNLI) [68] labeled for textual entailment across genre pairs, primarily consisting of premise and hypothesis sentence pairs. This task predicts the relationship between these sentences in three categories. The dataset includes 392,702 702 training and 9,815 validation instances, of which we used the matched case of the validation set. We conducted experiments by selecting two datasets from each GLUE benchmark based on their size scale. Additionally, specific details the fine-tuning methods can be found in Table 5.
semantically equivalent, utilizing the F1
16

---

Table 6: Detailed T5 experimental setup.
CATEGORY
DETAILS
MODEL SPECIFICATIONS
ARCHITECTURE
TRANSFORMER
PRE-TRAINING
T5-BASE
OPTIMIZER
ADAMW
LEARNING RATE
[[E-06, 1E-04]
BATCH SIZE
[32, 64]
GRADIENT ACCUMULATION STEP
2
EPOCHS
3.0
TASK SPECIEICATIONS
TASK NAME
Question Answering
INPUT TEXT
"question: [question] context: [context]"
LABEL TEXT
"[answer]"
DATASET
SQUAD2.0
MAX NEW TOKENS
10
For the T5-base model, we utilize the Stanford Question Answering Dataset (SQuAD 2.0) [52]. This dataset comprises 130,319 training pairs and 11,873 validation pairs of questions and answers. The dataset can be accessed through the Hugging Face datasets library.* Details our fine-tuning procedures are provided in Table 6. Furthermore, we assess the generated answers by adhering to the code established in the official SQuAD 2.0 repository."
A.2 Large Language Models
In experiments with the LLaMA2-7B* model, focused two tasks: summarization and dialogue generation. For the summarization task, we employed the Samsung Abstractive Messenger Summarization (SAMSum) dataset [19]. which consists of 14,732 training samples, 818 validation samples, and 819 test samples. For the dialogue generation task, we selected the End-to-End NLG Challenge (E2E) dataset [47]. This dataset includes 42,061 training samples, 4,672 validation samples, and 4,693 test samples. Details our fine-tuning process are provided Table 7. Notably, in the case of the E2E dataset, the test typically contains around five common inputs with variety of labels. To save time, we conducted a generate process for one common input and used the different labels multiple references calculate the metries. Consequently, for evaluation, the sentences generated by the model are based on unique label, totaling 630 sentences. This accounts for the discrepancy in experimental performance between our study and that presented in the original paper of the E2E dataset [47]. All metrics including BLEU, METEOR, and ROUGE were computed using the Huggingface evaluate library.?
To demonstrate that fine- Ane-ttunng is still necessary in specific domains and to show the effectiveness of method in finding the best model under these circumstances, conducted evaluations using the Korean Medical Multiple Choice Question Answering (KorMedMCQA) dataset [32]. For
"actps://rajjurkaa.. github Sacttuuu.i///uuuu io/SQuAD- 1o/SSuAD-explooree
7-
17

---

Table 7: Detailed LLaMA2-7B experimental setup.
CATEGORY
DETAILS
MODEL SPECIFICATIONS
ARCHITECTURE
TRANSFORMER
PRE-TRAINING
LLAMA-2-7B
LORA ALPHA
16
LORA DROPOUT
0.1
OPTIMIZER
ADAMW
LEARNING RATE
[1E-06, 1E-03]
BATCH SIZE
[16, 32]
GRADIENT ACCUMULATION STEP
EPOCHS
TASK SPECIFICATIONS
TASK NAME
SUMMARIZATION
PROMPT
"Summarize the following dialogue that is delimited with triple backticks."
DATASET
SAMSUM
TASK NAME
DIALOGUE GENERATION
PROMPT
"Generate natural language description for the following
restaurant attributes."
DATASET
E2E
NATURAL LANGUAGE GENERATION DETAILS
TOP-P
0.9
TEMPERATURE
1E-12
MAX NEW TOKENS
100
batch learning, we used text segments with a maximum sequence length not exceeding 512 tokens. Consequently, the train, test, and validation sets for doctors contained 1,890, 285, and 164 examples, respectively; for nurses, the train, test, and validation sets contained 582, 291, and 291 examples; and for pharmacists, the train, test, and validation sets contained 692, 614, and 300 examples, respectively. For in-context learning, tuning, we used a linear head. For this, we used the LLaMA3-8Bs model, the latest multilingual open-source large language model. This version was downloaded from this link. More specific details about the model and experiments can be found in Table 8.
provided examples within this length limit, and for classification fine-
A.3 Bayesian Optimization
Details of HPBO. In the HPBO experiments, the number of iterations varied depending on the size of each datuset. Specifically, 20 iterations were conducted for the RTE dataset, while 10 iterations
anccpe://hugginggaccoco com.t/accccoo//m....aaaa wat.-alaaaaaaaaaaa Nww.-eettaaaaaaaaeee-eaaaaaaaaaa 3-8B- Instruct
18

---

Table 8: Detailed LLaMA3-8B experimental setup.
Category
DETAILS
MODEL SPECIFICATIONS
ARCHITECTURE
TRANSFORMER
PRE-TRAINING
LLAMA-3-8B-INSTTUCCCII
LORA ALPHA
16
LORA DROPOUT
0.0
OPTIMIZER
ADAMW
LEARNING RATE
[1R-06, 1E-03]
BATCH SIZE
[8, 16]
GRADIENT ACCUMULATION
STEP
EPOCHS
TASK SPECIFICATIONS
TASK NAME
KOREAN MEDICAL QUESTION ANSWERING
PROMPT
420
4
A E de A.
1 1440 d To /
A. - (RECTOVAGINAL FISTULA)
B. 2 (ANAL FISSURE)
C. 4 (FECAL INCONTINENCE)
D. क
(FECAL IMPACTION)
E. Z
(RECTAL PROLAPSE)
DATASET
KORMEDMCQA
were carried out for the MRPC, SST2, and QNLI datasets. For the QQP and MNLI datasets, iterations were performed. In addition, the SQuAD 2.0. SAMSum, and E2E datasets each underwent 10 iterations. These iteration counts were determined based on the respective sizes of the datasets. For single metric tasks, the chosen objective was the single valid metric itself. Conversely, for multi-metric tasks, the objective was the sum of all valid metrics.
Details Sampling Fusion Members. We collected fusion members at step intervals ranging from 0.5 to 2.0 times the point of convergence identified in the training trajectory which represented B in Algorithm 1, adjusting the process yield approximately members in total. Additionally, for the ROBERTa model, we employed PyTorch's is official SWA scheduler with cosine annealing. For the T5 and LLaMA models, we do not use any additional scheduler for collecting SWA members.
Details of MOBO. and conducted iterations five times total number of fusion members. This approach follows common practice in BO of determining the initial points and the number of iterations based on the input dimension, allowing for the option to perform more iterations for improved performance. Furthermore, due to the differing scales of the loss and each metric, applied min-max normalization
In the case of MOBO, we initially provided the length of the fusion member
19

---

Algorithm Bayesian Optimization Model Fusion
Require:
Training set Dim, validation set Dval, initial pre-trained weights Omni initial hyperparame-
ters Ajni--
Ensure: Optimized hyperparameters X combination coefficients 8*
1: Initialize model M with mit Optionally freeze layers and cache intermediate features.
2: Initialize BO with GP model, starting with Ainit
and prior data Hu = (Ami Σx=1 f1k) (MM w (Aini))), Dvai)).
3: for 1 to / iter do
4: Define LogEl using current GP.
5: Find X by optimizing LogEI.
6: Training MM0mmi w) with (Dums X;).
7: Evaluate Σ z(k)
metric (M(O init. w w(A:). Dval).
8: Update GP model with new data (A,, E
[()] menii (M(0s, w(A:)). Den)).
9: end for
10: Collect X arg min, Ex=1 Fute (M(( w(A)), D
11: B s- arg ming meeiic ( (M(Omiit w wn (>\")) Dval)
12: S
3+11 0.
13: for j =1 to step do
Optimize M(Omiit: wy) with (Dun. X*).
15: i 0.5 then
SuSUM w,)
17: end if
end for
19:
Set reference roint
Linit +- 1- wr(dmi))) D Frn) (M(linits w(8iii)), Dval).
J(i)) (M(0 w(dmi)) )
Initialize MOBO with models, starting with Wini and prior data Ho (Winits innt Limi )
22: Compute initial Pareto optimal set Po using Ho.
23: for m = to iter do
24: Define EHVI using current GPS.
25:
Find 8m by optimizing EHVI. Equation 4
26:
L + [JannMMO ((6m)) Dvat), (M((0mm w6m))) Deal)."
z(K) (M(lmnic w(6m)); \
27:
Update GPs with new data (6m.2)
28:
Update P...
29:
end for
30:
Collect 5t arg max, L.,)
to adjust the scales, utilizing the lowest value of single model performance obtained after the convergence point in the trajectory collection of members, rounded the nearest value, as the minimum. The maximum values were determined by adding 0.1 for metric and 1.0 for loss respectively to this minimum value for use as the maximum. there is a more critical metric or criterion, one can freely modify the optimization by placing weights according the user's 's intent. No additional weights were applied in our experiments.
B Proposed Algorithm
In this section, we provide an overview of the complete process of BOMF encapsulated the algorithm. Algorithm systematically incorporates all three steps: 1) hyperparameter search via BO as detailed $ 4.2, 2) fusion member sampling as outlined 4.1, and 3) identification of optimal 8* and model fusion through MOBO in $ 4.3.
20

---

WWg
0.53
17.01
SWA
0.46
SWA
14.29
0.38
11.57
0.31
8.85
W1
Wg
W1
W1
0.23
(a) Loss surface of ViT-B/16.
(b) Metric surface of ViT-B/16.
Figure 4: Visualization of the loss landscape parameters (a) and the metric landscape parameters (b) for the vision task. The metric is accuracy error. We fine-tune the ImageNet-21K pre-trained ViT-B/16 [15] model on the Caltech-101 dataset. The members of the SWA for each figure are denoted as w W2, W3. Here we can see similar trends with the ResNet-50 case.
Table 9: Spearman's s rank correlation coefficient value of (a) ResNet18 model on the CIFARIO dataset, (b) ViT-B/16 model on the Caltech101 dataset, (c) ROBERTa-base model on the SST-2 dataset, (d) T5-basc model on the SQuAD2.0 dataset, and (c) LLaMA2-7B model on the E2E dataset. Here we used 15 fine-tuned weights for each task measure the Spearman's 's rank correlation. higher value indicates higher correlation between metric and loss value.
METRIC
(A)
(B)
(C)
(D)
(E)
SPEARMAN'S S RCC
0.6182
0.6558
0.10
0.2150
0.2286
C
Additional Experiments
In this section we demonstrate additional experiments not included in the main article.
C.1 Additional Experiments on Loss and Metric Landscapes
We also explored the potential loss metric discrepancy in PLMs, which might originate from the inherent features of transformer attention or from the use of adaptive optimizers. To analyze deeper, visualize the loss-metric surface of the VIT-B/16 model per-trained on ImageNet-2lk, using Adam optimizer which the same as the optimizer our language model. According to Figure 4, unlike in PLMS, the optimal points for loss and performance metrics in the ViT-B/16 model were aligned, indicating a distinct behavior between language and vision transformers in this context. Additionally, we have also undertaken the measurement of Spearman's 's rank correlation between loss and metric values across a range of models and tasks in both NLP and CV domains. Table clearly shows that the correlations in NLP tasks are less than in CV tasks.
C.2 Additional Experiments on Hyperparameter Alignment
In experiments utilizing LORA, aimed verify whether the optimal hyperparameters align when using a smaller rank to reduce costs, compared to using a larger rank. Figure demonstrates that the optimal batch size and learning rate align when the number of LoRA ranks varies. In addition to these hyperparameters, we investigated the potential alignment of the beta parameter of the Adam optimizer, which is the standard optimizer for training large language models, as well as the scheduler.
Figure 6 and Figure 7 indicate that, with the exception of the scenario where all layers are frozen, the optimal beta parameter of the Adam optimizer consistently aligns regardless the number of frozen layers the LoRA rank. Figure 9 and Figure 10 suggest that the optimal points of different learning rate schedules can be aligned according to the number of frozen layers.
These results indicate that by employing lightweight model, we can identify the optimal hyperpa- rameters, thereby simplifying the hyperparameter optimization process and reducing computational time.
21

---

Valid metric (%)
85.5
Valid metric (%)
91
Rank
Optimal values
90
85.0
Rank
Optimal values
89
84.5
Rank 16
84.0
Rank
87
86
83.5
85
83.0
8
10 10
10
Batch size
Learning rate
(a) Validation metric results for the varying batch size and the number of LoRA rank.
(b) Validatoin metric results for the varying learning rate and the number of LoRA rank.
Figure 5: Validation loss and metric (F1 score) results for the varying hyperparameter ((a) batch size, (b) learning rate) and the number of LoRA rank for the ROBERTA MRPC dataset. (a) and (b) indicate that the optimal hyperparameters consistently align well across different numbers of LORA rank.
75
0.68
70
Valid metric
0.66
Optimal values
1o55
65
Freeze layers
0.64
Freaze 2 layers
60
Freeze layers
0.62
Cotimal values
55
0.60
all layers Freeze layers
0.58
Freeze layers
50
fine-tuning
0.963
0.972
0.981
0.99
0.999
0.963
0.972
0.981
0.99
0.999
Adam Beta
Adam Beta
(a) Validation metric (Accuracy) for varying beta pa- rameters and frozen layers.
(b) Validation loss for varying beta parameters and frozen layers.
Figure 6: Results for the GoBERTa-base model on the RTE dataset, (a) and (b) indicate that the optimal hyperaarameters align well across different numbers of frozen layers, except when all pre-trained layers are frozen.
C.3 Additional Ablation Experiments
Metric Function in Multi-Objective Bayesian Optimization. In the 3, we pointed the issues of existing methods that perform fusion based solely on loss, particularly due to the discrepancy between metric and loss in language models. We demonstrated that our method outperforms the learned-swA method, which relies only on loss. To strengthen our claim, we additionally compared the loss and metric values between optimization processes with and without metrics. Specifically, we examined the loss and metric values of weights along the training trajectory between the initial point and the optimized point of Learned SWA and BOMF. Figures 9 and 10 show that BOMF generates complex trajectories for both loss and metric, exploring solutions based on both criteria. In contrast, Learned s SWA, relying solely on loss function, gets trapped local minima around the starting point, failing to discover the optimal solution. This suggests that BOMF's exploration property and the inclusion of metrics help escape local minima and discover more robust solutions.
Loss Function in Multi-Objective Bayesian Optimization. BOMF adopts both loss and metric as objectives when optimizing 6. This approach is based on the understanding that the loss provides macroscopic guide for overall metric performance. As seen Figure 1, solution SWA is optimal on the loss surface but not the metric surface. In contrast, BOMF performs well on both the loss and metric surfaces. These results indicate that BOMF ensures a high correlation between loss and metric values, preventing overfitting to either loss or metric during validation and enhancing the robustness
22

---

54
Optimal vallues
Rank
0.694
Rank
Valid metric
52
Rank
Valid loss
0.693
50
0.692
Optimal values
48
0.691
Rank
Rank 4
0.690
0.963
0.972
0.981
0.963
0.972
0.981
0.99
Adam Beta
Adam Beta
(a) Validation metric (Accuracy) for varying beta rameters and LoRA ranks.
(b) Validation loss for varying beta parameters and LoRA ranks.
Figure 7: Results for the RoBERTa-base model on the RTE dataset. (a) and (b) highlight that optimal hyperparameters align well across different numbers of LoRA ranks, emphasizing the importance of parameter tuning.
75
0.70
0.69
70
Valid metric
65
Valid loss
0.68
0.67
Optimal
Freeze all layers
0.66
layers
60
Optimal vallues
Freeze 4 layers
Freeze layers
0.65
Freeze 2 hyees
55
Freeze 2 layers
fine-turiing
layers
50
Freeze layers
0.63
Full fine-tune fine-tuning
0.62
cosine
linear
step
constant
cosine
linear Schedule
step
constant
Schedule
(a) Validation metric (Accuracy) for varying learning rate schedule methods and frozen layers.
(b) Validation loss for varying learning rate schedule methods frozen layers.
Figure 8: Results for the ROBERTa-base model the RTE dataset. (a) and (b) demonstrate that optimal hyperparameters are consistent across different numbers of frozen layers, indicating the critical role of hyperparameter choices.
of the weights on the test dataset. This clearly shows that including the loss function provides more useful guidance than optimizing exclusively in complex and sharp metric landscape.
Moreover, as demonstrated in Tables 10 and I, our approach, which utilizes both loss and metric, improves the correlation between them. This leads reaching the point of optimal performance, as shown in Tables 14, 15 and 17. These findings support that BOMF outperforms Bayesian optimization using a single metric without the loss function.
Multiple Metric Functions in Multi-Objective Bayesian Optimization. For tasks with multiple metrics, we optimized using all the available metrics. Therefore, it is important to investigate how multiple metrics impact the optimization process. In Table 4, we validated this through performance evaluations, but we also assessed changes in correlation. Table 11 shows that optimizing with single metric can weaken the correlation between the loss and other metrics.
Impact of the Better Trajectories in Model Fusion Performance BOMF aims construct the best-performing single model through model fusion within parameter space. As detailed in Š 4. different combinations fine-tuning hyperparameters (such as learning rate and batch size) yield varied generalization performances after the fine-tuning process. Therefore, to identify the best- performing single model after the model fusion, we must first determine the optimal hyperparameters that yield the best-performing single model before the fusion. In Table 14, confirm that the
23

---

Learned SWA
0.32
Learned SWA solution
0.31
BOMF
BOMF solution
0.30
Loss
0.29
0.28
0.27
0.26
0.25
10
20
40
50
Epochs
Figure 9: Visualization of the evolution loss throughout the optimization phases. We conduct experiments on the RTE dataset with the ROBERTa-base model. Learned SWA, an optimization process without metrics, tends to converge to local minima close to the starting point. Conversely, BOMF exhibits more successful exploration, incorporating metrics, and ultimately discovers more robust solution.
Learned SWA
93.75
Learned SWA solution
93.50
BOMF
F1 Score
93.25
x
solution
93.00
92.75
92.50
92.25
92.00
10
20
30
40
50
Epochs
Figure 10: Visualization of the evolution of metric throughout the optimization phases. We conduct experiments on the RTE dataset with the ROBERTa- base model. Like the loss, Learned SWA tends converge towards local minima without enhancing metric performance after the convergence. In contrast, BOMF effectively navigates through explorations to discover nigh-performing solution.
performance the final BOMF, executed after finding good hyperparameters through outerbo, better than the fusion performance executed with hyperparameters obtained from grid search.
Performance of Conventional Optimization Strategies for Each Task. Additionally, we con- ducted experiments to directly compare the original optimizers with BOMF. We found the optimal hyperparameters for the original optimizers using BO and then performed experiments on subsets of the GLUE dataset using the ROBERTa model and the SAMSum dataset using the LLaMMA-78 model. The results presented in Table 12 clearly demonstrate that BOMF outperforms the best performance achievable with any of the original optimizers.
Additional Experiments with BOMF Using LLM-BBsed Evaluation To validate BOMF's perfor- mance under diverse evaluation metrics, we conducted experiments using ChatGPT-3.5-Turbe Turbb--ab-d approach. This method involves scoring by asking the LLM to assess similarity between generated responses and ground-truth answers. Using this, BOMF was benchmarked against other models. As shown in Table 13, BOMF consistently outperformed these baselines, highlighting its capability to adapt not only to traditional metries but also to newer evaluation techniques.
24

---

Table 10: Spearman's rank correlation coefficient value between loss and metric for the various optimization processes in the middle scale tasks. Here, we assess the correlation of the RoBERTa-baa model on the MRPC dataset (a) and the RTE dataset (b). Additionally, between loss and metric of the T5-base model on the SQuAD2.. dataset using the Fl score (c) and the EM (d) metrics. We conduct evaluations for (c) and (d). Here, Loss BO SWA and Metric BO SWA denote the approach where we exclusively employ either the loss function the metric during the MOBO process, respectively.
examine the correlation
a total of 100 sampled sets for (a) and (b), and 30
METRIC
(A)
(B)
(C)
(D)
BASELINE (HPBO)
0.1640
0.3256
0.2076
0.3021
Loss BO SWA
0.1903
0.4334
0.4210
0.4710
METRIC BO SWA
0.2465
0.3755
0.5333
0.5000
BOMF
0.5189
0.5886
0.6500
0.6000
Table 11: Spearman's rank correlation coefficient value between loss and metric for the various optimization processes in large-scale tasks. We evaluate the LLaMA2-7B on the SAMSum dataset. Here, Loss BO and Metric BO SWA denote the approach where we exclusively employ either the loss function or the metric during the MOBO process, respectively.
METRIC
R1
R2
RI
AVG.
BASELINE (HPBO)
0,4024
0.2000
0.1030
0.2351
Loss BO SWA
0.5335
0.1306
0.1197
0.2613
METRIC BO SWA
0.5330
0.6060
0.6060
0.5817
BOMF
0.6445
0.6152
0.6137
0.6245
Table 12: Results of BOMF and Other Neural Network Optimization Strategies on Various Datasets. A higher value better for all the metrics.
GLUE (ROBERTA-BASH,
SAMSUM (LLAMA2-7B)
OPTIMIZATION STRATEGY RTE (ACC)
MRPC (F1) SST2 (ACC)
R1
R2
RL
SGD
54.37
88.90
90.11
50.63
19.34
41.24
ADAMW
78.49
92.72
94.41
52.25
27.52
44.03
SWA
76.70
91.73
94.75
52.21
27.58
44.04
SAM
77.34
92.56
94.77
52.75
28.55
44.16
BOMF
81.40
93.90
95.54
53.07
28.61
44.40
This robustness is further reinforced by BOMF's design, which combines loss with multiple metrics, enhancing its generalizability across unseen metrics. Notably, BOMF leverages BO for tuning combination coefficients, optimizing based on evaluation values rather than backward processing through the metrics themselves. This approach allows BOMF efficiently optimize coefficients across complex evaluation settings, as shown in the ChatGPT BOMF column in Table 13.
These findings underscore BOMF's adaptability, demonstrating its resilience across varied evaluation frameworks, including those generated by LLMs.
C.4 Full Experimental Results
In this section, we present full experimental results encompassing text classification tasks for the Masked Language Model (MLM) and question answering, summarization, and dialogue gen- eration tasks for the autoregressive LLM. In all tables, the best performance is indicated with boldfaced underline, and the second- value is represented with underline for methods that use the same best hyperparameters by $ 4.2. The final column 'Avg.' provides summary of overall results for each method across various datasets or metrics. The terms 'Full' and 'Freeze' in Tables 14 and 15 specify the exploration optimal hyperparameters using either the entire model or a model with half of its weights frozen, discussed in $ 4.2. Similarly, the terms 'Rank 64' and 'Rank 4' in Tables 16 and 17 denote that we use the Rank 64 or the lightweight Rank 4 version of LoRA model for the hyperparameter search, respectively.
25

---

Table 13: Evaluation Results Using ChatGPT-3.55Turie (a) Baseline, (b) SWA, (c) Greedy SWA, (d) Learned SWA, (e) BOMP, and (f) ChatGPT BOMF.
EVALUATION PROMPT EXAMPLE
YOU ARE AN AUTOMATED GRADING ASSISTANT HELPING A TEACHER GRADE STUDENT ANSWERS.
THE CORRECT SUMMARY FOR THIS TEXT IS: <GROUND-TRUTH>
A STUDENT SUBMITTED THE SUMMARY: <PREDICTION>
THE STUDENT'S SUMMARY MUST BE CORRECT AND SPECIFIC BUT NOT OVERCOMPLETE. SMALL DIFFERENCES FORMATTING SHOULD NOT BE PENALIZED. ON SCALE FROM O 100, WHERE 0 MEANS COMPLETELY INCORRECT AND 100 MEANS COMPLETELY CORRECT, HOW SIMILAR IS THE STUDENT'S SUMMARY TO THE GROUND TRUTH? PLEASE PROVIDE ONLY NUMERICAL SCORE WITHOUT ANY EXPLANATION.
(A)
(B)
(C)
(D)
(E)
(F)
GRADE 70.74
71.92
72.05
71.80
72.64
73.18 18
Table 14: Full Results on Text Classification Task Using RoBERTa- Results of BOMF and baseline methods with GLUE benchmark datasets. ACC and Fl denote metrics for each dataset, representing accuracy and F1 score, respectively.
AVG.
METHOD
RTE (ACC)
MRPC (FI)
SST (ACC)
QNLI (ACC)
QQP (FI)
MNLI (ACC)
GRID FINE-TUNE
77.78 79.66
92.39
94.87
92.62 92.79
88.16
87.41
88.93
GRID SEARCH INNER BO
92.48
HPBO (FULL.)
78.57
92.78
95.11
93.01
88. 58
87.46
89.22
SWA
78.62
92.24
95.42
92.81
88.49
87.41
87.
GREEDY SWA
80.70
92.83
95.54
93.16
88 64
87.45
89.72
92.94
TWA
81.2.
91.58
95.54
93.00
87.85
87.42
89. 44
(OURS)
93.37
94.83
90.30
HPBO (FREEZE)
78.49
92.72
94.41
92.71
88
87.45
SWA
76.70
91.73
94.75
93.21
88.35
87.44
88.70
GREEDY SWA
80.01
93.38
95.20
93.30
88.65 $8.67
87.84
89.73
78.79
93.56
93.03
TWA
78.29
92.16
94.87
92.86
88 .5
87.44
89.03
BOMF (OURS)
81.40
93.90
93.54
93.50
88.68
87.86
90.15
Text Classification. Table demonstrates the consistently better performance of BOMF over other baselines that employ the same best hyperparameters. These findings affirm the effectiveness of BOMF the context of single-metric NLP tasks.
Question Answering. Table 15 presents the complete experimental results for the question- answering task. BOMF consistently surpasses other baselines utilizing the same best hyperparameters. These outcomes prove the effectiveness BOMF the realm of multi-metric NLP tasks, improving both F1 and EM metrics concurrently.
Summarization. Table 16 provides empirical evidence that BOMF achieves the highest average performance across evaluated metrics. While Learned SWA and Greedy SWA exhibit the best performance results in RI and R2 metrics, respectively, they experience declines across other metrics. However, our approach demonstrates consistent improvement across all metrics. These results prove the efficacy BOMF multi-metric NLP tasks.
Dialogue Generation. Table 17 provides full experimental results for the dialogue generation task. The results for BOMF showcase the highest scores for all metrics in both cases of rank 64 and rank 4. Despite the conflicting correlations between BLEU and METEOR metrics [29, 1]. BOME achieves comprchensive improvements across all metrics, distinguishing itself from other basclines that fail to
26

---

Table 15: Full Results Question-Answering Task with T5-base. Results of BOMF and baseline methods with SQuAD2.0 dataset. F1 and EM denote the score and Exact Match, respectively.
METHOD
EM
AVG.
HPBO (FULL)
78.28
73.29
75.79
SWA
80.31
74.85
77.58
GREEDY SWA
80.63
75.44
78.03
LEARNED SWA
80.65
74.23
77.44
TWA
80.29
74.79
77.54
BOMF (OURS)
80.82
75.79
78.31
HPBO (FREEZE)
78.19
73.43
75.81
SWA
81.21
75.63
78.42
GREEDY SWA
81.73
76.20
78.97
LEARNED SWA
81.24
75.65
78.45
TWA
81.21
75.61
78.41
BOMF (OURS)
81.82
76.21
79.01
Table 16: Full Results on Summarization Task with LLaMA2-7B: Results of BOMF and baseline methods with SAMSum dataset. R1, R2, and RL denote Rouge-I, 1, Rouge-2, and respectiveseeetveeell
METHOD
R1
R2
RL
AVG.
HPBO (RANK 64)
52.66
28.22
44.33
41.73
SWA
51.81
27.61
43.55
40.99
GREEDY SWA
53.40
28.06
43.31
41,49
LEARNED SWA
52.93
28.97
44.04
41.98
BOMF (OURS)
53.40
28.78
44.38
42.19
HPBO (RANK 4)
52.25
27.52
44.03
41.27
SWA
52.21
27.58
44.04
41.28
GREEDY SWA
53.06
28.99
44.03
42.03
LEARNED SWA
52.10
26.76
44.13
41.00
BOMF (OURS)
53.07
28.61
44.40
42.03
Table 17: Full Results Dialogue Generation Task with LLaMA2-77 Results of BOMF and base- line methods with E2E dataset. B, M, and RL denote BLEU, METEOR, and Rouge-L, respectively.
METRIC
AVG.
HPBO (RaNK 64)
63.09
80.17
65.71
69.66
SWA
62.95
80.07
65.53
69.52
GREEDY SWA
63.86
80.00
66.63
70.
LEARNED SWA
63.65
79.76
66.34
69.92
BOMF (OURS)
64.70
80.91
67.53
71.05
HPBO (RANK 4)
63.46
81.26
67.09
70.60
SWA
63.50
81.02
67.08
70.53
GREEDY SWA
63.27
81.17
67.68
70.71
LEARNED SWA
64.42
79.04
66.09
69.85
BOMF (OURS)
64.81
81.28
67.70
71.26
achieve such comprehensive enhancements. This proves the efficacy of our multi-objective method in effectively considering multiple metrics with conflicting correlations.
27

---

D
Societal impact
BOMF does not directly have any positive or negative socictal impacts since our algorithm for fine-tuning and model fusion. However, in the sense of developing LLMs, we can argue the societal impacts of our work. On the positive side, our work can improve the productivity of human beings, e.g., reduction of repeating tasks, and discover new scientific knowledge, e.g., artificial intelligence for scientific discovery. On the other hand, as negative societal impacts, fine-tuning LLMS on downstream tasks can still consume significant compute resources, which leads to climate change. Moreover, since our fine-tuning process aims to optimize specific metrics, there can be a potential risk of optimizing towards malicious metrics such aggressiveness and violence. Therefore, should be aware of potential unethical outcomes and consider responsibility in selecting and optimizing these metrics.
E
Safeguards
We use publicly available benchmarks and open-source models widely recognized in the L.M research community. Additionally, we do not release proprietary or new datasets or models that could cause the risk of misuse. Although work potentially has possibility undertake inherent misuse that is derived from public benchmarks and open-source models, we think that our method itself does not pose high risk of misuse.
28

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: As outlined in $ 1, our approach proposes model fusion method that leverages RO techniques. The empirical results in 6 demonstrate that BOMF outperforms other baseline methods.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: discussed the limitations BOMF and proposed future research to provide theoretical reasoning for our findings.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
29

---

Answer: [NA]
Justification:
We don't have any theoretical results in our paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provided detailed explanation of the algorithm for BOMF in $ 4 and discussed experimental details in Appendix A. Also, we provide detailed algorithms in Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
cases, in which
5. Open access to data and code
30

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We offer comprehensive experimental settings, including details about the data, in Appendix C, and our code is provided in the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
(oglic/guides/Coeeeee
iereinsssooPoooli onPolicyl icy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ /nips. cc/public/// ht////uusss.../ee CCodeSubmemsa/ cerbbbmissioen ssi.nnoliccoooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide detailed experimental settings in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Due the need fine-tune Large Language Models across various tasks, conducting experiments multiple times would be excessively computationally expensive in our work.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
31

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
We provide our computational resources in Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: /tterr//neeu.....eee http://errpp.cccpplliiiccpc h
Answer: [Yes]
Justification: Our paper aligns with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: Yes]]
Justification:
We discuss the societal impacts regarding BOMF in Appendix D.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
32

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
We discuss the safeguards regarding BOMF in Appendix E.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We address the code, data, models, and packages utilized in experiments in Appendix A.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
33

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. cow datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification:
We do not provide any new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/codeemmodel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsoureing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not include crowdsourcing or research involving human sub- jects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation,
or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not include crowdsourcing or research involving human sub- jects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
34

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
35


Reviews:
[{"review_id": "nAHLlFmJYr", "reviewer": "Reviewer_ut18", "summary": "This paper introduces Bayesian Optimization Model Fusion (BOMF), a method for improving fine-tuning of pre-trained language models. BOMF addresses the challenge of selecting optimal models and hyperparameters by utilizing multi-objective Bayesian optimization to consider both loss and desired metrics during model fusion. It employs a two-stage approach, first optimizing hyperparameters for fine-tuning, then focusing on model fusion. Experiments across various NLP tasks demonstrate significant performance gains using BOMF, showcasing its effectiveness in both Natural Language Understanding and Generation.", "questions": "1) what if the metrics to be optimized can not be directly computed? How to deal with complex evaluation criteria (e.g. grounding, coherence) commonly seen in LLM evaluation? \n2) How the method perform in recent models such as Llama family?", "limitations": "Limited Performance Gain: The observed improvements from the proposed BOMF method are relatively small.\n\nOutdated Models and Datasets: The study relies on older base models (RoBERTa and T5) and simple datasets that may not reflect the current state-of-the-art in terms of model architectures, size, and task complexity. \n\nIncomplete Evaluation for Generative Tasks: The evaluation of generative tasks lacks the use of LLM-based evaluation or human evaluation, which are crucial for assessing the quality and diversity of generated text. Relying solely on computational metrics might not capture the nuances and potential biases of generated outputs.\n\nQuestionable Approach: The use of model fusion to address the discrepancy between loss and desired metrics is not necessarily the most effective solution. Alternative approaches like Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO) can directly optimize towards desired metrics, potentially bypassing the need for fusion altogether.", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Using Multi-Objective Bayesian Optimization (MOBO) that considers both metrics and loss functions for model fusion seems like an interesting idea.", "weaknesses": "Limited Performance Gain: The observed improvements from the proposed BOMF method are relatively small.\n\nOutdated Models and Datasets: The study relies on older base models (RoBERTa and T5) and simple datasets that may not reflect the current state-of-the-art in terms of model architectures, size, and task complexity. \n\nIncomplete Evaluation for Generative Tasks: The evaluation of generative tasks lacks the use of LLM-based evaluation or human evaluation, which are crucial for assessing the quality and diversity of generated text. Relying solely on computational metrics might not capture the nuances and potential biases of generated outputs.\n\nQuestionable Approach: The use of model fusion to address the discrepancy between loss and desired metrics is not necessarily the most effective solution. Alternative approaches like Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO) can directly optimize towards desired metrics, potentially bypassing the need for fusion altogether.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your constructive feedback. We have answered your questions and concerns in this response. Please let us know if you have any follow-up questions.\n\n**[W1] The observed improvements from the proposed BOMF method are relatively small**\n\nThe improvement of BOMF is not negligible compared to other baselines. First, we would like to emphasize that our method consistently outperforms various baselines across a wide range of task types (i.e., classification, question answering, and the medical domain) and models (i.e., T5, LLaMA3). While the degree of improvement might appear marginal in some cases, it is essential to recognize that the extent of improvement, whether small or large, is relative. The fact that our method consistently achieves superior performance in nearly every case serves as an absolute metric that is universally recognizable. Our method consistently outperforms the baselines in these diverse scenarios. Specifically, BOMF achieved a 13%, 14%, 10% error improvement in the GLUE tasks, SQuAD task and KorMedQA tasks, respectively. \n\nFurthermore, the recent fusion research [1,2,3] in various scenarios has shown similar tendency to our results, showing that even if the improvement in specific tasks seems marginal, they exhibit overall superior performance compared to the baseline, proving the necessity of such methods. In particular, even when compared to these methods, our method demonstrates improved performance in almost all experimental results, indicating its effectiveness.\n\n**[W2, Q2] The study relies on older base models (RoBERTa and T5) and simple datasets. How did the method perform in recent models such as the Llama family?**\n\nWe have already conducted experiments using Llama-2 and Llama-3 in Sec 6.2 of the main paper, and included the results for summarization, Korean multi-choice medical question answering, and dialogue generation. In most cases of these three experimental settings, our method outperforms other baseline methods. For detailed experimental setting and results, refer to Table 2 in the main text and Tables 15 and 16 in the appendices.\n\n\n**[W3, Q1] The evaluation of generative tasks lacks the use of LLM-based evaluation or human evaluation. How to deal with complex evaluation criteria commonly seen in LLM evaluation?**\n\nThank you for suggesting additional evaluations on generative tasks. We would like to address this question from two major perspectives:\n\n1. When a large language model (LLM) is used to evaluate the outputs of various models, BOMF demonstrates superior performance compared to the existing baselines.\n\n2. BOMF can utilize more complex evaluation criteria such as human evaluation or LLM assessment in the process of model fusion.\n\nRegarding the first one, following your suggestion, we conducted an evaluation with a ChatGPT-turbo-3.5-based approach. This evaluation method involves generating scores by asking the LLM to assess the similarity between the generated responses and the ground-truth answers. Using this, we compared BOMF's performance with other baselines. Here, BOMF refers to the model optimized with the R1, R2, and RL metrics. As shown in Table R.1, even when optimized with this specific metric, BOMF outperformed other baselines. This indicates that our approach not only excels with traditional metrics but also adapts well to the latest evaluation methods. \n\nAdditionally, the consideration of both loss and multiple metrics helps the model remain robust across various unseen metrics. Furthermore, since BOMF uses BO to optimize combination coefficients, it requires only the evaluation metric values corresponding to each set of combination coefficients to update them. This means that the optimization process does not rely on a backward process through the metric; as long as evaluation values are available, BOMF can optimize regardless of the complexity of the evaluation procedure. To illustrate this, we conducted optimization using metrics evaluated by LLMs, denoted as the column of ChatGPT BOMF in Table R.1. The result shows that BOMF can optimize the coefficients and achieve performance improvements even with these complex metrics. \n\n**[W4] RLHF and DPO can directly optimize towards desired metrics, potentially bypassing the need for fusion altogether.**\n\nWe agree that methods like RLHF can be used to optimize desired metrics. However, we want to emphasize that our method complements rather than contradicts methods like RLHF or DPO. For instance, the hyperparameters used in RLHF or DPO can be optimized more efficiently using BO with a lower rank or fewer layers. Also, when performing RLHF or DPO, multiple checkpoints can be generated during training procedure. Typically, the best validation performance checkpoint is selected from these. However, with BOMF, these checkpoints can be fused to produce better results. By combining multiple checkpoints through BOMF, better generalization performance can be achieved compared to selecting a single checkpoint. This capability is well illustrated in our experiments, as shown in Tables 2 and 4. \nAnother important point is that BOMF can consider multiple metrics simultaneously. While methods like RLHF or DPO require detailed preference settings for each metric in different scenarios, BOMF automatically balances various metrics and loss functions to find the optimal combination. To demonstrate that RLHF and BOMF can indeed be combined, we conducted additional experiments. According to Table R.2, performing various fusion methods on RLHF training checkpoints showed that BOMF outperformed other methods.\n\n**References**\n\n[1] Wortsman, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. ICML, 2022.\n\n[2] Weng, R., et al. G-tuning: Improving generalization of pre-trained language models with generative adversarial network. ACL, 2023.\n\n[3] Malladi, S., et al. Fine-tuning language models with just forward passes. NeurIPS 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y1K2ICxVxV", "reviewer": "Reviewer_csBg", "summary": "This paper proposes \"Bayesian Optimization Model Fusion\" (BOMF), which is a method to fuse model weights using Bayesian Optimization over a set of metrics for a given target task. The authors motivate the necessity of BOMF for model fusion in large language model fine-tuning by providing evidence that there exists a large misalignment between the loss and metric surface in NLP tasks for pre-trained models. Additionally, the authors demonstrate initially that, for a given target task, the optimal set of hyperparameters for fine-tuning appears to remain consistent across different levels of fine-tuning capacity (# of frozen layers or LoRA rank).\n\nDrawing on these insights, the total proposed method operates in 3 steps:\n- First, Bayesian Optimization is used to select the optimal set of hyperparameters for a given task and setting, optimizing the hyperparameters over the target-task metric, rather than validation loss. Additionally, drawing on their earlier observation, the authors perform this BO procedure on a lightweight model (freezing lower layers of the pre-trained model), and then applying the found hyperparameters to the larger model.\n- Then, the parameters are fine-tuned on the target-task, and various checkpoints during this fine-tuning procedure are uniformly sampled (towards the end of training) for model fusion.\n- Finally, to select the best set of weights to use when fusing the models together (i.e. taking a weighted average of the sampled checkpoint parameters), multi-objective Bayesian Optimization is used to optimize a set of weights that maximizes the Pareto frontier hyper-volume of a set of metrics (as well as the loss) of the target-task validation data.\n\nThe authors demonstrate that this procedure results in a fine-tuned model that outperforms standard fine-tuning and other comparable model fusion techniques. Additionally, the authors provide an ablation study, showing the benefit of optimizing for multiple objectives in the model fusion weights, and", "questions": "n/a", "limitations": "Yes", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- While the individual components of the proposed method are not novel in and of themselves, the proposed method results in a new technique for LLM fine-tuning that appears to convincingly improve upon previous relevant methods.\n- The authors provide ablation studies for their method, motivating each step of their method.\n- The paper provides some novel and interesting insights regarding the alignment between loss and metric surfaces for NLP tasks, and highlights a flaw in loss-based aggregation and fusion methods in LLM fine-tuning, before proposing a solution.", "weaknesses": "- The 3 individual components of the proposed method are fairly distinct and disconnected from one another. As a result, it feels as though there are two distinct contributions that are not necessarily related (hyperparameter BO using lower capacity models, and BO for model fusion).\n- Because of this disconnect, one of these contributions (hyperparameter transfer using BO) feels less important than the other (Multi-Objective BO for model fusion), resulting in the MOBO for MF feeling under-explored as the \"main contribution\" (for instance, it would be interesting to see how BOMF works for \"model soups\", models obtained from different training runs). Instead, both contributions get comparatively equal attention in this work.\n- The presentation of the insights used to motivate the proposed method (both the misalignment between metric and loss surface, and the alignment of optimal hyperparameters over capacity) are only showed qualitatively, i.e. over a single dataset and model setting. It is not clear whether these findings generalize well across other NLP tasks, although it could be argued that the success of the proposed method indicates that they generalize well.\n- While there are limitations due to the size of the model, unless I am mistaken, all results are taken from a single run, e.g. results are not averaged over different random seeds and it is therefore less clear how meaningful some of the improvements that BOMF makes over it's baselines are.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your constructive feedback. We have answered your questions and concerns in this response. Please let us know if you have any follow-up questions.\n\n**[W1, W2] It feels as though there are two distinct contributions that are not necessarily related. Because of this disconnect, one of these contributions feels less important than the other, resulting in the MOBO for MF feeling under-explored as the “main contribution”.**\n\nTo simplify the notations, we will refer to BO-based hyperparameter selection as `outer BO` and BO-based coefficient search as `inner BO`. The bottom line is that ***the outer BO process and the inner BO process are coupled with each other***, so they were not conducted for different purposes. \n\nAs outlined in Section 1, BOMF aims to construct the best-performing single model through model fusion within a parameter space. As detailed in Section 4.2, different combinations of fine-tuning hyperparameters (such as learning rate and batch size) yield varied generalization performances after the fine-tuning process. Figure 3 demonstrates that the generalization performances of models before and after fusion align well. Therefore, to identify the best-performing single model after the model fusion, we must first determine the optimal hyperparameters that yield the best-performing single model before the fusion. \n\nFor a simpler explanation, let us consider two fine-tuning hyperparameter configurations, H1 and H2. Figure 3 illustrates that if the performance of the fine-tuned solution from the training trajectory with H1 surpasses that of H2, then the performance after the model fusion with fusion members S1, sampled from the trajectory with H1, remains higher than that of the fusion members S2, sampled from the trajectory with H2. This finding, coupled with the non-differentiable nature of metric functions, underscores the necessity of identifying the best hyperparameter configurations through outer BO to achieve BOMF's ultimate objective with inner BO.\n\n\n**[W3] Both the misalignment between metric and loss surface, and the alignment of optimal hyperparameters over capacity are only shown over a single dataset and model setting.**\n\nIn our paper, we demonstrated that both full model fine-tuning and LoRA fine-tuning exhibit 1) metric and loss misalignment and 2) optimal hyperparameters alignment even when freeze layers or ranks change. However, we agree that extending these experiments to larger-scale models would strengthen our findings. Therefore, we conducted additional experiments on the Llama-2 model to verify if 1) and 2) still hold.\n\nFirstly, regarding point 2), the alignment of hyperparameters can be easily observed in Figures A.1. For point 1), the misalignment between loss and metrics is evident in Figure A.2. To further demonstrate the misalignment between loss and metrics, as well as among different metrics, we measured Spearman’s correlation using the loss and metrics of 20 sampled models. Also, we empirically demonstrate that BOMF significantly improves this correlation by considering both loss and metrics simultaneously. Please refer to Tables 9, 10, and 11 in Appendix C for full experimental results. We have included a portion of these results here. According to Table R.2 (c), (d), and (e), language tasks involving the Llama model show significantly lower correlation compared to vision tasks (a) and (b). Then, Table R.3 shows that BOMF successfully increases the correlation between loss and metrics compared to other baseline methods.\n\n**[W4] All results are taken from a single run.**\n\nThank you for your insightful comment regarding the use of random seeds. We agree that running multiple experiments with different seeds is crucial for ensuring the reliability and completeness of the results. Accordingly, we will provide experimental results obtained using multiple seeds for each experiment. Due to computational limitations during the rebuttal period, we were only able to add results for the medium-sized language models in Table 1 in the main paper, using five seeds per method and dataset. This result can be found in Table R.6 and Table R.7.\n\nAdditionally, we plan to conduct further experiments on large language models and update all tables with results from multiple seeds in the camera-ready version of this paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Bjz5KbdMVp", "reviewer": "Reviewer_WU7w", "summary": "The paper presents a novel approach to model fusion through Bayesian Optimization for fine-tuning pre-trained language models on downstream tasks. The authors address the challenges associated with hyperparameter selection and the discrepancy between loss and metric landscapes during the fine-tuning process. They propose a two-stage Bayesian optimization framework that first identifies optimal hyperparameters for fine-tuning and then uses multi-objective Bayesian optimization to find the best combination of models in the parameter space.  The paper provides a comprehensive experimental evaluation to demonstrate the properties and benefits of the proposed approach.", "questions": "See weakness.", "limitations": "The authors have adequately discussed the limitations of their work.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a novel approach to model fusion through Bayesian Optimization (BOMF), which is a creative application of Bayesian optimization to the problem of fine-tuning pre-trained language models. The originality lies in the development of a two-stage Bayesian optimization process that addresses the specific challenges of hyperparameter tuning and model combination in NLP tasks. The method is innovative as it optimizes both the loss and the desired metrics simultaneously, which is a new perspective in the context of model fusion. The empirical finding that a large discrepancy exists between loss and metric in fine-tuning LLM is valuable to the community.\n\n2. The quality of the paper is evident in its rigorous experimental design and comprehensive evaluation. The authors have tested BOMF across various NLP tasks and models, demonstrating its robustness and effectiveness. \n\n3. The paper is well-structured, with a clear presentation of the problem, the proposed solution, and the experimental setup. Overall, it is well-written and easy to follow.", "weaknesses": "1. There are no discussions on computational efficiency: Although the paper mentions a two-stage Bayesian optimization process, it could further elaborate on the computational efficiency of BOMF, especially when compared to traditional fine-tuning methods and the baselines. More detailed complexity analysis or comparisons could help readers understand the practicality of applying BOMF at scale.\n2. Hyperparameter sensitivity analysis: The paper could provide a more in-depth analysis of how sensitive BOMF is to the choice of hyperparameters for the Bayesian optimization process itself, e.g., the choice of kernel of Gaussian process regression. Understanding the stability of the method under different settings can be crucial for practitioners.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your constructive feedback. We have answered your questions and concerns in this response. Please let us know if you have any follow-up questions.\n\n**[W1] There are no discussions on computational efficiency: More detailed complexity analysis or comparisons could help readers understand the practicality of applying BOMF at scale**\n\nPlease refer to the general response for the analysis and additional experiments regarding the computational cost.\n\n**[W2] Hyperparameter sensitivity analysis: The paper could provide a more in-depth analysis of how sensitive BOMF is to the choice of hyperparameters for the Bayesian optimization process itself**\n\nSince the test suites used in this work require huge compute resources, it is difficult to provide hyperparameter sensitive analysis on Bayesian optimization processes. It is noteworthy that the hyperparameters of Bayesian optimization can be thought of as meta-hyperparameters, which should be tuned with multiple tasks and multiple domains. It implies that it necessitates utilizing more compute resources. In addition, from the standpoint of black-box optimization, the sensitive analysis assumes that the specifics of objective functions are known, which is generally infeasible in black-box optimization. Therefore, instead of analyzing the sensitivity of the hyperparameters of Bayesian optimization, we followed their conventional settings. All hyperparameters related to Gaussian processes are optimized via model selection. Matern 5/2 kernel for Gaussian processes and the logarithmic form of expected improvement are used.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yjQxQEnfUS", "reviewer": "Reviewer_JjKU", "summary": "The main contributions \n- Using Bayesian Optimization for hyperparameter search on LoRA’s for full-model fine-tuning \n- Using EVHI to learn coefficients to fuse models from within a checkpoint \n\nOther contributions \n- Finding discrepancy between metric and loss", "questions": "Major \n- How much extra time is required due to the sequential nature of HBPO compared to grid search? \n- Some discussion of the extra compute required for BOMF due to the extra evaluations. Does 75 iterations mean 75 evaluations are needed? \n- How is accuracy computed on RoBERTa? With a classifier head on top of the representations? Nowadays, for NLP, all tasks are cast as text-to-text and accuracy is done by comparing the log probs of the tokens corresponding to the different labels. \n- The claim that the best hyperparameters for fine-tuning LoRA can transfer to the full model are not that interesting for RoBERTa. Since RoBERTa is a pretty small model overall, LoRA is generally not needed and full-model fine-tuning is needed. LoRA is usually used for larger models, so showing this for the LLAMA-2 experiments would be interesting. \n\nMinor \n- Can you an algorithms description of MOBO in the main paper? Specifically, how is qNEHVI done? \n- Where is freeze in Table 1 and rank 4 in Table 2? (Mentioned in line 298/315) \n- What are the two variants of BOMF in Table 1 and 2?\n- Why are there are no grid fine-tune results in Table 2?", "limitations": "NA", "rating": 7, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Paper is well-written and easy to follow \n- Bayesian method for model fusion is interesting", "weaknesses": "- Empirical observations of discrepancy between metric and loss landscape are on one specific outdated model and not sure how accuracy is computed (but could be an outdated way). Same for hyperparameter alignment claim. See questions below \n- Extra compute required for BOMF \n- Performance improvements of BOMF are very small", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your constructive feedback. We have answered your questions and concerns in this response. Please let us know if you have any follow-up questions.\n\n**[W1, Q3, Q4] Empirical observations of discrepancy between metric and loss landscape, and hyperparameter alignment are performed on the RoBERTa model. Thus it requires results on Llama models. Also how does the accuracy computed on RoBERTa?**\n\nFirst, we want to clarify that we use a classifier head for the classification task because our task involves multiple-choice questions, allowing us to directly output an answer and measure accuracy. However, we want to emphasize that BOMF is not limited to this specific scenario. It can be applied to any model as long as the loss and metrics can be calculated. Since fusion does not require backpropagation, BOMF is equally applicable in situations where there is no classifier head. And in our paper, we demonstrated that both full model fine-tuning and LoRA fine-tuning exhibit 1) metric and loss misalignment and 2) optimal hyperparameters alignment even when freeze layers or ranks change. However, we agree that extending these experiments to larger-scale models would strengthen our findings. Therefore, we conducted additional experiments on the Llama-2 model to verify if 1) and 2) still hold.\nFirstly, regarding point 2), the alignment of hyperparameters can be easily observed in Figures A.1. For point 1), the misalignment between loss and metrics is evident in Figure A.2. To further show the misalignment between loss and metrics, as well as among different metrics, we measured Spearman’s correlation using the loss and metrics of 20 sampled models. Also, we empirically prove that BOMF significantly improves this correlation by considering both loss and metrics simultaneously. Refer to Tables 9, 10, and 11 in Appendix C for full experimental results. We have included a portion of these results here. According to Table R.2 (c), (d), and (e), language tasks involving the Llama model show significantly lower correlation compared to vision tasks (a) and (b). Then, Table R.3 shows that BOMF successfully increases the correlation between loss and metrics compared to other baseline methods.\n\n**[W2, Q1, Q2] How much extra time is required for BOMF.**\n\nPlease refer to our global response for answers.\n\n**[W3] Performance improvements of BOMF are very small**\n\nThe improvement of BOMF is not negligible compared to other baselines. First, we would like to emphasize that our method consistently outperforms various baselines across a wide range of task types (i.e., classification, question answering, and the medical domain) and models (i.e., T5, LLaMA3). While the degree of improvement might appear marginal in some cases, it is essential to recognize that the extent of improvement, whether small or large, is relative. The fact that our method consistently achieves superior performance in nearly every case serves as an absolute metric that is universally recognizable. Our method consistently outperforms the baselines in these diverse scenarios. Specifically, BOMF achieved a 13%, 14%, 10% error improvement in the GLUE tasks, SQuAD task and KorMedQA tasks, respectively.\n\nFurthermore, the recent fusion research [1,2,3] in various scenarios has shown similar tendency to our results, showing that even if the improvement in specific tasks seems marginal, they exhibit overall superior performance compared to the baseline, proving the necessity of such methods. In particular, even when compared to these methods, our method demonstrates improved performance in almost all experimental results, indicating its effectiveness.\n\n**[Q5] Can you provide an algorithm description of MOBO in the main paper? Specifically, how is qNEHVI done?**\n\nWe have already described the details of our algorithm in Sec A.1.3, B, and Algorithm 1 of the appendices. Due to the limited space of the NeurIPS submission, they should be located in the appendices. We will reorganize our contents in the final version.\n\nAs described in our manuscript, qNEHVI sequentially samples optimum candidates by maximizing the expected hypervolume improvement, which can be considered as an acquisition function in Bayesian optimization. At every iteration of MOBO, it chooses the maximizer of the alternative objective rather than one of multiple unknown target objectives. We will clarify it in the final version.\n\n**[Q6, Q7] Where is freeze in Table 1 and rank 4 in Table 2?**\n\nThank you for pointing out the confusion. The dagger symbol ($\\dagger$) in Tables 1 and 2 indicates whether low rank or freeze layers were used. Specifically, \"BOMF with dagger symbol ($\\dagger$)\" refers to cases where the best hyperparameters were determined using full rank or full layer settings, and these hyperparameters were then used for training before performing fusion. On the other hand, \"BOMF without a dagger symbol\" refers to cases where low rank or freeze layers were used to find the best hyperparameters. This information is clarified in the footnote on Page 8 of the paper.\n\n**[Q8] Why are there no grid fine-tune results in Table 2?**\n\nThe models used in Table 2 are LLMs, so even when tuning with LoRA, the forward path takes a considerable amount of time. This made it time-prohibitive to perform grid search, which requires a relatively large number of iterations to find the best-performing hyperparameters. However, to strengthen our argument in line with your suggestion, we are running the grid search in the remaining time and will present the results as soon as they are available.\n\n**References**\n\n[1] Wortsman, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. ICML, 2022.\n\n[2] Weng, R., et al. G-tuning: Improving generalization of pre-trained language models with generative adversarial network. ACL, 2023.\n\n[3] Malladi, S., et al. Fine-tuning language models with just forward passes. NeurIPS 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their valuable and constructive comments.

The reviewers acknowledge that the paper is well-written, easy to follow, and recognize the originality of using Bayesian optimization for model fusion in BOMF (R-ut18, R-csBg, R-WU7w, R-JjKU). Reviewers also recognize the paper for providing a thorough study of this method, motivating each step of the approach, and offering new and interesting insights into the alignment between loss and metric surfaces for NLP tasks (R-csBg, R-WU7w, R-JjKU). Additionally, reviewers agree that the authors clearly demonstrate the quality of the paper through rigorous experimental design and comprehensive evaluation. They acknowledge that the authors test BOMF across various NLP tasks and models, proving its robustness and effectiveness (R-csBg, R-WU7w).
All supplementary experimental outcomes and discussions will be incorporated into the final manuscript.

**Regarding the computational efficiency of BOMF**

In response to the reviewers' concerns regarding the computational efficiency of BOMF, we analyze these points as follows:

The additional computational cost incurred by BOMF can be analyzed in two ways: 1) in comparison to basic fine-tuning and 2) relative to other fusion baseline methods. 

First, compared to basic fine-tuning, the computational cost during hyperparameter tuning in BOMF is similar to that of grid search in fine-tuning. Given that we used the same number of iterations for the BO step and the same number of grid points for the grid search method. While grid search can theoretically reduce search time through parallelization with sufficient GPUs, the increasing memory and computational demands of large language models make it challenging to perform parallel grid searches without a significant number of GPUs. Consequently, the time difference between BOMF and basic fine-tuning may not be substantial. Thus, the additional computational cost of BOMF mainly consists of the cost associated with calculating the combination coefficient. Specifically, during the MOBO step, if performed K times, the additional computational cost includes K evaluations of the validation set metrics and losses, as well as the computations required for Bayesian Optimization.

When compared to other fusion methods, BOMF, like other baselines (excluding SWA), requires multiple forward passes to find the best combination coefficient. Methods like learned SWA, which optimize combination coefficients via backward passes, incur additional backward computation costs.

*Table R.5* shows the empirical results comparing the time required for combination coefficient optimization between BOMF and other fusion baselines. The experiments were conducted using the Llama-3 model on a Korean multi-choice medical question answering task to demonstrate the scenario with a large language model.  

Our additional experiments are included in the below PDF.

Meta-review: The paper presents a novel approach to model fusion through Bayesian Optimization for fine-tuning pre-trained language models on downstream tasks. The authors address the challenges associated with hyperparameter selection and the discrepancy between loss and metric landscapes during the fine-tuning process. They propose a two-stage Bayesian optimization framework that first identifies optimal hyperparameters for fine-tuning and then uses multi-objective Bayesian optimization to find the best combination of models in the parameter space. The paper provides a comprehensive experimental evaluation to demonstrate the properties and benefits of the proposed approach.

This paper has 3 accept scores (7, 7, 8) and 1 strong reject (2).

Let me start with the strong reject: the reviewer (ut18) has not responded to the authors rebuttal and my repeated requests (3) for feedback. His review is so generic and disconnected from the actual work that I suspect it is LLM-generated. Among the  “weaknesses” mentioned in the review, this one is especially meaningless:
“Questionable Approach: The use of model fusion to address the discrepancy between loss and desired metrics is not necessarily the most effective solution. Alternative approaches like Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO) can directly optimize towards desired metrics, potentially bypassing the need for fusion altogether.”
This work is about extracting the best model from a fine-tuning training run and is orthogonal to the source of supervision. Applying it to the supervised tuning phase of medium or large language models is the obvious priority. To please reviewer ut18, the authors also demonstrated their approach on RLHF, but there was no response.

Thus, this meta-review will only focus on the 3 accept reviews, all of them of high quality, with responses to the author’s rebuttal.
Among the strength, the reviewers mention that the paper is well-written and mention the originality of the approach, even if the individual components of the proposed method are not novel. The paper is  thorough in its methodology and experiments, motivates each step of the approach, and offers novel and interesting insights into the alignment between loss and metric surfaces for NLP tasks. 

Among the weaknesses, there were concerns about the computational efficiency of the method and the small size of the models. They were addressed in the rebuttal, with new results: performance on LLAMA-3, time cost, application to RLHF, etc… 
Remarkably, all 3 reviewers increased their score after this rebuttal.

This well written paper proposes an original method with significant improvements over basic fine-tuning, or other model averaging methods. The computational cost is reasonable, so the impact could be significant
Decision: Accept (spotlight)