## 5 Discussion  
The experimental results confirm the effectiveness of the model merging techniques. In particular, the model merging methods exhibited superior accuracy improvements compared to simple averaging within the FL context. Specifically, we observed that model merging resulted in a significant increase in accuracy for new tasks, highlighting its advantage in adapting to unseen data distributions. However, there was a trade-off between improving accuracy on new tasks and retaining previously acquired knowledge. Our experiments identified several key considerations for the practical application of continuous model merging methods. These insights are summarized as follows:  
* Challenges in Selecting Effective Methods: In this study, we applied a single model merging method iteratively to update the model. However, as illustrated in Figure 3,  
Figure 3: Cosine similarity of task vectors in first and last rounds when models are merged using task arithmetic. The darker the color, the higher the similarity between the task vectors; the lighter the color, the more orthogonal the task vectors are. \begin{table}
\begin{tabular}{l c c c c c c c c c c} \hline \hline  & **Cars** & **DTD** & **EuroSAT** & **GTSRB** & **KITTI** & **MNIST** & **RESISC45** & **SUN397** & **SVHN** & **Avg.** & **ImageNet** \\ \hline zero-shot & 59.6 & 44.0 & 46.9 & 32.5 & 27.6 & 48.5 & 60.7 & 63.1 & 31.6 & 46.1 & 63.4 \\ Averaging & 65.8 & 61.0 & 96.7 & 95.7 & 46.3 & 99.5 & 89.2 & **70.0** & 95.4 & 80.0 & 54.5 \\ PAINT & 63.4 & 54.8 & 93.1 & 88.2 & 39.9 & 99.1 & 83.5 & 68.1 & 91.8 & 75.8 & **58.1** \\ Task Arithmetic & 65.6 & 60.6 & 96.9 & 95.5 & 47.3 & 99.6 & 89.4 & 69.9 & 95.5 & 80.0 & 54.4 \\ TIES Merging & 63.3 & 55.3 & 93.4 & 88.4 & 39.2 & 99.2 & 83.8 & 68.1 & 92.0 & 75.8 & 57.8 \\ DARE & 66.6 & **63.9** & 97.6 & 95.5 & 46.7 & 99.5 & 88.6 & 69.2 & 95.9 & 80.4 & 51.8 \\ Breakcrumbs & **68.4** & 63.7 & **98.0** & **97.6** & **49.4** & 99.6 & **90.4** & 69.6 & 96.4 & **81.5** & 49.5 \\ MagMax & 51.4 & 37.8 & 92.1 & 91.8 & 30.0 & **99.7** & 82.9 & 65.8 & **97.4** & 72.1 & 39.2 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Classification accuracy obtained on each test dataset after 10 rounds in our experimental setting. The bold numbers in the table indicate the best results, and the underlined numbers represent the second-best results. the similarity between the task vectors varies significantly between the initial and final rounds of continuous model updating. This suggests that the interference between task vectors, which has conventionally been a concern in model merging, is not significant in later rounds. Consequently, techniques such as sparsification, which are typically employed to mitigate such interference, may become unnecessary in the later stages of training. Deeper analysis of the task vectors reveals that the model increasingly adapts to different datasets as learning progresses, which results in the task vectors becoming more orthogonal. The extent of interference between the task vectors decreases over time, potentially reducing the need for explicit sparsification mechanisms in later rounds. More broadly, this finding highlights a gap in existing research regarding the selection of appropriate model merging strategies for different training phases. Depending on the similarity between tasks and the stage of learning progression, different merging algorithms may be more effective. Therefore, exploring a variety of merging techniques is essential for enhancing the generality and adaptability of the proposed framework. * Challenges in Hyperparameter Optimization: In this study, the hyperparameters of each model merging method were determined empirically. However, as demonstrated in Figure 2, model accuracy is highly sensitive to the choice of hyperparameters. In task arithmetic, variations in the scaling factor resulted in accuracy differences of approximately 50%, highlighting the critical impact of hyperparameter selection.