Interfering with Interference: Blind Shuffling and Superposition for Better Multi-Model Compression

Authors: Hangyu Zhou, Aaron Gokaslan, Volodymyr Kuleshov, Bharath Hariharan
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: Task Arithmetic, Superposition, Model Merging, Multi-model Compression, Model Serving

Abstract:
We present two complementary random mechanisms to significantly reduce interference when eliminating cross-model redundancy for efficient multi-model serving: _Layer Shuffling_ and _Task Vector Superposition_. They work together to increase the orthogonality among interfering task vectors, forcing them into self-destruction without requiring any post-training learning or optimization. _Layer Shuffling_ randomly reorders layers of each individual models to reduce the alignment between interfering task vectors. While _Task Vector Superposition_ leverages random orthogonal transformations to decorrelate task vectors further. Together, these techniques drastically minimize interference, yielding improved performance across multiple tasks with effectively zero incremental memory cost when incorporating new models. Their data and model-independent nature also allows for seamless on-the-fly addition or removal of models, without requiring any re-computation, making them highly practical for real-world deployment scenarios.

Decision: reject
Reviewer scores: 6, 3, 6, 5

Reviews and discussion:
<thread>
creation date:
1730352768523

thread_type:
official_review

<thread_object>
creator: reviewer_jguu

<content>
summary:
This paper introduces Layer Shuffling and Task Vector Superposition, two random mechanisms to reduce interference in multi-model compression by increasing orthogonality between task vectors. The methods achieve near-identical accuracy to individual fine-tuned models while reducing storage costs by 4 times and enable seamless hot-swapping of models without recomputation.

strengths:
- The paper presents a novel approach to multi-model compression through random mechanisms

- The empirical evaluation is conducted across multiple benchmarks to demonstrate the effectiveness of the method

weaknesses:
- The current presentation and writing require significant improvements. For instance, the mathematical analysis is overly simplistic and does not warrant extensive explanation. Additionally, the proposed method lacks a rigorous proof demonstrating why Layer Shuffling specifically enhances orthogonality more effectively than other potential random transformations.

- The interaction between Layer Shuffling and Task Vector Superposition isn't thoroughly analyzed as it's unclear whether they're truly complementary or if one method dominates the benefits

- The experiments are not convincing because the models used for comparison are generally much smaller, leading to expected inferior performance from competitors. Meanwhile, the authors' model is significantly larger, resulting in better performance, which does not necessarily demonstrate an advantage.

questions:
see the above weakness part
</content>
</thread_object>

</thread>
<thread>
creation date:
1730550306968

thread_type:
official_review

<thread_object>
creator: reviewer_mt5t

<content>
summary:
The paper introduces two methods, Layer Shuffling and Task Vector Superposition, aimed at reducing interference when compressing multiple fine-tuned models into a single multitask model using task vector arithmetic. Layer Shuffling works by randomly reordering layers in each model before merging, reducing alignment between task vectors. Task Vector Superposition applies random orthogonal transformations to further decorrelate task vectors. Both techniques minimize interference and improve performance across tasks. Experiments with CLIP-ViT, Flan-T5, and GPT-2 show that this approach achieves higher accuracy than vanilla task arithmetic and other baseline methods.

strengths:
- The paper makes an observation that individual task vectors are too similar and successfully uses it to reduce task vector interference, leading to better multitask performance in model compression scenarios.
- Both proposed techniques operate without needing data, allowing flexible model addition or removal without retraining or optimization.
- The method achieves storage reduction compared to keeping individual models.
- The approach is shown to improve performance across diverse domains including image classification, text generation, and text classification.
- The method enables on-the-fly model integration, allowing seamless "hot-swapping" of models.
- The paper is very well written and clearly structured.

weaknesses:
- While the paper compares its method to several baseline techniques, it misses comparison with closely related recent works, particularly Guillermo Ortiz-Jimenez et al.'s work (mentioned in the paper) on task vector manipulation for model merging. Including these comparisons would strengthen the submission.
- Although the authors claim minimal memory overhead, additional context matrices and shuffled task vectors nearly double the memory requirement, which may not always justify the marginal performance gains over baselines like SMILE.
- LoRA results show that SMILE achieves a better tradeoff between accuracy and memory than the reported combination of the proposed methods.

questions:
Q1: Although randomization offers clear advantages like data independence, would a more systematic approach to orthogonalizing task vectors further improve the performance?
Q2: Did you observe any (in-)consistent performance variance due to randomness in shuffling and superposition?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730675196527

thread_type:
official_review

<thread_object>
creator: reviewer_jsqg

<content>
summary:
The paper proposes two stochastic mechanisms to improve performance for multi-task model merging by reducing task interference. First, the method takes advantage of the repeating structure of modern neural networks and randomly shuffles the same-module layer across blocks by first showing that the layers are mostly similar in the within-block across tasks. Second, the paper proposes random binary matrices to multiply parameter vectors to further reduce the task vector similarity. During inference, the inverse transforms are applied. The paper performs experiments across diverse benchmarks.

strengths:
1. The method is intuitive and simple. The motivation for both components is well written and properly ablated.
2. The method is scalable and memory efficient (modulo the duplication of model parameters) given that it only requires the storage of random seeds to retrieve the final model.
3. The experimental results are strong across benchmarks.

weaknesses:
1. The method has a limitation that is not discussed a lot, apart from the title: it requires the knowledge of the task id during inference. This needs to be underlined during the comparison with methods such as ties and task arithmetic for fairness.
2. lack of forward pass time complexity comparison. The proposed method introduces an overhead in the forward pass: layers need to be reshuffled in the correct order, signs need to be restored and the residual needs to be added to the pre-trained weights. Therefore, there should be a study of how much overhead all these operations incur.
3. Missing baselines: Given the parameter increase and the time complexity overhead, the paper should compare with the compression algorithm of [1].
4. The paper solely focuses on small models, base variants on ViT and Flan-T5, but the literature uses ViT-L/14 and T5-XXL regularly. It would also be interesting to check the performance of the method as tasks increase, see 14 and 20 task benchmarks from [1]. It would be interesting to also track the forward pass time metrics in the case of larger models.
5. L268-269: fix references for benchmarks: the vision one for instance comes from Ilharco et al. and not from FusionBench
6. Baselines and their categorization are not explained and the reader cannot understand why PSP ins included given its poor results or what WEMoE and SMILE are on their own category compared to everything else. It would be helpful for the reader to provide a brief description of each method as well as a high level overview of the categories to help the reader understand rather than deferring to the appendix where they are actually not discussed.
7. Extremely limited Related work: the quality of the paper is heavily undermined by the lack of proper references and discussion over related work.

Minor Comments


- L202: ontain → obtain
- Rephrase informal writing:
	- L217: “which we expect to be much lower”
	- L150: “but this balance has generally been tricky to achieve”

[1] Wang, K., Dimitriadis, N., Ortiz-Jimenez, G., Fleuret, F. and Frossard, P., 2024. Localizing Task Information for Improved Model Merging and Compression. *arXiv preprint arXiv:2405.07813*.

questions:
1. why is 5.03 the number of Gb attributed to fine-tuned? shouldn’t it be 8x the pre-trained model?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730719784652

thread_type:
official_review

<thread_object>
creator: reviewer_gjhr

<content>
summary:
This paper introduces two methods, layer shuffling and task vector superposition, aimed at reducing interference between task vectors in multi-model compression scenarios. The proposed methods work by increasing the orthogonality of task vectors, thus minimizing their interference during merging. By leveraging randomization, these methods require no additional training and can be applied across various models and tasks. Experiments on multiple benchmarks, including CLIP, Flan-T5, and GPT-2, demonstrate that this approach can achieve comparable performance to fine-tuned models while reducing storage costs, particularly for real-world deployment scenarios where adding or removing models on the fly is necessary.

strengths:
* Simplicity and Effectiveness: One of the major strengths of this paper lies in its approach’s simplicity. Layer shuffling and task vector superposition are straightforward yet powerful techniques that effectively reduce interference without needing additional training, optimization, or complex configurations. This simplicity not only enhances the practicality of the approach but also makes it easy to implement and adapt across various multi-model compression tasks, proving that even minimal adjustments can yield significant performance improvements.

* Effective Interference Reduction: The combination of layer shuffling and task vector superposition is innovative in addressing interference by increasing orthogonality among task vectors. This approach allows for a more effective merging process, yielding improved model accuracy without the need for additional optimization or training steps.

* Adaptability and Scalability: The proposed method’s flexibility is a clear strength. Its data and model-independent nature enables seamless additions and removals of models (hot-swapping) without re-computation, a valuable feature for dynamic applications. Moreover, the approach is efficient, doubling the memory footprint while providing significant accuracy improvements.

* Comprehensive Evaluation: The experiments cover a range of benchmarks and tasks, showcasing the model’s capability across various domains, from image classification to text generation. This breadth of evaluation helps establish the generalizability of the method across tasks and model architectures.

weaknesses:
* Lack of Detailed Performance Analysis Based on Shuffle/Superposition Levels: It would be useful to analyze the impact of different levels of shuffling and superposition, as these levels could influence task vector similarity and interference differently. This analysis would provide a clearer picture of optimal interference reduction strategies.
* Clarity Issues in Method Description: Some aspects of the method, such as the merged task vector formation in equation (8), could benefit from further clarification. Specifically, does shuffling task vectors in different layers cause mixing of task vectors across layers, for instance, between k-1 or k+1? Clarifying this would enhance understanding of how the shuffle affects layer-specific task vector alignment.
* Effectiveness Across Tasks: The effectiveness of either TA+Shuffle or STA appears to vary by task, yet the paper does not discuss why some tasks benefit more from specific strategies. A more in-depth analysis here would provide insights into optimizing methods based on task characteristics.
* Related Work Reference (PEFT): (Maybe, long shot) this paper is related? "Efficient Storage of Fine-Tuned Models via Low-Rank Approximation of Weight Residuals,"
* Minor Formatting Issues: There are some minor formatting errors in the document, such as incorrect Latex punctuation and inconsistent reference formatting. For example, equation 3 is mistakenly referenced in the context of equation 4, and parentheses are missing in certain citations. Additionally, clarifying what the values in parentheses mean in tables, such as in the average (%) and Bits (Gb) columns, would be helpful, as it currently requires reading the text to understand that they refer to relative performance to fine-tuned models.

questions:
Please see the above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732762269145

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Responses

comment:
We sincerely thank all reviewers for their thoughtful feedback and suggestions. Below, we address the key concerns regarding memory-performance tradeoffs, model scaling, and the interplay between shuffling and superposition. The revisions and corresponding results have been incorporated into the manuscript, with all changes highlighted in blue.


**[GR1] Memory Efficiency Clarification (Reviewer JsQG, mt5T, and jGUu)**

We appreciate the reviewers' valid concerns regarding the memory footprint of our methods. To address these concerns, we present new findings demonstrating that our approach requires saving only the pre-trained model and a merged task vector. Merging additional models involves storing just a single 8-bit random seed per model. Our method introduces less than 1s overhead during the forward pass for retrieving each model and maintains high performance. Note this runtime overhead is not dependent on the number of merged models.

1. **Compressing Additional Models into 8-bit Random Seeds**

Our methods—random layer shuffling and task vector superposition—are random operations independent of both data and model parameters. This property allows us to retrieve each merged model by reconstructing the layer shuffling orders and the inverses of the context matrices from a single random seed (takes 8-bit if less than 256 models are merged). Although we double the memory footprint due to the need to manipulate the merged task vector differently for each task, this initial investment remains fixed, regardless of the number of models being merged.

2. **Achieving Near-Fine-Tuned Performance with Reduced Memory Usage**

Our methods achieve performance close to that of fine-tuned models across all testing scenarios. Thanks to the feedback from Reviewer JsQG, we realized that when merging CLIP-ViT-B/32 models, our methods consume less memory than previously estimated, as we mistakenly counted the text encoder twice. Here is the updated Table 1 (truncated to highlight key comparisons):

**Performance comparison on eight image classification tasks with CLIP-ViT-B/32 models**
| **Method**             | **Avg.(%) ↑**     | **Bits(Gb) ↓**     |
| ---------------------- | ----------------- | ------------------ |
| Pre-trained            | 48.2 (53.4)       | 0.564 (1.00)       |
| Fine-tuned             | 90.3 (100)        | 2.84 (5.03)        |
| Task Arithmetic        | 69.8 (77.2)       | ***0.564 (1.00)*** |
| WEMoE                  | 89.2 (98.8)       | 2.27 (4.03)        |
| SMILE                  | **89.3 (98.9)**       | 1.23 (2.20)        |
| **STA+Shuffle (Ours)** | ***89.9 (99.6)*** | **0.89 (1.58)**    |

This updated table demonstrates that our algorithm achieves near-perfect accuracy while using significantly less memory compared to SMILE [3] and WEMoE [4].

Moreover, as shown in **[GR2]**, our STA+Shuffle algorithm maintains accuracy close to that of individually fine-tuned baselines when merging 14 or even 20 CLIP-ViT-L/14 models, with a constant memory footprint of less than twice that of a single model. In contrast, the state-of-the-art baseline TALL-masks uses 323% more memory than our method while achieving slightly lower performance when merging 20 CLIP-ViT-L/14 models. Finally, **[GR3]** shows that we can **halve** the current memory footprint by selecting a subset of layers to shuffle and superpose.

While methods like Task Arithmetic consume approximately 50% less memory than ours, their performance is significantly lower, with accuracy scores resembling those of zero-shot pre-trained models when more and larger models are merged, as noted in **[GR2]**.

3. **Minimal Runtime Overhead in Forward Pass for Model Retrieval**

To evaluate the overhead introduced by our method, we conducted experiments using a single random seed to retrieve models with shuffling and superposition applied to all layers. For the CLIP-ViT-B/32 model, the retrieval process executed on an Intel Xeon Gold 6448Y CPU over 10 repetitions yielded an average runtime of 292.70 ms. For the larger CLIP-ViT-L/14 model, the average runtime was 658.19 ms under the same conditions. We anticipate that further optimizations, such as utilizing GPUs, could reduce these runtimes further. As our method shows no increase in memory footprint as more models are merged, these overhead measurements also apply to scenarios where 14 or 20 CLIP-ViT-L/14 models are merged.

In conclusion, the new findings show that our algorithm can achieve near-perfect accuracy with constant memory use and minimal runtime overhead. It outperforms state-of-the-art methods in performance and memory efficiency, offering a simple yet powerful solution for model merging and serving.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The majority of the reviewers raised serious concerns regarding several aspects such as presentation clarify, methodological justification, empirical evaluation and contribution significance. The authors tried to response positively to those concerns, but failed to change the reviewer’s opinion. Overall, the paper may have potentials but need be improved a lot. This assertion also coincides with the author rebuttal, which indicates explicitly “To address your concern comprehensively, we will incorporate a task classifier to our method in the NEXT VERSION of our paper.”