Understanding Permutation Based Model Merging with Feature Visualizations

Authors: congshu  zou
geraldin  nanfack
stefan  horoi
eugene  belilovsky
Venue: NeurIPS_cc_2024_Workshop_UniReps
Published: 2024-09-23 19:24:54

Abstract: Linear mode connectivity (LMC) has become a topic of great interest in recent years. It has been empirically demonstrated that popular deep learning models trained from different initializations exhibit linear model connectivity up to permutation. Based on this, several approaches for finding a permutation of the model's features or weights have been proposed leading to several popular methods for model merging. These methods enable the simple averaging of two models to create a new high-performance model. However, besides accuracy, the properties of these models and their relationships to the representations of the models they derive from are poorly understood. 
In this work, we study the inner mechanisms behind LMC in model merging through the lens of classic feature visualization methods. Focusing on convolutional neural networks (CNNs) we make several observations that shed light on the underlying mechanisms of model merging by permute and average.

Main paper: 
 introduction: 
 Deep neural networks have achieved remarkable advancements in recent years, demonstrating their capability to perform various valuable tasks. Understanding the inner mechanisms of neural networks becomes critical, which leads to exploring the loss landscape. Frankle et al. [6] suggest linear mode connectivity (LCM) as a paradigm where two neural networks, randomly initialized and trained using stochastic gradient descent (SGD), are observed to be connected by a linear path in the parameter space with no or little loss barrier. The study of LMC can offer valuable insights into the complex structure of loss landscapes and training dynamics. It will also help various deep learning applications such as model merging, transfer learning, etc. However, naive LMC often fails due to the permutation invariance of neural networks [4] . Thus several algorithms are designed to linearly connect two trained models [1, 10] by first aligning one to another, and second by finding a permutation. This has been shown to lead to linear mode connectivity and models that can be merged (or averaged).However, why the LMC works with permutation models and what properties this induces on the average model is challenging to understand. In this work, we propose to study this through the lens of feature visualization. By maximizing the activation of certain neurons in a network, we can visually understand what features the model is detecting. If two models can be linearly interpolated, then they should activate similar features. In addition to visually inspecting the extracted features, we use some metrics to characterize the similarity of images extracted from two models [13] .In this study, we train multiple randomly initialized models with SGD optimization, then apply the REPAIR [10] method to permute the model weights space and visualize the top 10 images that most activate each neuron. We find that models after permutation share some similar top-activated features, but not all of them. We shed light on the features that don't match and highlight that they correspond to rare features in the models being aligned against. Finally, we show that despite the limitation of one-to-one permutation matching, features in a given model can be approximated by a sparse subset of features in another model, motivating recently proposed methods in model merging. Related Work Linear mode connectivity. Mode connectivity investigates the relationship between two optimal solutions found after training. It demonstrates that the optimal solutions for deep neural networks can be connected by very simple curves [7, 3] . This is further extended by Frankle et al. [6] as linear mode connectivity, which suggests that there exists a linear path between the two optima, where the loss barrier is low along the path. Later work by Entezari et al. [4] show that taking into account the permutation invariance, optimal solutions trained with SGD will likely have no barrier in the linear interpolation between them, i.e. linear mode connectivity [4] . Based on this conjecture, more recent studies have developed a few algorithms to find such a permutation [1, 10] . In particular, Jordan et al. [10] observe simple permutation alone does not yield a low loss barrier which is caused by a phenomenon called variance collapse. They introduce the REPAIR method by rescaling the parameters in the interpolated networks [10] . This paper uses REPAIR to permute our trained models.Feature Visualization. Visual interpretability methods provide an intuitive way to understand how neural networks function and make decisions. A saliency map highlights the area that has the most influence on the final model prediction [15] . Activation maximization finds input images that maximize the neuron activity by optimization [5, 14] . Through visualization, Li et al. [12] show that the representations learned by different networks converge to a set of features that are individually similar between networks [12] . In addition to visual-only qualitative analysis, Nanfack et al. [13] proposed Kendall-τ and CLIP-δ to characterize the similarity of images extracted from two models [13] . In this study, we visualize each layer of the trained model with top activated images from the training dataset and evaluate using Kendall-τ and CLIP-δ against the top images of each channel. 
experiments and results: 
 Our experiments focus on the AlexNet [11] and ResNet [8] architectures trained on the ImageNet dataset [2] . This choice is made largely because feature visualization methods work well and have been extensively studied for this model and dataset [11, 12] .We train five AlexNet models to use in our merging experiments. All model weights are randomly initialized, and each model is trained with 90 epochs. The validation accuracy for each model is 56.74%, 56.25%, 56.27%, 56.45%, and 56.27%. We also trained two AlexNet models with stratified ImageNet dataset, one model trained with 80% of 500 first image classes and 20% of the last 500 image classes, and another model trained on the rest data. Three ResNet18 [8] models are trained as well. Visualization results of standard trained AlexNet are included in the main paper. More results for stratified AlexNet and ResNet are provided in the Appendix, which follows similar trends.We take two trained AlexNet models, with one (Model 2) permuted to align with the other (Model 1). Then we feed the training dataset to both models and extract the 10 images that activate each convolutional channel most (top-10 feature visualizations). For a selected channel, top images of Observation 1: Many but not all permutation-matched features are similar. We informally define feature visualization similarity (FV) as a visual observer identifying the same structures and patterns (and sometimes the same images) between two visualizations of two different channels (typically found in two different models). We observe by looking through the visualizations over many channels that the permuted model 2 feature visualizations are often but not always similar to those from model 1 as illustrated in Fig 1 (bottom) . Thus despite being trained from different initialization, the training will often find similar features. This also matches observations in [12] .We now turn to analyzing the feature visualization of the merged model.Observation 2: Merged model feature visualization is dependent on the source models and whether there is an FV similarity after permutation matching. Analyzing the feature visualizations from many channels we make the following observations. In the case that the permuted model has similar top-activated images, the top images from the merged model are also similar. This is illustrated on the top in Fig 1 where the permuted model 2 shows similar features to model 1. However, when the images from 2 models are not similar after permutation (e.g. bottom of Fig 1 ), we observe different behavior on merged models top images. These behaviors were observed by inspection of all channels in AlexNet to fall into three general categories:1. Images from one model (either model 1 or 2) dominate the top images of the merged model.2. Images from both models appear in the top images of the merged model.3. Most top images of merged models do not belong to any of the 2 models.Since the matching of features is so critical to the visualization observed in the merged model, we ask whether the mismatched cases after permutation occur randomly or if they represent rare features in model 1 that are hard to match in model 2. To study this we perform the matching process multiple times with one fixed model being matched. Specifically, we permute the 4 of our AlexNet models (all trained from different initialization) to align with model 1. Top activated images are extracted for all 5 models. Observation 3: Certain features of the reference model are challenging to match in terms of FV similarity. Analyzing all channels in all layers of AlexNet we have the following cases (see Fig 2 ): 1. For a channel all 5 models have very similar top images (occurring (5 -10%))2. For a channel most models have similar top images except one or two (occurring (35-50%)).3. For a channel top images from all 5 models are dissimilar (40 -60%) We also study the mean and standard deviation of the metrics we use. It is clear that channels with high means (low for CLIP-δ scores) and low standard deviations usually have very similar top-activated images. On the other hand, if we have high standard deviation and low means, visually, the top images vary a lot.The results indicate that certain features are easily learned by the models, while some features are hard to learn. This is what we hypothesize, and it leads to cases of dissimilar FV after permutation (Observation 1).We observe that although per neuron or channel, feature visualization is based on the hypothesis that each channel represents a particular feature of the data, a given useful feature may be represented by a single channel in one network but can be represented in a distributed way by another network.We take all the features in a given layer of model 2 and regress them onto the features of model 1, building a separate regression model for each feature in model 1. Specifically, we get the activation of each convolutional channel from both models with 5000 images randomly selected from the ImageNet dataset, i.e., 5 images per class. Then for each layer, we find a linear regression model, such that for activation values of each channel in model 1, it is approximated by a linear combination of activation values from all channels in model 2. Afterward, we feed the entire dataset to model 2, apply the regression model on the activations, and extract the top 10 activated images of each channel.Observation 4: For each feature in the model there is a (often sparse) linear combination of features in model 2 that gives an FV similarity. In Fig 3,  we see the results of visualizing the regression model outputs at each layer. The results show that top-activated images are now visually aligned. Furthermore, we study the sparsity of the weights we got from the regression model. As shown in Fig 3,  we can see the top images do not change a lot when we only select the weights with top magnitude and set the rest to 0. This shows that the weights that impact the alignment are sparse. However we do find that for certain channels (particularly in the early layers), the top activated images change when we use less weights, and these channels are also visually hard to align channels when we compare the 5 models (Fig 3 ).Our results suggest that permutation-based merging can be substantially limited since it is too restrictive to assume a one-to-one matching between features. Indeed recent model merging methods specifically aim to address this restriction [16, 9] , showing promising new results. Our work helps to provide additional motivation for these new methods. 
conclusion: 
 By permuting trained models and visualizing top-activated images, we have gained valuable insights into the internal behavior of these models. Our findings visually demonstrate the effectiveness of model alignment by permutation and provide some insights into linear mode connectivity. Additionally, the use of linear regression to align models has shown promising results, and the sparsity of effective weights suggests an inner connection of the channels.  
 
 Appendix: We permuted 4 models against model 1. Visually, we can see that the similarity of the top images from 5 models varies from channel to channel. Some channels have similar images from all 5 models, and some have images totally dissimilar from all 5 models. There are also channels with similar images from 3 model and another type of similar images from 2 other models. Furthermore, We compute Model 2 is permuted to align with model 1. Both direct merge and permuted merged models are studied, as well as repaired merge model. Top activated images from original model 2 before permutation is also included. Fig 12 shows some selected results. We observe similar pattern as standard trained models. Although due to the unbalanced dataset, the visualization extracted shows less similarity compared to the models trained with full dataset.Figure 12 : Top activated images from selected channels for AlexNet models trained with stratified dataset. Results are similar to models trained with standard dataset. It is also observed that features are harder to align compared to standard trained models. This is probably because of the unbalanced training dataset.We trained 3 ResNet18 models. Each models are trained on full ImageNet Dataset for 100 epochs with SGD optimization. Learning rate starts at 0.1 and decays by a factor of 10 every 30 epochs. We observe some accuracy improvement with permuted only models, especially when α is less than 0.4 and greater then 0.6. After rescaling, the validation accuracies are improved significantly. Preprint. 


Decision: In light of the reviewers' feedback and relevancy of the submission, we are pleased to accept this paper for presentation at UniReps 2024. We kindly ask the authors to incorporate the reviewers' suggestions and feedback in the final camera-ready version of the manuscript.

Reviews: review: title: title: Understanding LMC and permutation matching by visualizing activation maps
main_review: main_review: The study of the permutation matching of two models by inspecting the feature activation map.

### Strengths

1. The paper is mainly easy to follow, and the idea of visually comparing the activation maps is intuitive and interesting. 

2. The authors make interesting observations using the feature maps, which I think is worth discussing in the workshop. However, there are still shortcomings which authors can improve on for the next version of the manuscript. 

### Weaknesses

1.
> We also trained two AlexNet models with stratified
ImageNet dataset, in which one model trained with 80% of 500 first image classes and 20% of the last
500 image classes, and another model trained on the rest of the data. 

The intuition behind this split is not clear.

2. AlexNet used in the experiments is a small model, and observations may not hold true for larger models.

3. Figure 1 is not clear, columns should be labeled. Permuting model 2 (column 2) should not change the top-10 activated images, so column 2 and column 3 (model 2 permuted) should have same images (but different order). However, that is not case and it is not clear why.
title: title: Interesting research questions, but current methodology/results are unconvincing.
main_review: main_review: I believe the research direction and approach is novel, and the paper is above the bar for a workshop submission in writing quality. The results, even for a workshop submission, were not very convincing to me however and I would suggest to the authors that their results would benefit much more from a more rigorous and quantitative analysis moving forward.

Strengths:
- Model merging itself is quite opaque still, and some of the author's questions are indeed of interest.
- We can can expect learned features/filters to differ between different models trained with different random initialization (even after weight or activation matching/permutation, which is not perfect) to differ, and of course expect merged models to learn different features, and I believe the best experimental directions proposed in this work align with the question of how these differ - I myself would love to see convincing results/analysis that would explain these areas further.
- Decent summary (for workshop paper) of LMC/permutation background, although I believe the wording of their main research question on line 26 should be improved.

Weaknesses
- The features/filters of a model before/after applying a permutation *should not differ* by definition, and likewise the images with maximum activations should also not differ, however we can expect the specific channel ordering of the filters learned to change. The author's results images focus on single channels unfortunately (e.g. Figure 1), and might even mislead some to believe model 2 v.s. model 2 permuted learn different filters. Only showing one channel gives a very incomplete intuition as to how similar the features learned by one model are v.s. another. This is true even after permutation, as weight matching/activation matching only approximate a good permutation (not perfect).
- Analysis is effectively qualitative (see below on metrics). The authors judge if models learn similar features/filters solely based on the top-10 images that maximally activate neurons (filters in CNN) in the corresponding models. The authors should take care to note however that even if two neurons share the same top-10 images, it is not necessarily the case that they have learned the same representation, unlike they suggest. This is however evidence that would support such a hypothesis, and I would suggest to the authors that it would be much more convincing if paired with other more quantitative evidence such as existing methods of comparing neural representations (e.g. "Similarity of Neural Network Representations Revisited", Kornblith et al., ICML 2019).
- The only metrics the authors use are used to compare images (i.e. the images that have maximum activation), rather than filters/features learned in the models, which does not address the issue I have pointed out to above, but is much better than only including visuals of channels of images. It is not clear to me however if the metrics are calculated over all channels or only a single channel.
- The images themselves are far too small to be the main results of the paper, it's very hard to make out individual images.
- Kind of strange to be using AlexNet for anything in 2024, although given the ResNet results I don't think this is a big weakness. I would suggest the authors find further, more modern architectures to compare however, although I realize weight/activation matching work better with wider models.
- I looked at some of the appendix out of curiosity, and didn't understand why there were class-unbalanced training results there - would suggest motivating those results even if they are in the appendix.
title: title: This article is mainly based on experiments and is meaningful.
main_review: main_review: The work focuses on visualizing model behaviors using feature visualization techniques on CNNs. The experimental results give insights into how models align after permutation and what impact this has on their performance and feature extraction.
1. The architecture of AlexNet and ResNet is investigated. It is better to include transformer-based architectures.
2. While the paper effectively demonstrates feature similarity after model permutation, the broader implications of this work for practical applications such as transfer learning or federated learning are absent.
title: title: Promising path with opportunities for refinement
main_review: main_review: ## Summary

The paper uses feature visualisation methods to explore how different randomly initialised image classifiers, trained on the same dataset, activate similar features and how these activations change when models are aligned using permutation or linear regression. Moreover, the paper produces visualisations that illustrate how merging of original and aligned models influences the features each channel responds to. The analysis reveals that the different models often produce channels that detect similar features, but this alignment is not guaranteed.

## Strengths and Weaknesses

Strengths:
- Feature Visualisations are a promising tool to understand when model alignment work and when it does not.
- The visualisations with increasing sparser regressions can provide a good intuition on how much sparsity is needed and what are the effects of having more or less sparsity.

Weaknesses:

- Both the abstract and the conclusion are very vague in terms of what can be concluded from the results.
- Only feature visualisations are shown. These are good to highlight particular interesting cases, but they fail to communicate the statistic relevance of the visualisations and what is the overall model behaviour. Including aggregate metrics like CLIP-$\delta$ and Kendall-$\tau$ across multiple layers would strengthen the statistical grounding of the results.
- Figure 3 should also show the model 2 after permutation so that the reader can compare that method with the regression.

## Questions

- The figures caption and legend could be more clear. What does the number of “features” means in the title? In Figure 1, the CLIP-$\delta$ and the Kendal-$\tau$ are a measure between which images?


Soundness: 2

Presentation: 2

Contribution: 2