Model Fusion of Heterogeneous Neural Networks via Cross-Layer Alignment
Authors: Dang Nguyen
Khai Nguyen
Nhat Ho
Dinh Phung
Hung Bui
Venue: ICLR 2022 Submitted (2022)
Keywords: Model Fusion
Cross-layer Alignment
Knowledge Distillation
Model Compression
Model Transfer

Abstract: Layer-wise model fusion via optimal transport, named OTFusion, applies soft neuron association for unifying different pre-trained networks to save computational resources. While enjoying its success, OTFusion requires the input networks to have the same number of layers. To address this issue, we propose a novel model fusion framework, named CLAFusion, to fuse neural networks with a different number of layers, which we refer to as heterogeneous neural networks, via cross-layer alignment. The cross-layer alignment problem, which is an unbalanced assignment problem, can be solved efficiently using dynamic programming. Based on the cross-layer alignment, our framework balances the number of layers of neural networks before applying layer-wise model fusion. Our synthetic experiments indicate that the fused network from CLAFusion achieves a more favorable performance compared to the individual networks trained on heterogeneous data without the need for any retraining. With an extra finetuning process, it improves the accuracy of residual networks on the CIFAR10 dataset. Finally, we explore its application for model compression and knowledge distillation when applying to the teacher-student setting.

Under review as a conference paper at ICLR 2022
MODEL FUSION OF HETEROGENEOUS NEURAL NET- WORKS VIA CROSS-LAYER ALIGNMENT
Anonymous authors Paper under double-blind review
ABSTRACT
Layer-wise model fusion via optimal transport, named OTFusion, applies soft neuron association for unifying different pre-trained networks to save computa- tional resources. While enjoying its success, OTFusion requires the input networks to have the same number of layers. To address this issue, we propose a novel model fusion framework, named CLAFusion, to fuse neural networks with a dif- ferent number of layers, which we refer to as heterogeneous neural networks, via cross-layer alignment. The cross-layer alignment problem, which is an unbalanced assignment problem, can be solved efficiently using dynamic programming. Based the cross-layer alignment, our framework balances the number of layers of neural networks before applying layer-wise model fusion. Our synthetic experi- ments indicate that the fused network from CLAFusion achieves a more favorable performance compared to the individual networks trained on heterogeneous data without the need for any retraining. With an extra finetuning process, it improves the accuracy of residual networks on the CIFARIO dataset. Finally, we explore its application for model compression and knowledge distillation when applying to the seacher-studen setting.
INTRODUCTION
The ubiquitous presence of deep neural networks raises the question. "Can we combine the knowledge of two and more deep neural networks improve performance?" As one of the earliest successful methods, ensemble learning aggregates the output over a collection of trained models to improve the generalization ability (Hansen Salamon, 1990; Zhou, 2012). However, it is expensive in terms of computational resources because it requires storing and running many trained neural networks during inference. To overcome the previous hardness, we need to find a single neural network that can inherit the knowledge from multiple pre-trained neural networks and have small size at the same time. To our best knowledge, there are two popular approaches for this purpose: knowledge distillation and model fusion.
The first approach, knowledge distillation, a machine learning technique that transfers knowledge from large networks (teachers) into smaller one (student) (Hinton al., 2015). The smaller network is trained by the task-specific loss and additional distillation losses that encourage the student networks to mimic the prediction of the teachers. The distillation process, however, is computationally expensive because it trains the student network from scratch. In addition, at each training epoch knowledge distillation runs forward inference for all networks and demands sharing the large training data. In some situations sharing data is prohibited due to critical issues such as data security or network bandwidth constraints.
The second line of works is called model fusion which is the problem of merging collection of pre-trained networks into a unified network (Wang et al., 2020; Singh Jaggi, 2020). The simplest technique in model fusion is vanilla averaging (Utans, 1996; Smith & Gashler, 2017), which computes the weighted average of pre-trained network parameters without the need for retraining. Their methods fuse neural networks with the same architecture without considering the permutation invariance nature of neural networks. To mitigate this problem, the idea of solving a neuron alignment problem before applying weight averaging is discussed in Ashmore & Gashler (2015); Li et al. (2015). Recently, there are two concurrent works that benefit from the idea of neuron alignment. FedMA (Wang al., 2020) formulated and solved the assignment problem to find a permutation

---

Under review as a conference paper at ICLR 2022
matrix that undoes the permutation of weight matrices. While OTFusion (Singh & Jaggi, 2020) viewed the problem through the lens of optimal transport (Monge, 1781; Kantorovich, 2006) and utilized the transport map align the weight matrices. Though both methods can fuse multiple neural networks, their applications are limited to networks with the same number of layers.
Contributions. In this paper, we propose model fusion framework for fusing heterogeneous neural networks, namely, unequal width and unequal depth neural networks, which is named as Cross-Layer Alignment Fusion (CLAFusion). CLAFusion consists of three parts: cross-layer alignment, layer balaneing method, and layer-wise model fusion method. We first formulate the cross-layer alignment as an assignment problem using layer representation and layer similarity, then propose dynamic programming based algorithm to solve it. Next, we present two natural and fast methods to balance the number of layers between two networks. Finally, our experiments demonstrate the efficiency of CLAFusion on three different setups. In skill transfer, our framework successfully fuses two heterogeneous neural networks trained on heterogeneous data and improves the performance over the individual networks without retraining. In addition, the fused model from CLAFusion serves as an efficient initialization when training residual networks. Furthermore, CLAFusion shows potential applications for model compression and knowledge distillation in the teacher-student setting,
BACKGROUND
In this section, we first recall the layer representation and layer similarity that have been widely used in applications. Then, we review available model fusion methods and their limitations. Finally, we discuss OTFusion and the challenge of applying this framework to heterogeneous neural networks.
2.1 LAYER REPRESENTATION AND LAYER SIMILARITY
common layer representation is the matrix of activations (Kornblith et al., 2019). Activation matrix, also known as activation map, can be used as a type of knowledge to guide the training in knowledge distillation (Gou et al., 2021). Another choice is the weight matrix of the trained neural networks, which has been shown perform well
netronketwork comparison tasks Neill et al. (2020).
The neural network exhibits the same output when its neurons in each layer are permuted, this is so-called the permutation invariance nature of neural network parameters. Due to the permutation invariance property, a meaningful layer similarity index should be invariant to orthogonal transforma- tions (Kornblith et al., 2019). Their proposed Centered Kernel Alignment (CKA) effectively identifies the relationship between layers of different architectures.
2.2 MODEL FUSION
Despite showing good empirical results, vanilla averaging (Utans, 1996; Smith & Gashler, 2017) only works in the case when the weights of individual networks are relatively close in the weight space. As an effective remedy, several works on model fusion considered performing permutation of neurons in hidden layers before applying vanilla averaging. FBA- IBA-Wagging (Ashmore & Gashler, 2015), Elastic Weight Consolidation (Leontev et al., 2019), and FedMA (Wang et al., 2020) formulate the neuron association problem as an assignment problem and align the neurons. Nevertheless, those variants are not generalized to heterogeneous neural networks.
There is limited literature devoted the discussion of fusing heterogeneous neural networks. Neu- ralMerger (Chou et al., 2018) is one of the few attempts to deal with this setting. However, there are two key differences in NeuralMerger from our CLAFusion. First, their cross-layer alignment is hand-crafted and dedicated to specific neural network architectures. Secondly, they decompose weights into lower dimensions and use vector quantization to merge the weights of two networks. On the other hand, we introduce a systematic way to solve the cross-layer alignment problem, and combining weights is done using the layer-wise model fusion method.
2.3 MODEL FUSION VIA OPTIMAL TRANSPORT
Recently, Singh & Jaggi (2020) proposed OTFusion, which solves the neuron association problem based on free-support barycenters (Cuturi & Doucet, 2014). OTFusion leads to noticeable improve-
2

---

Under review as a conference paper at ICLR 2022
Mocel
Model
Model
Model
Model
Model
Fused
Cross
Layer belandinng
Layer model fusion
algemme
Input
Cupul
Figure 1: CLAFusion strategy: In the first step. the cross-layer alignment problem is solved for two pre- trained models. Two corresponding layers are surrounded by two rounded rectangles of the same color. Based on the optimal mappings obtained in the previous step, CLAFusion balances the number of layers (adding layers in this figure). The red circles and lines indicate the newly added neurons and weights (Zero weights are omitted). Finally, CLAFusion applies layer-wise model fusion method to produce the final output.
ment over vanilla averaging when all individual networks have the same architecture. One advantage of OTFuuion over other vanilla averaging based model fusion methods is that it can fuse networks with different widths (i.e., number of neurons) thanks to the nature of optimal transport.
The challenge of applying OTFusion to heterogeneous neural networks. OTFusion, however, still a layer-wise approach that cannot be directly employed in heterogeneous neural networks setting. We need to find two corresponding layers, matching their neurons before averaging weight matrices. There are two challenges to this scheme. Firstly, the one-to-one mapping between layers not available in advance. A naive one-to-one mapping of layers with the same index the layer-wise approach may cause a mismatch when the numbers of layers are different. For example, we analyze two VGG configurations (Simonyan & Zisserman, 2014, Table 1) with different depths: VGG11 and VGG13. The second convolution layer of VGG11 has 128 channels and an input image of size 112 x 112. While that of VGG13 has 64 channels and an input image of size 224 224. Secondly, assume that we have successfully found a one-to-one mapping, there still necessitates a special treatment for remaining layers that has counterparts. Simply removing those layers has adverse effect on the deeper network, this may cumulatively degrade the performance of the fused model.
CROSS-LAYER ALIGNMENT MODEL FUSION
To overcome the challenges of OTFusion, in this section, we propose Cross-layer Alignment Model Fusion (in short, CLAFusion) framework for fusing heterogeneous neural networks.
Notation. We define [r]]]]]]]] as a set of integers { n) and L,, as an identity matrix of size n n. We use A, B to denote the individual models and F to denote the fused model. indicates the layer index. Layer& of model has ( neurons and activation function p( The pre-activation and activation vector layer of model A are denoted PA1 x
The
(1) ( RP' respectively. The weight
matrix between layers I and - of model A is W (t.-1) The following equations hold between two consecutive layers of model A (we omit the bias terms here).
A
1 (= - 1 (w (1-1))
We stack the pre-activation vector over samples to form a f-row pre-activation matrix. Let (0 _ denote a matrix of pre-activations at layer I of model for samples, and z e gxx!! denote a matrix of pre-activations at layer of model B for the same t samples. The activation matrices x() X (!) E TRexp' are obtained by applying the corresponding activation functions clement-wise to the pre-activation matrices.
Problem setting. Hereafter, we consider fusing two feed-forward networks A and B the same architecture family. Two networks have the same input and output dimension but different number of hidden layers. In each network, the hidden layer has an activation function ReLU, which is the general setting in modern architectures. Additional work is required to handle bias terms and the batch normalization layer properly so we leave them for future work. Let m and n be the number of hidden layers of models A and B, respectively. Taking the input and output layers into account, the
3

---

Under review as a conference paper at ICLR 2022
Model
Model
Model
Model
Input
2
3
16
1
2
9
2
4
3
4
4
0
Output
(a) Pretrained models
(b) Cost matrix
(c) Optimal mapping
Figure 2: Cross-layer alignment example: Two pre-trained neural networks given in (a). Model A has 4 hidden layers of size 3, while model B has hidden layers of size 3, 2, 1. (b) shows the cost matrix (squared Euclidean distance) between layer representations (number neurons) networks. Three color cells represent the solution of the CLA problem. Note that the upper-left and lower-right cells are automatically chosen by the constraints the first and last hidden layers. (c) visualizes the optimal mapping. Two rounded rectangles of the same colors represent corresponding layers in the optimal CLA.
hidden layers two
numbers of layers are 2 and n + 2, respectively. Without loss of generality, assume that 2 The layer index of model A and are {0, 1,....... m + 1} and {0, I......., n + 1}. respectively.
General strategy. Our framework has three components, which are illustrated in Figure 1. The first part is a cross-layer alignment which is a one-to-one mapping from hidden layers of model B to hidden layers of model A. The second part is layer balancing method that equalizes the number of layers of two models based on the cross-layer alignment. The last part is a layer-wise model fusion method that is applied to same-sized models. The third part of our framework adopts any model fusion methods that can fuse two neural networks with the same number of layers. Next, we give more details about the first and second parts as well as provide some concrete methods.
3.1 CROSS-LAYER ALIGNMENT
Cross-layer alignment (CLA) is nontrivial problem that also arises in the context of the feature- based knowledge distillation approach (Zagoruyko & Komodakis, 2016; Tung & Mori, 2019; Chen et al., 2020). CLA requires finding one-to-one function that matches similar layers between two models. Out a few model fusion approaches considering CLA, NeuralMerger based on the hand-crafted layer association, which causes lack of generalization. To address this matter, we introduce efficient method for solving the CLA problem.
Because the input and output layers of the two models are identical in both dimension and functionality, we only align their hidden layers. In addition, the first and last hidden layers usually play a critical role in the model performance (Zhang al., 2019). Therefore, hidden layers of the two models. The importance of this constraint in CLA is studied in Appendix C. feed-forward, so the mapping is necessary keep the sequential
directly match the first and last
Furthermore, two networks order of layers. Taking all into consideration, we formulate the CLA problem as follows.
Definition 1. Assume that we have two layer representations for hidden layers of two models as, mm) and LB = (T. ..... I.(²) .....I.Ig }. Let Ci.j denote the cost matrix between them, (i) where d a layer dissimilarity. The optimal CLA a strictly increasing and minimizing Σ Ca(i).
LA (T. (1)
i.e. C.2, = d(L L
mapping [n] + [m]] satisfying a(1) 1, a,)()
The above problem is a special case of the (linear) unbalanced assignment prob- lem (Ramshaw & Tarjan, 2012). The condition of strictly increasing ensures the sequential order of layers when matching. If we have two increasing layer representations in 1-D, the problem can be interpreted as the Partial Optimal Transport in I-D problem with uniform weights. It can be solved efficiently in Ommn) using the proposed method in Bonneel & Coeurjolly (2019). However, the increasing constraint on layer representations is quite strict and layer representations might not be in 1-D in general. On the other hand, the unbalanced assignment problem can be solved using the generalization of the Hungarian algorithm (Kuhn, 1955). The time complexity of the Hungarian algorithm in our CLA problem is O(mm?).
Discussion.
4

---

Under review as a conference paper at ICLR 2022
Algorithm 1 Cross-layer alignment algorithm
Input: C [Caples
S(i. Li=1 C.. [m]
S( + 0.j E [m]]
S(i, <- 0, [n]
i 111=tto to do
for = to m do
Si. + min -)i
end for end for
a(1) < 1;a(n) + m; - n 1:j < m
while i 2 2 do
while 22i4 and S(i,)) S(i,j - 1) do
j J+j= j-1
end while
a(i) = =j +- i jjj+j-1
end while Output:
Cross-layer alignment algorithm. We propose efficient algorithm based dynamic program- ming to solve the CLA problem. Algorithm details the pseudo-code for our algorithm. Given the cost matrix, the time complexity of Algorithm 1 is only ((m)). Note that the optimal alignment from Definition is not necessarily unique so we choose one which obtained by backtracking.
Layer representation. An important process of computing the cost matrix is to design an appropriate representation for the hidden layer. We list three possible layer representations for hidden layers of two models. The first representation is the number of neurons in each layer. This encourages two layers that have similar number of neurons to match with each other. The second choice is the activation matrix which is widely used for cross-layer comparison. We can also use the pre- activation matrix instead of the activation matrix. The third alternative is the weight matrix of the pre-trained model. Next, we discuss the choice of the cost function (layer dissimilarity). For the first representation, either the Euclidean distance or its squared version is sufficient. To measure the discrepancy between two activation matrices we choose the dissimilarity index based on CKA while another option is the Wasserstein distance. Finally, the cosine distance, Kullback-Leible divergence, and Wasserstein distance can be used in the weight matrix representation (Neill et al., 2020).
CLA for Convolutional neural network (CNN). Following NeuralMerger, we do not match fully connected layers and convolution layers. It is natural that a convolution layer has different functionality from fully connected layer. In addition, there is a lack of appropriate way to combine the weights of a fully connected layer and a convolution layer. For the rest of this paper, consider CNN architecture which consists of consecutive convolution layers followed by fully connected layers such as VGG (Simonyan & Zisserman, 2014) and RESNET (He et al., 2016). The key idea is to solve the CLA for same-type layers separately then combine two mappings. Note that the combination of two strictly increasing mappings is also a strictly increasing mapping, thus it satisfies the strictly increasing requirement. We further break the convolution layers part into smaller groups and find the mapping for each pair of groups. In particular, we divide into 5 groups separated by the max max-pooling pooolpooin layers in VGG. RESNET, we group consecutive blocks with the same number of channels, also known stage, and in each stage, align blocks instead of layers. For the block representation, we choose the (pre-)activation matrix of the second convolution layer each block.
3.2 LAYER BALANCING METHODS
As mentioned above, some layers in model A may have counterparts in model B in the optimal mapping. Naturally, we have two opposite directions: reduce layers in model A or add layers into model B. Assume that we have already balanced the number of layers up layer of model B. At layer + 1 of model B, there are two possibilities. If a(l + 1) a(l) = 1, the layer-wise fusion method is ready apply up layer I + 1. If a(l + 1) a(!) > can either merge layers between
5

---

Under review as a conference paper at ICLR 2022
Model
Model
Model
Model
Input
2
3
Oulput
(a) Adding layers
Merging layers
Figure 3: Layer balancing examples: In (a), new layer is added between hidden layers 1 and 2 of model B. Newly added neurons and weights are in red (We omit zero weights). The new weight matrix between layers I and 2 of model B is an identity matrix whose size equals the number of neurons at layer While the old weight matrix between layers and 2 model B becomes the new weight matrix between layers and 3. In (b), hidden layer model is removed. The red lines indicate the new weight matrix which is calculated based on two old weight matrices and the activation matrix of the removed layer.
all) and all + 1) of model or add layers between and ++ model B. Next, we discuss two approaches in the case al + 1) a(l) = 2. When all + 1) a(l) 2, we can repeat the same approach a(I + 1) (I) times.
Add layers into model B. We add layer 1' between layer and I 1 of model B. The new weight matrices are defined as w
( and w +- ( TRP xP The new
(1+))
B Pg activation function is defined as fB ) pla(1)+1), which ReLU. Because FB = for all I [n]] from Equation we have
(I)
1 (''))) (W'""""00 "200) = ReLU
Therefore, the information of model B remains unchanged after adding layer 1'. Note that the new layer just an identity mapping. which is trick that has been used in RESNET and NET2NET (Chen et al., 2015). Also discussed in NET2NET, the adding layers method is a function function-preserring transformation that allows us to generate a valuable initialization for training larger network. We later examine this hypothesis in our experiments.
Merge layers in model A. We merge layer a(I)+ + into layer a(l) of model by directly connecting layer a to layer all + 1). Because f p(a(1)+1) is ReLU, the new weight matrix can be written as w (a(++)),a(0) w (=(++1),a((++1) D ( W (a(()+1,(())
where (1) E TRA
(a(1+1) is an input-dependent diagonal matrix with Os and 1s on its diagonal. The 7th entry in the diagonal has a value if the ith entry of x (a(I)+1) is positive and otherwise. Because the actual sign of neuron at layer a() + varies by the input, we provide simple estimation. Given that the neuron E [p (a()+))) has a pre-activation vector over samples as Z., E R which is the t column of the pre-activation matrix (o(1)+1) We estimate the sign of neuron i using either sign of sum (sgn Σ zj,z) or sign of majority (i.e., if at least t/2 samples have positive activation values, O otherwise).
Pros and cons of balance methods. An advantage of merging layers is that the fused model has fewer layers. However, merging layers degrades the accuracy of model A and it slower than adding layers method that does not involve any complex calculations. On the flip side, adding layers does affect the accuracy of model but results in deeper fused model. Surprisingly, merging layers shows comparatively better performance than adding layers in experiments on MLP. Performance comparison between the two methods will be provided in Appendix C and D. Note that we can use more sophisticated model compression methods instead of the merging layers method. Similarly, it is possible to replace the adding layers method with a more advanced network expanding technique (Wei et al., 2016). Here, just introduce two natural and fast ways demonstrate our framework.
Balancing the number of layers for CNN. The same merging method does not work in the case of CNN. On the other hand, the adding layers method be easily applied for convolution layers.

---

Under review as a conference paper at ICLR 2022
90
88
Global bol. acoarwey
82-
80-
MF
Me
0.0 0.2
0.2
0.5
0.8
1.0
Weight propartion of model
Figure 4: The performance (global test accuracy on the whole MNIST dataset) of the fused model when fusing the specialist model A and the generalist model B in various weight proportions of model B. All results are averaged across 5 seeds. Detailed results given in Table 8.
For VGG, we can set all filters of a new convolution layer to identity kernels. For RESNET, we add block in which all filters of two convolution layers become zero kernels while the short-cut connection remains as the identity mapping. A block without short-cut connection can be simply replaced by identity convolution layers as in VGG.
4 EXPERIMENTS
Outline. Our method is first applied for two MLPs trained the synthetic MNIST dataset (LeCun et al., 1998). We then fuse RESNET34 and a RESNET18 trained on CIFARIO 10 dataset (Krizhevsky al., 2009). The last one teacher-student setting in which VGG architecture is utilized. The detailed settings including training hyperparameters, used assets, and computational resources are specified in Appendix B. We further conduct ablation studies to validate the effectiveness of cross- layer alignment and compare two layer balancing methods in Appendix C. The experimental results for different seeds can be found in Appendix D)
4.1 SKILL TRANSFER
Settings. In skill transfer, we aim to obtain single model that can inherit both overall and specialized skills from the two individual models. We adopt the same heterogeneous data-split technique in (Singh & Jaggi, 2020, Section 5.1) for two MLPs A and B. The MNIST dataset is separated into two datasets. One dataset has all images of label and 10% images of other labels while the other contains the rest. We train MLP with 3 hidden layers of size 400, 200, 100 for model and MI.P with 4 hidden layers of size 400, 200, 100, 50 for model B. The pre-activation matrix is obtained by running inference for 400 samples. We use the number of neurons layer representation and adding layers as the balancing method. The weight proportion of model B, which is referred to as WB is swept over 11 different values: wB {0.0, 0.1,..... 1.0).
Results. Figure 4 shows that when the weight proportion of model B is small (0.1 or 0.2), the fused model improves over both individual models. The reason is that the specialist model A has higher accuracy than the generalist model B. When distributing the weight proportion fairly (0.5), the fused model performs worst due to the effect of heterogeneous training data. The fused model has the best accuracy of 91 .31 %. which is higher than those of models A (90.76%) and B 87.5..). This claims that the knowledge has been transferred successfully to the fused model without retraining.
4.2 GENERATE AN EFFICIENT INITIALIZATION
Settings. Following the experiments conducted in (Singh & Jaggi, 2020, Section 5.3), CLAFusion can be used to generate an initialization when training the larger neural network. We apply CLAFusion to pre-trained RESNET34 and RESNETI8 on CIFAR10 dataset. Layer representation is the pre- activation matrix of 200 samples while the cost function is calculated by subtracting the linear CKA from 1. After fusing, we also perform finetuning improve the accuracy as observed in previous
7

---

Under review as a conference paper at ICLR 2022
MA
Mp
Ensemble Learning
Finetune
MA
MB
My
Mg depth-aligned
Random RESNET34
93.31
92.92
93.81 #0.21
65.58
93.52
93.29
93.60
93.22
92.00
#0.13
+0.20
+2.61
#0.09
#0.14
#0.14
#0.13
#0.17
Table 1: The results of fusing and finetuning residual networks CIFARIO dataset. The results shown are mean #std. deviation over seeds. The detailed results are reported in Table 11.
Method
HMT
HMT + Naive avg
HMT + OTFusion
HMT + CLA + OTFusion
Transfer
10.79 #1.01
37.49 +4.59
18.32 #1.76
55.22 +3.30
Finetune
92.93 #0.34
91.34 +4.00
93.66 #0.14
93.70 #0.05
Table 2:
The results of heterogeneous model transfer from RESNETI8 to RESNET34. Experiments are run 5 times and details can be found in Table 12.
model fusion works (Chou et al., 2018; Singh & Jaggi, 2020). The finetuning hyperparameters, which are adopted from (Singh & Jaggi, 2020, Section S3.1.3), are reported in Table
The idea of transferring the knowledge from a pre- pre-trained network to a deeper network before continuing training is similar to NET2NET. The term MB depth-aligned referred to the model state after adding layers into the shallower model but before fusing with the deeper model. We use NET2NET to generate a RESNET34 model (Mg depth-aligned) which preserves the performance of MB. Note that MB depth-aligned has the same accuracy the smaller model and the same architecture as the fused model. To compare with our method, we also finetune the pre-trained models, MM depth-aligned, and train RESNET34 from scratch. As reference, we report the result of ensemble learning that calculates the average predictions over all individual models.
Baselines.
Results. As can be seen from Table retraining from the fused model gains accuracy of 93.60, which is the highest among all initializations. This demonstrates the advantage of CLAFusion over NET2NET when initializing a large model from pre-trained small model. Our method allows combining the knowledge from the pre-trained large model to generate better initialization, rather than solely relying on the knowledge of the small model. Although the result is still lower than the accuracy of ensemble learning, the fused model almost halves the computational resources.
4.2.1 HETEROGENEOUS MODEL TRANSFER
Settings. Heterogeneous model transfer (Wang et al., 2021) is branch of model transfer that deals with heterogeneous neural networks. is contrasted homogeneous model transfer, also known as transfer learning, in which the same network architecture is used in both the pre-tru pre-training and finetuning phase. In this experiment, we apply CLAA and OTFusion to the heterogeneous model transfer method (in short, HMT). We transfer the pre-trained RESNETIS 88 to the pre-trained RESNET34 using different methods, then finetune and compare their performance. We use the same pre-trained models, layer representation, layer dissimilarity, and finetuning hyperparameters as in Section 4.2.
Baselines. The heterogeneous model transfer method (HMT) in the original paper (Wang et al., 2021) used. Model parameters of the pre- treitra RESNETI8 18 are transferred to the pre-trained RESNET34. The weights that are not transferred are set to the weights of the pre-trained RESNET34. In the second method (HMT + Naive avg). after transferring we apply naive averaging to the transferred RESNET34 and the pre-trained RESNET34. Naive averaging is replaced with FT'uusion in the third method (HMT OTFusion).. Different from the third method, the last method utilizes the optimal mapping from the CLA problem instead of the longest chain for layer-to-layer transfer.
Results. Table demonstrates the average performance of different transfer methods. The second row reports the model accuracy before finetuning while the third row shows the final performance. Using CLA improves the performance of model transfer significantly. After finetuning, the combination of CLA and OTFusion leads to an improvement of 0.77 compared to using HMT only.
4.3 TEACHER-STUDEN FUSION
Settings. In this experiment, we transfer the knowledge from the pre- pre-traine teacher model to the student model. We train two VGG models on CIFAR 10 dataset: model A has VGG13 architecture
8

---

Under review as a conference paper at ICLR 2022
# Params
Teacher
Students
Finetune
(Ma, Mp. My)
MA
MB
My
MA
Mp
My
MB aaph-aliggedd
(33M, 3M, 3M)
92.70
89.92
82.66
92.65
89.89
90.96
90.84
Table 3: The results of fusing teacher and student VGG models CIFARIO 10 dataset. More results can be found in Table 13.
Distillation initialization
Random MB
MB
Mf
Ms depth
Random My
Best
89.10
90.98
91.43
91.24
88.84
Average
88.41 #0.77
90.73
+0.43
91.16 #0.25
91.09 #0.19
88. #0.68
Table 4: Knowledge distillation from teacher MA into student model Mg. Results in the last row are mean #std. deviation over different temperatures. The detailed results are reported in Table 14.
with double number of neurons at each layer while model B has VGGI1 architecture with half number of neurons at each layer. After fusing the teacher and student models, we retrain the fused model and compare it the retraining result of the student model. The layer measure and the layer dissimilarity are identical to Section 4.2. We adopt the best hyperparameters as reported in (Singh & Jaggi, 2020, Section S11) for retraining, All finetuning hyperparameters are reported in Table 10.
Results. The classification accuracies are summarized in Table 3. After retraining, the fused model yields better performance than retraining other student models. The compression ratio between the fused model and the teacher is about 11 at a cost of reducing 1.74% accuracy. This suggests that CLAFusion can act network pruning method compress heavy model into lightweight one.
4.3.1 KNOWLEDGE DISTILLATION
Settings. As discussed in OTFusion, the fused model is efficient initialization for knowledge distillation. To examine the same application of CLAFusion, we perform knowledge distillation to the above pre-trained VGG models. We employ the method in Hinton et al. (2015) match the logit distribution of the student model to the teacher model. For initialization, we consider five different choices for the student model: (a) randomly initialized MB. (b) MB, (c) My, (d) MB depth-aligned. (e) randomly initialized My. We sweep over set of hyperparameters to choose the best combination that maximizes the accuracy of the student model. The sets of hyperparameters for distillation also follows the setting in (Singh & Jaggi, 2020, Section S12): temperature T {20, 10, 4, } and loss-weight factor = {0.05, 0.1, 0.5, 0.7, 0.95, 0.99). J. For a fair comparison, the hyperparameters for training student models are identical to finetune hyperparameters in Section 4.3.
Results. The results for retraining and knowledge distillation are reported in Table 4, further results can be found in Table 14. The first row shows the best performance for all combinations of hyperparameters. The second row is the mean of the best accuracies obtained at different temperatures. Using the fused model initializations. It showcases the application of CLAFFsion as an efficient initialization for know medledge distillation. If averaging over different temperatures, retraining the fused model (90.96) works better than vanilla distilling from the teacher model into the original student architecture (88.41. 488.41, 90.73). Even the best accuracy of vanilla distillation (90.98) is slightly higher than that of retraining the fused model but comes with the cost of hyperparameter tuning. It further strengthens the claim that CLAFusion in combination with finetuning can act as a model compression method.
initialization achieves the best performance among different choices of
CONCLUSION
In the paper, we have presented framework for model fusion. Our CLAFusion extends layer-wise model fusion methods to the setting of heterogeneous neural networks by solving a cross-layer alignment problem, followed by a layer balancing step. CLAFusion has been successfully applied in the setting of heterogeneous data from CLAFusion achieves a better accuracy for RESNET trained on CIFAR 10 dataset. Furthermore, it shows potential applications for model compression and knowledge distillation.
skill transfer method. In addition, finetuning the fused network

---

Under review as a conference paper at ICLR 2022
Reproducibility Statement: We provide source codes for all experiments in the paper in the supplementary of the paper. The details of experimental settings, computational infrastructure, and other used public libraries are given in Appendix B. Neural network architectures and finetuning process of all experiments in the main text can be found in Appendix D. We also provide links to pre-trained models for each experiment in the README file of our source code.
Ethics Statement:
We expect that there are no negative social impacts from our work.
REFERENCES
Stephen Ashmore and Michael Gashler. A method for finding similarity between multi-layer perceptrons by forward bipartite alignment. In 2015 International Joint Conference on Neural Networks (LJCNN), pp. 1-7. IEEE, 2015. (Cited on pages and 2.)
Nicolas Bonneel and David Coeurjolly. SPOT: Sliced partial optimal transport, ACM Trans. Graph., 38(4), July 2019. ISSN 0730-0301. doi: 10. 115553330644633333021 URL https: /doiiii..///////// org/ 2...115533065
3323021. (Cited page 4.)
Defang Chen, Jian-Ping Mei, Yuan Zhang, Can Wang, Zhe Wang, Yan Feng, and Chun Chen. Cross- layer distillation with semantic calibration. arXiv preprint arXiv:2012.032.30 2020. (Cited on page 4.)
Tianqi Chen, Ian Goodfellow, and Jonathon Shlens. Net2net: Accelerating learning via knowledge transfer. arXiv preprint 1rtxiv::511.0000 2015. (Cited on pages 6 and 16.)
Yi-Min Chou, Yi-Ming Chan, Jia-Hong Lee, Chih-Yi Chiu, and Chu-Song Chen. Unifying and merging well-trained deep neural networks for inference stage. arXiv preprint arXiv: 1805: 04980, 2018. (Cited pages and 8.)
Marco Cuturi and Arnaud Doucet. Fast computation of Wasserstein barycenters. In International conference on machine learning, pp. 685-693. PMLR, 2014. (Cited on page 2.)
Alessio Figalli. The optimal partial transport problem. Archive for rational mechanics and analysis, 195(2):533-555) 560, 2010. (Cited on page 13.)
Rémi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Alaya, Aurélie Boisbunon, Stanislas Chambon, Lactitia Chapel, Adrien Corentlos, Kilian Fatras, Nemo Fournier, Léo Gautheron, Nathalie T.H. Gayraud, Hicham Janati, Alain Rakotomamonjy, levgen Redko, Antoine Rolet, Antony Schutz, Vivien Seguy, Danica J. Sutherland, Romain Tavenard, Alexander Tong, and Titouan Vayer. Pot: Python optimal transport. Journal of Machine Learning Research, 22(78):1-8, 2021. URL http:// / fmlr. org/ http//paprss/22// 20- 451 html. (Cited page 14.)
Jianping Gou, Baosheng Yu, Stephen J. Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision, Mar 2021. ISSN 1573-1405. doi: 10. s1 1226--021-1143-- URL http:/ dx. doi. . org/ 1t1111/011111100000111111 (Cited on page 2.)
Lars Kai Hansen and Peter Salamon. Neural network ensembles. IEEE transactions pattern analysis and machine intelligence, 12(10)9933-1011 1990. (Cited on page 1.)
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016. (Cited on page 5.)
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv: 550.002333 2015. (Cited on pages and 9.)
Gaofeng Huang and Andrew Lim. A hybrid genetic algorithm for three-index assignment problem. In The 2003 Congress on Evolutionary Computation, 2003. CEC*03., volume 4, pp. 2768. IEEE, 2003, (Cited on page 13.)
Leonid V Kantorovich. On the translocation of masses. Journal of mathematical sciences, 133(4): 1381-1382, 2006. (Cited page 2.)
10

---

Under review as a conference paper at ICLR 2022
Jun Kitagawa and Brendan Pass. The multi-marging marginal optimal partial transport problem. In Forum of Mathematics, Sigma, volume 3. Cambridge University Press, 2015. (Cited on page 13.)
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference Machine Learning, pp. 3519- 3529. PMLR, 2019. (Cited on page 2.)
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Master's s thesis, Department of Computer Science, University of Toronto, 2009. (Cited on page 7.)
Harold W Kuhn. The Hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83-97, 1955. (Cited on page 4.)
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278 22222222,2 1998. (Cited page 7.)
Mikhail Iu. Leontev, Viktoriia Islenteva, and Sergey V. Sukhov. Non-iterative knowledge fusion in deep convolutional neural networks. Neural Processing Letters, 51(1):1-22, Jul 2019, ISSN 1573-773X. doi: 10. 1007/s 1063-019-1007444
URL http: /tpp//xxx///// doi wwwp:o.o........../
s11063- 1110633019-1007- 100-4-00..... (Cited on page 2.)
Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John E Hoperoft. Convergent learning: Do different neural networks learn the same representations? In FE@ NIPS, pp. 196-212, 2015. (Cited on page 1.)
Gaspard Monge. Mémoire sur la théorie des déblais et des remblais. Histoire de l'Académie Royale des Sciences de Paris, 1781. (Cited page
James O' Neill, Greg Ver Steeg, and Aram Galstyan. Compressing deep neural networks via layer fusion. arXiv preprint arXiv:2007. 14917, 2020. (Cited on pages 2 and 5.)
Lyle Ramshaw and Robert Tarjan. On minimum-cost assignments unbalanced bipartite graphs. HP Labs, Palo Alto, CA, USA, Tech. Rep. HPPL-2012400P 2012. (Cited page 4.)
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogmition. arXiv preprint arXiv:1409.1556 2014. (Cited on pages 3 and 5.)
Sidak Pal Singh and Martin Jaggi. Model fusion via optimal transport, Advances in Neural Information Processing Systems, 33, 2020. (Cited on pages 7, 8, 9, 13, and 14.)
Joshua Smith and Michael Gashler. An investigation of how neural networks learn from the experi- ences of peers through periodic weight averaging. In 2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA), pp. 731-736. IEEE, 2017. (Cited pages and
Frits CR Spicksma. Multi index assignment problems: complexity, approximation, applications. In Nonlinear assignment problems, pp. 1-12. Springer, 2000. (Cited on page 13.)
Frederick Tung and Greg Mori.
knowledge distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 1365-1374, 2019. (Cited on page 4.)
Joachim Utans. Weight averaging for neural networks and local resampling schemes. In Proc. AAAI-96 Workshop on Integrating Multiple Learned Models. AAAI Press, pp. 133-138. Citeseer, 1996. (Cited pages and 2.)
Guangcong Wang, Jianhuang Lai, Wenqi Liang, and Guangrun Wang. Heterogeneous model transfer between different neural networks. Submission at International Conference on Learning Repre- sentations, 2021. URL https: //tp////e/eeeeieeeeeeeeee net wow.mood=dmmmdmaaaa. FKtV. (Cited
on pages 8 and 14.)
Hongyi Wang, Mikhail Yurochkin, Yuckai Sun, Dimitris Papailiopoulos, and Yasaman Khazacni.
Federated learning with matched averaging. arXiv preprint arXiv:2002.06440, 2020. (Cited on pages and 2.)
Tao Wei, Changhu Wang, Yong Rui, and Chang Wen Chen. Network morphism. In International Conference Machine Learning, pp. 364-572. PMLR, 2016. (Cited page 6.)
11

---

Under review as a conference paper at ICLR 2022
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the perfor- mance of convolutional neural networks via attention transfer. arXiv preprint arXiv: 1612.03988 2016. (Cited page 4.)
Chiyuan Zhang, Samy Bengio, and Yoram Singer. Are all layers created equal? arXiv preprint 192.09990,.01996 2019. (Cited on page 4.)
Zhi-Hua Zhou, Ensemble methods: Foundations and algorithms. CRC press, 2012. (Cited on page 1.)
12

---

Under review as a conference paper at ICLR 2022
Supplement to "Model Fusion of Heterogeneous Neural Networks via Cross-Layer Alignment"
In this supplementary material, we first present some thoughts on the application of CLAFusion to multiple models scenario Appendix A. The common settings, used assets, and computational resources are described in Appendix B. Next, we study the importance of constraints on the first and last layers as well as the efficiency of CLA in Appendix C. Finally, we provide additional results of experiments in Appendix D.
A CLAFUSION FOR MULTIPLE NEURAL NETWORKS.
The cross-layer alignment problem for multiple neural networks is a non-trivial task. It can be formulated multi-index assignment problem (Spieksma, 2000; Huang & Lim, 2003). In addition, it has some connections to the multi-marginat optimal partial optimal transport (Figalli, 2010; Kitagawa & Pass, 2015). it is an interesting direction for future work. Nevertheless, there are two naive approaches to extend CLAFusion to the case of multiple networks. Consider K pre-trained models [M In the first approach, we can apply CLAFusion for two models iteratively K times to fuse K models. In each iteration, we fuse pair of models and replace them with their fused model, thus the number of models reduces by 1. The second approach similar to what they did in OTFusion (Singh & Jaggi, 2020). Starting with an estimation of the fused model My. We apply the first and second parts of CLAFusion K times to align pre- trained models with respect to the fused model. Finally, average the weights of those aligned networks to produce the final weights for the fused model. The choice of My plays an important role in this approach. However, is unclear how it is chosen in OTFusion.
Results. We tried two naive approaches to fuse multiple networks. Unfortunately, we could not achieve desirable outcome. As for the first approach, cumulatively deteriorates the performance of the fused networks after each iteration. For the second approach, we could not achieve a good initialization for the fused model as in OTFusion. It important to stress that our settings are more difficult than previous model fusion methods because we consider two networks with different numbers of layers.
B EXPERIMENT SETTINGS
Following OTFusion, the layer-wise model fusion is computed using the activation- bbased alignment strategy in which the pre-activation matrix instead of the activation matrix used for the neuron measure. The hyperparameters for pre- pre-trained models summarized in Table 5. Without any further specification, the random seed for training or retraining is set to its default value. The accuracy of pre-trained MIP is the accuracy of the last epoch. While the accuracies of VGG and RESNET reported as the best performing checkpoint. For all finetuning experiments, we always chose the best record among all epochs.
MLP
VGG
ResNet
Number of epochs
10
300
300
Training batch size
64
128
256
Test batch size
1000
1000
1000
Optimizer Initial LR
SGD
SGD
SGD
0.01
0.05
0.1
Momentum
0.5
0.9
0.9
Weight decay
0.0005
0.0001
LR decay factor
2
10
LR decay epochs
30,60. 2,70
150,250
Default seed
42
42
5 seeds
0,1,2,3,4
40,41,42,43,44
40,41,42,43,44
Table 5: The hyperparameters for training different network architectures.
13

---

Under review as a conference paper at ICLR 2022
Pair Layer representation
Cost function
Both
Last layer constraint
First layer constraint
None
Number of neurons
squared Euclidean
[1. 5]
[1. []
[1.3]
[1. 31
activation matrix
1 linear CKA
[1,5]
[1.
[1,
2]
Pre-activation matrix
Wasserstein distance
[1.5]
[2.
[1.3]
[2.3]
Number of neurons
squared Euclideun
2. 51
11.
[1. 31
[1,
2
Pre-activation matrix
1 linear CKA
21
[1.2,5]
[1,
[1,
Pre-activation matrix
Wassenstein distance
[1.
[1,
[1,
[1,2,3]
Table 6: The optimal CLA for different combinations of layer representation and cost function.
Open source code. We adapt the official implementation of OTFusion' (Singh & Jaggi, 2020) in our implementation. For computing the optimal transport, we use Python Optimal Transport (POT) library" (Flamary al., 2021). For model transfer, the source code of the original paper* (Wang et al., 2021) is utilized.
Computational resources. All experiments are done 1 Tesla V100 GPU.
C ABLATION STUDIES
Settings. We compare the performance of different combinations of layer representation and layer balancing methods in our framework. To prove the efficiency of the CLA step, we compare the result of fusing two models using different mappings. We also consider removing the constraint the first and last hidden layers. We train two pairs of MLP in 5 different seeds. In both two pairs. model A has 5 hidden layers of size 400, 200, 100. 50, 25. Model has 2 hidden layers of size 400, 100 in the first pair while it has 3 hidden layers of size 400, 200, 100 in the second pair.
Results of the optimal mappings. The cross-layer mappings obtained using Algorithm with different combinations are given in Table 6. According to Definition 1 in Section 3.1, the optimal mapping is strictly increasing mapping a [[n]]] + [m]]]] where n and m are the number of layers of the shallower and deeper networks, respectively. In this case, we have = 5, = 2 for the first pair, and n for the second pair. The optimal mapping a is represented as [a(1), o a-n)]. For example, [1, 2, 5] means that a(1) 1, a2) 2, and a(3) 5. We observe that changing the random seed barely affects the result of CLA in this setting. When keeping both first and last layer constraints, all three combinations yield the same mappings in both pairs. For the second pair, all three combinations share the same mappings are placed (When the random seed is 2, the optimal mapping for pre-activation matrix + I- linear CKA is [1, 2, 5] instead of [1, 2, 3J). On the other hand, removing both constraints in the first pair makes each combination have different mapping.
out seeds even none of the two constraints
The elfectiveness of the optimal mapping. We fuse each pair of MLPs for all 10 possible mappings between two models A and B. For the layer balancing method, we try both adding layers and merging layers (the sign of sum estimation). The average performance of pre-trained models and the fused model across random seeds are summarized Table 7. In three out of four scenarios, the best accuracy of the fused model is achieved when imposing both the first and last layer constraints. Removing the last layer constraint decreases the performance of the fused model. When removing the first layer constraint, the accuracy drops even more significantly. Because an input image may consist of negative cell values due to normalization, adding a new layer as the first hidden layer does not maintain the accuracy of the shallower model. Therefore, it is necessary to match the first and last hidden layers of the two models. In addition, the mappings obtained from the CLA step generally result in higher accuracy than other mappings, proving the efficiency of the CLA. Comparing between layer balancing methods, the merging layers method runs slower but leads to a higher accuracy than the adding layers in both pairs.
' com/ssdaa//offfsiiin h httphh/www.oooooyoooooeoeeeeee
https://anooymoos.oppmmoo.cc wwtp://pwpwococcoooeeeeeeeee had hetero, model, transfrpra
14

---

Under review as a conference paper at ICLR 2022
Pair
Mapping
MA
MB
Mf
Add
Merge
[1
91.91
95.11
[1.3]
91.95
94.20
[1,4]
92.26
93.30
[1.5]
92.38
93.18
[2,
79.20
62.93
1
96.95
97.59
[2,4]
81.73
57.09
[2
81.05
58.17
[3.4]
75.42
56.48
[3.5]
75.28
58.34
[]
72.68
56.00
[1, 2, 3]
92.33
93.90
[1, 2, 4]
92.48
93.41
[1,
92.49
94.00
[1.3.4]
90.69
92.21
[.,
90.88
93.04
2
96.95
97.75
[1,4.5]
90.98
93.06
3,
82.21
51.26
[2, 3.
83.14
56.56
[2, 5]
83.00
54.16
[3, 4.
79.49
56.31
Table 7: Performance comparison between different combinations of mapping and balancing methods.
Seed
W/B
0.2
0.3
0.5
0.6
0.8
0.9
86.10
88.23
78.64
69.70
70.45
81.38
85.36
91.60
91.81
91.23
85.63
77.20
77.99
81.59
84.85
86.88
87.48
87.29
90.80
92.07
92.46
84.06
82.53
83.21
85.01
86.83
87.76
88.01
87.83
Adding layers
79.58
78.31
85.79
87.01
87.51
93.34
93.31
91.56
81.70
77.45
78.46
82.15
85.00
86.62
87.37
87.58
Avg
90.76
91.31
90.97
81.92
76.50
76.23
79.50
83.24
85.69
87.05
87.58
86.10
87.19
87.10
78.51
63.66
56.74
$3.31
53.88
57.65
66.55
76.35
91.60
92.23
91.83
89.08
78.69
76.04
77.34
81.44
84.87
86.39
90.80
91.98
92.30
86.00
81.27
79.42
80.09
82.95
85.46
85.61
85.04
Merging layers
91.94
92.66
92.76
88. 13
71.65
65.90
64.20
67.39
72.91
76.06
77.40
93.34
93.18
90.03
79.27
82.52
83.91
Avg
90.76
91.37
91.43
86.35
75.10
71.47
71.19
73.64
76.96
79.86
82.10
Table 8: Performance comparison between layer balancing methods on skill transfer task. Experiments are run 5 times.
D ADDITIONAL EXPERIMENT RESULTS
D.1 SKILL TRANSFER
Results. The results of skill transfer for different random seeds are reported in Table 8. The best accuracy for each seed is written in bold font. The special list model A generally has higher accuracy than the generalist model seeds because has been trained on all 10 labels. In out of seeds, the fused model improves over both individual models when the weight proportion of model is small (wB- = (0.1, 0.2)).
Comparison between two layer balancing methods. For comparison purposes, further conduct the same procedure but replace adding layers by merging layers (the sign of sum estimation). Similar to the previous experiment, the fused model attains the best performance when the weight proportion of model B is small (w = {0.1, 0.2)). Figure illustrates the average performance of two layer balancing methods the skill transfer task. When averaging over 5 random seeds, both methods
15

---

Under review as a conference paper at ICLR 2022
52.3
90.3
88-
87.5
86-
Globd test aecurMc
85.0
82.5-
844
60.0-
122
77.3
80-
70.0
75-
MA
Mip digged
72.3
Mo digud
0.11
0.2
0.4
0.6
1.0
0.11
0.2
144
0.6
0.8
Wegght posoetiie of mode
Wiigtt proportlon model
(a) Adding layers
(b) Merging layers
Figure 5: The average performance of skill transfer using (a) adding layers and (b) merging layers the layer balancing method.
Number of epochs
120
Number of epochs
120
Training batch size
256
Training hatch size
128
Test hatch size
1000
Test batch size
1000
Optimizer
SGD
Optimizer
SGD
Initial LR
0.1
Initial LR
0.01
Momentum
0.9
Momentum
0.9
Weight decay
0.0001
Weight decay
0.0005
LR decay factor
2
decay factor
LR decay epochs
20,40,60,80,100
LR decay epochs
20,40,60,80,100
Table 9: Finetuning RESNET hyperparame- ters.
Table 10: Finetuning teacher-student models hyperparameters.
Seed
MA
Ma
Ensemble
MF
Finetune
Learning
MA
M
M,
Mr depth-aligned
Random RESNET34
40
93.42
93.21
94,10
67.83
93.50
93.47
93.60
93.25
91.95
41
93.37
92.90
93.81
63.16
93.61
93.26
93.73
93.43
91.93
42
93.31
93.05
93.94
68.44
93.61
93.36
93.77
93.19
92.25
43
93.39
92.78
93.71
66.60
93.50
93.30
93.47
93.20
91.76
93.06
92.65
93.49
61.86
93.37
93.05
93.41
93.01
92.12
Avg
93.31
92.92
93.81
65.58
93.52
93.29
93.60
93.22
92.00
Table 11: The results of finetuning RESNET from different choices of initialization across 5 different seeds.
perform best if wg = 0.1. The best accuracy of merging layers (91.57) is slightly higher than that of adding layers (91.31) even though the accuracy of MB depth-aligne drops from 87.58 to 82.10.
D.2 GENERATE AN EFFICIENT INITIALIZATION
Results. Finetuning from the fused model always improves the accuracy of the pre-trained models as illustrated Table 11. In comparison with other initializations, it yields the best performance in out of 5 seeds. MB depth-aligned, which is the result of NET2NET (Chen et al., 2015) operation, only performs better than finetuning from scratch and be comparable to that of HMT. Starting from the results of NET2NET and HMT are worse than that from the smaller pre- pre-trained model. (Note that MR + MA divergences with LR 0.1 so its accuracy is 10, which equals to random guessing.)
16

---

Under review as a conference paper at ICLR 2022
Method
Seed
HMT
HMT Naive avg
HMT OTFusion
HMT CLA OTFusion
Transfer
40
9.55
37.80
20.17
60.72
41
10.13
37.66
17.66
52.09
42
10.45
43.44
19.25
55.94
43
12.44
39.24
15.20
55,82
44
11.37
29.33
19.30
51,55
Avg
10.79
37.49
18.32
55.22
Finetune
40
93.29
93.63
93.76
41
93.14
93.28
93.69
93,69
42
92.48
93.16
93.78
93,68
43
93.20
93.27
93.80
93.62
44
92.93
83.35
93.41
93.75
Avg
92.93
91.34
93.66
93.70
Table 12: The results of model transfer from MB to MA across 5 different seeds.
Seed
Ma
B
Ensemble Learning
Mf
Retrain
MA
Mg
M,
My depth-aligned
40
92.71
89.68
92.53
81.66
92.67
89.81
90.64
90.54
92.34
89.61
92.68
82.59
92.50
89.57
90.59
90.42
42
92.70
89.92
92.86
82.66
92.65
89.89
90.96
90.84
92.79
89.78
92.82
81.79
92.77
90.10
90.55
90.61
44
92.70
89.62
92.51
81,43
92.56
89.46
90.44
90.19
Avg
92.65
89.72
92.68
82.03
92.63
89.77
90.64
90.52
Table 13:
Finetuning teacher-student VGG across different seeds. The results reported in Table are run with the random seed of 42.
D.3 HETEROGENEOUS MODEL TRANSFER
Results. Table 11 details the results of 4 different model transfer methods. Combining CLA and OTFusion with HMT always results in great boost (at least 17.73) in the accuracy of the transferred model. After finetuning, it gives the best performance in 3 out of 5 random seeds.
D.4 TEACHER-STUDERT FUSION
Table 13 illustrates the finetuning result of teacher and student models across different seeds. The fused model M, is the most productive initialization student models 4 out of seeds. Retraining the fused model always yields higher accuracy than continuing training the student model.
Results.
Discussion. At first glance, our approach may resemble the NET2NET operations. Although both approaches increase the depth of the student network and use the deeper student as a good initialization, there are two major differences between ours and NET2NET. Firstly, we present systematic way of finding the location to add the identity mappings for different types of network architectures while NET2NET manually designed for specific type of network. Secondly, we do not enlarge the width of the student network (equivalently, only NET2DEEPNET operation) so that the student network remains more compact than the teacher network.
D.5 KNOWLEDGE DISTILLATION
The results of distilling the teacher model into the student model are given Table 14. Knowledge distillation from the fused model leads to the best accuracy in 4 out of temperatures. In addition, both its average and best performance are the highest among all initializations. MB depth-aligned, which is obtained from the NET2NET operation, is the second-best choice, followed by the pre-trained student model MB. Naive knowledge distillation into either random initialized student models, however, yields much lower accuracy. This suggests that CLAFusion can serve as an efficient initialization knowledge distillation.
Results.
17

---

Under review as a conference paper at ICLR 2022
Temperature
Distillation initialization
Random MB
MB
MB depth-aligned
Random Mf F
88.99 (0.70)
90.94 (0.70)
91.29 (0.10)
91.14 (0.70)
88.20 (0)10)
10
89.10 (0.70)
90.97 (0.70)
91.23 (0.50)
91.23 (0.10)
88.84 (0.50)
8
88.91 (0.70)
90.98 (0.70)
91.43 (0.70)
91.24 (0.70)
88.76 (0.50)
87.89 (0.70)
90.81 (0.99)
91.14 (0.50)
91.10 (0.50)
87.65 (0.05)
87.14 (0)
89.97 (0.10)
90.69 (0.05)
90.73 (0.05)
87.04 (0.05)
Best
89.10 10
90.98
91.43
91.24
88.84
Avg
88.41
90.73
91.16
91.09
88.10
Table 14: Distillation results for different temperatures. Each entity is the best accuracy obtained by varying the boss-weight factor. The last two rows report the best and average performance across different temperatures.
18


Reviews:
[{"review_id": "meXKqLVtfrE", "reviewer": "Reviewer_1twh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper presents an algorithm named CLAFusion. The goal is to fuse two networks with different architectures so that the fused one has reasonable computational complexity and works favorably compared to the separate ones. Experiments on CIFAR10 and small neural networks show the effectiveness.\n\nI have the following concerns about this paper.\n\nFirst, I am thinking of the practical value of this algorithm, as well as the OTFusion algorithm. If the cost of training multiple to-be-fused networks is acceptable, the baseline that simply uses all these costs to train the target network should be compared. For example, if training a ResNet34 needs 10 hours and a ResNet18 needs 6 hours, then what is the performance of training a ResNet34 for 16 hours? Many prior works showed that simply extending the training epochs can improve classification accuracy.\n\nSecond, the experimental part only shows CIFAR10 results. This is far below expectation. Since training-from-scratch is not required, I would expect ImageNet results to be offered, or at least CIFAR100 can be tested. We know that CIFAR10 is a very small and relatively simple dataset, and more importantly, the classification accuracy on this dataset often suffers ~0.5% random noise, so using it as the single testbed is not acceptable to me. BTW, the deviation of different algorithms should be reported, especially when the gain is smaller than 0.5% (e.g. Table 1).\n\nThird, I tried to read the paper multiple times, but I cannot find the technical description that how two models with the same number of layers are combined. Section 3 ends with adding layers to the shallow network or merging layers in the deep network, but I cannot find the next procedure. Is it missing in the paper?\n\nBTW, I cannot say that the paper is well organized. It spent too much space in the introduction and background parts, but many details (e.g. how to compute the cost matrix or how to combine the networks) are not discussed.", "summary_of_the_paper": "This paper presents an algorithm named CLAFusion. The goal is to fuse two networks with different architectures so that the fused one has reasonable computational complexity and works favorably compared to the separate ones. Experiments on CIFAR10 and small neural networks show the effectiveness.", "summary_of_the_review": "1. The practical value is questionable.\n2. The description of the approach is not complete.\n3. Experiments are only performed on very small datasets.\n\nOverall, this paper is not well prepared for publication.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Y6bk12A0GPK", "reviewer": "Reviewer_pWLE", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I have two main concerns about this paper.\n(1)Neural networks have many layers and neural networks are non-linear. It is unclear that simply computing average models over two models ($A$ and $B^{‘}$) leads to better performance compared with $A$. For example, for layer $l$, $A$ and $B$ have weights $W_A^l$ and $W_{B^{‘}}^l$, the input of this layer is denoted as $x$, we obtain $W_A^lx$ and $W_{B^{‘}}^lx$. It is hard to understand to fuse $f(W_A^lx)$ and $f(W_{B^{‘}}^lx)$ as $f((W_A^l+W_{B^{‘}}^l)x)$ where $f(\\cdot)$ is a non-linear activation function.\n\n(2)The experiments are very weak. For Table 1, $M_A$ obtains $93.52\\%$ and the $M_F$ obtains $93.60\\%$. For Table 2, HMT+CLA+OTFUSION obtains $93.70\\%$ against $93.52\\%$. For Tables 3 and 4, $M_F$ obtains about $1.07\\%$ and $0.43\\%$ improvement, respectively. But only VGGNet is provided. Extending the experiments to state-of-the-art networks and larger datasets are suggested. \n\n(3) For Table 3 and Table 4, why does $M_F$ have 3M parameters as $M_B$, and $M_A$ have 33 M parameters. Do the authors use merging layers here? As mentioned before, the authors claim that the same merging method does not work in the case of CNN. ", "summary_of_the_paper": "This paper proposes a layer-wise model fusion method that fuses neural networks with a different number of layers by cross-layer alignment. Given two networks $A$ and $B$ ($A$ is deeper than $B$), the framework first expands $B$ as $B^{’}$ via cross-alignment such that the structure of $B^{‘}$ is the same as that of $A$ while keeping mapping unchanged. Then it performs fusion for $B^{‘}$ and $A$. The idea is interesting. The experiments show slight improvements compared with baselines.", "summary_of_the_review": "(1) Model fusion is a challenging problem in deep learning. The topic is interesting.\n(2) The explanations and analyses of the proposed model fusion are not convincing.\n(3) More expensive experiments are needed.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "S-pn7seLB6w", "reviewer": "Reviewer_HQUA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- Strengths \n\n-- The paper is relatively easy to understand, especially the  high-level motivation for more efficient model ensembling.\n\n- Weakness\n\n-- I'm having a hard time to convince myself why the proposed method can work at all, as it basically assumes, the different networks learn similar parameter statistics, which is an invalid assumption to me, given different architectures differ significantly, for example, we know that, using residual allows to train very deep networks easily, while if not using residual, it will require many tricks to make it work, it clearly shows the behaviour of different networks varies significantly, also, the network initialisation will matter a lot. \n\n-- Please could the authors demonstrate some quantitative evaluation or theoretical proof, to prove me wrong ?\n\n-- Experiment-wise, it's not convincing either, the demonstrated experiments are too toyish, and it seems only work on synthetic MNIST. I don't understand why it is not working for CIFAR, as it should not require finetuning either. Also, since the idea doesn't not require re-training the model, can we show the effectiveness on ImageNet ?\n\n-- I can't find know how different models are combined.\n\n-- In Figure 2, why the size of the the hidden layers are of 4,4,2,1 and 4,2,1 ? should it be 5,4,3,1 and 3, 2, 1 ?", "summary_of_the_paper": "- The paper considers to look for a single neural network that can inherit the knowledge from multiple pre-trained neural networks and have a small size at the same time.\n\n- The authors propose a model fusion framework for fusing heterogeneous neural networks, *Cross-Layer Alignment Fusion* (CLAFusion)\n\n- CLAFusion consists of three parts, cross-layer alignment, layer balancing method, and layer-wise model fusion method. Generally, speaking, it aims to find the correspondence between layers in different networks, and then add or merge the layers in different models to balance the number of layers, and fuse the parameters eventually.\n\n", "summary_of_the_review": "Overall, I think the paper is below the acceptance threshold, and my main question is really about the fundamental assumption the authors have made, why networks of different architecture can be combined in that manner ?", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "PVya3-GN19j", "reviewer": "Reviewer_LJwg", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Pros:** \n- Relatively simple bag of tricks that are reasonably motivated and that enable OTFusion to handle different number of layers, which importantly reduces a key shortcoming of OTFusion.\n- Empirical demonstration on several settings is thoroughly done and shows a decent advantage of the fusion approach.\n- It seems that some ablations are also carried out for the particular choices made in their framework. E.g., enforcing the mapping of first and last hidden layers. But this is all in the supplementary and the description could be made more clear. \n\n**Cons:**\n\n*No demonstration for >2 networks:* The main drawback of this approach is that its empirical application is limited to the case of only 2 networks. It is not clear how components of their approach like Algorithm 1 will behave when applied to the case of > 2 networks. While the authors provide a descriptive account in the appendix of a possible way around, but the concrete application is still up in the air. It is important that authors provide at least some, perhaps preliminary, evidence in this regard. (This is unlike most related work on model fusion which shows a much thorough demonstration for multiple-model setting)\n\n*Clarity, presentation, empirical details:* This is another very important aspect that the authors should really work hard upon, as currently, the paper is hard to read.  \n1. Many important empirical details are missing in the main text, and one has to go hunting in the appendix to understand what's going on. This is despite that the paper is not as space-constrained either. E.g., how two layers are adjudged to be similar, heterogenous model transfer, etc. Similarly, the table captions could be a bit more detailed. \n2. Several references are made to OTFusion, but it gets clarified only when referenced on page 3. Please make this clear upfront. \n3. The reference for model fusion on page 2 to (Claici et al, 2020) is inaccurate in this particular context. Please use something more appropriate like Wang et. al. 2020 and/or Singh & Jaggi 2020.\n4. 'M_B aligned' is not as clear or even slightly misleading. Perhaps an alternative could be 'M_B depth-aligned'? This is because M_B is still not aligned with M_A in the sense of permutations. \n5. This paper heavily relies on OTFusion, and thus I think perhaps a more informative abbreviation is  'CLOTFusion'. \n6. I suppose that what you refer as 'HMT' is using Wang et. al., 2021 to first fuse Resnet18 and Resnet34 into a fused ResNet34 model 'HMT', which you then further combine with the original ResNet34 via naive avg, OTFusion, CLAFusion? Besides, what is the 'TRANSFER' in Table 2 and why does it have so dismal performance as compared to 'FINETUNE'?\n7. Can you explain how Section 4.2 is inspired from Net2Net (whose reference, btw, is missing)? To me, it seems to be more similar to the setting in OTFusion of Tables 1-3. \n8. Tables 6, 7 in the appendix are quite hard to read. Can you define and better explain the notational usage there?\n\n*Language issues:* This is a minor but understandable issue. Yet, strictly speaking, on many occasions, it prevents a precise understanding of what is being articulated. Please have the text be proofread or use something like Grammarly. Some examples of weird/ungrammatical language constructs include: \"still needs to be secure\" -> \"still needs to be secured\", \"stark development\", \"It was empirically found that vanilla averaging combines\", \"Because vanilla averaging did not study the permutation invariance nature\". Besides, in many places, appropriate punctuation marks are missing. \n\nI am happy to improve my score if the above concerns are adequately addressed. ", "summary_of_the_paper": "This work extends the OTFusion approach, for fusing multiple neural networks into a single neural network, to handle the case of networks with different number of layers. To this end, the authors first find a mapping between layers of the shallower network to that of the deeper network (by a dynamic programming approach, reminiscent of longest-subsequence like problems), then balance the number of layers (by either adding identity connections in one or merging layers), after which OTFusion is called upon to do the rest. The authors provide an empirical demonstration of their approach in similar settings as in the OTFusion paper, but where they can now successfully handle networks with different numbers of layers (but in the setting of two input networks).", "summary_of_the_review": "The extension of model fusion (OTFusion) to heterogeneous settings (in the sense of different depths) seems to be well-executed. There is still room for many of the choices to be more streamlined or automated, but nevertheless does show fairly satisfactory results across various settings. The other important drawback is that the current empirical demonstration is limited to the case of 2 networks. The lack of clarity in the presentation, at some places, does not help either.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: While fusing multiple heterogeneous neural networks into a single network looks like an interesting exploration, there are many major concerns raised by the reviewers:
1) The motivation why the proposed method works is not convincing. In other words, under what conditions the proposed would work or would not work is not clear.
2) The authors failed to provide either theoretical analysis or convincing empirical studies of the proposed method. In the rebuttal, the authors did not address the critical issues raised by the reviewers.
3) There are many other detailed problems about the proposed method as well as the experimental setup.

Therefore, by considering the above concerns, this submission does not meet the standard of publication at ICLR.
Decision: Reject