Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: - The paper considers to look for a single neural network that can inherit the knowledge from multiple pre-trained neural networks and have a small size at the same time.

- The authors propose a model fusion framework for fusing heterogeneous neural networks, *Cross-Layer Alignment Fusion* (CLAFusion)

- CLAFusion consists of three parts, cross-layer alignment, layer balancing method, and layer-wise model fusion method. Generally, speaking, it aims to find the correspondence between layers in different networks, and then add or merge the layers in different models to balance the number of layers, and fuse the parameters eventually.


main_review: - Strengths 

-- The paper is relatively easy to understand, especially the  high-level motivation for more efficient model ensembling.

- Weakness

-- I'm having a hard time to convince myself why the proposed method can work at all, as it basically assumes, the different networks learn similar parameter statistics, which is an invalid assumption to me, given different architectures differ significantly, for example, we know that, using residual allows to train very deep networks easily, while if not using residual, it will require many tricks to make it work, it clearly shows the behaviour of different networks varies significantly, also, the network initialisation will matter a lot. 

-- Please could the authors demonstrate some quantitative evaluation or theoretical proof, to prove me wrong ?

-- Experiment-wise, it's not convincing either, the demonstrated experiments are too toyish, and it seems only work on synthetic MNIST. I don't understand why it is not working for CIFAR, as it should not require finetuning either. Also, since the idea doesn't not require re-training the model, can we show the effectiveness on ImageNet ?

-- I can't find know how different models are combined.

-- In Figure 2, why the size of the the hidden layers are of 4,4,2,1 and 4,2,1 ? should it be 5,4,3,1 and 3, 2, 1 ?
summary_of_the_review: Overall, I think the paper is below the acceptance threshold, and my main question is really about the fundamental assumption the authors have made, why networks of different architecture can be combined in that manner ?
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work extends the OTFusion approach, for fusing multiple neural networks into a single neural network, to handle the case of networks with different number of layers. To this end, the authors first find a mapping between layers of the shallower network to that of the deeper network (by a dynamic programming approach, reminiscent of longest-subsequence like problems), then balance the number of layers (by either adding identity connections in one or merging layers), after which OTFusion is called upon to do the rest. The authors provide an empirical demonstration of their approach in similar settings as in the OTFusion paper, but where they can now successfully handle networks with different numbers of layers (but in the setting of two input networks).
main_review: **Pros:** 
- Relatively simple bag of tricks that are reasonably motivated and that enable OTFusion to handle different number of layers, which importantly reduces a key shortcoming of OTFusion.
- Empirical demonstration on several settings is thoroughly done and shows a decent advantage of the fusion approach.
- It seems that some ablations are also carried out for the particular choices made in their framework. E.g., enforcing the mapping of first and last hidden layers. But this is all in the supplementary and the description could be made more clear. 

**Cons:**

*No demonstration for >2 networks:* The main drawback of this approach is that its empirical application is limited to the case of only 2 networks. It is not clear how components of their approach like Algorithm 1 will behave when applied to the case of > 2 networks. While the authors provide a descriptive account in the appendix of a possible way around, but the concrete application is still up in the air. It is important that authors provide at least some, perhaps preliminary, evidence in this regard. (This is unlike most related work on model fusion which shows a much thorough demonstration for multiple-model setting)

*Clarity, presentation, empirical details:* This is another very important aspect that the authors should really work hard upon, as currently, the paper is hard to read.  
1. Many important empirical details are missing in the main text, and one has to go hunting in the appendix to understand what's going on. This is despite that the paper is not as space-constrained either. E.g., how two layers are adjudged to be similar, heterogenous model transfer, etc. Similarly, the table captions could be a bit more detailed. 
2. Several references are made to OTFusion, but it gets clarified only when referenced on page 3. Please make this clear upfront. 
3. The reference for model fusion on page 2 to (Claici et al, 2020) is inaccurate in this particular context. Please use something more appropriate like Wang et. al. 2020 and/or Singh & Jaggi 2020.
4. 'M_B aligned' is not as clear or even slightly misleading. Perhaps an alternative could be 'M_B depth-aligned'? This is because M_B is still not aligned with M_A in the sense of permutations. 
5. This paper heavily relies on OTFusion, and thus I think perhaps a more informative abbreviation is  'CLOTFusion'. 
6. I suppose that what you refer as 'HMT' is using Wang et. al., 2021 to first fuse Resnet18 and Resnet34 into a fused ResNet34 model 'HMT', which you then further combine with the original ResNet34 via naive avg, OTFusion, CLAFusion? Besides, what is the 'TRANSFER' in Table 2 and why does it have so dismal performance as compared to 'FINETUNE'?
7. Can you explain how Section 4.2 is inspired from Net2Net (whose reference, btw, is missing)? To me, it seems to be more similar to the setting in OTFusion of Tables 1-3. 
8. Tables 6, 7 in the appendix are quite hard to read. Can you define and better explain the notational usage there?

*Language issues:* This is a minor but understandable issue. Yet, strictly speaking, on many occasions, it prevents a precise understanding of what is being articulated. Please have the text be proofread or use something like Grammarly. Some examples of weird/ungrammatical language constructs include: "still needs to be secure" -> "still needs to be secured", "stark development", "It was empirically found that vanilla averaging combines", "Because vanilla averaging did not study the permutation invariance nature". Besides, in many places, appropriate punctuation marks are missing. 

I am happy to improve my score if the above concerns are adequately addressed. 
summary_of_the_review: The extension of model fusion (OTFusion) to heterogeneous settings (in the sense of different depths) seems to be well-executed. There is still room for many of the choices to be more streamlined or automated, but nevertheless does show fairly satisfactory results across various settings. The other important drawback is that the current empirical demonstration is limited to the case of 2 networks. The lack of clarity in the presentation, at some places, does not help either.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a layer-wise model fusion method that fuses neural networks with a different number of layers by cross-layer alignment. Given two networks $A$ and $B$ ($A$ is deeper than $B$), the framework first expands $B$ as $B^{’}$ via cross-alignment such that the structure of $B^{‘}$ is the same as that of $A$ while keeping mapping unchanged. Then it performs fusion for $B^{‘}$ and $A$. The idea is interesting. The experiments show slight improvements compared with baselines.
main_review: I have two main concerns about this paper.
(1)Neural networks have many layers and neural networks are non-linear. It is unclear that simply computing average models over two models ($A$ and $B^{‘}$) leads to better performance compared with $A$. For example, for layer $l$, $A$ and $B$ have weights $W_A^l$ and $W_{B^{‘}}^l$, the input of this layer is denoted as $x$, we obtain $W_A^lx$ and $W_{B^{‘}}^lx$. It is hard to understand to fuse $f(W_A^lx)$ and $f(W_{B^{‘}}^lx)$ as $f((W_A^l+W_{B^{‘}}^l)x)$ where $f(\cdot)$ is a non-linear activation function.

(2)The experiments are very weak. For Table 1, $M_A$ obtains $93.52\%$ and the $M_F$ obtains $93.60\%$. For Table 2, HMT+CLA+OTFUSION obtains $93.70\%$ against $93.52\%$. For Tables 3 and 4, $M_F$ obtains about $1.07\%$ and $0.43\%$ improvement, respectively. But only VGGNet is provided. Extending the experiments to state-of-the-art networks and larger datasets are suggested. 

(3) For Table 3 and Table 4, why does $M_F$ have 3M parameters as $M_B$, and $M_A$ have 33 M parameters. Do the authors use merging layers here? As mentioned before, the authors claim that the same merging method does not work in the case of CNN. 
summary_of_the_review: (1) Model fusion is a challenging problem in deep learning. The topic is interesting.
(2) The explanations and analyses of the proposed model fusion are not convincing.
(3) More expensive experiments are needed.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents an algorithm named CLAFusion. The goal is to fuse two networks with different architectures so that the fused one has reasonable computational complexity and works favorably compared to the separate ones. Experiments on CIFAR10 and small neural networks show the effectiveness.
main_review: This paper presents an algorithm named CLAFusion. The goal is to fuse two networks with different architectures so that the fused one has reasonable computational complexity and works favorably compared to the separate ones. Experiments on CIFAR10 and small neural networks show the effectiveness.

I have the following concerns about this paper.

First, I am thinking of the practical value of this algorithm, as well as the OTFusion algorithm. If the cost of training multiple to-be-fused networks is acceptable, the baseline that simply uses all these costs to train the target network should be compared. For example, if training a ResNet34 needs 10 hours and a ResNet18 needs 6 hours, then what is the performance of training a ResNet34 for 16 hours? Many prior works showed that simply extending the training epochs can improve classification accuracy.

Second, the experimental part only shows CIFAR10 results. This is far below expectation. Since training-from-scratch is not required, I would expect ImageNet results to be offered, or at least CIFAR100 can be tested. We know that CIFAR10 is a very small and relatively simple dataset, and more importantly, the classification accuracy on this dataset often suffers ~0.5% random noise, so using it as the single testbed is not acceptable to me. BTW, the deviation of different algorithms should be reported, especially when the gain is smaller than 0.5% (e.g. Table 1).

Third, I tried to read the paper multiple times, but I cannot find the technical description that how two models with the same number of layers are combined. Section 3 ends with adding layers to the shallow network or merging layers in the deep network, but I cannot find the next procedure. Is it missing in the paper?

BTW, I cannot say that the paper is well organized. It spent too much space in the introduction and background parts, but many details (e.g. how to compute the cost matrix or how to combine the networks) are not discussed.
summary_of_the_review: 1. The practical value is questionable.
2. The description of the approach is not complete.
3. Experiments are only performed on very small datasets.

Overall, this paper is not well prepared for publication.
review_title: null

Meta-review: While fusing multiple heterogeneous neural networks into a single network looks like an interesting exploration, there are many major concerns raised by the reviewers:
1) The motivation why the proposed method works is not convincing. In other words, under what conditions the proposed would work or would not work is not clear.
2) The authors failed to provide either theoretical analysis or convincing empirical studies of the proposed method. In the rebuttal, the authors did not address the critical issues raised by the reviewers.
3) There are many other detailed problems about the proposed method as well as the experimental setup.

Therefore, by considering the above concerns, this submission does not meet the standard of publication at ICLR.
Decision: Reject