Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method for neural network model fusion based on the concepts of Wasserstein barycenter, and Gromov-Wasserstein barycenter. The method uses OT couplings from each previous layer to construct couplings between the subsequent layer that minimize a cost based off the previous layers' couplings. Significant empirical validation of this method is given, including applications to model fusion in data hetergenous settings, data homogenous settings, as well as diverse model architectures and some exploration of the linear mode connectivity hypothesis.
strength_and_weaknesses: Strengths:
- Model fusion is an important problem
- Applying optimal transport to this problem is of significant interest
- Some of the empirical performance is significantly better than existing work

Weaknesses:
- The empirical performance is not convincingly improved on the whole
- There seems to be only marginal innovation over the previous work Singh & Jaggi 2019
- There is no theoretical development of their ideas
clarity,_quality,_novelty_and_reproducibility: The work is clearly written. Code is provided. Originality is addressed above.
summary_of_the_review: This work addresses an important problem and proposes a method with extensive empirical validation. However, the empirical results are not completely convincing, and the method is only marginally novel compared to previous work. Therefore, I think it is slightly below the threshold of acceptance.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a new unified mathematical framework for neural network (NN) model fusion, which is based on the Wasserstein/Gromov-Wasserstein barycenters (WB/GWB), i.e., formulating as a series of optimal transport (OT) problems. The proposed mathematical framework is universal and can be applied to a broad class of NN architectures. The authors experimentally demonstrate that the framework is effective at fusing different types of NNs, including RNNs and LSTMs, and interpret the fusion by visualizing the aggregation of two neural networks in a 2D plane. This may be new experimental evidence about the linear mode connectivity of the loss landscape.
strength_and_weaknesses: #### Strengths
1. I like the idea of using optimal transport and Wasserstein/Gromov-Wasserstein barycenters to model the model weight fusion, and to my best knowledge, I think it is novel. The description of the corresponding fusion algorithm is also clear.
2. I like the idea of visualizing the model fusion and loss landscape in 2D (Figure 3 and related paragraphs).

#### Weaknesses
1. One major concern is that (please correct me if I am wrong), I think the proposed WB/GWB-based model fusion lack of sufficient & in-depth theoretical analysis (or any theoretical guarantees) on how the test performance is preserved/affected by the proposed fusion method. In terms of experiments, the advantages to baselines like OT fusion and FedMA in terms of test performance are not consistent or significant. So I think a more careful theoretical and experimental comparison to the baselines is also needed.
2. In terms of the relation to the linear mode connectivity phenomenon/conjecture. Currently, I think only from the visualizations in Figure 3; it is confusing how the experiments on WB/GWB model fusion bring new insights or contribute to our understanding of the linear mode connectivity phenomenon. Is the model fusion setup a special case considered in Entezari et al.(2021) or other related literature? I understand Figure 3 can be thought of as some empirical evidence, but what is the new insight that contributes to that line of research? If the contribution is somehow limited to empirical support, I think it is a bit overclaimed and misleading to address this contribution in the title, abstract, and introduction. Another question is: will other model fusion baselines yiled similar behavior when visualized similarly to Figure 3?

#### Minor Issues
1. The authors should use `\citep` and `\citet` for in-text citations of different formats. 
clarity,_quality,_novelty_and_reproducibility: 1. The authors did a good job of clearly describing the WB/GWB-based fusion algorithm. And to my best knowledge, I agree with the novelty of the proposed method.
2. The theoretical and empirical soundness and significance of the proposed method are somehow limited by the lack of in-depth analysis and more experimental comparisons.
3. The reproducibility depends on whether the authors will release the code (conditioned on the acceptance) and cannot be judged now.
summary_of_the_review: Overall I recommend the rejection for this current manuscript. The major reason is that (1) The theoretical and empirical soundness of the proposed WB/GWB model fusion method is somehow limited by the lack of in-depth theoretical analysis and more experimental comparisons, (2) the claim that this work brings new insight, and contribute to the understanding to the linear mode connectivity phenomenon, is a bit over-claimed. It is possible I missed some important details as I am not an expert in the model fusion line of research, and I definitely welcome the authors to correct me if I made some factual mistakes and get involved in the discussion.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a mathematical formalism for model fusion motivated by the TLP distance which is interesting. They illustrate the performance of their method by fusing various networks on both homogenous and heterogenous task settings, where they seem to perform slightly better than the OTFusion method upon which the given paper builds. Also, they show additional results by extending OTFusion to handle RNNs and LSTMs by using the Gromov-Wasserstein distances. Further, they also provide interesting visualizations and touch upon the Linear Mode Connectivity (LMC) conjecture in NLP tasks.  Despite all of this,  it must be said that nevertheless, it is but a formal rewrapping of the existing model fusion approach OTFusion --- which is obviously also considering Wasserstein barycenters. Furthermore, their empirical comparisons seem to be off and it seems that they have not been performed fairly, which casts a doubt on the seeming advantage of their method.
strength_and_weaknesses: ### Strengths:
- Their formalism of defining a measure for each neuron and the TLP interpretation is interesting and could be useful for future extensions. 
- Support RNNs and LSTMs via Gromov-Wasserstein barycenters extends the model fusion direction further. 
- Some evidence is provided for LMC conjecture on RNNs and LSTMs. 


### Weaknesses:
- **Novelty:** The reality of the introduced formalism is that it basically devolves into the model fusion known as OTFusion (Singh & Jaggi, 2019). I happened to read the given paper's Appendix D, which made me compare the two more in detail and which makes this aspect quite clear. The reason is that many of the considered elements in the formalism when instantiated reduce their whole abstract framework to something pretty obvious, for e.g., the function w_j defined from the previous layer to the reals basically turns out to be the weight vector (as the domain is discrete). Apart from the row-wise scaling factors, I don't see much difference between the resulting cost functions in eqn 26 and 27; and this should also not change the resulting transport map much. The paper mentions that OTFusion introduces this without a derivation but that is false; see section S9 of their paper where they derive the barycentric projection. All of this, i.e., the similarities as well as the differences, should be discussed in detail and within the main text so that reader can better realize these aspects.

- **Incomplete discussion & comparison**: The question then arises of where does the empirical difference come from. I believe this can be reconciled by looking at the choice of ground cost used, which is based on comparing the weight vectors. Now, the thing is that under this cost, the fusion problem is a bilinear assignment problem (Ainsworth et al., 2022), and so the transport map at each layer depends on the previous. Thus it makes sense that the current paper has to run more iterations of the OTFusion procedure under this ground metric. However, this paper hides the fact that many of the results in OTFusion are based on what is called 'activations-based alignment'. To repeat Ainsworth et al., 2022, now the problem is a linear assignment problem since the transport map at each layer can be computed independently and thus there is indeed no need of running more iterations. But, unfortunately, this paper seems to ignore this completely. 
 - **Improper empirical comparisons**: In fact, the empirical comparisons themselves seem to be rather off as well. For instance, in Figure 2 where they discuss the fusion performance for homogenous and heterogenous tasks, it seems that OTFusion implementation is done strangely. This can be seen if we take a look at the results from that paper in the given settings (which this paper follows), namely Figure 2 and 4 of the OTFusion paper. There OTFusion seems to have much smaller variance and is rather stable, with clear gain in performance. I suspect the difference comes from not using activations based alignment as specified in the caption of that paper. Additionally, for most of the results in OTFusion they seem to use exact OT instead of Sinkhorn since they get a better performance with the former. So before the resulting gains from the paper's method are to be evaluated they should at least ensure that their comparisons are proper, and the experiments fairly reproduced and implemented (such as activations, exact OT, and likewise for results in Table 1 & 2 besides Figure 2). 

- To properly show if their method is significantly different from OTFusion, another strategy could be to do the visualizations as in Figure 3 where they consider OTFusion (both the choices of ground cost, weights and activations), their own method, and vanilla average (or something else that is appropriate). I think this should clarify if there is indeed much difference from OTFusion.

- The **discussion of the related work** should be corrected. (i) Firstly, as per the above points, the similarities and differences wrt OTFusion should be properly acknowledged in the section where they introduce their paper. Next, naming their method as 'Wasserstein Barycenters'  WB fusion is plainly incorrect and deceptive. OTFusion, although uses OT in its acronym and not WB, makes it explicitly clear that they use Wasserstein Barycenters to fuse the model. They define WB and the variational problem underneath, it is obvious from their algorithm that they are solving this variational problem, and even more they explicitly discuss the exact update as Barycentric projection. I suggest the authors necessarily rename their method to something which is more representative of what they propose, e.g., TLPFusion. (ii) Besides, OTFusion, it should also be mentioned that there have also been attempts in the literature to handle the quadratic assignment in the context of model fusion Liu et al., 2022, https://proceedings.mlr.press/v162/liu22k/liu22k.pdf. Their approach might be better able to handle the case of RNNs/LSTMs. Furthermore, it was already mentioned explicitly in Wang et al. 2020 https://arxiv.org/pdf/2002.06440.pdf that a Gromov-Wasserstein barycenter based idea could be used in this precise context --- this should also be acknowledged in this work. 
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and provides useful illustrations to convey their point.  The discussion of related work is a bit misleading and should be fixed. For more, see above. 
summary_of_the_review: In general, I liked the paper at first glance. However, when one digs deep, it becomes rather clear that presently both the discussion and the experiments are misleading and not properly carried out. Nevertheless, I am willing to increase my score towards acceptance, when all of the above-stated points have been adequately addressed.

---- POST-REBUTTAL ----
I thank the authors for their rebuttal. It becomes quite clear from the rebuttal, as the authors themselves admit, that there should not be visible differences between the baselines on FCNs and VGG networks. Likewise, the authors also make it clear that when using the EMD solver, the default choice in the prior work OTFusion, their method performs similarly. However, these aspects will hardly be clear to most readers. It is completely alright not to focus on activation-based costs for the reasons mentioned, but that does not mean it is fair to provide an incomplete/inaccurate picture. Then about the novelty of formalism: it is again totally fine to have the TLP formalism, but all you need to emphasize is the differences/similarities to prior work. In particular, these cannot be relegated to supplementary or openreview discussions. Also, regarding 'WB', as elaborated before I disagree with the authors and surely the argument cannot rely on merely looking at the abstract instead of the entire main paper. Further, specifying 'WB" for their algorithm is just plain misrepresentation, to be frank. 

Presently, as the authors admit, there are many significant and extensive changes yet to be made. So, in this highly-inaccurate shape, the paper cannot be accepted. Moving forward, I would recommend the authors truly depict and represent their own work wrt prior works, and then focus on consolidating their own novel contributions (like GW, theoretical analysis etc.) in a future submission. Thus, I maintain my score.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper considers the fusion algorithm to aggregate different neural network models trained locally. The Wasserstein/Gromov-Wasserstein barycenter is the main technique used in the construction. Extensive numerical experiments demonstrate the good performance of the proposed fusion algorithm, and provide empirical evidence for the conjecture made in Entezari et al. (2021).
strength_and_weaknesses: Strength:
- the proposed fusion algorithms allow aggregating models within a variety of NN architectures 
- by formulating the fusion problem as a series of Wasserstein/Gromov-Wasserstein barycenter problems, bridge the NN fusion problem with computational OT
- empirically demonstrate the effectiveness of fusing different types of networks, including RNNs and LSTMs

Weaknesses:
- Although the usage of Wasserstein/Gromov-Wasserstein barycenter in NN fusion seems to be new, the Wasserstein barycenter problem itself is well studied, and thus the contribution of this work is rather limited
- The proposed fusion algorithm is performed layer by layer, in which the architecture of the whole network seems to be unused.



clarity,_quality,_novelty_and_reproducibility: This paper is overall well-written and easy to follow, with sufficient preliminaries provided. The novelty is limited, and the author provided the code so the results should be reproducible.
summary_of_the_review: Overall I like the idea of applying Wasserstein/Gromov-Wasserstein barycenter for NN fusion, which is actually a natural idea in federated learning, such as [1][2] (which might be related to this work). And extensive numerical results are provided to justify the good performance. However, it is hard to judge the novelty of this work as barycenters are not new techniques and their usage in federated learning also exists.  

[1] Nguyen, T. A., Nguyen, T. D., Le, L. T., Dinh, C. T., & Tran, N. H. (2022). On the Generalization of Wasserstein Robust Federated Learning. arXiv preprint arXiv:2206.01432.
[2] Farnia, F., Reisizadeh, A., Pedarsani, R., & Jadbabaie, A. (2022). An Optimal Transport Approach to Personalized Federated Learning. arXiv preprint arXiv:2206.02468.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a new model fusion method of deep neural networks that utilizes Wasserstein/Gromov-Wasserstein barycenter computation. The effectiveness of the proposed method is investigated through extensive numerical experiments. The experiments also give a result that supports the Linear Mode Connectivity (LMC) conjecture.

Strength: Utilizing the Wasserstein/Gromov-Wasserstein barycenter for model fusion is a natural idea. It would be one of promising directions in the literature.   
Weakness: On the other hand, the paper lacks (1) sound theoretical justification (pointed out by shzY, fEwc), (2) the numerical experiments are not strong (pointed out by shzY, obb2), and (3) its comparison to existing work should be more convincing (pointed out by shzY, fEwc, obb2, CYR8). In particular, the last concern (3) is pointed out by all the reviewers. The novelty compared to the work should be properly specified and the authors should describe more clearly what is the new insight that contributes to the line of research.   

For these reasons, this paper's quality is not yet sufficient for publication in ICLR.  
justification_for_why_not_higher_score: The paper should contain more novel contents in their theoretical analyses and numerical experiments. Since the numerical experiments do not show much improvement of the performance, they are not so much convincing. More thorough discussions about the comparison with existing work are required. Then, the paper requires major revision to be accepted.
Decision: Reject