Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper brings up an ambitious and pioneering paradigm, termed Deep Model Reassembly (DeRy), that reuses the pretrained neural networks as building blocks for new model construction to address the transfer learning problem. A two-stage solution is proposed to jointly search for the optimal model architecture and weight for the reassembled model. DeRy first partitions the pre-trained networks jointly via a cover set optimization, and then assemble blocks to customize networks subject to hard constraints via solving an integer program backed up with a training-free proxy. Experimental results on ImageNet and 8 down-stream tasks validate that the reassembled model can achieve higher performance than any candidate models in the model zoo.
strengths_and_weaknesses: [Strengths]
- Overall, this is a well-written paper with an interesting and extensible idea at its core: considering the knowledge transfer as finding optimal layer-wise stacking of pre-trained blocks (In Definition 1). Comparing to the typical NAS problem with a fixed search space like operation or topology, the DeRy involves a dynamic search space (The partition of the pre-trained networks is unknown beforehand), which is much more challenging.

- New Findings: This study strives to point out that, arbitrary trained networks are largely reassembleable, even though the models may have diverse architecture and source tasks. The grafted pre-trained models potentially could provide satisfying performance. 

- Overall, I am convinced with the intuition of the method: first dissecting the networks into swappable equivalence blocks, then constructing new models with the best performance. The dedicated solution draws inspiration from the conventional discrete optimization community, thus with a good convergence guarantee and low search complexity. I personally favor the current solution.

- Evaluation is sufficient to support the argument of this paper. Section 4.1 validates the pipeline designs and lists several intriguing findings for partitioned blocks

[Weaknesses]
- If my understanding is correct, the current methodology may result in an over-complicated pipeline for the model reassembly, which may not be scalable to more sophisticated tasks (detection, segmentation and generation) or model zoo setting. I am not expecting the authors to resolve this issue in this paper, but it would be great to have a discussion over this matter in future work part.

- The current solution still needs a long training time, like 100~300 epochs for ImageNet. This is OK since this is the first attempt along the line, but again it would be great if the authors can provide a discussion on this issue. 

- Despite the authors already including quite a significant number of experiments, it would be better if more pretraining tasks and datasets be included in the evaluations.

questions: - Add more literature review on NAS. Given that there is a focus on joint search for architecture and model weights with zero-cost proxy in this paper, the literature review should be extended beyond a handful of papers currently listed. What is the main difference between the proposed DeRy and the traditional NAS problem?

- Although the staged solution largely eliminates the combinational search for a search space, however, the authors pose strong heuristics in the model partition stage. The blocks with high ``functional similarity'' are grouped and the partition should maximize the overall group-ability. Why the ``functional similarity'' introduced in this paper gives rise to swap-ability? 

- Typos:
1. Line 170: a path graphs -> a path graph
2. Line 179 and Eq (3): What does N_g stand for? Is it the number of equivalence sets? Should it be the same as K?
3. Line 207: process -> possess
4. Line 229: architecture -> architectures


limitations: Yes, limitations have been discussed. I do not find the potential negative societal impact of this work.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a novel knowledge-transfer task, termed as Deep Model Reassembly (DeRy), for general-purpose model reuse. Specifically, given a collection of heterogeneous models pre-trained from distinct sources and with diverse architectures, authors first dissect each model into distinctive building blocks, and then selectively reassemble the derived blocks to produce customized networks under both the hardware resource and performance constraints. Experiments on ImageNet dataset show the promising performance of the proposed method under different settings.
strengths_and_weaknesses: Strengths:

* The paper is very well written and easy to follow.

* The problem of model reassembly is interesting with many practical use-cases.

* The proposed method is novel and the two step procedure makes sense.

* Empirically demonstrated that the proposed method obtains competitive performance on ImageNet dataset.

Weaknesses:

Overall, I liked the problem definition and proposed solution to address the interesting model reassembly task. However, I would like the authors to address my following concerns/questions to further improve the quality of the work.

* Why the proposed approach focuses on fusing heterogeneous models, can it be applied to homogenous models? If yes, how does that compare to the existing model fusion methods that work only for fusing homogenous models, e.g.,  [54, 14, 55, 59]. I would suggest the authors to include experiments for this to strengthen the paper.

* How does the size of model zoo effect the performance of the proposed method? In particular, experiments and discussion along the direction of scalability of the proposed method wrt to number of models in the model zoo would be interesting.

* Can model reassembly help in reducing the training cost of the large models? A discussion on potential applications of model reassembly besides knowledge transfer would be good for demonstrating a broader impact of the work.

* Authors mention that "We manually identify the atomic node to satisfy our line graph assumption. Each network is therefore a line graph composed of atomic nodes." This is not clear from the current descriptions. Can authors explain this?

* What is the effect of partition number on the final performance? How does it increase the complexity of the method?

* How is the proposed method comparable to Model Fusion of Heterogeneous Neural Networks via Cross-Layer Alignment? A comparison and discussion should be included in the paper to verify the advantage of the method over similar methods.

* Can the proposed approach DeRy be applied to other computer vision tasks beyond image classification? 
questions: Address weaknesses. I would like the authors to address my questions regarding experiments and comparisons as described above.
limitations: Given that DeRy combines different models, what about the bias of the resulting model? Is there any way to control the biases of the combined models so that they won't be reflected on the final model? While this is beyond the scope of the current paper, a discussion on limitations wrt to biases would be interesting.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes to reuse the building blocks of pretrained neural networks for new tasks by resembling them under given computation constraints. The proposed approach, DeRy, first learns to partition the layers of the base networks jointly into equivalent sets via a cover set optimization, then selects and stitchs the optimal blocks into a new network by solving an integer programming problem. DeRy is evaluated on ImageNet1K and 9 other transfer-learning benchmarks via linear probing and finetuning.
strengths_and_weaknesses: Strengths

i) the paper introduces a new task, deep model reassembly. The task is ambitious!

ii) it is interesting to learn that the task, when defined under specific assumptions, has a feasible solution that works to a certain extent.

iii) the writing is clear in general


Weaknesses

The formulation is not as general as I would expect. I have concerns about the practical usefulness of the method and whether it could scale.

i) The model space is small. DeRy assumes the base model forms a path graph (L170-171), which excludes architectures with heavy skip connections, e.g. UNets. The granularity of the partition is small, i.e. K = 4. The number of complexity levels, i.e. 5 param/gflop levels,  is also small, which excludes efficient model architectures (e.g. < 1 gflops) that are of great interest to the community. 

ii) Performance: on ImageNet1k, DeRy shows no advantage. On the transfer-learning datasets, DeRy shows marginal advantages with further pretraining on ImageNet1k. From L311-312, the hyperparameters for DeRy were tuned for each model-task combination, I wonder if the hyperparameters for the base models were also tuned in this way?

iii) Overhead. I would guess building the s(*,*) table involves computation of R * (N * N * K * K)/20 epochs (from L237, 1/20 training samples are used). Even with a small granularity (N=28, K=4, R=200), the computation looks huge?

iv) Insights:  from Fig.1 in the suppl. , the partition tends to group blocks of the same stage id, with the current K (i.e. K=4), it seems the partition algorithm obtains a trivial solution? If this is the case, do we need the partition step at all?



Minor issues

i) Typos, L73 trasnfer → transfer L110 (k) → (l), L170 graphs → graph, L248 ImagNet → ImageNet, L308 testure –> texture, L89 (suppl): #praram → #param
ii) L121 “It is clear that no single model universally dominants in transfer evaluations. It builds up our primary motivation to reassemble trained models rather than trust the “best” candidate. ” Arguably, DeRy does not provide a universally dominant model either.

questions: i) What is used as the s(*,*) metric. From the paper, I would guess it is linear CKA [33]?

ii) The overhead (complexity) of building the s(*,*) index, partition, and stitching. Table 2 shows the GPU time. I wonder if it includes building the s(*,*) index.

iii) I wonder if the s(*,*) table needs to be re-built for each dataset? If not, I wonder what is the main dataset used for computing the s(*,*) table (e.g. the GPU time in Table 2 is for what dataset)

iv) I wonder if the partition and stitching need to be re-run for each dataset? (e.g. if the DeRy models in Table 7 and Fig 11 are the same).

v) the list of the 28 base models used

vi) It seems Fig. 11 only shows a subset of the models (less than 28 models). I wonder what are the models shown in Fig. 11.

vii) the detailed architectures of the obtained DeRy models	
			
limitations: The limitations and the potential negative societal impact are NOT discussed in the submission.
review_title: null

Meta-review: metareview: 
 This paper proposes an interesting new way to think about how to use a model zoo of pre-trained models: extract modular building blocks that are swappable from the networks and then stitch them together. To do the former, a cover set optimization methods is proposed, and the blocks can then be combined in a way that respects various resource and performance constraints. The idea is both interesting and ambitious, and has the potential to open up various avenues of research if done well. 

The paper lives up to the task: It is well-written (k5P1, 3qgQ, q2yK), conducts experiments to validate if the such stitched networks can do well, and proposes an intuitive principled method to extract the blocks (3qgQ, k5P1). The reviewers did express some concerns about scalability/generalizability to other tasks (k5P1, 3qgQ), larger zoos (k5P1 ,3qgQ), other architectures (all reviewers), and computation (q2yK) as well as several other potential issues such as limited performance improvements. The authors provided strong rebuttals to these, including some new experiments. At the end of the process, the reviewers were all satisfied with most of the concerns, and the overall consensus on the paper seems to be with high scores. 

  Given the potentially high-impact, novel perspective as well as the solid execution, I highly recommend this paper for acceptance. 
confidence: Certain
Decision: Accept