Deep Model Reassembly
Authors: Xingyi Yang
Zhou Daquan
Songhua Liu
Jingwen Ye
Xinchao Wang
Venue: NeurIPS 2022 Accept (2022)
Keywords: Transfer Learning from Model Zoo
Neural Network Reassembly
Representation Similarity

Abstract: In this paper, we explore a novel knowledge-transfer task, termed as Deep  Model Reassembly (DeRy), for general-purpose model reuse.
Given a collection of heterogeneous models pre-trained from distinct sources and with diverse architectures, the goal of DeRy, as its name implies, is to first dissect each model into distinctive building blocks, and then selectively reassemble the derived blocks to produce customized networks under both the hardware resource and performance constraints. Such ambitious nature of DeRy inevitably imposes significant challenges, including, in the first place, the feasibility of its solution. We strive to showcase that, through a dedicated paradigm proposed in this paper, DeRy can be made not only possibly but practically efficiently. Specifically, we conduct the partitions of all pre-trained networks jointly via a cover set optimization, and derive  a number of equivalence set, within each of which the network blocks are treated as functionally equivalent and hence interchangeable. The equivalence sets learned in this way, in turn, enable  picking and assembling blocks to customize networks subject to certain constraints, which is achieved via solving an integer program backed up with a training-free proxy to estimate the task performance. The reassembled models give rise to gratifying performances with the user-specified constraints satisfied. We demonstrate that on ImageNet, the best reassemble model achieves 78.6% top-1 accuracy without fine-tuning, which could be further elevated to 83.2% with end-to-end fine-tuning. Our code is available at https://github.com/Adamdad/DeRy.

Deep Model Reassembly
Xingyi Yang'
Daquan Zhou 1.2
Songhua Liu'
Jingwen Ye! Xinchao Wang'
National University of Singapore
Byytedance
(xyang, daquan zhou, songhua. 1uu) nus edu, (jingweny, xinchao)@@iin edu. ag
Abstract
In this paper, we explore novel knowledge-transfer task, termed as Deep Model Reassembly (DeRy), for general-purpose model reuse. Given collection of heterogencous models pre-trained from distinct sources and with diverse architectures, the goal of DeRy, as its name implies, to first dissect each model into distinctive building blocks, and then selectively reassemble the derived blocks to produce customized networks under both the hardware resource and performance constraints. Such ambitious nature of DeRy inevitably imposes significant challenges, including, in the first place, the feasibility of its solution. We strive to showcase that, through dedicated paradigm proposed in this paper, DeRy can be made not only possibly but practically efficient. Specifically, we conduct the partitions of all pre-trained networks jointly via cover set optimization, and derive number of equivalence set, within each of which the network blocks are treated functionally equivalent and hence interchangeable. The equivalence sets learned in this way, in turn, enable picking and assembling blocks to customize networks subject to certain constraints, which is achieved via solving an integer program backed up with a training-fiee free proxy to estimate the task performance. The reassembled models, give rise to gratifying performances with the user-specified constraints satisfied. We demonstrate that on ImageNet, the best reassemble model achieves 78.6% top-1 accuracy without fine-tuning, which could be further elevated to 83.2% with end-to-end training. Our code is available at https://gi http://tttttub...c com hwtmm//dmddd/ad// Deny
Introduction
The unprecedented advances of deep learning and its pervasive impact across various domains are partially attributed to, among many other factors, the numerous pre-trained models released online. Thanks to the generosity of our community, models of diverse architectures specializing in the same or distinct tasks can be readily downloaded and executed largely alleviates the model reproducing effort. The sheer number of pre-trained models also enables extensive knowledge transfer tasks. such as knowledge distillation, which the pre-trained models can be reused to produce lightweight or multi-task students.
plug-and and-play manner, which, turn,
In this paper, we explore novel knowledge transfer task, which coin Deep Model Reassembly (DeRy). Unlike most prior tasks that largely focus on reusing pre-trained models as whole, DeRy, as the name implies. goes deeper into the building blocks of pre-trained networks. Specifically, given a collection of such pre-trained heterogeneous models or Model Zoo. DeRy attempts to first dissect the pre-trained models into building blocks and then reassemble the building blocks to tailor models subject to users' specifications, like the computational constraints of the derived network. As such, apart from the flexibility for model customization, DeRy expected to aggregate knowledge from heterogeneous models without increasing computation cost, thereby preserving or even enhancing the downstream performances.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Scance
M POTC
2
Superised Self-sperviei
Model Zoo
Network Partition
Network Reasserably
Figure 1: Overall workflow of DeRy. It partitions pre-trained models into equivalent sets of neural blocks and then reassemble them for downstream transfer. Both steps optimized through solving constrained programs.
Admittedly, the nature of DeRy per makes it highly challenging and ambitious task; in fact, it even unclear whether a solution is feasible, given that no constraints are imposed over the model architectures in the model zoo. Besides, the reassembly process, which assumes the building blocks can be extracted in the first place, calls for a lightweight strategy to approximate the model performances without re-training, since the reassembled model, apart from the parametric constraints, expected to behave reasonably well.
We demonstrate in this paper that, through dedicated optimization paradigm, DeRy can be made not only possible by highly efficient. At the heart of our approach a two-stage strategy that first partitions pre-trained networks into building blocks to form equivalence sets, and then selectively assemble building blocks to customize tailored models. Each equivalence set, specifically, comprises various building blocks extracted from heterogeneous pre- pre-trained models, which are treated to be functionally equivalent and hence interchangeable. Moreover, the optimization of the two steps is obtained and fixed, they can readily serve
purposely decoupled, so that once the equivalence as the basis for future network customization.
We show the overall workflow of the proposed DeRy in Figure 1 It starts by dissecting pre-trained models into disjoint sets of neural blocks through solving a cover set optimization problem, and derives number of equivalence sets, within each of which the neural blocks are treated as functionally swappable. In the second step, DeRy searches for the optimal block-wise reassembly a training-free manner. Specifically, the transfer-ability of candidate reassembly is estimated by counting the number of linear regions in feature representations [55], which reduces the searching cost by 10* times compared to training all models exhaustively.
The reassembled networks, apart from satisfying the user-specified hard constraints, give rise truly encouraging results. We demonstrate through experiments that, the reassembled model achieves > 78% top-1 accuracy on Imagenet with all blocks frozen. we allow for finetuning, the performances can be further elevated, sometimes even surpassing any pre-trained network in the model zoo. This phenomenon showcases that DeRy is indeed able to aggregate knowledge from various models and enhance the results. Besides, DeRy imposes constraints on the network architectures in the model zoo, and may therefore readily handle various backbones such as CNN, transformers, and MLP.
Our contributions are thus summarized as follows.
1. We explore a new knowledge transfer task termed Deep Model Reassembly (DeRy), which enables reassembling customized networks from zoo of pre-trained models under user-specified constraints.
2. We introduce a novel two-stage strategy towards solving DeRy, by first partitioning the networks into equivalence sets and then reassembling neural blocks customize networks. The two steps are modeled and solved using constrained programming, backed up with training-free performance approximations that significantly speed up the knowledge-transfer process.
3. The proposed approach achieves competitive performance on series of transfer learning benckmarks, sometimes even surpassing than any candidate in the model zoo, which, in turn, sheds light on the the universal connectivity among pre-trained neural networks.
2 Related Work
Transfer learning from Model Zoo. A standard deep transfer learning paradigm to leverage single trained neural network and fine-tune the model the target task [85]
87
45
35
98
96
33

---

No need retrain
Adaptive Architecture
No Additional Computation
Utlize All Heterogeneous Models
Problem
Architecture
Single Model Transfer
Zoo Transfer by Selection
Zoo Transfer by Ensemble
X
X
Zoo Transfer by Parameter Fusion
Neural Architecture Search
DeRy
Table 1: Comparison of a series of transfer learning tasks and our proposed Deep Model Reassembly.
The availability of large-scale model repositories brings about a new problem of transfer learning from a model zoo rather than with single model. Currently, there are three major solutions. One line of works focuses on select one best model for deployment, either by exhaustive fine-tuning [39. 74 87 or quantifying the model transferability [94 92 57
or impart the knowledge to other models B1 88
48
G 41 on the target task. However, due to the unreliable measurement of transferability, the best model selection may be inaccurate, possibly resulting in a suboptimal solution. The second idea was to apply ensemble methods [99 99 977, which inevitably leads to prohibitive computational costs at test time. The third approach is to adaptively fuse multiple pre-trained models into a single target model. However, those methods can only combine identical [71] [8] (78) or homogeneous [72, 58 network structures, whereas most model zoo contains diverse architectures. contrast to standard approaches in Table [ DeRy dissects the pre-trained models into building blocks and rearranges them in order to reassemble new pre-trained
73 76
models.
Neural Representation Similarity. Measuring similarities between deep neural network representations provide a practical tool to investigate the forward dynamics of deep models. Let X E Raxdi and Y E R2xd denote two activation matrices for the same n examples. A neural similarity index sXX, Y) is a scalar to measure the representations similarity between X and Y, although they do not necessarily satisfy the triangle inequality required of proper metric. Several methods including linear regression [86] 3T], canonical correlation analysis (CCA) [65] 27 (64),
centered kernel aligmment (CKA) [40], generalized shape metries [81]. In this study, leverage the representations similarity towards function level to quantify the distance between two neural blocks.
Neural Architecture Search. Automatic neural archtecture search (NAS) has achieved promising performance-effictnn efficiency trade-offs as well as reducing human efforts. With a pre-defined search space 775 47 82 69], the designing problem of the optimal architecture is formalized as a discrete optimization, where the best solution could be found with reinforcement learning (RL) [TOO], evolutionary algorithms (EA) [66] gradient-baser search [47. Because it is costly to measure the performance of each candidate, several surogate methods like one-shot NAS [62
5 47 predictor-
based NAS [46] 51 [0) and zero-shot NAS (55
9 1 are proposed to accelerate the evaluation
process. In this paper, we similarly formalize the network reassembly as a search problem; However, compared with NAS that searches random initialization, DeRy is searching the optimal structure combination along side with network weights. In addition, the search space of DeRy is not preset heuristically, but determined by the network partition results.
Network Stitching. Initially proposed by 44 model stitching aims to "plug-in" the bottom layers of one network into the top layers of another network, thus forming stitched network [6] It provides an alliterative approach to investigate the representation similarity and invariance of neural networks. A recent line of work achieves competitive performance by stitching visual transformer on top of the ResNet 77). Instead of stitching two identical-structured networks in a bottom-top manner, in our study, we investigate to assemble arbitrary pre- pre-trained networks by model stitching.
3 Deep Model Reassembly
In this section, we dive into the proposed DeRy. We first formulate DeRy, and then define the functional similarity and equivalent sets of neural blocks to partition networks by maximizing overall groupbility. The resulting neural blocks are then linked by solving an integer program.
3

---

Backhone
Init.
(PPrrams (ccc))())
telamiim
Mected
inlk sup
23.71
84.67
ResNet50
inar2021 sup
23.71
82.57
ResNet50
(nste 1&22)
inti(Staee M44)
23.98
85.30
ResNet50
23,71
84.67
inlk sup
27.60
85.56
ResNerf0(Stge 1&22 Swin-T(Stage 384)
inlk sup
27.94
85.77
Figure 2: The top-1 accuracy difference between "off-the toff-the- shelf" pre-trained models on 4 down-stree wwwn-tteaam tasks.
Table 3: Accuracy on CIFAR- 100 with the pre- trained networks and their reassembled ones.
3.1 Problem Formulation
Assume we have a collection of N pre-trained deep neural network models 2 = (M,) that each composed of L, N layers of operation [F:(k) J!=1" therefore M; pP(2), FT(2)... F(L.) Each model can be trained on different tasks or with varied structures. We call Z a Model Zoo. We define learning task T composed of labeled training set Der {x), -= and a test set Dr. {x ト
Definition (Deep Model Reassembly) Given a task T, our goal is to find the best-pere rest-performed L-layer compositional model M on T, subject hard computational or parametric constraints.
We therefore formulate it as optimization problem
M (M)) M
F s.t.M F(l1) F((2) F.L) |MI
(I)
where p(()) is the 1-th layer of the i-th model, (.M indicates the performance on T, and [M] C denotes the constraints. For two consecutive layers with dimension mismatch, we add a single stitching layer with 1 x convolution operation to adjust the feature size. The stitching layer structure described in Supplementary.
No Single Wins For All. Figure provides a preliminary experiment that 8 different pre-trained models are fine-tuned on 4 different image classification tasks. It is clear that no single model universally dominants in transfer evaluations. It builds up our primary motivation to reassemble trained models rather than trust "best" candidate.
Reassembly Might Win. Table compares the test performance between the reassembled model and its predecessors. The bottom two stages the ResNet50 iNaturalist0221 (inat2021 sup) 77 are stitched with ResNet50 ImageNet-1k 1k (inlk sup) stage 3&4 to form new model for fine-tuning on CIFARI00. This reassembled model improves its predecessors by 0.63% accuracy respectively, Similar phenomenon is observed on the reassembled model between ResNet50 inlk and Swin-T in ikl Despite its simplicity, the experiment provides concrete evidence that the neural network reassembly could possibly lead to better model in knowledge transfer.
Reducing the Complexity. From the overall M = Σi Lg layers, the search space Eq I is of size L-permutations of M PMM, L), which is undesirably large. To reduce the overall search cost, we intend partition the networks into blocks rather than the layer-wise-diidder setting. Moreover, time-consuming to evaluate each model on the target data through full-time fine-tuning. Therefore, we hope to accelerate the model evaluation, even without model training.
Based on the above discussion, the essence of DeRy lies two steps (1) Partition the networks into blocks and (2) Reassemble the factorized neural blocks. In the following sections, we elaborate on "whal is a good partition?" and "whal is good assembly?".
3.2 Network Partition by Functional Equivalence
A network partition 21 21
is division of neural network into disjoint sub-nets. In this study, we refer specifically to the partition of neural network M2 along depth into K blocks (B(*)) Jk=1 so that each block is stack of layers A²a) F(!) F(++)) ... (((+>) and l is its stage index. Inspired by the hierarchical property of deep neural networks, we aim to partition the neural networks according to their function level, for example, dividing the network into "low-level" block that identifies curves and a "high-level" block that recognizes semantics. Although we cannot strictly differentiate "low-level" from "high-level", it is feasible to define functional equivalence.

---

Definition 2 (Fumctional Equivalence) Given two functions B and B' with same input space X and output space y. d y x y + I. is the metric defined on Y. For all inputs x E X, if the outputs are the equivalent dl (x)()); B' (x)) = 0, we say B and B' are functional equivalent.
A function is then uniquely determined by its peers who generate the same output with the same input. However, we can no longer define functional equivalence among neural networks, since network blocks might have varied input-output dimensions. It is neither possible to feed the same input to intermediate blocks with different input dimensions, nor allow for a mathematically valid definition for metric space B 7
when the output dimensions are not identical. We therefore resort to recent measurements on neural representation similarity [27 40] and define the functional similarity for neural networks. The intuition is simple: two networks are functionally similar when they produces similar outputs with similar inputs.
Definition 3 (Functional Similarity for Neural Networks) Assume we have neural similarity index
s(;-)) and two neural networks B X' E Buxddn + Raxdous and B': : X E RRnxd.... y Baxdo... For any two batches of inputs X S X and X' S X' with large similarity sXX X 6 the functional similarity between B and B' are defined as their ouiput similarity s (B(()., B (X)).
This definition generalizes well to typical knowledge distillation (KD) [1 when din. d'n which we will elaborate in the Appendix. We also show in Appendix that Def.3 provides a necessary and insufficient condition for two identical networks. Using the method of Lagrange multipliers, the conditional similarity in Def.3 can be further simplified to S = s (BB(X), (x))))+ (X, X'), which is summation of its input-output similarity. The full derivation is shown in the Appendix.
Finding the Equivalence Sets of Neural Blocks. With Def.3, we are equipped with the math tools to partition the networks into equivalent sets of blocks. Blocks in each set are expected to have high similarity, which are treated to be functionally equivalent and hence interchangeable.
With graphical notion, we represent each neural network as path graph G(V, E) 25 with two nodes of vertex degree I, and the other n 2 nodes of vertex degree 2. The ultimate goal is to find the best partition of each graph into K disjoint sub-graphs along the depth, and the dissected sub-nets are concurrently grouped into K functional equivalence sets, that sub-graph within each group has maximum internal functional similarity S(B, B'). In addition, we take a mild assumption that each
sub-graph should have approximately similar size |B < (1+ e) LMal, where | indicates the model size and is coefficient controls size limit for each block. We solve the above problem by posing a tri-level constrained optimization with joint clustering and partitioning (Clustering)
N
K
K
max Bay
J(A, (BA (0.11)
= max
A(RJ))
S(B (Ba,)
(2)
K
s.t.
Σ A(kk)) 1, (B ]== arg max
Σ ((tk))((BB Ba,))
(Partition)
(3)
((K)
j=1
B(a)) B(2)...o
B ((a) = Mi, B(ki) n B(ta) = 0, Vki + k2
s.t.
...o
(4)
|B(0| < (1 +
MM
k = 1 1,...K K
(5)
K
Where A E NKNNK is the 0-1 1 assignment matrix, where A(tk.)) = denote the B block belongs to the j j-th equivalence set, otherwise O. Note that each block only belongs to one equivalence set, thus each column sums up to 1, ΣØ= A(sk.)) = 1. Ba, is the anchor node for the j-th equivalence set, which has the maximum summed similarly with all blocks in set j. B((bi) n B((aa) = 0 refers to the fact the no two blocks has overlapping nodes.
The inner optimization largely resembles the conventional set cover problem [32 or (K, 1 + c) graph
partition problem (36) that directly partition graph into k sets. Although the graph partition falls exactly in NP-hard (28 problem, heuristic graph partitioning algorithms like Kernighan-Lin (KL) algorithm (38) and Fiduccia (FM) algorithm 23 can be applied to solve our problem efficiently. In our implementation, we utilize a variant KL algorithm. With a random initialized network partition ( for M at O. we iteratively find the optimal separation by
swapping nodes (network layer). Given the two consecutive block ((,)/2 ((l) ....... .... (!l-pe) and
5

---

B(k+t)/ F p((tputt).... El(+pa+tptt)) at time t, we conduct forward and a backward neural
network layer swap between successive blocks, whereas the partition achieving the largest objective value becomes the new partition
(B(k) 1+1,
B(k+2)/+41)) (441)) = arg max max (((+t))))))))))))))))(11 J(B I/. JBB B, (x+ 12))
(*)
F. m Bltt)) (B B f = B(b)
where (B. le B ())= B00,,
B (((+1)
For the outer optimization, we do K-Means (52 style clustering. With the current network partition {B(*)* TK=1- alternate between assigning each block to a equivalence set Gj. and identifying the anchor block within each set Bay E G It has been proved that both KL and K-Means algorithms converge a local minimum according the initial partition and anchor selection. We repeat the optimization for R = 200 runs with different seeds and select the best partition as our final results.
3.3 Network Reassembly by Solving an Integer Program
As we have divided each deep network into K partitions, each belongs to one of the K equivalence sets, all we want now is to find the best combination of neural blocks as a new pre-trained model under certain computational constraints. Consider K disjoint equivalence sets GI...., Gk of blocks to be reassembled into a new deep network of parameter constraint Cparam and computational constraint CELOPs. the objective is to choose exactly one block from each group Gj as well as from each network stage index such that the reassembled model achieves optimal performance on the target task without exceeding the capacity. We introduce two the binary matrices X(sk.j) and Y(sk.)) to uniquely
identity the reassembled model MMXX X(sk.)) takes on value 1 if and only if B(ar is chosen in
group Gj. and Y(RR.) = if (*()) comes from the k-th block. The selected blocks are arranged by the block stage index. The problem is formulated as Pr (MM(X, Y)) X,Y
(8)
s.t.
(M(X C puram: FL.OPs S CrLOrS
(9)
((ak.)) 1, X(ak,) E 10,1 = I....,K
(10)
Y(u.)) = 11, Y, E k0 =1, I,...,K K
(11)
where Pr is again the task performance. Equation IraiiionOOoaaa and 11 innicates that each model only possesses single block from each equivalence set and each stage index. As such, the reassembled blocks are automatically ordered through its stage index in their original model. The problem falls exactly into a 0-1 Integer Programminng [60] problem with non-linear objective. Conventional methods train each M(X. Y) to obtain Pr. Instead of training each candidate till convergence, we estimate the transfer-ability of network by counting the linear regions in the network as a training-free proxy.
The number of linear region [56] 26 is theoretical-groundee tool to describe the expressivity of neural network, which has been 10]. We, therefore, calculate the data-dependent linear region to estimate the transfer performance of each model-task combination. The intuition is straightforward: the network can hardly learn to distinguish inputs with similar binary codes.
Estimating the Performance with Training-Free Proxy.
successfully applied on NAS without training [
We apply random search to get generation of reassembly candidates. For a whole inputs. we feed them into each network and binarilize the features vectors using a sign function. Similar to NASWOT [55], we compute the kernel matrix K using Hamming distance d(.,-) and rank the models using log(det K). Since the computation of K requires nothing more than a few batches of network forwarding, we replace Py. in Equation 8 with NASWOT score for fast model evaluation.
of
4 Experiments
In this section, we first explore some basic properties of the the proposed DeRy task, and then evaluate our solution on series of transfer learning benchmarks to verify its efficiency.

---

Suave RaNaašo Stage
ReaNe50 Stgge
log
Tarper Stage
Tinget Stage
Figure 4: FROZEN-TUNING accuracy on ImageNet by Figure 5: Pair-wise Linear CKA between pre-trained replacing the 3rd and 4th stage of R50 to target blocks.
R50 and (1) RI01 (2) RX50 and (3) Reg8G.
Model Zoo Setup. We construct our model zoo by collecting pre-trained weights from Torchvision
t timm and MpenMMlab Fl We includes series of manually designed CNN models like ResNet 30
and (e) as well as
architectures like RegNetY [63] and MobileNetv3 [3
Due to recent popularity of vision transformer, we also take several well-known attention-based
architectures into consideration, including Vision Transformer (ViT) (20] and Swin-Transformer 49
In addition to the differentiation of the network structure pre-trained on ImageNet, we include models with a variety of pre-trained strategies, including SimCLR 8 MoCov2 [I]] and BOL [24] for ResNet50, MoCov3 [1] and MAE 229 for VTT-B. Those models are pre- trainrrieed on ImageNetlk (68), ImageNet21K 667 Xrays [5] and iNaturalist2221 [77], Finally we result in 21 network architectures, with 30 pre-trained weights in total. We manually identify the atomic node to satisfy line graph assumption. Each network is therefore a line graph composed of atomic nodes.
Implementation details. For all experiments, we set the partition number K = 4 and the block size coefficient e = 0.2. We sample 1/20 samples from each train set to calculate the linear CKA representation similarity. The NASWOT (55] score is estimated with 5-batch average, where each mini-batch contains 32 samples. We set 5 levels of computational constraints, with Oparam E {10,20, 30, 50, 90} and CFLOPs [3, 5, 6, 10, 20), which is denoted DeRy(K, Cperam-CFLOPs). For each setting, we randomly generated 500 candidates. Each reassembled model is evaluate under 2 protocols (I) FROZEN-TUNING
We freeze all trained blocks and only update the parameter for the stitching layer and the last linear classifier and (2) FULL-TURNING. All network parameter are updated. All experiments similarity calculation cost, we construct the similarity table offline on ImageNet. The complexity analysis and full derivation are shown in the Appendix.
conducted
8x RTX 3090 server. To reduce the feature
4.1 Exploring the Properties for Deep Reassembly
Figure/ validates our functional similarity, reassembled block selection, and its effect on the model performance. For the ResNet50 trained on ImageNet, we replace its 3nd and 4th stage with a target block from another pre-trained network (ResNet101, Xt50 and RegNet Y&G), connected by a single stitching layer. Then, the reassembled networks are re-trained on ImageNet for 20 epochs under PROZEN-TURNIE protocol. The derived functional similarity in Section 3.2 shown the diameter of each circle. We observe that, the stitching position makes a substantial difference regarding the reassembled model performance. When replaced with target block with the same stage index, the reassembled model performs surprisingly well, with 2 70% top-1 I accuracy, even if its predecessors are trained with different architectures, seeds, and hyperparameters. It is also noted that, though function similarity is not numerically proportional to the target performance, it correctly reflects the performance ranking within the same target network. suggests that our function similarity provides reasonable criteria to identify equivalence set. In sum, the coupling between the timilarity-positionn pot-piion-perorrmaaaee performance explains our design to select one block from each equivalence set as well as the stage index. We also visualize the linear CKA [40] similarity between the R50 and the target networks in Figure 5 An interesting finding is that diagonal pattern for the feature similarity. The representation at the same stage is highly similar. More similarity visualizations are provided in the Appendix.
Similarity, Position and Reassembly-ability
Partition Results. Due to the space limitation, the partition results of the model zoo are provided in the Appendix. Our observation is that, the equivalent sets tend to cluster the blocks by stage index.
http:://yyooch..ayyiiiooshhheeee P hhups://githubb hhans:/ggttnn.cmm//vigtnaaaa/
7

---

0.38
Plowerii
timm Pretrained ImageNet 0.42
Top Accurccy
155 160 1d NASWOT scom
X4 1s5
LE5 160 183 IXX NASWOT ccem
155 1s0 165 150 112
NASWITI soer
NASWIT sorea
Cars T 0.33
UTD T 0:47
Pate - 0.33
CUB T 0.51
15. JA :63 17a
155
14. 113
115 14
130 NASWOT score
Figure 6: Plots of NASWOT [55] score and test accuracy for (Left) 10 pre-trained model on 8 downstream tasks and (Right) timm model zoo on ImageNet. T is Kendall's Tau correlation.
For example, all bottom layers of varied pre-trained networks are within the same equivalence set. It provides valuable insight that neural networks learns similar patterns at similar network stage.
Architecture or Pre-trained Weight. Since DeRy searches for the architecture and weights concurrently, natural question arises that "Do both architecture and pre-trained weights lead to the final improvemen?? Or only architecture counts?" We provide the experiments in the Appendix that both factors contribute. It is observed that training the DeRy architecture from scratch leads to substantial performance drop compared with DeRy model with both new structures and pre-trained weights. It validates our arguments that our reassembled models benefit from the pre-trained models for efficient transfer learning.
Verifying the training-free proxy. As the first attempt to apply the NASWOT to measure model transfer-ability. we verify its efficacy before applying it to DeRy task. We adopt the score to rank 10 pre-traaeeee rre-trained models on 8 image classification tasks, as well as the timm model zoo ImageNet, shown in Figure 6 We also compute the Kendall's Tau correlation [37 between the fine-tuned accuracy and the NASWOT score. It is observed that the NASWOT score provides reasonable predictor for model transfer-ability with a high Kendall's Tau correlation.
4.2 Transfer learning with Reassembled Model
Evaluation on ImageNetlk. We first compare the reassembled network on ImageNet 68] with current best-performed architectures. We train each model for either 100 epochs as SHORT-TRAINING or a 300 epochs as FULL-TRAINING, Except for DeRy, all models are trained from scratch. We optimize each network with Adam (50) alongside a initial learning rate of le 3 and cosine Ir-decay, mini-batch of 1024 and weight decay of 0.05. We apply RandAug [17, Mixup (95 and CutMix [ as data augmentation. All model are trained and tested on 224 image resolutions.
Table 7 provides the Top-1 accuracy comparison on Imagenet with various computational constraint. We underline the best-performed model in the model zoo. First, It is worth-noting that DeRy provide very competitive model, even under FROZEN-TURNING TURNING or SHORT-TRAINEE manages to reach 78.6% with 1 1.27M parameter trainable, which provides convincing clue that the heterogeneous trained model are largely graftable. With only SHORT-TRAINING DeRy models also match up with the full-time trained model in the zoo. For example, DeRy(4, 10,3) gets to 76.9% accuracy within 100 epochs' training, surpassing all small-sized models. The performance can be further improved towards 78.4% with the standard 300-epoch training. Second, DeRy brings about faster convergence. We compare with ResNet-50 and Swin-T under the same SHORT-TR, SHORT-TRAINING setting in Table 9 and Figure 10) It is clear that, by assembling the off- off-the-self pre-trained blocks, the DeRy models can be optimized faster than the it competitors, achieving 0.9% and 0.2% accuracy improvement over the Swin-T model with less parameter and computational requirements. Third, as showcased in Figure g our DeRy is able to search for diverse and hybrid network structures. DeRy(4, learns to adopt light- blocks like MobileNetv3, while DeRy(4.90,20) gets to a large CNN-Swin hybrid architecture. Similar bybrid strategy has been proved to be efficient in manual network design [54]]] 83).
protocol. DeRy(4,90,p))
Transfer Image classification. We evaluate transfer learning performance on 9 natural image datasets. These datasets covered a wide range of image classification tasks, including 3 object classification
8

---

Architecture
#Train/All Params (M)
FLOPS (G)
Top-1
RSB-ResNet-18
11.69/11.60
182
70.6
RegNerY-S00M
6.306630
0.8
76.3
VIT-T16
5.7/5.7
1.3
74,1
De83(4.10.3)-FF
1,0277,83
2.99
41.2
Da8y(4,10.3)-TT'
7.83/7.83
2.99
76.9
DeRy(4.10.1)FFT
7.83/7.83
2.99
78.4
RSB-ResNet-50
25.56/25,56
4.12
79.8
RegNetY-4GF
20.60/20600
4,0
79.4
VIT-S16
22.0/22.0
4.6
79.6
Figure 8: Reassembled structures on ImageNet.
Swin-T
28.29/28.29
4.36
81.2
DeRG(4.30.6)-FZ
1.57/24.89
4.47
60.5
Anchitecture
Purams FM)Ps (G)
Top. 1 |TTpp
DeRg(4.30.6) FT
24.89/24.89
4.47
79.6
Swin-1
ResNet-50
25.50 56
4.12
76.8
93.3
DT
24.89/24.89
4.47
81.2
DeRy(30, 6)-F
24.89
4.47
79.6
94.8
RSB-ResNet-101
44,55/44.55
7.85
81.3
ResNer-10 101
Swin-S
44.33
49.61
8.52
7.3
80.8
79.0
94.5
95.7
RegNetY-SGF
39, 39.20/39.20 2000
81.7
DaRy(50, F0
40.41
6.43
81.2
95.6
Swin-S
49.61/49.1
8.52
82.8
Table 9: Top-1 and Top-5 Accuracy for the ImageNet 100-epoch FULL-TUNING experiment.
DeRy(4.50.10)-F2
3.92/4044
6.43
72.0
1o8)(4.50.10)FFT
40.41/40.41
6.43
81.3
D88)(4.50.10-FT
40.41/40.41
6.43
82.3
RegNetY-16GF
83.6/83.6
16.0
82.9
VIT-B16
86/66/6.86
33.03
79.8
Swin-B
87.77/87.77
15.14
83.1
De8y(4.90.00-F2
1.27/ 80.66
13.29
78.6
D68)(4.90.0)-FFT DeRg(4.90,20. FT
80.66/ 80.66
13.29
82.4
80.66/ 80.66
13.29
83.2
Table 7: Top-1 accuracy of models trained on ImageNet. f means the model is trained for 100 epochs. "FZ" and "FT" denote the reassembled blocks are frozen finc-tuned, Trainable parameters are marked in red.
Table 10: (Left) Test accuracy and (Right) Train loss comparison under the 100-epoch training on ImageNet.
CIEARI00
CIBARIO
Cabechto
Flooes
Clrrs
8 4
59
25
25
Pemmecter (M)
Paramdeer (M)
Parmmiio (M)
Parmmcics (MI
Parmmntees (M)
Aircraft
DID
Pas
CUB
X
Meski Zoe Runa tak. Madel tie Prematsd
Deily
Dally
modk Medid Zoo Lightll mooi)
Transfer performance on image classification tasks with the model zoo and our DeRy. Each blue or orange point refers to a single model trained from scratch or pre-traineh weights.
Figure 11:
tasks CIFAR- 10
43 CIFAR- 100 43 and Caltech- 101
22): 5 fine-grained classification tasks
Flower- 59, Stanford Cars 42 FGVC Aircraft Oxford-IIIT Pets [6]] and CUB-Bird [79]
and texture classification task DTD 14]. We FULL-TUNE all candidate networks in the model zoo and compare them with our DeRy model. Two model selection strategies LogME [92 and LEEP (57
are also taken as our baselines. For fair comparison, we further train the reassembled network on ImageNet for 100 epochs to further boost the transfer performance. Following (8) 41), we perform hyperparameter tuning for each model-task combination, which are elaborated in the Appendix.
Figure II compares the transfer performance between our proposed DeRy and all candidate models. By constructing models from building blocks, the DeRy generally surpasses all network trained from scratch within the same computational constraints, even beats pre-trained ones on Cars, Aireraft, and Flower. If allowing for pre- vre-trainin on ImageNet (DeRy+In we can further promote the test accuracy, even better than the best-performing candidate in the original model zoo (highlighted by x). The performance improvement rises up as parameter constraints increase, which demonstrates the scalability of the proposed solution. Model selection approaches like LogME and LEEP may not necessarily get the optimal model, thus failing to release the full potential of the model zoo. These

---

findings provide encouraging evidence that DeRy gives rise to an alternative approach to improve the performance when transferring from a zoo of models.
4.3
Ablation Study
To support the effectiveness of the DeRy pipeline, we further verify influence of each design in our solution through ablation studies.
Partition and Reassembly Strategy. First, we conduct experiments by replacing the (1) cover set partition and (2) training-free reassembly with random partition or search. For the partition ablation, we randomly dissect each network into K partitions and reassemble the blocks in an order-less manner using our training-free proxy. For the reassembly ablation, we retain the cover set partition and fine-tune each randomly reassembled network for 100 epochs. Due to the computation limitation, we can only evaluate 25 candidates for reassembly ablation. We report the 100-epoch TROZIN-TUNING top-1 accuracy and the search time on ImageNet in Table 2 include the similarity computation time into our account since it is computed offline. We see that the majority of the search cost comes from the fine-tuning stage. The training- free proxy largely alleviates the tremendous computational cost by 10s times, with marginal performance degradation. On the other hand, the cover set model partition not only improves the transfer performance but also reduces the reassembly search space from O (k=))) to O(1). Both stages are crucial.
Cover Set Partition
Train-Free Reassembly
Search Cost
Acc (%)
(GPU days)
72.0
0.23
X
70.5
1.48
x
73.5
135
X
x
72.2
135
Table 2:
Ablation study
partition and
reassembly strategy.
under the De,, 10) setting. Note that we do not
Granularity of Partition. To see the impact of partition number K, here we show the experimental results with different K E {4, 5, 6}. We set the configuration to DeRy(K, 30, 6). The reassembled network is trained with FULL-TUNING setting on ImageNet for 100 epochs. We report the parameter size, FLOPs as well as their top-1 and top-5 accuracy. As demonstrated in the Table 3 We notice that, as the partition number K increases, the performance of the reassembled model remains quite stable or slightly increase. It suggest that DeRyis highly flexible with different granularity of partitioning.
Partition Number
Param
FLOPPs
Top-1
Top-5
K = 4
24.89
4.47
79.6
94.8
K = 5 K 6
21.14
5.53
79.7
94.9
23.38
5.39
79.8
95.0
Table 3: Ablation study on partition granularity with K (4, (4)5.
5 Conclusion
In this study, we explore novel showledge-transtee task called Deep Model Reassembly (DeRy). DeRy seeks to deconstruct heterogeneous pre-trained neural networks into building blocks and then reassemble them into models subject to user-defined constraints. We provide proof-of-concept solution to show that DeRy can be made not only possible but practically efficient. Specifically, pre- trained networks are partitioned jointly via cover set optimization to form a series of equivalence sets. The learned equivalence sets enable choosing and assembling blocks to customize networks, which is accomplished by solving integer program with training-free fask-performance proxy. DeRy not only achieves gratifying performance on series of transfer learning benchmarks, but also sheds light on the functional similarity between neural networks by stitching heterogeneous models.
Acknowledgement
This research supported by the National Research Foundation, Singapore under its Al Singapore Programme (AISG Award No: AISG2-RP-2021-40 Xinchao Wang is the corresponding author.
10

---

References
[1] Mohamed S Abdelfattah, Abhinav Mehrotra, Eukasz Dudziak, and Nicholas Donald Lane. Zero-cost proxies for lightweight [nas]. In International Conference on Learning Reprexentations, 2021.
[2] Andrea Agostinelli, Jasper Uijlings, Thomas Mensink, and Vittorio Ferrari. Transferability metrics for selecting source model ensembles. arXiv preprint ar a1Xiv:2111.13011, 2021.
[3] Yamini Bansal, Preetum Nakkiran, and Boaz Barak. Revisiting model stitching to compare neural representations. Advances in Neural Information Processing Systems, 34, 2021.
[4] Yajie Bao, Yang Li, Shao-Lun Huang, Lin Zhang, Lizhong Zheng, Amir Zamir, and Leonidas Guibas. An information-thcoreee approach to transfcrabilitt in task transfer learning. In 2019 IEEE International Conference on Image Processing (ICIP), pages 2309-2313. IEEE, 2019.
[5] Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoc Le. Understanding and simplitying one-shot architecture search. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 Proceedings of Machine Learning Research, pages 550-559. PMLR, 10-15 15 Jul 2018.
[6] Daniel Bolya, Rohit Mittapalli, and Judy Hoffman. Scalable diverse model selection for accessible transfer learning. Advances in Neural Information Processing Systems, 34, 2021.
[7] Valeril Vladimirovich Buldygin and IU V Kozachenko. Metric characterization random variables and random processes, volume 188. American Mathematical Soc., 2000.
[8] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[9] W. Chen, X. Gong, and Z. Wang. Neural architecture scarch imagenet in four gpu hours: A theoretically inspired perspective, 2021.
[10] Wuyang Chen. Xinyu Gong, and Zhangyang Wang. Neural architecture search on imagenet four gpu hours: A theoretically inspired perspective. In International Conference on Learning Representations. 2021.
[11] Xinlci Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint 2xXiv:2003.0429) 2020.
[12) Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9640 9649,96499790 2021.
[13] B. Choudhary. The Elements of Complex Analysis. New Age International Publishers, 1992.
[14] M. Cimpoi, S. Maji. L. Kokkinos, S. Mohamed, and A. Vedaldi. Deseribing textures in the wild. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2014.
[15] Joseph Paul Cohen, Joseph D. Viviano, Paul Bertin, Paul Morrison, Parsa Torabian, Mattco Guarrera, Matthew P Lungren, Akshay Chaudhari, Rupert Brooks, Mohammad Hashir, and Hadrien Bertrand. CorchXRayVisioos A library of chest X-ray datasets and models. In Medical Imaging with Deep Learning. 2022.
[16] Adrián Csiszárik, Péter Kórösi-Szabó, Ákos Matszangosz, Gergely Papp, and Dániel Varga. Similarity matching neural network representations. Advances in Neural Information Processing Systems, 34, 2021.
[17] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoe V Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops, pages 702-703, 2020.
[18] Dong Dai and Tong Zhang. Greedy model averaging. Advances in Neural Information Processing Systems, 24, 2011.
[19] Thomas G Dietterich. Ensemble methods in machine learning. In International workshop multiple classifier systems, pages 1-15. Springer, 2000.

---

[20] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani. Matthias Minderer. Georg Heigold, Sylvain Gelly, et al. An image worth 16x16 words: Transformers image recognition at scale. arXiv preprint arXiv:2010.1199) 2020.
[21] Tomas Feder, Pavol Hell, Sulamita Klein, and Rajeev Motwani. Complexity of graph partition problems. Proceedings of the thirty-first annual ACM symposium on Theory of computing, pages 464-472, 1999.
[22] Fei-Fei, Rob Fergus, and Pictro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recognition workshop, pages 178-178. IEEE, 2004.
[23] Charles M Fiduccia and Robert M Mattheyses. A linear-time heuristic for improving network partitions. In 19th design automation conference, pages 175-18 181. IEEE, 1982.
[24] nan-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Beraard Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bootstrap your own latent-a new approach to self-supervised learning. Advances in Neural Information Processing Systems, 13:21271-21284 2020.
[25] Jonathan I. Gross, Jay Yellen, and Mark Anderson. Graph theory and its applications. Chapman and Hall/CRC, 2018.
[26] Boris Hanin and David Rolnick. Complexity of linear regions in deep networks. In International Conference Machine Learning, pages 2596-2604, PMLR, 2019.
[27] David R Hardoon, Sandor Szedmak, and John Saywe-Taylor. Canonical correlation analysis: An overview with application to learning methods. Neural computation, 16(12)2633-2666 2004.
[28] Juris Hartmanis. Computers and intractability: a guide the theory of np-completeness (michael garey and david s. johnson). Siam Review, 24(1) 1982.
[29] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv:2111.06377 2021.
[30] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[31] Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in neural network. arXiv preprint arXiv: 153.001503.02531, 2(7), 2015.
[32] Dorit S Hochba. Approximation algorithms for np-hard problems. ACM Sigact News, 28(2):40-52, 1997.
[33] Qibin Hou, Daquan Zhou, and Jiashi Feng. Coordinate attention for efficient mobile network design. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13712-13722 2021.
[34] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chich Chich Chen, Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1314-1324, 2019.
[35] Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. arXiv preprint arXivv1801.00 801006146666..1 2018
[36] George Karypis and Vipin Kumar. Multilevel w-w hypergraph partitioning. VLSI design, 11(3):285-300 2000.
[37] Maurice G Kendall. A new measure of rank correlation. Biometrika, 30(1/2):81-95 1938.
[38] Brian W Kernighan and Shen Lin. An efficient heuristic procedure for partitioning graphs. The Bell system technical journall, 49(2):291-307, 1970.
[39] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigserver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In Eurupean conference on computer vision, pages 491-507. Springer, 2020.
[40] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning, pages 2019.
PMLR,
12

---

[41] Simon Kornblith, Jonathon Shlens, and Quoe V Le. Do better imagenet models transfer better? In Pruceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2661-2671, 2019.
[42] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In 4th International TEEE Workshop on 3D Representation and Recognition (3dRR-13), Sydney, Australia, 2013.
[43] Alex Krizhevsky, Geoffry Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[44] Karel Lenc and Andrea Vedaldi. Understanding image representations by measuring their equivariance and equivalence. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 991-999, 2015.
[45] Xingjian Li, Haoyi Xiong, Hanchao Wang, Yuxuan Rao, Liping Liu, Zeyu Chen, and Jun Huan. Delta: Deep learning transfer using feature map with attention for convolutional networks. arXiv preprint arXiv:/901.000 09929999,9.9.......999 2019.
[46] Chenxi Liu, Barret Zoph. Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei. Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
[47] Hanxiao Liu, Karen Simonyan, and Yiming Yang- DARTS: Differentiable architecture search. In International Conference on Learning Representations, 2019.
[48] Huihui Liu, Yiding Yang, and Xinchao Wang. Overcoming catastrophic forgetting in graph neural networks. In Proceedings of the AAAI conference on arrificial intelligence, 2021.
[49] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang. Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. International Conference on Computer Vision (ICCV), 2021.
[50] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint a7Xiv: 1.05 2017.
[51] Rengian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie- Yan Liu. Neural architecture optimization. In NIPS 2018. Microsoft, December 2018.
[52] James MacQueen et al. Some methods for classification and analysis of multivariate observations. In Proceedings of the fifth Berkeley symposium on mathematical statistics and probability, volume I, pages 281 281-291 181- Oakland, CA, USA, 1967.
[53] Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft. Technical report, 2013.
[54] Sachin Mehta and Mohammad Rastegari. Mobilevit: weight, general-purpose, and mobile- friendly vision transformer. arXiv preprint arXiv:2110.02178. 2021.
[55] Joe Mellor, Jack Turner, Amos Storkey, and Elliot Crowley. Neural architecture search without training. International Conference on Machine Learning, pages 7588-7598. PMLR, 2021.
[56] Guido F Montufar, Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio. On the number of linear regions of deep neural networks. Advances in neural information processing systems, 27, 2014.
[57] Cuong Nguyen, Tal Hassner, Matthias Seeger, and Cedric Archambeau. Leep: A new measure to evaluate transferability of learned representations. In International Conference on Machine Learning, pages 7294-7305. PMLR, 2020.
[58] Dang Nguyen, Khai Nguyen, Dinh Phung, Hung Bui, and Nhat Ho. Model fusion of heterogeneous neural networks via cross-layer alignment. arXiv preprint a0. 155338 2021.
[59] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over large number of classes. In 2008 Sixth Indian Conference on Computer Vision. Graphics & Image Processing, pages 722-77 729, IEEE, 2008
[60] Christos H Papadimitrion and Kenneth Steiglitz. Combinatorial optimization: algorithums and complexity. Courier Corporation, 1998.
[61] O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. V. Jawahar. Cats and dogs. In IEEE Conference on Computer Vision and Pattern Recognition, 2012.
13

---

[62] Hieu Pham, Melody Guan, Barret Zoph. Quoe Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume of Proceedings of Machine Learning Research, pages 4095 4104, PMLR, 10-15 Jul 2018.
[63] Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces, In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 10428-10436, 2020.
[64] Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Sveca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. Advances in neural information processing systems, 30, 2017.
[65] JO Ramsay, Jos ten Berge, and GPH Styan. Matrix conrelation. Psychometrika, 49(3):003-433, 1984.
[66]
Esteban Real. Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc Le, and Alexey Kurakin. Large-scale evolution of image classifiers. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 2902-2911. PMLR, 06-11 Aug 2017.
[67] Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor. Imagenet-21k pretraining for the masses, 2021.
[68] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. Internationall journal of computer vision, 115(3):21 1-25,, 2015.
[69] Mark Sandler. Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chiehe Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2018.
[70] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of hert: smaller, faster, cheaper and lighter. arXiv preprint 1910.00 108, 2019.
[71] Yang Shu, Zhi Kou, Zhangjie Cao, Jianmin Wang, and Mingsheng Long. Zoo-tuning: Adaptive transfer from zoo of models. In International Conference on Machine Learning, pages 9626-9637. PMLR, 2021.
[72] Sidak Pal Singh and Martin Jaggi. Model fusion via optimal transport. Advances in Neural Information Processing Systems, 33:22045-22055, 2020.
[73] Jie Song. Yixin Chen, Xinchao Wang. Chengehao Shen, and Mingli Song. Deep model transferability from attribution maps. Advances Neural Information Processing Systems, 2019.
[74] Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How train your vit? data, augmentation, and regularization in vision transformers. arXiv preprint arXiv:2106. 10270, 2021.
[75] Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoe V. Le. Mnasnet: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[76] Anh T Tran, Cuong Nguyen, and Tal Hassner. Transferability and hardness of supervised classification tasks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1395-1405. 2019.
[77) Grant Van Hom, Elijah Cole, Sara Beery, Kimberly Wilber, Serge Belongie, and Oisin Mac Aodha. Benchmarking representation learning for natural world image collections. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 12884-12893 2021.
[78] Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni. Federated learning with matched averaging. arXiv preprint arXtv:2002.064440 2020.
[79] Peter Welinder, Steve Branson, Takeshi Mita. Catherine Wah, Florian Schroff, Serge Belongie, and Pietro Perona. Caltech-ucsd birds 200. Technical Report CNS-TR-201, Caltech, 2010.
[80] Wei Wen, Hanxiao Liu, Yiran Chen, Hai Li, Gabriel Bender, and Pieter-Jan Kindermans. Neural predictor for neural architecture search. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm. editors, Computer Vision ECCV 2020, pages 660-676, Cham, 2020. Springer International Publishing.
14

---

[81] Alex Williams. Erin Kunz, Simon Kornblith, and Scott Linderman. Generalized shape metrics on neural representations. Advances in Neural Information Processing Systems, 34, 2021.
[82] Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recoggition (CVPR), June 2019.
[83] Tete Xiao, Mannat Singh, Eric Mintun, Trevor Darrell, Piotr Dollár, and Ross Girshick. Early convolutions help transformers see better. Advances in Neural Information Processing Systems, 34:30392-30400, 2021.
[84] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492-1500, 2017.
Aggregated residual
[85] LI Xuhong, Yves Grandvalet, and Franck Davoine. Explicit inductive bias for transfer learning with convolutional networks, In International Conference on Machine Learning, pages 2825-2834, PMLR, 2018.
[86] Daniel LK Yamins, Ha Hong, Charles F Cadieu, Ethan A Solomon, Darren Seibert, and James J DiCarlo. Performance-oppe optimmzed hierarchical models predict neural responses in higher visual cortex. Proceedings of the national academy of sciences, 1 111(23):8619 8624, 2014.
[87] Xingyi Yang, Xuehai He, Yuxiao Liang, Yue Yang. Shanghang Zhang, and Pengtao Xie. Transfer learning or superviid learning? a tale of two pretraining paradigms. arXiv preprint arXiv:2007.0422 042344 2020.
[88] Xingyi Yang, Jingwen Ye, and Xinchao Wang. Factorizing knowledge in neural networks. European Conference on Computer Vision, 2022.
[89] Yiding Yang, Zunlei Feng, Mingli Song, and Xinchao Wang. Factorizable graph convolutional networks. Advances in Neural Information Processing Systems, 33:20286-20296, 2020.
[90] Yiding Yang, Jiayan Qiu, Mingli Song, Dacheng Tao, and Xinchao Wang. Distilling knowledge from graph convolutional networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[91] Jingwen Ye, Yixin Ji, Xinchao Wang, Kairi Ou, Dapeng Tao, and Mingli Song. Student becoming the master: Knowledge amalgamation for joint scene parsing. depth estimation, and more. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2829-2838, 2019.
[92] Kaichao You, Yong Liu, Jianmin Wang, and Mingsheng Long. Logme: Practical assessment of tre-trained models for transfer learning. In International Conference on Machine Learning, pages 12133-12143, PMLR, 2021.
[93] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6023 6032, 2019.
[94] Guojun Zhang, Han Zhao, Yaoliang Yu, and Pascal Poupart. Quantifying and improving transferability in domain generalization. Advances in Neural Information Processing Systems, 34, 2021.
[95] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint 1rXiv:1710.09412. 2017.
[96] Daquan Zhou, Qibin Hou, Yunpeng Chen, Jiashi Feng, and Shuicheng Yan. Rethinking bottleneck structure for efficient mobile network design. In European Conference on Computer Vision, pages 680-697. 680-697..pringerr, 2020.
[97] Daquan Zhou, Xiaojie Jin, Xiaochen Lian, Linjie Yang. Yujing Xue, Qibin Hou, and Jiashi Feng. Autospace: Neural architecture search with less human interference. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 337-346, 2021.
[98] Daquan Zhou, Bingyi Kang. Xiaojie Jin, Linjie Yang, Xiaochen Lian, Zihang Jiang, Qibin Hou, and Jiashi Feng. Deepvit: Towards deeper vision transformer. arXiv preprint arXiv:2 1186, 2021.
[99] Hua Zhou. Ensemble learning. In Machine learning, pages 181-210. Springer, 2021.
[100] Barret Zoph and Quoe Le. Neural architecttre search with reinforcement learning. In International Conference Learning Representations, 2017.
15


Reviews:
[{"review_id": "dwlyT1q7-9k", "reviewer": "Reviewer_k5P1", "summary": "This paper brings up an ambitious and pioneering paradigm, termed Deep Model Reassembly (DeRy), that reuses the pretrained neural networks as building blocks for new model construction to address the transfer learning problem. A two-stage solution is proposed to jointly search for the optimal model architecture and weight for the reassembled model. DeRy first partitions the pre-trained networks jointly via a cover set optimization, and then assemble blocks to customize networks subject to hard constraints via solving an integer program backed up with a training-free proxy. Experimental results on ImageNet and 8 down-stream tasks validate that the reassembled model can achieve higher performance than any candidate models in the model zoo.", "questions": "- Add more literature review on NAS. Given that there is a focus on joint search for architecture and model weights with zero-cost proxy in this paper, the literature review should be extended beyond a handful of papers currently listed. What is the main difference between the proposed DeRy and the traditional NAS problem?\n\n- Although the staged solution largely eliminates the combinational search for a search space, however, the authors pose strong heuristics in the model partition stage. The blocks with high ``functional similarity'' are grouped and the partition should maximize the overall group-ability. Why the ``functional similarity'' introduced in this paper gives rise to swap-ability? \n\n- Typos:\n1. Line 170: a path graphs -> a path graph\n2. Line 179 and Eq (3): What does N_g stand for? Is it the number of equivalence sets? Should it be the same as K?\n3. Line 207: process -> possess\n4. Line 229: architecture -> architectures\n\n", "limitations": "Yes, limitations have been discussed. I do not find the potential negative societal impact of this work.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths]\n- Overall, this is a well-written paper with an interesting and extensible idea at its core: considering the knowledge transfer as finding optimal layer-wise stacking of pre-trained blocks (In Definition 1). Comparing to the typical NAS problem with a fixed search space like operation or topology, the DeRy involves a dynamic search space (The partition of the pre-trained networks is unknown beforehand), which is much more challenging.\n\n- New Findings: This study strives to point out that, arbitrary trained networks are largely reassembleable, even though the models may have diverse architecture and source tasks. The grafted pre-trained models potentially could provide satisfying performance. \n\n- Overall, I am convinced with the intuition of the method: first dissecting the networks into swappable equivalence blocks, then constructing new models with the best performance. The dedicated solution draws inspiration from the conventional discrete optimization community, thus with a good convergence guarantee and low search complexity. I personally favor the current solution.\n\n- Evaluation is sufficient to support the argument of this paper. Section 4.1 validates the pipeline designs and lists several intriguing findings for partitioned blocks\n\n[Weaknesses]\n- If my understanding is correct, the current methodology may result in an over-complicated pipeline for the model reassembly, which may not be scalable to more sophisticated tasks (detection, segmentation and generation) or model zoo setting. I am not expecting the authors to resolve this issue in this paper, but it would be great to have a discussion over this matter in future work part.\n\n- The current solution still needs a long training time, like 100~300 epochs for ImageNet. This is OK since this is the first attempt along the line, but again it would be great if the authors can provide a discussion on this issue. \n\n- Despite the authors already including quite a significant number of experiments, it would be better if more pretraining tasks and datasets be included in the evaluations.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "XjfgzI9s2s8", "reviewer": "Reviewer_3qgQ", "summary": "This paper presents a novel knowledge-transfer task, termed as Deep Model Reassembly (DeRy), for general-purpose model reuse. Specifically, given a collection of heterogeneous models pre-trained from distinct sources and with diverse architectures, authors first dissect each model into distinctive building blocks, and then selectively reassemble the derived blocks to produce customized networks under both the hardware resource and performance constraints. Experiments on ImageNet dataset show the promising performance of the proposed method under different settings.", "questions": "Address weaknesses. I would like the authors to address my questions regarding experiments and comparisons as described above.", "limitations": "Given that DeRy combines different models, what about the bias of the resulting model? Is there any way to control the biases of the combined models so that they won't be reflected on the final model? While this is beyond the scope of the current paper, a discussion on limitations wrt to biases would be interesting.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n* The paper is very well written and easy to follow.\n\n* The problem of model reassembly is interesting with many practical use-cases.\n\n* The proposed method is novel and the two step procedure makes sense.\n\n* Empirically demonstrated that the proposed method obtains competitive performance on ImageNet dataset.\n\nWeaknesses:\n\nOverall, I liked the problem definition and proposed solution to address the interesting model reassembly task. However, I would like the authors to address my following concerns/questions to further improve the quality of the work.\n\n* Why the proposed approach focuses on fusing heterogeneous models, can it be applied to homogenous models? If yes, how does that compare to the existing model fusion methods that work only for fusing homogenous models, e.g.,  [54, 14, 55, 59]. I would suggest the authors to include experiments for this to strengthen the paper.\n\n* How does the size of model zoo effect the performance of the proposed method? In particular, experiments and discussion along the direction of scalability of the proposed method wrt to number of models in the model zoo would be interesting.\n\n* Can model reassembly help in reducing the training cost of the large models? A discussion on potential applications of model reassembly besides knowledge transfer would be good for demonstrating a broader impact of the work.\n\n* Authors mention that \"We manually identify the atomic node to satisfy our line graph assumption. Each network is therefore a line graph composed of atomic nodes.\" This is not clear from the current descriptions. Can authors explain this?\n\n* What is the effect of partition number on the final performance? How does it increase the complexity of the method?\n\n* How is the proposed method comparable to Model Fusion of Heterogeneous Neural Networks via Cross-Layer Alignment? A comparison and discussion should be included in the paper to verify the advantage of the method over similar methods.\n\n* Can the proposed approach DeRy be applied to other computer vision tasks beyond image classification? ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "CwuGsPiWqjA", "reviewer": "Reviewer_q2yK", "summary": "The paper proposes to reuse the building blocks of pretrained neural networks for new tasks by resembling them under given computation constraints. The proposed approach, DeRy, first learns to partition the layers of the base networks jointly into equivalent sets via a cover set optimization, then selects and stitchs the optimal blocks into a new network by solving an integer programming problem. DeRy is evaluated on ImageNet1K and 9 other transfer-learning benchmarks via linear probing and finetuning.", "questions": "i) What is used as the s(*,*) metric. From the paper, I would guess it is linear CKA [33]?\n\nii) The overhead (complexity) of building the s(*,*) index, partition, and stitching. Table 2 shows the GPU time. I wonder if it includes building the s(*,*) index.\n\niii) I wonder if the s(*,*) table needs to be re-built for each dataset? If not, I wonder what is the main dataset used for computing the s(*,*) table (e.g. the GPU time in Table 2 is for what dataset)\n\niv) I wonder if the partition and stitching need to be re-run for each dataset? (e.g. if the DeRy models in Table 7 and Fig 11 are the same).\n\nv) the list of the 28 base models used\n\nvi) It seems Fig. 11 only shows a subset of the models (less than 28 models). I wonder what are the models shown in Fig. 11.\n\nvii) the detailed architectures of the obtained DeRy models\t\n\t\t\t", "limitations": "The limitations and the potential negative societal impact are NOT discussed in the submission.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\ni) the paper introduces a new task, deep model reassembly. The task is ambitious!\n\nii) it is interesting to learn that the task, when defined under specific assumptions, has a feasible solution that works to a certain extent.\n\niii) the writing is clear in general\n\n\nWeaknesses\n\nThe formulation is not as general as I would expect. I have concerns about the practical usefulness of the method and whether it could scale.\n\ni) The model space is small. DeRy assumes the base model forms a path graph (L170-171), which excludes architectures with heavy skip connections, e.g. UNets. The granularity of the partition is small, i.e. K = 4. The number of complexity levels, i.e. 5 param/gflop levels,  is also small, which excludes efficient model architectures (e.g. < 1 gflops) that are of great interest to the community. \n\nii) Performance: on ImageNet1k, DeRy shows no advantage. On the transfer-learning datasets, DeRy shows marginal advantages with further pretraining on ImageNet1k. From L311-312, the hyperparameters for DeRy were tuned for each model-task combination, I wonder if the hyperparameters for the base models were also tuned in this way?\n\niii) Overhead. I would guess building the s(*,*) table involves computation of R * (N * N * K * K)/20 epochs (from L237, 1/20 training samples are used). Even with a small granularity (N=28, K=4, R=200), the computation looks huge?\n\niv) Insights:  from Fig.1 in the suppl. , the partition tends to group blocks of the same stage id, with the current K (i.e. K=4), it seems the partition algorithm obtains a trivial solution? If this is the case, do we need the partition step at all?\n\n\n\nMinor issues\n\ni) Typos, L73 trasnfer → transfer L110 (k) → (l), L170 graphs → graph, L248 ImagNet → ImageNet, L308 testure –> texture, L89 (suppl): #praram → #param\nii) L121 “It is clear that no single model universally dominants in transfer evaluations. It builds up our primary motivation to reassemble trained models rather than trust the “best” candidate. ” Arguably, DeRy does not provide a universally dominant model either.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept