Title: Modularized Morphing of Neural Networks
Authors: Tao Wei
Changhu Wang
Chang Wen Chen
Venue: ICLR 2017
Keywords: Deep learning
Computer vision

Abstract: In this work we study the problem of network morphism, an effective learning scheme to morph a well-trained neural network to a new one with the network function completely preserved. Different from existing work where basic morphing types on the layer level were addressed, we target at the central problem of network morphism at a higher level, i.e., how a convolutional layer can be morphed into an arbitrary module of a neural network. To simplify the representation of a network, we abstract a module as a graph with blobs as vertices and convolutional layers as edges, based on which the morphing process is able to be formulated as a graph transformation problem. Two atomic morphing operations are introduced to compose the graphs, based on which modules are classified into two families, i.e., simple morphable modules and complex modules. We present practical morphing solutions for both of these two families, and prove that any reasonable module can be morphed from a single convolutional layer. Extensive experiments have been conducted based on the state-of-the-art ResNet on benchmark datasets, and the effectiveness of the proposed solution has been verified.


Review discussion:
Definition 1 reads: "We call such an M as a module. If there exists a process that we are able to morph M0 to M, then we say that module M is morphable, and the morphing process is called modular network morphism."
It sounds tautologic and is unclear whether the morphism should happen through a sequence of atomic morphisms as desribed in figure 1 obeying the equations 3.1 (1) and (2)?
Module M is morphable <=> there exits a process that morph M0 to M. This process is called modular network morphism. "A sequence of atomic morphisms as described in figure 1 obeying the equations 3.1 (1) and (2)" is only one way of achieving the morphism. As an example for another way of morphing, in Section 3.6, we show that irreducible modules can achieve the morphism by Algorithm 2.
The paper presents an interesting incremental approach for exploring new convolutional network hierarchies in an incremental manner after a baseline network has reached a good recognition performance.
The experiments are presented for the CIFAR-100 and ImageNet benchmarks by morphing various ResNet models into better performing models with somewhat more computation.
Although the baselines are less strong than those presented in the literature, the paper claims significant error reduction for both ImageNet and CIFAR-100.
The main idea of the paper is to rewrite convolutions into multiple convolutions while expanding the number of filters. It is quite unexpected that this approach yields any improvements over the baseline model at all.
However, for some of the basic tenets of network morphing, experimental evidence is not given in the paper. Here are some fundamental questions raised by the paper:
- How does the quality of morphed networks compares to those with the same topology trained from scratch?
- How does the incremental training time after morphing relate to that of the network trained from scratch?
- Where is the extra computational cost of the morphed networks come from?
- Why is the quality of the baseline ResNet models lag behind those that are reported in the literature and github? (E.g. the github ResNet-101 model is supposed to have 6.1% top-5 recall vs 6.6 reported in the paper)
More evidence for the first three points would be necessary to evaluate the validity of the claims of the paper.
The paper is written reasonably well and can be understood quite well, but the missing evidence and weaker baselines make it looks somewhat less convincing.
I would be inclined to revise up the score if a more experimental evidence were given for the main message of the paper (see the points above).
We appreciate the recognition and detailed comments from the reviewer, and have the following responses to address the concerns.
1) How does the quality of morphed networks compares to those with the same topology trained from scratch?
The proposed learning scheme produced much better results than the learning from scratch scheme.
For example, on the CIFAR10 dataset, for morph110_1c1, the absolute performance improvement is up to 2.66%, and the relative improvement is up to 32.6%. On the CIFAR100 dataset, for morph110_1c1, the absolute performance improvement is up to 5.13%, and the relative improvement is up to 16.1%. These results have been updated in this paper with, a) two newly added tables (Table 2 & 4), b) one newly added paragraph (the second to the last one at the end of Section 4.2).
2) How does the incremental training time after morphing relate to that of the network trained from scratch?
In this work, we adopted a uniform training procedure for both of these two learning scheme. The setup is the same as [He2015].
Therefore, the incremental training time is the same as training from scratch in this research.
But the incremental training time after morphing can be greatly speeded up since the weights are already learned. From experience, the incremental training does not necessarily need a full training. Currently we are inspecting on this topic.
3) Where is the extra computational cost of the morphed networks come from?
The extra computational cost comes from the newly added parts of the network.
More precisely speaking, it is the computational cost of the part that the network morphs to, minus the computational cost of the part that the network morphs from. As an example for ResNet, please refer to Fig. 3 (a) and (b), where the newly added part is shadowed in green.
4) Why is the quality of the baseline ResNet models lag behind those that are reported in the literature and github? (E.g. the github ResNet-101 model is supposed to have 6.1% top-5 recall vs 6.6 reported in the paper)
There might be some misunderstanding. 6.61% is the top-1 error rate for ResNet-110 on CIFAR10 ([He2015] (Table 6)), while 6.1% is the top-5 error rate for ResNet-101 on ImageNet.
The paper proposes a methodology for morphing a trained network to different architecture without having to retrain from scratch.
The manuscript reads well and the description is easy to follow.
However, the results are not very convincing as the selected baselines are considerably far from the state of the art.
The paper should include comparisons with state of the art, for example wide residual networks.
Tables should also report number of parameters for each architecture, this would help fair comparison.
We appreciate the recognition and detailed comments from Reviewer #2, and have the following responses to address the concerns.
1) “The results are not very convincing as the selected baselines are considerably far from the state of the art. The paper should include comparisons with state of the art, for example wide residual networks.”
In this paper, we target at introducing a general learning scheme, instead of a specific neural network model. Owning to the strict definition of this learning scheme and the proposed morphing algorithms, it has potential to build a more powerful model based on an (arbitrary) existing parent network. Definitely, it could be better if we are able to include comparison results with all state-of-the-art networks beyond residual networks.
However, due to the computational resource limitation, we decided to select the residual networks, the most representative network architecture, as the parent network to morph. Notice that, almost all recently proposed network architectures, such as wide residual networks, fractal networks, stochastic depth networks, and densely connected networks, are variants of or quite related to the residual networks. The success of the proposed learning scheme on residual network architectures indicates that it is very likely to also work well on other newly proposed state-of-the-art networks related to residual architectures, but more experiments about which are needed and considered as our future work.
2) “Tables should also report number of parameters for each architecture, this would help fair comparison.”
We have added the number of parameters for each architecture in Table 1 & 3. As can be seen, the morphed network architectures have less than 1.2x parameters comparing against the original ones. Proper discussions about these results are also added at the end of the second paragraph in Section 4.2.
The score is updated after revision.
This paper presents works on neural network / CNN architecture morphing.
Results are not reported on ImageNet larger ResNet and new network architecture such as Xception and DenseNet - which are maybe too new!
Also most results are reported in small datasets and network, which do not offer confidence in the usability in production systems.
My biggest issue is that computational time and effort for these techniques is not mentioned in detail. We always want to be able to quantify the extra effort of understanding and using a new technique, especially if the results are minor.
We appreciate the recognition and detailed comments from Reviewer #1, and have the following responses to address the concerns.
1) “Results are not reported on ImageNet larger ResNet and new network architecture such as Xception and DenseNet - which are maybe too new!
Also most results are reported in small datasets and network, which do not offer confidence in the usability in production systems.”
In Section 4.4, we reported the results of the morphed network morph18_1c1 and the original ResNet-18 on ImageNet. In the updated version, we also added the results of the original and morphed 34-layer networks on ImageNet. These results are consistent with an absolute performance improvement of 1% and a relative improvement up to 4%. The results of the 34-layer network were added in the second paragraph of Section 4.4, Table 5, and Fig. 6.
As for the newly proposed network architectures, definitely, it would be better if we are able to provide comparison results with all state-of-the-art networks beyond residual networks.
However, in this paper, we target at introducing a general learning scheme, instead of a specific neural network model. Owning to the strict definition of this learning scheme and the proposed morphing algorithms, it has potential to build a more powerful model based on an (arbitrary) existing parent network. Due to the computational resource limitation, we decided to select the residual networks, the most representative network architecture, as the parent network to morph.
2) “My biggest issue is that computational time and effort for these techniques is not mentioned in detail. We always want to be able to quantify the extra effort of understanding and using a new technique, especially if the results are minor.”
It has been verified in our previous work [Wei2016] that VGG16 could be easily morphed into a 19-layer network (NetMorph-VGG16), with a 15x speedup compared with training from scratch. (NetMorh-VGG16 performed better than both VGG16 and VGG19.) In this research, we are expecting to use a minimal extra cost to achieve a maximum possible performance improvement. As shown in Table 1 and 3, both the number of parameters and the computational cost (measured in FLOP) of the morphed models are only slightly larger than the original models (around 1.1x), yet with noticeable performance improvements (absolute performance improvements up to 2% and relative performances up to 20%). In this work, our focus is to study the macroscopic problem of network morphism, i.e., modularized morphing.
Thus we adopted a uniform training procedure for both of these two learning schemes.
Therefore, the training time for network morphism is the same as training from scratch in this research.
But the training can be greatly speeded up as in [Wei2016], which however is not our focus in this research.
This paper studies knowledge transfer problem from small capacity network to bigger one. This is a follow-up work of Net2Net (ICLR 2015) and NetMorph(ICML 2016).
Comments
- 1) This paper studies macroscopic problem, with the morphing process composed by multiple atomic operations.
While the atomic operations are proposed in Net2Net and NetMorph, there has not been study of the general modularized process principally.
Thus this paper asks a novel question.
- 2) The solution by composing multiple atomic transformations seems to be quite reasonable.
- 3) In the “related work” section, it is better to change “network morphism” to “knowledge transfer” or in the subsection title, most of these works are known as knowledge transfer and it helps to connect to the existing works.
- 4) The author shows experiments on variants of ResNet.
While the experiment shows that initializing from ResNet gives better error rate than the ones trained from scratch, it is unclear what the source This paper studies knowledge transfer problem from small capacity network to bigger one. This is a follow-up work of Net2Net (ICLR 2015) and NetMorph(ICML 2016).
Comments
- 1) This paper studies macroscopic problem, with the morphing process composed by multiple atomic operations.
While the atomic operations are proposed in Net2Net and NetMorph, there has not been study of the general modularized process principally.
Thus this paper asks a novel question.
- 2) The solution by composing multiple atomic transformations seems to be quite reasonable.
- 3) In the “related work” section, it is better to change “network morphism” to “knowledge transfer” or in the subsection title, most of these works are known as knowledge transfer and it helps to connect to the existing works.
- 4) The author shows experiments on variants of ResNet.
While the experiment shows that initializing from ResNet gives better error rate than the ones trained from scratch, it is unclear what the source is.
- 5) One major advantage of this type of knowledge transfer (Net2Net, NetMorph) is to speedup training and model exploration. There seems to be no experiments demonstrate such advantage (possibly due to the lose initialization of BatchNorm). This is the major drawback of this paper.
-6)  The method proposed by the author can in principle do quite complicated transformation, e.g. transform  an entire resnet from a single conv layer, the experiment only consists of simple module transformations, which in some way can be covered by atomic operations. It would be more interesting to see what the results of more complicated transformations are (even if they are not as effective).
In summary, this paper studies a novel problem of knowledge transfer in a macroscopic level. The method could be of interest to the ICLR community. The experiments should be improved (comment 5) to make the results more convincing and practically useful and I strongly encourage the authors to do so.
We appreciate the recognition and detailed comments from Reviewer #4, and have the following responses to address the concerns.
3) In the “related work” section, it is better to change “network morphism” to “knowledge transfer” or in the subsection title, most of these works are known as knowledge transfer and it helps to connect to the existing works.
We have made the change as suggested.
4) The author shows experiments on variants of ResNet.
While the experiment shows that initializing from ResNet gives better error rate than the ones trained from scratch, it is unclear what the source is.
When available, the results were cited from their original papers.
For example, the accuracies of ResNet on the CIFAR10 dataset in Table 1 were from [He2015]. For other ones that cannot be found in any existing paper, we trained the models by ourselves.
5) One major advantage of this type of knowledge transfer (Net2Net, NetMorph) is to speedup training and model exploration. There seems to be no experiments demonstrate such advantage (possibly due to the lose initialization of BatchNorm). This is the major drawback of this paper.
It has been verified in our previous work [Wei2016] that VGG16 could be easily morphed into a 19-layer network (NetMorph-VGG16), with a 15x speedup compared with training from scratch. (NetMorh-VGG16 performed better than both VGG16 and VGG19.) In this work, our focus is to study the macroscopic problem of network morphism, i.e., modularized morphing.
Thus we adopted a uniform training procedure for both of these two learning schemes.
Therefore, the training time for network morphism is the same as training from scratch in this research.
But the training can be greatly speeded up as in [Wei2016], which however is not our focus in this research.
It is much cheaper to adopt network morphism to explore and design a new effective network architecture.
For example, for the network architectures illustrated in Fig. 4, we simply split the branches to compose new modules, which does not require domain-specific knowledge.
However, designing such a new network architecture via training from scratch requires significant insight to the network architectures, and it is also hard to tell whether the newly designed network will achieve a better performance until it is fully trained.
Another advantage when exploring new network architectures with network morphism is that, one can quickly check whether a morphed architecture deserves further exploration by continuing to train the morphed network in a finer learning rate (e.g. 1e-5), to see if the performance is improved.
Hence, one does not have to wait for days or even months of training time to tell whether the new network architecture will achieve a better performance. This could significantly save human time for deciding which network architecture is worth for exploring.
For BatchNorm, it is not a problem in our proposed algorithms.
While it will cause the morphed network not to exactly preserve the network function, this small perturbation introduced by BatchNorm actually will not affect the performance of the morphed network. We observed almost identical accuracies for the original network and the morphed network before continual training. This was stated in the last paragraph of Section 3.7.
6) The method proposed by the author can in principle do quite complicated transformation, e.g. transform an entire resnet from a single conv layer, the experiment only consists of simple module transformations, which in some way can be covered by atomic operations. It would be more interesting to see what the results of more complicated transformations are (even if they are not as effective).
This is a very good suggestion. Theoretically, an entire ResNet could be morphed from a single convolutional layer.
However, intuitively, this will not be that effective as the change to the network architecture is too large.
Instead, we proposed to grow the network depth in an exponential order by network morphism. It was demonstrated to be very effective. This is not trivial topic, and we have organized the findings in another paper which is under review.
Quotes from the author
" Another advantage when exploring new network architectures with network morphism is that, one can quickly check whether a morphed architecture deserves further exploration by continuing to train the morphed network in a finer learning rate (e.g. 1e-5), to see if the performance is improved.
Hence, one does not have to wait for days or even months of training time to tell whether the new network architecture will achieve a better performance. "
This is exactly one benefit of my previous comment(5) and is the major reason why such method is used.
However, while the claim seems to be true, it should be reflected in the experiment section.
One can continue to train the morphed network in a finer learning rate, and a better-performing model could be achieved with significant shortened training time.
However, we also found that a morphed network with a full training process in multiple learning rates as the learning from scratch scheme could achieve a slightly better result.
On the other hand, in this work, we focused more on the model capability, instead of training time.
Therefore, for a fair comparison, we adopted a uniform training strategy for both the morphing scheme and the learning from scratch scheme.
While not being a focus, as suggested, we add a new paragraph at the end of Section 4.2, to make clear this point. We leave the thorough study on how to systematically leverage the new learning scheme to speed up the model training and exploration as our future work.
skip question

Meta-review:
role: PC
data: comment: The paper investigates the problem of morphing one convolutional network into another with application to exploring the model space (starting from a pre-trained baseline model). The resulting morphed models perform better than the baseline, albeit at the cost of more parameters and training. Importantly, it has not been demonstrated that morphing leads to training time speed-up, which is an important factor to consider when exploring new architectures starting from pre-trained models. Still, the presented approach and the experiments would be of interest to the community, so I recommend the paper for workshop presentation.

Decision: Invite to Workshop Track