Better Together: Resnet-50 accuracy with $13 \times $ fewer parameters and at $3 \times $ speed
Authors: Utkarsh Nath
Shrinu Kushagra
Venue:  (2021)
Keywords: Deep neural networks
Theory of deep networks
deep regularization
Neural network compression

Abstract: Recent research on compressing deep neural networks has focused on reducing the number of parameters. Smaller networks are easier to export and deploy on edge-devices. We introduce Adjoined networks as a training approach that can regularize and compress any CNN-based neural architecture. Our one-shot learning paradigm trains both the original and the smaller networks together. The parameters of the smaller network are shared across both the architectures. We prove strong theoretical guarantees on the regularization behavior of the adjoint training paradigm. We complement our theoretical analysis by an extensive empirical evaluation of both the compression and regularization behavior of adjoint networks. For resnet-50 trained adjointly on Imagenet, we are able to achieve a $13.7 \times$ reduction in the number of parameters and a $3 \times$ improvement in inference time without any significant drop in accuracy. For the same architecture on CIFAR-100, we are able to achieve a $99.7 \times$ reduction in the number of parameters and a $5 \times$ improvement in inference time. On both these datasets, the original network trained in the adjoint fashion gains about $3\%$ in top-1 accuracy as compared to the same network trained in the standard fashion. 

Under review as a conference paper at ICLR 2021
BETTER TOGETHER: RESNET-50 ACCURACY WITH 13x FEWER PARAMETERS AND AT 3x SPEED
Anonymous authors Paper under double-blind review
ABSTRACT
Recent research on compressing deep neural networks has focused on reducing the number of parameters. Smaller networks are easier to export and deploy on edge-devices. We introduce Adjoined networks as a training approach that can regularize and compress any ENN-based neural architecture. Our one-shot learning paradigm trains both the original and the smaller networks together. The param- eters of the smaller network are shared across both the architectures. We prove strong theoretical guarantees on the regularization behavior of the adjoint training paradigm. We complement our theoretical analysis by an extensive empirical evalu- ation of both the compression and regularization behavior of adjoint networks. For resnet-50 trained adjointly on Imagenet, we are able achieve 1 reduction in the number of parameter: aaaameter=aadd and a 3x improvement in inference time without any significant drop in accuracy. For the same architecture on CIFAR- 100, we are able to achieve 99.7x reduction in the number of parameters and a 5æ improvement in inference time. On both these datasets, the original network trained in the adjoint fashion gains about 3% in top-1 accuracy compared to the same network trained in the standard fashion.
INTRODUCTION
Neural networks have achieved state-of-the art performance on computer vision such classifica- tion, object detection Redmon et al.] (2016), image segmentation Badrinarayanan et al.) (2017) and many more. Since the introduction of (rexnet(Krizhewsss et al. (2012), neural architectures have progressively gone deeper with an increase the number of parameters. This includes architectures like Resnet He al. (0166) and its many variants (xresnet [He et al. (2019), resnext Xie al. (2017);
Hu et al. (2018) etc.), Densenet Huang et al. (2017). Inception networks Chen et al. (2017) and many others.
While these networks achieve exceptional performance on many tasks, their large size makes it difficult to deploy on many edge devices (like mobile phones, iot and embedded devices). Unlike cloud servers, these edge devices are constrained in terms of memory, compute and energy resources. large network performs lot of computations, consumes more energy and is difficult to transport and to update. A large network also has a high prediction time per image. This is constraint when real-time inference is needed. Thus, compressing neural networks while maintaining accuracy has received significant attention in the last few years.
Pruning These techniques involve removing parameters (or weights) which satisfy some criteria. For example, in weight pruning, all the parameters whose values are below some pre-determined threshold are removed Han et al. (2015). A natural extension of this is channel pruning Liu et al. (2017) and filter pruning Li et al. (2016) where entire convolution channel or filter is removed according to some criteria. However, all of these methods involve multiple passes of pruning followed by fine-tuning and require very long time compress. Moreover, weight pruning doesn't 't give the benefit of faster inference times unless there is hardware support for fast sparse matrix multiplications. In this paper, we propose one-shot compression procedure (as opposed multiple passes). To the best of our knowledge, pruning techniques have not been successfully applied (without significant accuracy loss) to large architectures like Resnet-50 considered in this paper.
For size comparison, ignore the parameters in the last linear layer as it varies by dataset and are typically dropped during fine-tuning. Else, the reductions 11.5x and 95xr for imagenct and cifar-101 100 respectively.

---

Under review as a conference paper at ICLR 2021
Larger
y
Shared parameters
P
mp
Smaller
loss fn
Figure 1: Training paradigm based on adjoined networks. The original and the compressed version of the network are trained together with the parameters of the smaller network shared across both. The network outputs two probability vectors (original network)and (corresponding the smaller network).
Quantization and now-precision training- In quantization- based techniques, multiple parameters are quantized such that share the same value. Hence, only the effective values and their indices need to be stored Han et al. 2015). This method of scalar quantization can be extended to vector quantization where a group of parameters share values Saureira-Perpinaa & Idelbayev (2017). In recent work. [Stock et al. (2019) used a clustering based approach for vector quantization of resnet architectures. A similar approach is low-precision training where the goal is to train networks with integer ternary or binary weights instead of floating point numbers SharerrSShrrrar al. (017)) McDonnell (2018); Courbariaux et al. (2015). As before, these techniques assume the availability of special hardware which supports fast inference.
Small architectures - Another approach is to design smaller architectures which can run efficiently on edge devices. SqueezeNet (aaeezooee landola et al. (216)), MobileNet (andler (2018) and EfficientNet Tan & Le (2019) to name few. In this paper. our goal is to design paradigm which can compress any architecture. Hence, the direction of architecture search is orthogonal to our approach.
To summarize, most of the current approaches suffer from one of the two problems. (1) Require the availability of special hardware to support fast inference. (2) Require large training time as they alternate between pruning and fine-tuning. In this work, we propose novel training paradigm based on adjoined networks which can compress any neural architecture, provides inference-time speedups and works at the application layer (does not require any specialized hardware).
As shown in Fig. T in the adjoint training paradigm, both the original and the compressed network trained together at the same time. The parameters of the larger network are super-set of the parameters of the smaller network. Details of our design, how it supports fast inference and relationship with other architectures (feacher-student Hinton al. (015)) siamese networks Bertinetto et (2016), slimmable networks Yu et al. (2018). deep mutual learning Zhang et al.) (2018) and lottery ticket hypothesis Frankle & Carbin (2018) are discussed in Section 2 In our training paradigm, we get two outputs and corresponding to the original and smaller networks respectively. We train the two networks using novel iinedeepedeett loss function, adjoint loss described in Section 3 The adjoint loss not only trains the smaller network but also acts as a regularizer for the bigger (original) network. In Section we provide strong theoretical guarentees adjoint training. We also show that training and regularizing in the adjoint fashion is better than other regularization techniques like dropouts Srriaastaaa al. (2014).
the regularization behaviour of
In Section 5 we describe our results. We run several experiments various datasets like ImageNet Russakovsky et al. (2015) and CIFAR-10 and CIFAR- 100 Krizhevsky et al. (2009). For each of these datasets, we consider different architectures like resnet-50 and resnet-18. On CIFAR-100, the adjoint training paradigm allows to compress resnet-50 architecture by 99.7x without losing any accuracy. The compressed architecture has inference speed of 5z when compared against the original, bigger architecture. Moreover, the original network gains 3.58% in accuracy when compared against the same network trained in the standard (non-adjoint) fashion. On the same dataset, for resnet-18 18,
2

---

Under review as a conference paper at ICLR 2021
imput
output
conv filters
VS
impl
out
out
inp2
only a-fraction of conv filters are non-zero for inp2
Figure 2: (Top) Standard 2d-convolution operation. The inputs, outputs and convolution filters are all 3d volumes. (Bottom) Adjoint convolution operation. The convolution layer receives two inputs inp1 and imp2. Standard convolution operation is applied on inp I to get out For the second input, only a fraction of the filters used (rest can be treated
omitted) get out2.
acheive a compression factor of 49r an inference speed of 3r with the bigger network gaining 5.55% accuracy (over the non-adjoint counterpart). On Imagenet, for resnet-50, we are able to compress by 13.7x an inference speed of 3x. In this case, the bigger network gains 2, 2.43% over its non-adjoint cousin clearly showing that it is better to train the networks together.
2 ADJOINED NETWORKS
In our training paradigm the original (bigger) and the smaller network are trained together. The motivation for this kind of training comes from the principle that good teachers lifelong learners. Hence, the bigger network which serves as a teacher for the smaller network should not be frozen (as in standard teacher-student architecture designs Hinton al. (2015)). But rather both should learn together in 'combined learning environment', that is, adjoined networks By learning together both the networks can be better together.
We are now ready to describe our approach and discuss the design of adjoined networks. But before that, let take re-look the standard convolution operator. Let Rxxxxee be the input convolution layer with weights W E R Rous k where Cin. Cout denotes the number of input and output channels, i the kernel size and h, the height and width of the image. Then, have that
y = conv(x, W)
In the adjoint paradigm, a convolution layer with weight matrix W and a binary mask matrix M e [0,1] x receives two inputs x1 and x2 of size yı and defined below.
Gin and outputs two vectors
(I) Here M is of the same shape as W; and represents element-wise multiplication. note that the parameters of the matrix M are fixed before training and not learnt. The vector x represents an input the original (bigger) network while the vector X2 the input to the smaller, compressed network. For the first convolution layer of the network equal for the deeper convolution layers (Fig. D}. The mask matrix M serves to zero-out some of the parameters of the convolution layer thereby enabling network compression. In this paper, we consider matrices
Y1 conv W)
Y2 conv W M)
But the two vectors are not necessarily
the following form.
M = Ga = matrix such that the first Cout rows are all and the rest
(2)
Throughout the paper, the adjoined networks and adjoint training paradigm interchangeahly.
3

---

Under review as a conference paper at ICLR 2021
Consider another matrix r3 of the same shape M which is such that /3 fraction of the entries (selected uniformly at random) are zero and the rest are one. We achieve further compression. by multiplying a, by rg. In Section [ we run experiments with M a, for {4, 8, 16) and for various choices 3 E {0.1,... can be converted and trained in adjoint fashion by replacing the standard convolution operation by the adjoint convolution operation (Eqn. [ Since the first layer receives single input (Fig. D two copies are created which are passed the adjoined network. The network finally gives two outputs corresponding to the original (bigger or unmasked) network and q corresponding to the smaller (compressed) network where each convolution operation is done using a subset of the parameters described by the mask matrix M (or M.). We train the network using a novel time-dependent loss function which forces p and q to be close to one another (Defn. D. Before we describe our loss function in detail, we first compare our approach to other similar approaches and ideas in the literature.
0.9) Putting this all together, we see that any CNN-based architecture
Similar architectures The idea of training two networks together was recently considered by Zhang ctt al. (2018). Rather than using teacher to train student, they let a cohort of students train together consider train teacher and student together
using distillation loss function. In this paper, rather than a cohort of students. We also use a novel time-dependent loss function. Moreover, we also provide theoretical guarantees on the efficacy of our approach. Another related architecture was considered in the work of Slimmable Networks Yu et al. (2018). Here different networks (or architectures) are switched on one-al-ar ane-at-a-time and trained using the standard logistic loss function. By contrast, in this work, both the networks are trained together at the same time using novel loss function (adjoint-loss). Moreover, this work, we show compression and regularization results on large-scale datasets like Imagenet and for large architectures like Resnet-50.
Siamese networks is neural network design that used to compare two input vectors. Also called twin networks, they consist of two networks with identical parameters. Given two input vectors, the network returns two output vectors which are then used to compute the similarity between the input vectors Bromley et al. (1994). More recently, siamese networks have been used in face verification Taigman et al. (2014). The design of adjoined network also uses two architectures, However, both the architectures are not identical. Rather one is super-set of the other. Also, rather than working on two different input vectors, adjoined networks work on a single input.
Lottery ticket hypothesis Frankle & Carbin (2018) says the following. A neural network contains within it a subset of parameters which are 'special'. These 10 - 20% special parameters won the lottery during the network initialization process and it is possible prune and train the original network to using only these special parameters. Thus, some weights in the network are just 'lucky' that they happened to be initialized in the 'right' way which made training possible with only these subsets of parameters, In our design, rather than relying on random initialization, we "force" some parameters to be special through the use of the mask matrix M and then training the two networks together through the adjoint- (Defn. 1.
3 ONE-SHOT REGULARIZATION AND COMPRESSION
In the previous section, we looked at the design on adjoined networks. For one input (X, E Rxxxxen [0, 1]"e, the network outputs two vectors p and q E [0, 1]a where n. denotes the number of classes and Cin denotes the number of input channels (equals 3 for RGB images).
Definition (Adjoint loss). Let y be the ground-truth one-hot encoded vector and p and q be output probabilities by the adjoined network. Then
C(y.p.9) = -y logp + +A(t) At).L(pp.))
(3)
where KL(p,q) = E, log Ps is the measure of difference between two probability measures q. Kullback & Leibler (1951). The regularization term X : [0, 1] + is a function which changes with current epock
the number of epochs during training. Here Total number of spocks equals zero at the start of training
and equals one at the end.
In our definition of loss function, the first term is the standard entropy loss function which trains the bigger network. To train the smaller network, we use the predictions from the bigger network as a soft ground-truth signal. We use kl-divergence to measure how far the output of the smaller network
4

---

Under review as a conference paper at ICLR 2021
is from the bigger network. This also has regularizing effect as it forces the network to learn from smaller set of parameters. Note that, in our implementations, we use [ q) = EEPt ps log Pitc to g:=e
avoid rounding and division by zero errors where -0-
At the start of training, p is not a reliable indicator of the ground-truth labels. To compensate for this, the regularization term À changes with time. In our experiments, we used X(t) min (4t², 1]. Thus, the contribution of the second term in the loss is zero at the beginning and steadily grows to one at 50% training. We experiment with different choices of the regularization function À the results of which are in the appendix.
Comparison with dropout- Our idea of using only subset of the parameters has some similarity to using dropouts where fraction of the parameters are initialized to zero at training time. In this case, only the output of the smaller network is used train the parameters. In our case, we use both the outputs. Our experiments show that training the networks together is a much more effective regularization strategy as compared to dropouts.
4 THEORY: REGULARIZATION BEHAVIOR
To study the regularization behavior of adjoint training paradigm (or the loss functionn, we consider the following neural network model. A multi-layer neural network that has only convolution or linear fully-connected layers with two possible choices for activation function, ReLU or the linear function. Although neural networks used in practice often have other types of layers max- pooling. batch-normalizatioo etc. and other possible activation functions, the proposed model is often studied as a first step towards understanding the dynamics and behavior of deep networks (for example in Phuong & Lampert (2019), (aaxe et al. (2013), Kawaguchi (2016) and Hardt & Ma (2016)). We are now ready to state the main theoretical result of this paper.
Theorem 4.1. Given a deep neural network which consists of only convolution and linear layers. Let the network use one of fxx) min 0} (relu) f(x) = (linear) as the activation function. Let the network be trained using the adjoint loss function as defined in Eqn. 3 Let be the set of parameters of the network A which is shared across both the smaller and bigger networks. Let Y be the set of parameters of the bigger network not shared with the smaller network. Let p be the output of the larger network and lel q be the output of the smaller network. Then, the adjoint loss function induces a data-dependent regularizer with the following properties.
For all E X, the induced L2 penalty is given by Ex Pi (log' Pi pi- llo' q:)² For all E Y, the induced L2 penalty is given by Σ, Pi (log' p.)²
Proof sketch To analyze the induced L2 penalty, we consider the second order taylor approximation of the ki-divergence term for all parameters x E and for all y E Y. By treating, P; (and q:) as functions of the given parameter (or y) and careful analysis gives us the result of the theorem. The detailed derivations are the appendix.
Thm. A.I shows that for that shared parameters adjoint loss penalizes those parameters whose rate of change between the larger and smaller networks is large (on the log scale). Hence, it encourages' parameters that have similar behavior (rate of change) on both the networks. Similarly, for the perrpparamteerized weights of the model y, the regularization imposed by the adjoint loss such that if these parameters change a lot (on the log scale) then the penalty imposed on such parameters more. Thus, adjoint loss "encourages" such parameters not to change by lot thereby enabling compression.
EXPERIMENTS
We are now ready to describe our experiments detail. We run experiments on five different datasets.
(1) Imagenet an image classification dataset Russakovsky al. (2015) with 1000 classes and about
1.2M images (2) CIFAR- /0 - a collection of 60k images in 10 classes. (3) CIFAR-100 same as
cifar- 10 Krizheeeyy et al. (2099) but with 100 classes. (4) Imagewoof A proxy dataset Howard
(2019); (Shleifer & Prokop 2019) containing 10 different dog breeds from imagenet. (5) Oxford-
Pets a dataset Parkhi al. (2012) of pets containing 37 classes with approximately 200 images per
5

---

Under review as a conference paper at ICLR 2021
Adjoint-small vs teacher student vs standard
Mask matrix (M)
Speed gain
Adjoint (small)
Teacher student
Network
Dataset
#params
Standard
Resnet-50
Cifar-10
ag T0.9
88.9x
4x
90.05
89.44
90.25
100
016 * To.9
99.6x
4.7x
64.54
63.62
65.31
Imagenet
04
13.7x
3.1x
71.84
70.55
73.4 F
Imagewoof
04 ro.8
44.8x
3.1x
84.76
83.88
85.2
Pets
04 7o.5
24.15x
3.1x
85.38
84.97
85.31
Resnet-18
Cifar-10
016 T0.9
68.7x
2.6x
87.67
88.43
88.17
Cifar-100
To.9
48,9x
2.1x
61.42
60.69
61.39
Imagewoof
04 ro.5
21.8x
2.1x
82.81
82.61
83.35
Pets
02 ro.5
7.2x
1.7x
83.62
80.98
84.84
Table 1:
The #params column is the ratio of number of parameters of the smaller network compared against the standard full network. The speed gain column denotes the ratio of inference time. The last three columns compare the accuracies (in %) of the the smaller network trained using the adjoint paradigm vs the the accuracy of the same network trained using the teacher student paradigm against the accuracy of the standard full network. dn defined in Eqn. 2 with 3 fraction of its entries are zero, * denotes element-wise dot-product. Detailed results can be found in the appendix.
and ry denotes random matrix
class. For each of these datasets, we use standard data augmentation techniques like random-resize cropping, random flipping etc. The details provided in the appendix.
We train two different architectures on all the above datasets. Namely, resnet50 and 8 The detailed architecture diagram be found in supplementary material. On each dataset, first train these architectures in the standard non-adjoint fashion using the cross-entropy loss function. We will refer it by the name standard standard-full. Next, we train the adjoint network, obtained by replacing the standard convolution operation by the adjoint convolution operation, using the adjoint loss function. In the second step, we obtain two different networks. In this section, we refer to them by ajoint-full and the adjoint-smale small networks. We compare the performance of the adjoint-full and adjoint-smal networks against the standard network. One point to note that we do not replace the convolutions in the stem layers but only those in the res-blocks. Since most of the weights are in the later layers, this leads to significant space and time savings while retaining competitive accuracy.
We ran experiments gpu enabled machine using pytorch. We trained all networks using the adam optimizer with a cosine learning rate schedule with gradual warm-up. The parameters of the network were randomly initialized. Unless otherwise specified, train both the standard and adjoint networks for the same number of epochs. We have also open- sourcse our implementation
show the strong regu- larizing effect of adjoint training. In Scction 5.3 we compare our approach to dropouts, a popular approach regularizing deep neural networks. In total, we close 150 different experiments covering different datasets, different masking matrices etc. The detailed results are included in the appendix.
In Section 5.1
describe results for compression. In Section 5.2,
5.1 COMPRESSION
Table 1 compares the performance (top 1% accuracy)of the adjoint-small network against the per- formance of standard-full network. We use the aa as the masking matrix (defined in Eqn. ( The mask is such that the last (1 1) filters are zero. Hence, these can be pruned away to support fast inference. Using aa suffices for speed optimization. But, we can get further reduction in the number of parameters, by multiplying the adjoint matrix by another random matrix This matrix is such that only fraction of its entries are non-zero. Hence, using such a mask matrix further reduces the model size.
The code can be found at
http:/thww..coooouarraaananttt// http:///ww.ooottttoo......k. giit
WWe train imagenet using the standard pytorch code Facebook (022)). The accuracy the same as reported for multi-gpa training imagnet hy nvidia Nvidia 2020)

---

Under review as a conference paper at ICLR 2021
Valid loss vs #Epochs
Valid loss #Epochs
12
Adgand hal
3.00
Stanalard
Standerr 1.
2.2
2.50
1035
2.25
2.4
CE
2.30
3.5
1.0
123
220
Epoch
Epoch
Figure 3: (Left) Plot of validation cross-entropy loss of the adjoint-16 full and standard resnet50 network on CIFAR-100. (Right) Plot of validation cross-entropy loss of the adjoint-4 full and standard resnet50 network on Imagenet
Adjoint-full vs standard
Mask
Network
Dataset
Adjoint-full
Standard
matrix (M)
Resnet-50
Cifar-10
016
91.34
90.25
Cifar-100
416
69.07
65.31
Imagenet
d1
75.84
73.41
Imagewoof
04 * 7o.9
86.3
85.2
Pets
02
87
85.31
Resnet- -8
Cifar-10
as
90.26
88.17
Cifar- 100
0.1
66.84
61.39
Imagewoof
04
84.16
83.35
Pets
08
86.33
84.84
Table 2:
The last two columns show the accuracies (in %) of the network trained in the adjoint fashion and the same network trained in the standard way. In all cases the adjoint network exceeds the accuracy of the standard full network. aa ng are as in Table Detailed results can be found in the appendix.
also observe that resnet-50 is a bigger network and can be compressed more. Also, different datasets can be compressed by different amounts. For example, on cifar-10 10 and 100 datasets, the network can compressed by factors 90x while for other datasets ranges from 7a 44x. Our goal is to compare the adjoint models against the standard models. Note that in all the cases, the drop in accuracy is small, a maximum -1.4% over all the datasets. In some cases, the smaller network outperforms the bigger network. We also compare the performance of the smaller network trained in the adjoint fashion against the same network trained using the standard teacher-student paradigm. On all the datasets and for all architectures (except one) that we considered, adjoint training paradigm out-performed teacher student training.
5.2 REGULARIZATION
Table 2 compares the performance of the adjoint-full network against the performance of the cor- responding standard-ful -fll network. We see consistent trend that the network trained adjointly outperforms the same network trained in the standard way. We see maximum gains on cifar-100, exceeding accuracies by as much as 5.5%. Even imagenet, we see gain of about 2.5%. Fig. shows the plot of validation cross-entropy loss as a function of the number of epochs. For a fair comparison, we look at the -y log (Eqn. 3 term of the adjoint loss function. By regularizing the loss function, we are able to train longer while decreasing the loss function. On the other hand, the network trained in the standard fashion starts to over-fit fit after while (as evident in the loss profile). Fig. shows the loss plot on one architecture for two datasets. Similar plots on other datasets and for other architectures are available in the appendix.
7

---

Under review as a conference paper at ICLR 2021
Valid loss vs #Eoochs
Valid loss #Epochs
102
Standard
14:00-16-4 fal
6.-
Standard fat
Aq fal
2.75
10-
2.50
1.0
CE 1055
2.75
2.5
2.30
248
1.90
125
Epoch
Epoch
Figure 4: (Left) Plot of validation cross-entropy loss the adjoint-16 full and standard dropout resner50 network on CIFAR-100. (Right) Plot of validation cross-entropy loss of the adjoint-4 full and standard dropout resnet50 network on Imagenet
Adjoint-full vs standard + dropouts
Mask
Network
Dataset
Full
Std dropout
matrix (M)
Resnet-50)
Cifar-10
016
91.34
90.11
Cifar-100
Q16
69.07
66.66
Imagenet
44
75.84
69.54
Imagewoof
dg To.9
86.3
82.76
Pets
04
86.73
84.7
Resnet-18
Cifar-10
(g
90.26
89.04
Cifar- 100
0.1
66.09
64.3
Imagewoof
0.4
84.16
82.1
Pets
Ag
86.33
82.61
Table 3: The last columns show the accuracies (in %) of the network trained in the adjoint fashion vs the same network trained in the standard way using dropouts. In all cases the adjoint network exceeds the accuracy of the standard one. aas rg are as in Table Details can be found in the appendix.
5.3 COMPARISON AGAINST DROPOUTS
Table 3 compares the performance of the adjoint-full network against the performance of the corre- sponding standard- -uul network where the standard network was trained using dropouts. We see consistent trend that the network trained adjointly outperforms the network trained with dropout regularization. We see maximum gains on imagenet, exceeding accuracies by as much as 6.3%. On cifar and imagewoof, we gains of about 3%. As is evident from our experiments, dropouts very effective regularizers on the imagenet dataset. Fig. 4 shows the plot of validation cross-entropy loss a function of the number of epochs. As before. we only look the -y log p (Eqn. 3 term. The above results show that adjoint training is much more effective regularization strategy compared against dropouts. Similar plots on other datasets and for other architectures are available in the appendix.
6 CONCLUSION
In this work, we introduced the paradigm of Adjoined network training. We showed how this approach to training neural networks can allow us to compress large networks like Resnet-50 by 13x, (even going up to 99.7xr on some datasets) while retaining the accuracy. We showed that the idea of adjoining two networks together can used to regularize any architecture. We showed that adjoining large and small network together enables the large network to significantly exceed its own accuracy (when trained in the standard way) and a much more effective regularization strategy than dropouts.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. Segnet: A deep convolutional encoder- decoder architecture for image segmentation. IEEE transactions on pattern analysis and machine intelligence, 39(12):2481 2)22411-2495, 2017.
Luca Bertinetto, Jack Valmadre, Joao Henriques, Andrea Vedaldi, and Philip HS Torr. Fully- convolutional siamese networks for object tracking. In European conference on computer vision, pp. 850 865. Springer, 2016.
Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. Signature verification using a" siamese" time delay neural network. In Advances in neural information processing systems, pp. 737-744, 1994.
Miguel A Carreira-Perpinet and Yerlan Idelbayev. Model compression constrained optimization, with application to neural nets. part ii: Quantization. arXiv preprint arXivv1707.04339, 2017.
Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:/706.05587, 2017.
Matthieu Courbariaux, Yoshua Bengio, and Pienre David. Binaryconnect: Training deep neural networks with binary weights during propagations. In Advances in neural information processing systems, pp. 3123-3131 2015.
Facebook. Imagenet training pytorch code.
http://hhw...o//pottoto//////aa/ blob/master/ http://////mmmeeeeeeeaeeea py 2020. Accessed: 2020-10-01.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv: /80300380 2018.
Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00144 2015.
Moritz Hardt and Tengyu Ma. Identity matters in deep learning. arXiv preprint 161 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li. Bag of tricks for image classification with convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 558-567, 2019.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.025.1 2015.
Jeremy Howard. Imagenette. https:/giiuuh.co/ffsttaaaaff com/fussta//mmagenette 2019.
URL: Github
repository with links todataset.
Jie Hu, Li Shen, and Gang Sun. sibrezz--andeexitttti networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132-7141, 2018.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
Forrest N landola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size. arXiv preprint arX1v:1602.07360, 2016.
Kenji Kawaguchi. Deep learning without poor local minima. In Advances in neural information processing systems, pp. 586-594, 2016.

---

Under review as a conference paper at ICLR 2021
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar- 10 and cifar- 100 datasets. URI: https://www. cs. toronto. edulkriz/cifar. html, 6, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffry Hinton. Imagenet classification with deep convolu- tional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathemati- cal statistics, 22(1):79-88) 86, 1951.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint arXiv:1608.087700 2016.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learn- ing efficient convolutional networks through network slimming. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2736-2744422 2044, 2017.
Mark McDonnell. Training wide residual networks for deployment using a single bit for each weight. arXiv preprint arXiv:/802.088530 2018.
Nvidia. Mixed -recissoe training of deep neural networks. [tttps://developeer nvidia mi wtwp:ppeeee.siitaaaaaaaeeeeeeeee
2020. Ac-
cessed: 2020-10-01.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pp. 3498-3505. IEEE, 2012.
Mary Phuong and Christoph Lampert. Towards understanding knowledge distillation. In International Conference Machine Learning, pp. 1142-5151, 2019.
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 779-788, 2016.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115 1-252, 2015. doi: 10.1001 1263-015-0816-
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chich Chich Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520, 2018
Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprini arXiv: 13/.61220, 2013.
Oran Shayer, Dan Levi, and Ethan Fetaya. Learning discrete weights using the local reparameteriza- tion trick. arXiv preprint arXiv: 2017.
Sam Shleifer and Eric Prokop. Using small proxy datasets to accelerate hyperparameter search. arXiv 2019.
preprint 1906.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-199 2014.
Pierre Stock, Armand Joulin, Rémi Gribonval, Benjamin Graham, and Hervé Jégou. And the bit goes down: Revisiting the quantization neural networks. arXiv preprint arxiv:1907.05686, 2019.
Yaniv Taigman, Ming Yang, Mare' Aurelio Ranzato, and Lior Wolf. Deepface: Closing the gap to
human-level performance in face verification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1701-1708, 2014.
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural networks. arXiv preprint arXiv: 1905. 1946, 2019.
10

---

Under review as a conference paper at ICLR 2021
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1492-1500, 2017.
Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas Huang. Slimmable neural networks. arXiv preprint 1812:0881808228 2018.
Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5320-43388 2018.
A REGULARIZATION THEORY
Theorem A.1. Given a deep neural network A which consists of only convolution and linear layers. Let the network use one of f(x) min{z, 0} (relu) or f(x) = (linear) as the activation function. the network be trained using the adjoint loss function as defined Eqn. 3 Let be the set of parameters of the network A which is shared across both the smaller and bigger networks. Let Y be the set of parameters of the bigger network not shared with the smaller network. Let p be the output of the larger network and let q be the output of the smaller network. Then, the adjoint loss function induces a data-depenan dependent regularizer with the following properties.
For all X, the induced L2 penalty is given by Σ. Pi log' p. - log' q:)²
For all y E Y, the induced L2 penalty is given by E. pi log's p.)²
Proof. We interested in analyzing the regularizing behavior of the following loss function. -ylogp+ + (p, q) y is the ground truth label, p is the output probability vector of the bigger network and g the output probability vector of the smaller network. Recall that the parameters of smaller network are shared across both. We will look at the second order taylor expansion for the kl-divergence term. This will give us insights into regularization behavior of the loss function.
Let be parameter which is common across both the networks and be parameter in the bigger network but not smaller one.
D(x) Σ px(z) log (x) log 4 (x)) and D Σ (1) log q.)
For the parameter is constant. Now, computing the first order derivative, we get that
D'(x) Σ px( (x) - log)(z)) + +((x)) (x()))(())
((xx)
D'(y) Σ (l(w))(ogpp(d) log + (v)
Now, computing the second derivative for both the types of parameters, we get that
p((x)
g((x)
D²(x) = Σ (z) - log q: (x)) + (x)
+ (x(
(t(x) (x(x)
((()))))))) (z)
(x(x)
log + p((v)d(()) pi(w)
D²(y) = Σ
+ (v)
p((x))((z) 2p(())((x)) P((x)
(())g(()))))
p((x)) ((x)) px(x) (xx)
(x) Σ
Ep((t)
(x(x)
(2(x)

---

Under review as a conference paper at ICLR 2021
Σ pi Plol' p: log' ):)²
Similarly, for the parameters only in the bigger network, we get that
D²(v)= = Σ p((W)p(()) (:(1)
Σ Pillog' pi)²
(5)
Note that y represents the over over-parameeeee parameterized weights of the model. The equations above show that the regularization imposed by the KI--divergence term these parameters is such that if these parameters change lot (on the log scale) then the penalty imposed on such parameters is more. Thus, the kl-divergence term encourages such parameters not change by a lot.
B DATA AUGMENTATION
We use different data-augmeenationn techniques for different datasets. Below are the details.
Cifar-10, Cifar- and Oxford pets
We apply the following set of transforms for these datasets. (1) We flip the image with probability 0.5. (2) With probability 0.75, we rotate the image by degree d chosen uniformly random from (-mar,, maxa). maz, 25.0 for the cifar datasets and maxa 10.0 for the pets dataset. (3) With probability 0.75, we apply the contrast and brightness transforms.
(4) With probability 0.75, we apply the warp and zoom transforms. (5) We normalize
the image by first dividing all the pixel values by 255. and then subtracting the mean (0.485, 0.456, 0.406] and dividing by the variance [0.229, 0.224, 0.225].
Imagenet and Imagewoof
On these two datasets, apply the following set fo transforms. (1) Random resize cropping Crop a rectangular region with aspect ratio in [3/4, 4,444 4/3] (selected uniformly at random) with area in (0.08, 1.0) of the original area. (2) Flip the image horizontally with probability 0.5. (3) Normalize the image by dividing all pixel values 255.0.
For cifar-10 and cifar- 100, input size is 32 x 32 for all the other datasets, the input size is 224 x 224. The above transforms are applicable for the training dataset. For validation,
center crop select the center of the image with 85% area, followed by a normalization step. Note that our data augmentation are not heavily optimized for accuracy. Rather our goal compare adjoint training with standard training. Hence, we use the same data augmentation steps for both the trainings. For standard training, our accuracies are still comparable to the accuracies reported in the literature on these datasets using the resnet 8 and resnet50) architectures. However, the adjoint training methodology proposed in this paper outperforms the network trained in the standard way.
More details of the autaentnmentttion can be found in the code provided with the supplementary materials.
C RESNET ARCHITECTURE DIAGRAM
conv(3, 16)
conv(16, 64) conv(64, 64) stem
+ MorPool - Res BesBBloc(16,, 64) -> ResBlock(64, 128) - Res ResBkock 256)
at
ResBlook 512) - Adepoge Linear
23
Figure 5: Architecture diagram for resnet50 network used in this paper. conw(ni, no) is combination of convolution layer with ni input and no output channels followed by batch norm and relu layer. The ResBlocks defined in Fig.
12

---

Under review as a conference paper at ICLR 2021
conv(4ni, no) conv(no, no) (nov(no, 4no)
"conv(4no, no) conv(no, no) conv(no, 4no) x 1)
Figure 6: A ResBlock with / layers and input ni and output no.
The architecture for resnet50 is depicted in Figs. and 6 Each cone layer is actually a combination of three layers. A standard convolution layer followed by batch normalization layer followed by relu activation. The Res Block refers to the residual blocks in resnet architecture. Note that the skip connections are not shown these diagrams. For a resnet architecture, each resblock repeated twice instead of 3, 4, 6 and 3 times. Also, the resblock does not have factor four in the convolution input and output.
For adjoint networks, the convolution parameters of all the last three resblocks shared across both the original and the smaller architecture. Note that both the networks have different parameters for the batch-norm layers.
D DETAILED EXPERIMENTAL RESULTS
D.1 EXPERIMENTS ON CIFAR-10
Valid loss Vs Epochs
Valid loss #Epochs
Staddrd
Standier
10-
loss
CE lo5s
84-
04-
Epoch
Epoch
(a) Without random mask
(b) With random mask
Figure 7: Validation loss for the various training paradigms for resnet50 trained Cifar-10. All the adjoint plots correspond to the bigger network.
Valid loss vs # Epochs
Valid loss vs #Epochs
Arg
Arg
loss
0.8
38
54-
20
80
30
120
20
60
80
100
Epoch
Epoch
(a)
(b)
Figure 8: Validation loss for the various training paradigms for resnet50 trained on Cifar-10 10. Valida- tion loss for the various training paradigms for resnet50 trained on Cifar- 10, All the adjoint plots correspond to the bigger network.
13

---

Under review as a conference paper at ICLR 2021
Valid loss vs #Epochs
Valid loss vs # #Epochs
Minurh
144 1asded
Mianb
14-4
Alammb
dispout
1.0
lo85
CE
CE
0.6
64-
20
69
100
20
40
120
Epoch
Epoch
(a) Without random mask
(b) With random mask
Figure 9: Validation loss for the various training paradigms for resnet trained on Cifar-10. 10. All adjoint plots correspond to the bigger network.
Valid loss vs # Epochs
Valid loss #Epochs
r4 Swnt--
Standert
Standard
10-
CE loss
CE
Fpach
Fpach
(b)
Figure 10: Validation loss for the various training paradigms for resnet18 trained on Cifar- 10. All the adjoint plots correspond to the bigger network.
Training with resnet50 on cifar-10 10
Training paradigm
Masking matrix (M)
top-1 full
top-5 full
top-1 small
top-5 small
Standard-full
90.25
99.67
Standard-ful + dropout (0.75)
90.11
99.68
Adjoint-4
04
90.83
99.7
90.25
99.6
Adjoint-8
08
91.06
99.67
89.77
99.63
Adjoint-16
01 6
91.34
99.71
89.88
99.61
Adjoint-4
0. 7o.5
90.56
99.75
90.13
99.73
Adjoint-4
04 70.9
91.12
99.69
90.1
99.7
Adjoint-8
0g 70.5
91.01
99.77
89.64
99.71
Adjoint-8
0g 70.9
91.31
99.73
90.05
99.61
Adjoint-16
016 * 7o.5
91.13
99.59
89.81
99.63
Adjoint-16
016 To.9
91.2
99.68
89.42
99.61
Table Accuracy for the various training paradigms for resnet50 trained on Cifar-10
14

---

Under review as a conference paper at ICLR 2021
Training with 8 on cifar-10
Training paradigm
Masking matrix (M)
top-1 full top-5 full
top-1 small
top-5 small
Standard-full
88.17
99.41
Standard-full dropout
89.04
99.55
(0.5)
Adjoint-2
02
88.75
99.54
88.06
99.41
Adjoint-4
04
89.8
99.73
88.74
99.64
A
90.26
99.72
88.62
99.62
Adjoint-16
016
89.58
99.61
87.94
99.57
Adjoint-4
01 To.9
89.88
99.65
88.59
99.53
Adjoint-4
T0.9
89.75
99.56
88.34
99.48
Adjoint-8
Og Po.5
89.96
99.61
87.85
99.52
Adjoint-8
Og 1o.9
89.4
99.56
87.61
99.51
Adjoint-16
016 * To.9
89.66
99.52
87.67
99.35
Table 5: Accuracy for the various training paradigms for resnet18 trained on Cifar-10
D.2 EXPERIMENTS ON CIFAR-100
Valid loss vs #Epochs
Valid loss vs #Epochs
3.5
Aquin1
3325
A4 4 fund 0.5
Afsint
A4j sani Ca
3.0
44-44 sasi-o07
2.75
dispoutt
Sand-o.4
2.50
105s
2.8
1.9
325
1.90
20
00
:00
:20
580
80
120
Epoch
Epoch
(a) Without random mask
(b) With random mask
Figure 11: Validation loss for various training paradigms for resnet50 trained on Cifar-100, All the adjoint plots correspond to the bigger network.
Valid loss vs # #Epochs
Valid loss vs # #Epochs
322
3.2
6 Tavi-n
3.0
Aq Tand-0.7
Aspid 1and-0.7
arp- tastio
2.75
A4 tand 0a
A4-4 Tant-0.8
Agpid tans-Cul
Agp-8 tund-03
2.50
Standond
Standars
1oss
3.22
CE
2.30
3.7
2.3
1.28
175
175
120
Epoch
Epoch
(b)
Figure 12: Validation loss for the various training paradigms for resnet50 trained on Cifar-100 All the adjoint plots correspond the bigger network.
15

---

Under review as a conference paper at ICLR 2021
Valid loss vs #Epochs
Valid loss vs # #Epochs
16
Mianb
10-
dispout
CE 1055
23
2.5
20
100
60
80
100
320
140
Epoch
Epoch
(a) Without random mask
(b) With random mask
Figure 13: Validation loss for various training paradigms for resnet18 trained on Cifar-100, All the adjoint plots correspond to the bigger network.
Valid loss vs # #Epochs
Valid loss vs # #Epochs
15
6 Taaiin
Mi tanr-0 6
arp d tastio
A4 1and-5.7
Aspie 1and-03
A4-4 Tand-0.
2 tand 0a
1..
Agpcd tans-Cul
Agj-8 1uns-00
Standond
Standare
CE 1o55
CE 1oss
120
Epoch
Epoch
(b)
Figure 14: Validation loss for the various training paradigms for resnet18 trained on Cifar-100. All
the adjoint plots correspond the bigger network.
16

---

Under review as a conference paper at ICLR 2021
Training with resnet50 on cifar-100
Training paradigm
Masking matrix (M)
top-1 full top-5 full
top-1 small
top-5 small
Standard-full
65.31
87.35
Standard-full dropout
66.66
88.24
(0.75)
Adjoint-4
04
67.15
89.04
64.73
87.6
Adjoint-8
ag
68.75
90.79
65.37
88.88
Adjoint-16
01 6
69.07
91.11
65.11
89.08
Adjoint-4
04 70.5
67.47
89.78
64.71
88.12
Adjoint-4
01 T0.6
67.45
89.22
64.56
87.56
Adjoint-4
70.7
67.35
89.69
64.26
87.84
Adjoint-4
04 70.8
67.82
89.71
64.71
88.13
Adjoint-4
04 1o.9
67.57
89.99
65.04
88.24
Adjoint-8
0g 70.5
68.7
90.57
65.35
89.05
Adjoint-8
Og 70.6
68.86
90.68
65.47
89.27
Adjoint-8
Og T0.7
68.57
90.71
64.76
88.6
Adjoint-8
0g 70.8
67.82
89.84
64.76
88.01
Adjoint-8
0g T0.9
69.09
90.99
65.11
89.13
Adjoint-16
016 T0.5
68.59
91.13
64.53
89.1
1djoint-16
016 T0.6
68.04
90.79
64.55
88.66
Adjoint-16
016 To.7
68.85
91.03
64.76
88.99
Adjoint-16
016 T0.8
68.52
90.93
64.35
89.29
Adjoint-16
016 * T0.9
68.89
90.79
64.54
89.48
Table 6: Accuracy for the various training paradigms for resnet50 trained on Cifar-100
17

---

Under review as a conference paper at ICLR 2021
Training with resnet 8 on cifar-100
Training paradigm
Masking matrix (M)
top-1 full top-5 full
top-1 small
top-5 small
Standard-ful
61.39
85.38
Standard-full dropout
64.3
88.13
(0.5)
Adjoint-4
04
66.84
89.2
62.73
87.4
Adjoint-8
0g
66.09
89.43
61.38
87.12
Adjoint-16
016
64.48
89.02
57.84
85.24
Adjoint-4
04 70.5
66.54
89.56
62.64
87.38
Adjoint-4
01 T0.6
65.81
89.49
62.26
87.52
Adjoint-4
70.7
65.88
89.75
61.7
87.53
Adjoint-4
04 70.8
66.13
89.55
61.91
87.08
Adjoint-4
04 1o.9
66.29
89.55
61.42
86.89
Adjoint-8
0g 70.5
65.63
89.28
60.64
86.79
Adjoint-8
Og 70.6
65.14
89.08
60.08
86.39
Adjoint-8
Og T0.7
64.49
89.03
58.93
85.63
Adjoint-8
0g 70.8
64.88
89.17
59.45
86.09
Adjoint-8
0g T0.9
64.34
88.54
58.17
85.15
Adjoint-16
016 T0.5
63.66
88.38
56.22
84.68
1djoint-16
016 T0.6
64.07
88.27
56.83
84.22
Adjoint-16
016 To.7
63.14
88.02
54.78
83.65
1d
016 T0.8
63.26
87.85
54.52
83.2
Adjoint-16
016 * T0.9
62.32
87.37
52.88
82.13
Table 7: Accuracy for the various training paradigms for resnetl 8 trained on Cifar-100
D.3 EXPERIMENTS ON IMAGENET
Valid loss vs # #Epochs
4.0
Adjoint-4
Adjoint
Standard
3.5
loss
2.5
2.0
1.0
20
40
60
800
100
Epoch
Figure 15: Validation loss for the various training paradigms for resnet50 trained on Imagenet. All the adjoint plots correspond the bigger network.
18

---

Under review as a conference paper at ICLR 2021
Training with resnet50 on imagenet
Training paradigm
Masking matrix (M)
top-1 full top-5 full
top-1 small
top-5 small
Standard-ful
73.41
91.07
Standard-full dropout
69.54
88.88
(0.75)
Adjoint-4
Q4
75.84
92.77
71.84
90.42
Adjoint-8
Gg
73.46
91.52
64.7
85.96
Table 8: Accuracy for the various training paradigms for resnet50 trained on imagenet
D.4 EXPERIMENTS ON IMAGEWOOF
Valid loss vs #Epochs
Valid loss vs #Epochs
Aduint
Sand 0.3
Alsint
Ag sand 0.
Standard
Adl-t sand-o.a
Stanclard diopoast
Aut land-00 0/9
Standerd
CF locs
CE loss
Epoch
Epoch
(a) Without random mask
(b) With random mask
Figure 16: Validation loss for the various training paradigms for resnet50 trained on imagewoof. All the adjoint plots correspond the bigger network.
Valid loss vs #Epochs
Valid loss vs #Epochs
6 Tastin
uiul tanr-i 6
arpl tastid
Aq Tand-0.7
Asple + lans-0
A4-4 Tant-0.8
Standars
Ad 2 tand 0.
Standoond
CF 1o0s
CF locs
Epoch
Epoch
(b)
Figure 17: Validation loss for the various training paradigms for resnet50 trained on imagewoof. All
the adjoint plots correspond the bigger network.
19

---

Under review as a conference paper at ICLR 2021
Valid loss vs #Epochs
Valid loss vs #Epochs
144 1asded
Mianb
Atcemb22
drspout
CE loss
loss
20
100
20
100
Epoch
Epoch
(a) Without random mask
(b) With random mask
Figure 18: Validation loss for the various training paradigms for resnet18 8 trained on imagewoof. All the adjoint plots correspond to the bigger network.
Valid loss # #Epochs
Rando
Shandard
CE loss
Epoch
Figure 19: Validation loss for the various training paradigms for resnet18 trained on imagewoof. All the adjoint plots correspond to the bigger network.
Training with resnet18 on imagewoof
Training paradigm
Masking matrix (M)
top-1 full
top-5 full
top-1 small
top-5 small
Standard-full
83.35
98.3
Standard-full dropout
82.1
98.09
(0.5)
Adjoint-4
0.1
84.16
98.38
82.96
98.3
Adjoint-8
0g
84.11
98.54
81.25
98.02
Adjoint-16
016
83.59
98.43
79.37
97.78
Adjoint-4
04 70.5
84.14
98.48
82.81
97.99
Adjoint-4
04 70.9
83.82
98.17
80.54
98.02
Adjoint-8
Ag 7o.5
84.27
98.69
81.69
98.28
Adjoint-8
10.9
84.06
98.41
78.8
97.78
Table 9; Accuracy for the various training paradigms for resnet18 trained on imagewoof
20

---

Under review as a conference paper at ICLR 2021
Training with resnet50 on imagewoof
Training paradigm
Masking matrix (M)
top-1 full
top-5 full
top-1 small
top-5 small
Standard-full
85.2
98.4
Standard-full dropout
82.76
98.38
(0.75)
Adjoint-4
04
85.52
98.3
85.26
98.17
Adjoint-8
0g
85.62
98.59
84.11
98.35
Adjoint-16
016
85.28
98.54
82.94
98.12
Adjoint-4
04 r0.5
85.75
98.48
85
98.48
Adjoint-4
01 r0.6
85.75
98.69
85.15
98.54
Adjoint-4
r0.7
85.31
98.38
84.5
98.35
Adjoint-4
04 r0.8
85.65
98.64
84.76
98.43
Adjoint-4
04 r0.9
86.3
98.59
84.27
98.48
Adjoint-8
0g +0.5
85.72
98.67
83.95
98.28
Adjoint-8
Og r0.6
85.62
98.64
84.01
98.41
Adjoint-8
Og r0.7
85.62
98.48
83.3
98.35
Adjoint-8
0g +0.8
85.49
98.51
82.52
98.38
Adjoint-8
0g 70.9
85.65
98.75
83,41
98.33
Adjoint-16
016 r0.5
86.01
98.56
83.09
98.25
Adjoint-16
016 r0.8
85.39
98.61
82.73
98.12
Adjoint-16
016 r0.9
85.33
98.67
82.26
98.38
Table 10: Accuracy for the various training paradigms for resnet50 trained on imagewoof
D.5 EXPERIMENTS ON OXFORD-PETS
Valid loss vs #Epochs
Valid loss vs #Epochs
Mirett
64-4 + 3ant-0.5 Qhaviant
Ziascladd
Stanciand dispout
CF loss
CF locs
Epoch
Epoch
Without random mask
(b) With random mask
Figure 20: Validation loss for the various training paradigms for resnet50 trained on oxford-pets All the adjoint plots correspond the bigger network.
21

---

Under review as a conference paper at ICLR 2021
Valid loss #Epochs
CE loss
Epoch
Figure 21: Validation loss for the various training paradigms for resnet50 trained on oxford-pets. All
the adjoint plots correspond the bigger network.
Valid loss vs #Epochs
Valid loss vs #Epochs
44-4 1ait-0.5
64-4 lastid
Standard
Ziascland
Stancord dispout
CF loss
CF locs
Epoch
Epoch
(a) Without random mask
(b) With random mask
Figure 22: Validation loss for the various training paradigms for resnet18 trained on oxford-pete -ptts. All the adjoint plots correspond the bigger network.
Valid loss vs #Epochs
Valid loss vs #Epochs
6 Tastin
arp' lastid
Standerd
Standars
1055
CF locs
1.3
Epoch
Epoch
(b)
Figure 23: Validation loss for the various training paradigms for resnet18 trained on oxford-ped All the adjoint plots correspond the bigger network.
22

---

Under review as a conference paper at ICLR 2021
Training with resnet50 on oxford-pets
Training paradigm
Masking matrix (M)
top-1 full
top-5 full
top-1 small
top-5 small
Standard-full
85.31
97.63
Standard-full dropout
84.7
98.1
(0.5)
Adjoint-2
02
87
98.03
86
98.1
Adjoint-4
04
86.73
98.24
85.58
98.03
A
85.85
98.24
84.03
98.1
Adjoint-2
02 70.5
86.6
97.22
85.85
97.36
Adjoint-2
0g To.9
85.52
98.17
85.58
97.69
Adjoint-4
70.5
86.
97.9
85.38
97.56
Adjoint-4
04 7o.9
85.25
97.63
83.69
97.69
Adjoint-8
Og Po.5
86.33
98.37
84.64
97.63
Adjoint-8
To.5
85.85
98.37
81.93
98.51
Table 11: Accuracy for the various training paradigms for resnet50 trained oxford-pets
Training with resnet 8 on oxford-pets
Training paradigm
Masking matrix (M)
top-1 full
top-5 full
top-1 small
top-5 small
Standard-full
84.84
97.76
Standard-full + dropout
82.61
97.09
(0.5)
Adjoint-2
02
84.5
97.56
83.89
97.36
Adjoint-4
04
86.33
98.3
83.08
97.76
Adjoint-8
0g
85.52
98.71
82.61
97.97
Adjoint-2
02 7o.5
84.91
97.69
83.62
97.63
Adjoint-2
0g 70.9
83.76
97.76
80.71
97.22
Adjoint-4
04 70.5
85.65
97.9
82.74
97.83
Adjoint-4
0. To.9
84.84
97.83
80.51
97.09
Adjoint-8
ag 7o.5
85.31
97.9
82.07
97.63
Adjoint-8
10.9
83.33
97.9
76.72
96.48
Adjoint-16
0, 6 To.5
84.5
97.83
77.46
96.75
Adjoint-16
0,6 * T0.9
82.27
97.76
72.05
95.26
Table 12: Accuracy for the various training paradigms for resnet18 trained on oxford-pets
E CHOOSING THE REGULARIZATION FUNCTION
Finally, we compare different choices of regularization function for the adjoint loss (Eqn. 3). For all the previous experiments, the architecture, dataset and mask matrix as resnet18, cifar100 and Adj-4 respectively and vary the regularization function. We look at different functions which includes exponential and trigonometric
the 'quadratic' function X(t) min 1 1). In this section, fix
23

---

Under review as a conference paper at ICLR 2021
Valid loss vs # #Epochs
33
Ouadh abc
Ce l0s5
2.-
65
20
309
Epoch
Figure 24: Validation cross entropy loss for various regularization functions. The networks were trained using Adj-4 mask matrix on cifar- 100 using resnet-18.
Adjoint trained with different regularization functions
Regularization function X(t)
top-1 I full
top-1 I small
cos(t)
2286
-3.44
0.1
0.15
min (onf442,, 11 exp(t) -1
0.00
0.00
-0.37
+0.38
Table 13: The effect of training with differrntt regularization functions the top-1 accuracies of the bigger and the smaller networks. The quadratic function min (4t², 2 113 is used as the base for comparison.
functions. Table 13 and Fig. 24 both show the same trend. The cos function performs the worst while the rest have similar performance. We conjecture that any function that is close to zero for / <- and grows to one eventually should be a reasonable choice for the regularization function. Note that throughout our discussion, have used X(t) e emin 4t², 1} with 1. Depending on the dataset, other values of maybe more appropriate.
24


Reviews:
[{"review_id": "wDUbpxnubD0", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes “Adjoined networks” as a training approach that can regularize and compress any CNN-based neural architecture in a manner of one-shot learning paradigm. The proposed technique deals with the main issues of current compression approaches. That is (1) Requiring the availability of special hardware to support fast inference and (2) Requiring large training time as they alternate between pruning and fine-tuning.   \n\n1)\tThe section 2 of Adjoined networks is not clearly illustrated. The author says that the small network selects a fraction of the convolution filters of the large one as its filters. However, the channel of random selection filters may be inconsistent with the feature maps of the deeper layers. How to implement the convolution in small network needs to be further illustrated?\n2)\tThe author needs to clarify whether the M is optional expect the setting as 16/8/4/2 in the work. A binary mask matrix M is used to compress parameters. However, the M is fixed before training and is not learnt. In the experiments, M is such a matrix that the first 16/8/4/2 rows are all 1 and the rest 0. In table 9, with M zeros-outing parameters incrementally, the results do not follow the same trendy. \n3)\tThe proposed “Adjoint loss” is similar with the loss of existing knowledge distillation architectures. The author needs to illustrate the difference between them.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "rIorrIDzV7U", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors introduce the concept of \"Adjoined Network\" training, where a CNN network is trained concurrently to a much smaller network that is composed of a subset of the parameters of the larger network. The parameters are shared between the two, and the loss --aside from the standard cross-entropy of the larger network-- also incorporates the KL-divergence between the outputs of the two architectures. This is so the trained small network will be able to simulate the larger one with a fraction of the parameters, which makes for less memory and faster inference at deploy time.\n\nThe paper additionally shows that the concurrent training also acts as a regularizer for the larger network, and empirically demonstrates to be more effective than dropout.\n\nIn general, I like the general idea and how this adjoined training encourages the large network to be more careful at optimizing a specific subset of its weights. The improvement on the accuracy of the large network when trained adjoined is also a nice byproduct.\n\nHowever, I have the following concerns:\n- From a practical standpoint, this approach requires to train the large network alongside the small one. So, at least as described, this approach cannot be used with existing pre-trained networks which is the more typical scenario. It'd be interesting to see if a similar approach could be applied to obtain a small network by fine-tuning a pre-trained architecture.\n- The results are marginally better than teacher-student, but the only architecture size being presented is the one for which the performance is comparable to that of the full network. I believe it's important to understand how the small adjoined network compares to the teacher-student network as a function of changing the desired size.\n\nMinor: it would help the reader to add a paragraph describing intuitively which pieces of the larger CNN are shared by the smaller one.  From what I understand, in each layer, a small subset of the output channels is selected and additionally a random subset of such weights is not used?\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Sq-o8X2LRmJ", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors present a very interesting idea of training a large network and a small network simultaneously with an interesting new loss function. The authors show that this can lead to a much smaller network with good accuracy (compared to the original network); thus this may be a good technique for sparsification.\n\n**Theory**\n\nPlease clarify what induced L2 penalty means. The use of primes to indicate differentiation was ambiguous. Please be explicit about what you are differentiating with respect to in the notation.\n\nCan you motivate why we should focus on the second order term during training?\n\nAlso if at all possible would be good to show the relevant quantities during training to see if the intuition is correct.\n\n**Experimental Evidence**\n\nError bars from multiple runs to get a sense of variation relative to the difference being measured in Table 1 seems critical. \n\nIn addition, showing data from the sparsity literature on what other methods are able to accomplish (particularly weight magnitude pruning) would be very useful. For example, [Blalock and Gonzalez: What is the State of Neural Network Pruning?] seems to have data for ResNet-50, but it may not exactly apply, so the ideal thing would be to provide data by doing the baseline comparison yourself. (That paper also has a checklist of things to be careful of when doing pruning research.)\n\nAlso, a claim in the introduction is that training is faster this way since pruning and finetuning don’t need to be iterated. Would be good to back this up with actual numbers.    \n\n**Writeup**\n\nThe write-up could do with some more attention e.g. use \\citep instead of \\citet in most places. The mask M should be a tensor not a matrix. Also in your context, “speed” could refer to training time or inference latency. Would be nice to use these more canonical terms instead of just speed.\n\n**Future Work**\n\nOne question that occurred to me while reading your paper (particularly the theory section) is what happens if you do the joint training as you describe, but instead of the adjoint loss, you use some sort of regular weight loss and different amounts of regularization penalties for the common weights and the weights unique to the large model. Would that give similar results?\n\nFor practical use of course, ResNet50 may not be the best target for sparsification since EfficientNets and MobileNets are already so much smaller. Please see the [Blalock and Gonzalez] paper (Sec 3.3) for perspective on this. \n\n**Post Rebuttal**\n\nThanks for the response. I still didn't get an answer for why the second order term and not the first order term is of interest in the Taylor expansion. Also the paper does not seem to be updated with the promised changes, particularly error bars. So I am leaving my score as is. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "KckNCqCCjBJ", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a training policy for jointly learning parameters of both supernets (or big teacher models) and subnets (or small student models). The student models have the similar architectures with teacher models, but have different numbers of convolution filters. For jointly training teacher models and student models, an adjoint loss consisted of cross-entropy loss and KL-div is defined. The experiments are conducted on several image benchmarks using ResNet-18 and ResNet-50.\n\nStrengths:\n+: The experimental results show the proposed method can significantly reduce model complexity with slight performance loss. \n+: The proposed method seems simple and easy to implement.  \n\n\nWeaknesses:\n-: Lacking more clarification on significances of this work.\n(1) No state-of-the-art model compression method is compared for verifying the effectiveness of the proposed method. For filter pruning, many recently proposed methods [r1, r2, r3, r4] are presented to compress different sizes of CNN models (e.g., VGG, ResNet18, ResNet34, ResNet50, ResNet101, ResNet152) on various image benchmarks (e.g., CIFAR-10, CIFAR-100 and ImageNet). The corresponding discussion and comparison are missing in this work.\n[r1] Discrimination-aware Channel Pruning for Deep Neural Networks. NIPS, 2018.\n[r2] Gate Decorator: Global Filter Pruning Method for Accelerating Deep Convolutional Neural Networks. NIPS, 2019.\n[r3] Channel Pruning via Automatic Structure Search. IJCAI, 2020.\n[r4] HRank: Filter Pruning using High-Rank Feature Map. CVPR, 2020.\n\n(2) The idea of training subnets from supernets shares similar philosophy with one-shot NAS. Particularly, one-shot NAS based on knowledge distillation has been studied [r5], which supernets and subnets are jointly trained. Meanwhile, [r5] also shows the searched small student models can achieve comparable or better performance than big teacher models. Therefore, what are advantages of the proposed method over [r5]?\n[r5] Neural Architecture Search by Block-wisely Distilling Architecture Knowledge. CVPR, 2020.\n\n(3) I wonder that why the results (73.41%) of ResNet-50 on ImageNet in this paper is inferior to those of other works (~75.1%). Besides, why the authors did not report the results of ResNet-18 on ImageNet?\n\n(4) For fully verifying the effectiveness of the proposed method, the authors would better report the results using more CNN models, rather than those on various benchmarks. In particular, Imagewoof shares similar philosophy with ImageNet.\n\n-: The writing is too colloquial, and there are too many typos. \n(1) $13.7x$ -> $13.7 \\times$\n(2) kl -> KL\n(3) the the smaller -> the smaller\n(4) dropouts -> Dropout\n\n-: I wonder that the meaning of Table 3. For the common settings, ResNet does not adopt Dropout. Therefore, is it fair or necessary for comparing ResNet with Dropout?\n\n-: How about the transfer (generalization) ability of the proposed adjoined networks?  \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "HBtu9YFg0Qa", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a method for training two networks jointly. This resembles the teacher-student approach where the teacher (large) network is trained first and used as an instructor for training the student network (smaller). The proposed paper implements the training of the two networks in a joint procedure and proposes a new kind of loss function \"adjoint loss\" that consists of two terms: (1) the prediction loss of the teacher network, and (2) the Kullback-Leibler divergence between the predictions of the student and the teacher. Moreover, the weights of the student are a subset of those of the teacher. \n\nThe method is similar to classical knowledge distillation approach (student-teacher paradigm by Hinton et al 2015). The key difference seems to be in weight sharing across the two networks. This idea is partly novel, but should refer to the 2018 paper \"Rocket Launching: A Universal and Efﬁcient Framework for Training Well-Performing Light Net\" by Zhou et al, who also consider weight sharing but use a different (but similar) loss function.\n\nThe results should be clarified. The impression is that the comparison is primarily against the conventional teacher-student setting (Hinton, 2015). This comparison is not relevant, as there are over 4500 citations to the original work. Also in more general terms, the proposed method should better reflect the already lengthy history of knowledge distillation, and position the current work better in the landscape.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors proposed to train a large network and a small network simultaneously with a new loss function. The parameters are shared between the two networks, and the loss also incorporates the KL-divergence between the outputs of the two models. In this way, the authors claim that one can train a small network with similar accuracy to the large one, while using less memory and having faster inference speed.

The reviewers think the papers is at the borderline. It has some interesting results, however, it also has quite a few problems:
1)	The technical novelty of the paper as compared to the teacher-student models is not adequate. 
2)	There are many missing references and baselines, since model compression has been a long-studied problem.
3)	Experiments on more different NN models are preferred in order to verify the generalization ability of the proposed approach
4)	The compatibility with the pretraining framework is not very clear

The authors provided their rebuttals to the review comments. However, according the discussions among the reviewers, their concerns were not fully addressed yet and most of them would like to stand on their original scores. As a result, we do not think the paper should be accepted in its current form.

Decision: Reject