ParaDiS: Parallelly Distributable Slimmable Neural Networks
Authors: Alexey Ozerov
Anne Lambert
Suresh Kirthi Kumaraswamy
Venue: ICLR 2022 Submitted (2022)
Keywords: convolutional neural networks
efficient inference
distributed inference
parallel distribution
slimmable neural networks
flexible neural networks

Abstract: When several limited power devices are available, one of the most efficient ways to make profit of these resources, while reducing the processing latency and communication load, is to run in parallel several neural sub-networks and to fuse the result at the end of processing. However, such a combination of sub-networks must be trained specifically for each particular configuration of devices (characterized by number of devices and their capacities) which may vary over different model deployments and even within the same deployment. In this work we introduce parallelly distributable slimmable (ParaDiS) neural networks that are splittable in parallel among various device configurations without retraining. While inspired by slimmable networks allowing instant adaptation to resources on just one device, ParaDiS networks consist of several multi-device distributable configurations or switches that strongly share the parameters between them. We evaluate ParaDiS framework on MobileNet v1 and ResNet-50 architectures on ImageNet classification task. We show that ParaDiS switches achieve similar or  better accuracy than the individual models, i.e., distributed models of the same structure trained individually. Moreover, we show that, as compared to universally slimmable networks that are not distributable, the accuracy of distributable ParaDiS switches either does not drop at all or drops by a maximum of 1 % only in the worst cases.

Under review as a conference paper at ICLR 2022
PARADIS: PARALLELLY DISTRIBUTABLE SLIMMABLE NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
When several limited power devices are available. one of the most efficient ways to make profit of these resources, while reducing the processing latency and com- munication load, is to run in parallel several neural sub-networks and fuse the result at the end of processing. However, such combination of sub-networks must be trained specifically for each particular configuration devices (charac- terized by number of devices and their capacities) which may vary over different model deployments and even within the same deployment. In this work we intro- duce parallelly distributable slimmable (ParaDiS) neural networks that are split- table in parallel among various device configurations without retraining. While inspired by slimmable networks allowing instant adaptation resources just one device, ParaDiS networks consist of several multi-device distributable config- urations switches that strongly share the parameters between them. We evaluate ParaDiS framework on MobileNet v1 and ResNet--50 architectures on ImageNet classification task and WDSR architecture for image super-resolution task. We show that ParaDiS switches achieve similar or better accuracy than the individual models, i.e., distributed models of the same structure trained individually. More- over, we show that, as compared to universally slimmable networks that are not distributable, the accuracy of distributable ParaDiS switches either does not drop at all or drops by maximum of % only in the worst cases. Finally, once dis- tributed over several devices, ParaDiS outperforms greatly slimmable models.
INTRODUCTION
Neural networks are more and more frequently run on end user devices such as mobile phones or Internet of Things (IoT) devices instead of the cloud. This is due to clear advantages in terms of better processing latency and privacy preservation. However, those devices have often very limited (runtime and memory) resources. While manually designing lightweight networks (Howard et al.)
2017
Zhang et al. 2018) or using model compression schemes (Cheng et al. 2018 (e.g., pruning [eCCun al. 1990) knowledge distillation (Hinton et al., 2015 Xie al. 2020 allow satis- fying particular constraints, these approaches are not well suited when those resources vary across devices and over time. Indeed, particular model is handcrafted compressed for given resource constraints, and once those constraints change, a new model needs to be created. To this end, a new family of approaches that are able to instantly trade off between the accuracy and efficiency was introduced. We call these approaches (or models) flexible (Ozerov & Duong [202]). and they include among others early-exit models (Huang et al. 2018), slimmable neural networks (Yu al.
2019 Yu & Huang, 2019), Once-for-All (OFA) networks (Cai et al. 2019 Sahni et al. 2021). and neural mixture models (Ruiz& Verbeek 2020).
An alternative path to gain in efficiency under constrained resources setting, consists in distributing the neural network inference over several devices (if available) (Teerapittayanon et al., 2017). The simplest way is to distribute a neural network sequentially, e.g., in case of two devices by execut- ing early layers on the first device and the remaining layers on the second one, while transmitting the intermediate features between the two devices over network (see Fig. | (B)). Depending on the transmission network bandwidth, the features might undergo a lossless or a more or less severe lossy compression (Choi et al.) 2018). Neural network (A) (see Fig. [T) might be either distributed as it is or, in case the features are distorted by compression, after a tne-ttunng or retraining (Choi et al. 2018). Though sequential distribution might overcome issues of limited memory and com-

---

Under review as a conference paper at ICLR 2022
puting power, it does not decrease the processing latency and might only increase it because of the additional communication delay. This might be overcome via parallel distribution (Hadidi et al. 2019). illustrated Fig. (C). However, maintain the functionality of the original neural network all convolutional and fully connected layers need to communicate, which requires a con- siderable communication burden and increases processing latency. To overcome this the model may be distributed parallelly without intermediate communication (Bhardwaj et al. 2019 Asif et al,, 2019), as Fig. (D), where the only communication needed is to transmit the data each device and the final result from each device for a fusion. In this work we chose to focus on this parallel distribution without communication and call it simply parallel distribution hereafter, since it leads to much smaller processing latency, as compared to sequential distribution; does not suffer from high communication burden of communicating parallel distribution, and also easy to deploy. The price to be paid for these advantages consists in potentially decreasing performance and need of retraining due to the loss connections between the intermediate layers.
neural network
(8] sequential
(C) parallelwith communication
[D] parallel communication
device 1
network deptr
network distribution
device
deecice
desice?
device
"landscape"
"landsrape"
"Iandscaee"
"landscape"
Figure 1: Different ways of distributing a neural network (A) over two devices: tion (B), parallel distribution with communication (C), and parallel distribution without communi- cation (D). Bold red arrows indicate communication over the transmission network.
sequential distribu-
The question instantly and near-optimally Tb parallelly distributed over several devices, regardless of the mum- ber of devices and their capacities? In this paper introduce new framework called parallelly distributable slimmable neural networks or ParaDiS short that allows such functionality. As its name suggests, ParaDiS is inspired by slimmable neural networks framework (Yu et al.) (019). where the full model variants (or so-called switches) consist of the model sub-networks of different widths, and all variants are trained jointly while sharing the parameters. For example, the switch denoted as 0.25x is sub-network with each layer composed of the first 25 % of the channels of the full network. Similarly, in ParaDiS framework we consider several variants or switches. However, ParaDiS switches include configurations distributable on several devices illustrated on Figure 2 As such, we represent each configuration or switch as a list of widths. For example, (.55.0.5] x denotes a configuration consisting of two parallel networks extracted from the first and the second halves of channels of the full network, respectively. All ParaDiS model switches are trained jointly, while strongly sharing most of their parameters. Once set of available devices is known, a suit- able configuration may be selected and instantly deployed. Moreover, a copy the full model available each device, the configuration may be changed instantly without re-loading new model with a new configuration. For example, if one device has got more computational resources for reason (e.g., some other process has finished) configuration (0.5, 0.25, 0.25) x may be in- stantly changed to [0.5, 0.5] by simply changing the sub- sub-models executed on each device. Finally, similar slimmable framework. ParaDiS framework is applicable for most modern convolutional neural network (CNN) architectures and for different tasks (c.g., classification, detection, identifica- tion, image restoration and super-resolution).
trying answer in this work is: Can have neural network that may
'The term "near-optimally" means here that each distributed configuration makes profit of all available resources and that it performs on par with similar configuration trained specifically for this setup.
2

---

Under review as a conference paper at ICLR 2022
Dog
[1.0]K
[0.5. 0.5|x
[0.5, 0.25, 0.25
[0.25, 0.25, 0.25, 0.25]x
Figure 2:
Illustration of ParaDiS model consisting of switches. While white circles denote the neu- rons, gray circles denote either network inputs (e.g.. image pixels) or its outputs. All the parameters are shared between switches (except for batch normalization statistics) and trained jointly.
has been noticed in Yuu et al. 2019) that slimmable network switches may share all the parameters except the global statistics of batch normalization (BN) blocks. Individual BN statisties adopted instead, which does not lead to any serious drawback. Indeed, individual BN statistics may be computed within simple calibration phase after the model is trained (Yu & Huang 2019). and they represent a very tiny overhead in terms of the total number of parameters. Since ParaDiS an extension of the slimmable framework, statistics. As for training, while it is inspired by the training procedure for universally slimmable (US) networks (Yu & Huang 2019), where all switches are trained jointly, the full model trained from the data and all other switches are distilled via knowledge distillation (KD) (Hinton et al. 2013) from the full model. We introduce two important differences. First, instead of distilling the knowledge from the full model (switch [1.0]x) we distill it from a wider model (e.g., switch [1.2|x) called simply wide model hereafter. Second, we change the KD procedure by distilling the feature activation maps before the final fully connected layer in addition to the output predicted by the wide model. We have shown that these new ingredients are important for training ParaDiS.
adopt in this work the same strategy of individual BN
We investigate the proposed ParaDiS framework on ImageNet classification task using MobileNet vl (Howard et al.) 2017) and ResNet-50 (He et al.) (016)) as underlying architectures and on image super-resolution task using WDSR architecture Yu et al. 2020a). We compare ParaDiS models with the corresponding parallelly distributed configurations trained individually and with the US models Yu & Huang 2019) that not distributable. First, we find that ParaDDiS model performs good as and in many cases better than the individually trained distributed configurations. Second, we observe that distributable ParaDiS switches perform almost as good as non-distributable US model switches of the same overall complexity. Third, we show that, once distributed overs several devices, ParaDiS outperforms greatly the US models. Finally, conduct an exhaustive ablation study to show the importance of knowledge distillation, using wide model and distilling activations.
RELATED WORK
Compressed and flexible models. Model compression schemes (Cheng al., 2018 (e.g., prun- ing (LeCun et al. 1990) or knowledge distillation (Hinton et al.) (015)) allow satisfying particular memory and processing constraints, though they do only allow producing fixed model for every particular setting. To go further, new family of so-called flexible approaches allowing to instantly trade off between the accuracy and efficiency was introduced. These approaches include among oth- early-exit models (Huang al. 2018), slimmable neural networks Yuu et al. 20019] Yu & Huang. 2019, Once-for-All (OFA) networks (Cai et al. 2019: Sahni et al. 2021), and neural mixture mod- els (Ruiz& & Verbeek 2020). Moreover, flexible models like OFA networks have been shown to be useful for efficient neural architecture search (Yu et al., (2205)) and for developing dynamic infer- approaches (Li et al. 2021). However, unlike our approach, all these approaches are within one device non-distributed settings.
Sequentially distributed models. In Neurosurgeon (Kang et al. 2017). the authors propose a dynamic partitioning scheme of an existing deep neural network between client (end-user device,
3

---

Under review as a conference paper at ICLR 2022
for example) and a distant server (edge or cloud) (see also Fig. Fig.[(B) (B)). The neural network can only be split between layers to form a head and a tail and the optimal partition is obtained by estimating the execution time of each part on the different devices. The main drawback of this approach is the added communication time occurring between the different partitions. To this end other optimization strategies have been developed and intermediate feature compression schemes have been proposed (Matsubara et al.) 2021). However, as we have discussed above, distributing a model sequentially does not allow making use of plurality of devices to reduce processing latency.
Parallelly distributed models. In contrast to sequential distribution, distributing model parallelly with no communication (see Fig. (D)) allows reducing processing latency without increasing com- munication burden at the same time. Several recently proposed approaches (Bhardwaj et al. 2019 Asif et al. 2019) suggest creating such efficient parallel models via knowledge transfer from a big- ger single pre-traines model. [Wang al. (2020) revisit model ensembles that are parallellizable as well, and show that they are almost as efficient as individual bon-distributable models of the same complexity. However, in contrast to proposal, none of those models are sharing parameters.
3 PARADIS NEURAL NETWORKS
3.1 SWITCHABLE BATCH NORMALIZATION STATISTICS
Batch normalization (BN) loffe& & Szegedy 2015) is an essential block present nowadays in most popular deep CNNs. BN is very simple and consists in normalizing CNN features in each layer and channel with mean and variance of the current mini-batch. While during training the statistics of the current mini-batch are considered, during testing these statistics are replaced by global statistics computed as moving averages over the whole or part of the training set. Because of this inconsis- tency between training and testing, the BN parameters cannot be simply shared in slimmable neural networks YYu et al.) 2019; Yu & Huang 2019). As such, it was first proposed in (Yu et al., 2019) to consider so-called switchable BN block having separate set of parameters for every switch. was then shown that it is sufficient to consider only switchable BN statistics, while the remaining parameters of the BN block might be shared between switches. Finally, it was proposed (Yu & Huang, 2019) to compute switchable BN statistics in a so-called calibration phase after training. For each switch the calibration consists simply in a forward pass over the corresponding network over subset of the training set, so to compute global BN statistics. Since the proposed ParaDiS framework is based on slimmable networks, we here adopt switchable BN statistics together with the calibration procedure.
3.2 TRAINING PARADIS NETWORKS
The switches of ParaDiS network are trained jointly using training procedure inspired by US networks training YYu& & Huang 2019), though with several modifications we introduced that we have shown to be important for achieving good performance. Recall that US networks training (Yu & Huang 2019) consists in updating all switches jointly (though sampling a limited number of switches per epoch), while only the full 1.0x model is trained from data and all other switches trained from the output of the full model via a knowledge distillation (KD) (Hinton et al.) 2015). This procedure is referred
as implace knowledge distillation (IPKD) since all the switches strongly sharing their parameters. More specifically, assuming here without loss of generality an image classification task, the full 1.0x model in US training (Yu & Huang 2019) is updated based on the true labels using the conventional cross- cross-entropy (CE) loss
1 E Ye log(v() loss Lor v) = T
(1)
where C' the number of classes, and y and y/ are C-dimensional hot vectors of true and predicted labels, respectively. All other switches are trained by optimizing the following KD loss 2
loss LCr(D.v'),
2 As compared to Eg. 2 more sophisticated KD-losses are often considered Hinton et al. 2015 Ozeroy & Duong 202T), including those that take into account the true labels y as well. However, keep here this simple formulation since it was used in US training YYu & Huang 2019.
4

---

Under review as a conference paper at ICLR 2022
where and y are hot vectors of labels predicted by the current switch and by the full model, respectively. We adopt this approach to train the ParaDiS networks, while improving it with two novel add-ons (that are important for training) explained just below and depicted on Fig.
Wide model (1.2jx
True labels
y
"landscape"
y
layer
Softmai
Ø
loss = CCE(y'
model (1.0)x:
IPKD
FC layer
Softmax
loss (C
IPKD-A
All remainingswitches
3/
x
FC
Softmax
loss Lox (Ú, a')+ + 3 MLmss(ā
x
à
FC
Figure 3: Visualization of the ParaDiS network training scheme, also described by Alg.
Knowledge distillation from wider model. Within early experiments training ParaDiS MobileNet v1 network with 4 switches (as shown on Fig. [ on ImageNet dataset we tried apply- ing directly the US training framework, but this did not lead to a satisfactory result. Indeed, the full [1.0] x model was not training well enough under-performing by about 1 % the individual Mo- bileNet v1. Since all the other switches were distilled from the full model, their performances were degraded by at least the same amount, as compared to the expected ones. We believed that this is because all the switches span the full 1.0x network width, while sharing their parameters, and thus the full [1.0] x model is not "free enough" to train well. In fact, this is not the case for slimmable (Yu et al., 2019) or US (Yu & Huang 2019) networks, where all other switches have smaller widths compared to the full model. This gave the idea perform IPKD from wider (c.g., [1.2] x) network, which simply called hereafter wide network. Indeed, the wide network is always wider than all other ParaDiS switches, and thus it is not constrained and train better. Note also that the wide network is only needed for training, as such its additional parameters (as compared to the full network) may be simply discarded after training. Moreover, another alternative, any other big network (completely independent from the ParaDiS network) may be also used as wide network.
Distilling feature activation maps. Another issue we have noticed is that the training of ParaDiS network distributed switches is not always very stable. Indeed, since the fusion occurs only after the last layer (see, e.g., Fig. 3 bottom) training of one sub-network depends on the outputs of all other sub-networks, and thus very "independent". To make this IPKD training more "sub-model dependent", we modify the IPKD to distill as well the full [1.0]x model feature activation maps just before the final fully connected (FC) layer to all other sub-models. More precisely, we propose to modify the KD-loss 2 as follows:
loss = Lce(..)) + BCusE(a, C N lla - a'll3 112
(3)
5

---

Under review as a conference paper at ICLR 2022
where 3 is constant parameter, a' is an N-dimensional vector containing vectorization of the fea- ture activation maps before the final FC layer. and is the vector including the respective activation maps of the corresponding switch (sce Fig. 3). We believe that this should stabilize training since when |3 grows training of each sub-network becomes more and more independent of the outputs of other sub-networks within switch. Such kind of regularisation of KD-loss has been already sidered in (Bhardwaj et al. (2019) (see Eq. (6) in the paper), but not in the case of inplace knowledge distillation as we consider here. We refer this proposed activation distillation IPKD-A.
Note also that in contrast to US training Yu & Huang 2019), where random selection of switches updated at each iteration, we update all the switches at each iteration. The resulting procedure is schematized on Figure 3 and described in detail by Algorithm
Algorithm 1: Training ParaDiS network M (see also Fig.
Require: Define list of switches, e.g., as on Fig. 2
[[1.0] x, [0.5, (.5,0.55.,,... ..... [0.25, 0.25, 0.25, 0.25) x]
Require: Define wide model, e.g.. as [1.2]= x switch.
Initialize training settings of the main shared network M. for t .....,TTtters.. do
Get next mini-batch of data æ and label y. Clear gradients, optimizer zerograd().
Execute wide model, y' = (x. wide model).
Compute loss, loss Lce . y) (Eq. Accumulate gradients, loss. booksabcck
Stop gradients of y' as label, y' y .detach().
Execute full model, (g. a²) M (x. [1.0]x), where a' is the activation maps vector.
Compute loss, loss CcE (ū. v') (Eq. 2
10
Accumulate gradients, loss. backkardkw)))
11
12
Stop gradients of a' as activation, a' = .ettachh) for switch in list of switches do switch /= [1.0]x x then
13
14
Execute current switch, (0, = Mx, switch), where à is the activation maps vector..
Compute loss, loss LCE v/) + BLMSE((, (Eq. 33)
17
Accumulate gradients, Joss.backwarddyy
Update weights, optimizer.step()
18
4 EXPERIMENTS
We evaluate our approach on the ImageNet classification dataset (Deng et al. 2009) with 1000 classes. We conducted experiments on one wight- model (MobileNet v1) and one large model (ResNet-50). For both models 4 different switches are trained at each epoch: [4 0.25) x [ 00.50.55 x, [1.0] x and [1.2] x. We compare our performance with US model and individually trained models (4 x 0,25]x, [0.5, 0.5]x, [0.5, 0.25, 0.25]x x and [1.0]x). Note that for fair com- parison the reference US model, similarly to our ParaDiS model, was trained while distilling from a wide [1.2]x model instead of the full [1.0] model as in the original work YYu & Huang 2019). The corresponding approach is then referred to as US Wide (some results on the effect of wide distillation for US and slimmable Nets are in Appendix BB.
We evaluate the accuracy (1 - top-1 error) on the center 224 x 224 crop of images in the validation set. For all models, we show the complexity in terms of Millions of Multiply-Adds (MFLOPs). For readability the results of wide [1.2]x x model as well as of small switches for the reference models are not shown on the plots, however detailed results can be found in the corresponding appendices.
Since switch of a ParaDiS network does not necessary span the range of the full model (c.g., might be [0.25, 0.25)), vector à in 3 may he shorter than this case we assume that à simply padded by zeros to have the same length.
[4 x 0.25] x shorter notation for [0.25, 0.25, 0.25, 0.25)

---

Under review as a conference paper at ICLR 2022
However, this representation in terms of total MFLOPs does not represent any gain of ParaDiS over US and Slimmable models in terms of latency. Also, it is not trivial since such gain depends on the available devices and their capacities. To represent this gain, we add additional curves to the main graphs as follows. We assume a situation when all the devices have equal capacities, and we have any number of them. In that case the US switch 0.5x and the ParaDiS switch [0.5, 0.5) x will run with the same latency (neglecting the communication latency that is small) and the same number of MFLOPs per device, while [0.5, 0.5) x has superior accuracy. In summary, we add the corresponding graphs below a curve tagged as "distributed" that shows the accuracies of switches [0.25, 0.25, 0.25, 0.25] x. [0.5, 0.5] [1.0]x as a function of MFLOPs for one sub-model (and not sum of MFLOPs for all sub-models) This curve is representative of a deployment with several equal capacities devices and shows the real advantage of ParaDiS over US and Slimmable models.
In order to diversify the tasks and architectures, we also investigated ParaDiS for image super- resolution using WDSR model (Yu et al., 2020a). Detailed results can be found Appendix [ they are inline with those for ImageNet classification.
4.1 MOBILENET VI
We use the default training and testing settings (Howard et al. 2017) for MobileNet v1 with the same adjustments as proposed in US-Nets paper (Yu Huang 2019): 250 epochs, stochastic gradient descent optimizer (instead of RMSProp), linearly decaying learning rate from 0.5 to and batch size 1024 images.
The results are shown on Figure 4 (see Appendix C for details). First, may that ParaDiS performs on par with or in some cases (4400.25) x and (0.5, 0.5] better than the individual models. This is inline with similar findings for slimmable and US models (Yu et al., 2019 Yu & Huang 2019), and this is mostly thanks to implicit KD (parameters sharing) and explicit KD. Second, yet more striking the fact that the distributable ParaDiS switches perform as good as non-distributable US switches of the same overall complexity. Finally, while [0.5, 0.25, 0.25] switch has not been explicitly trained all, it performs well. This is possibly because spans the full width, as other switches, and all its sub-models were trained within other switches. Thanks to the "distributed" curve we see also great improvement of distributed ParaDiS over US model.
4.2 RESNET-50
For ResNet-50, we use the same parameters as slimmable networks (Yu et al. 2019): 100 epochs with a batch size of 256 images, learning rate of 0.1 divided by 10 at 30, 60 and 90 epochs. We use stochastic gradient descent (SGD) as optimizer, Nesterov momentum with a momentum weight of 0.9 without dampening, and weight decay of lc-4 for all training settings.
The results are shown on Figure (see Appendix C for details), where we also compare with slimmable and slimmable wide, since they perform better than US wide for ResNet-50. Almost the same conclusions as for MobileNet v1 may be drawn with the following differences. First, the improvement of ParaDiS over individual models is yet more significant. Second, the performance of ParaDiS [4 x 0.25] x switch drops little as compare to the similar US switch.
5 ABLATION STUDY
In this section we show the impact of different training settings such as Inplace Knowledge Distilla- tion (IPKD), the use of wide network for the IPKD and the importance of distilling the activation. We use MobileNet v1 and Resnet-50 trained on ImageNet to measure the contribution of these dif- ferent parameters (additional plots may be found in Appendix E)
First, results in Table D indicaae the importance of IPKD and of the wide model for ParaDiS training. At the same time it shows that distilling activations (either with 3 0.1 or 3 1) does not help for training ParaDiS MobileNet vl. However, results in 2 the importance of distilling activations for training ParaDiS ResNet-50.
7

---

Under review as a conference paper at ICLR 2022
6:251x
10.5:0.25.0.255 0...5
10.5:0.918
74
25:02550055
72
25:00
Accuracy (%)
68
66
US Wide
62
ParaDiS Wide IPKD ParaDiS Wide IPKD (distributed) Single Configuration
60
100
200
300
400
500
600
Millions of Multiply-Ades (MFLOPs)
Figure 4: Complexity in MFLOPs accuracy of MobileNet v1 trained Imagenet for differ- ent switches (US-Nets and ParaDiS). The vertical dotted lines indicate the complexity of specific switches. The diamond points individual models trained for particular switch. The square blue points are switches explicitly trained, while the empty blue circle is never trained, only tested.
78
0.25%
5.5%
20..25500222 25
25:0.25xx
00.59.
[1.01%.
25:0
76
(0.25,0 25:0
Accuracy (%)
74
72
Slimmable
Slimmable Wide IPKD
70
US Wide IPKD
ParaDiS Wide IPKD-A
ParaDiS Wide IPKD-A (distributed)
Single Configuration
68
500
1000
1500 2000 2500 3000 3500 4000 4500
Millions of Multiply-Adds (MFLOPs)
Figure 5: Complexity in MFLOPs vs accuracy of ResNet-50 trained on Imagenet for different switches (US-Nets, Slimmable and ParaDiS). The vertical dotted lines indicate the complexity of specific switches. The diamond points are individual models trained for particular switch. The square blue points are switches explicitly trained, while the empty blue circle never trained, only tested.
8

---

Under review as a conference paper at ICLR 2022
[1.0|x
0.5) x
0.25, 0.255 |5 x
0.25 x
ParaDiS Wide IPKD-A 3=1
72.64%
71.77%
68.23%
66.32%
63.66%
ParaDiS Wide IPKD-A 3=0.1
73.04%
71.99%
68.59%
66.36%
64.18%
ParaDiS Wide IPKD
73.18%
72.13%
69.10%
66.49%
64.43%
ParaDiS IPKD
N/A
71.52%
68.21 %
66. 10%
63.51%
ParaDiS
N/A
70.83%
67.67%
64.41%
63.28%
Table I: Impact of different hyperparameters the training of ParaDiS (MobileNet ImageNet).
1.2 x
1.0 x
0.5; x
0.5; 0.25; 0.25
[4 0.25 x
ParaDiS Wide IPKD-A 3 =
76.48%
76.25%
74.56%
72.87%
71.17%
ParaDiS Wide IPKD
76.48%
76.06%
74.38%
73.02%
71.34%
Table 2: Impact of the distillation of the activation when training Resnet-50 for ImageNet.
6 SCALABILITY ISSUES
Within the proposed ParaDiS networks there is potentially many more possible switches than in slimmable or US nets. Indeed, each ParaDiS switch sub-network might be of any width. As such, seems quite important be able scale the approach to more switches than the used in our experiments.
We have shown our experiments that some switches might perform satisfactory though they were never explicitly trained, e.g., switch [0.5, 0.25, 0.25) x (see Figs. [nd possibly because it spans the full width, as other switches, and all its sub-models were trained within other switches. Based on this observation,
conjecture that if we have smaller sub-models, e.g., by considering wider full model, we may have more non-tramn non-trained satisfactory switches "for free". For example, we split the full model in 8 sub-models and train [1.2] x, [1.0] x, [0.5, 0.5) x, [4 x 0.25] x and [8 x 0.125 x, we would have other switches coming for free. We show experimentally in Appendix F. that this path to scale up ParaDiS framework is viable.
Alternatively, we have also tried training ParaDiS with switches, where we also vary the width spanned by the switch. The results we obtained (see Appendix F.2 for details) are very unsatisfac- tory: the accuracy of each switch is at least 4 % below the accuracy of the corresponding US net switch. What remains unclear and should be subject of a further study is: Is this performance drop because all these switches cannot ex-exist within the shared ParaDiS parameters setting or because we have not found yet how to train them correctly?
CONCLUSION
In this work we have introduced parallelly distributable slimmable (ParaDiS) neural networks. These networks are splittable in parallel between various device configurations without retraining. While so-called flexible networks allowing instant adjusting to a given complexity have been already widely studied, the kind of "feexiblity-to-diittibuuue property of ParaDiS networks is considered for the first time to our best knowledge. ParaDiS networks are inspired by one-device flexible slimmable networks and consist of several distributable confi coniiuurations or switches that strongly share the pa- rameters between them. We evaluated ParaDiS framework on MobileNet v1 and ResNet-50 archi- tectures on ImageNet classification task and on WDSR architecture for image super-resolution task. First, we have shown that ParaDiS models perform as good as and in many cases better than the individually trained distributed configurations. Second, we have found that distributable ParaDiS switches perform almost good non non-distributable universally slimmable model switches of the same overall complexity. Finally, we have shown that once distributed over several devices, ParaDiS outperforms greatly slimmable models.
very important topic for further study would be scaling the ParaDiS networks to more configura- siits/s than 4. As have demonstrated in Appendix [.I one way to achieve that consists in considering more sub-models at the last level of split with "for frree""non "oon--nnnnnned switches as a result. To yet improve that, might also consider width-overlapping sub-models.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Umar Asif, Jianbin Tang. and Stefan Harrer. Ensemble knowledge distillation for learning improved and efficient networks. arXiv preprint arXiv: wwXiv:/99.000090 2019.
Kartikeya Bhardwaj, Ching- Yi Lin, Anderson Sartor, and Radu Marculescu. Memory-and communication-as aware -ware model compression for distributed deep learning inference on iot. ACM Transactions Embedded Computing Systems (TECS), 18(5s):1-22, 2019.
Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once-for-all: Train one network and specialize it for efficient deployment. In International Conference on Learning Representations, 2019.
Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. Model compression and acceleration for deep neural networks: The principles, progress, and challenges. IEEE Signal Processing Magazine, 35 (1):126-136, 2018.
Jongwon Choi, Hyung Jin Chang, Tobias Fischer, Sangdoo Yun, Kyuewang Lee, Jiyeoup Jeong, Yiannis Demiris, and Jin Young Choi. Context-aware deep feature compression for high-speed visual tracking. In Proceedings of the IEEE Conference Computer Vision and Pattern Recog- nition, pp. 479-488, 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. leee, 2009.
Ramyad Hadidi, Jiashen Cao, Micheal S Ryoo, and Hyesoon Kim. Collaborative execution of deep neural networks internet of things devices. arXiv preprint
2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.0253 2015.
Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 777..044444111 2017.
Gao Huang, Danlu Chen, Tianhong Li, Felix Wu, Laurens van der Maaten, and Kilian Weinberger. Multi-scale dense networks for resource efficient image classification. In International Confer- ence on Learning Representations, 2018
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv: 2015.
Yiping Kang, Johann Hauswald, Cao Gao, Austin Rovinski, Trevor Mudge, Jason Mars, and Lingjia Tang. Neurosurgeon: Collaborative intelligence between the cloud and mobile edge. volume 45, pp. 615-629, ACM New York, NY, USA, 2017.
Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. In Advances in neural information processing systems, pp. 598-605, 1990.
Changlin Li, Guangrun Wang. Bing Wang. Xiaodan Liang, Zhihui Li, and Xiaojun Chang. Dynamic slimmable network. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8607-8617, 2021.
Yoshitomo Matsubara, Marco Levorato, and Francesco Restuccia. Split computing and early exiting for deep learning applications: Survey and research challenges. arXiv preprint arXiv:2 103.20000000 103.04505, 2021.
Alexey Ozerov and Ngoc Duong. Inplace knowledge distillation with teacher assistant for improved training of flexible deep neural networks. In 29th European Signal Processing Conference, EU- SIPCO 2021, 2021.
10

---

Under review as a conference paper at ICLR 2022
Adrià Ruiz and Jakob Verbeek. Distilled hierarchical neural ensembles with adaptive inference cost. arXiv preprint arXiv:2003.0147. 2020.
Manas Sahni, Shreya Varshini, Alind Khare, and Alexey Tumanov. Compofa: Compound once-for- all networks for faster multi-platform deployment. arXiv preprint arXiv:2104. 2242, 2021.
Tim Salimans and Durk Kingma. Weight normalization: simple reparameterization accelerate training of deep neural networks. Advances in neural information processing systems, 29:901- 909, 2016.
Surat Teerapittayanon, Bradley McDanel, and Hsiang-Tsung Kung. Distributed deep neural net- works over the cloud, the edge and end devices. In 2017 IEEE 37th International Conference Distributed Computing Systems (ICDCS), pp. 228-339. IEEE, 2017.
Radu Timofte, Eirikur Agustsson, Luc Van Gool, Mina-Hsuan Yang, and Lei Zhang. Ntire 2017 challenge on single image super-resolution: Methods and results. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops (CVPRW), pp. 114-125, 2017.
Xiaofang Wang, Dan Kondratyuk, Eric Christiansen, Kris M Kitani, Yair Movshovitz-Attias, and Flad Eban. On the surprising efficiency of committee-based models. arXiv preprint
arXiv:2012.01988, 2020.
Qizhe Xie, Minh- Thang Luong, Eduard Hovy, and Quoc V Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 10687-10698, 2020.
Jiahui Yu and Thomas Huang. Universally slimmable networks and improved training techniques. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1803-1811, 2019.
Jiahui Yu, Linjie Yang. Ning Xu, Jianchao Yang, and Thomas Huang. Slimmable neural networks.
In 7th International Conference on Learning Representations, ICLR 2019, 2019.
Jiahui Yu, Yuchen Fan, and Thomas Huang. Wide activation for efficient image and video super- resolution. In 30th British Machine Vision Conference, BMVC 2019, 2020a.
Jiahui Yu, Pengchong Jin, Hanxiao Liu, Gabriel Bender, Pieter-Jan Kindermans, Mingxing Tan, Thomas Huang, Xiaodan Song, Ruoming Pang, and Quoc Lc. Bignas: Scaling up neural archi- tecture search with big single-stage models. European Conference on Computer Vision, pp- 702-717. Springer, 2020b.
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6848-6855 2018

---

Under review as a conference paper at ICLR 2022
A
PARADIS AND SLIMMABLE NEURAL NETWORKS REPRESENTATION
Slimmable Neural Nets
1.0x
0.75x
0.25x
ParaDiS Neural Nets
[1.0]x
[0.5, 0.5|x
[0.5, 0.25, 0.25|K
[0.25, 0.25, 0.25, 0.25]x
Figure graphical representation of Slimmable and ParaDiS switches. For each switch, the surface of the blue area is roughly equal to the relative number of parameters, as compared to the full 1.0x model, and the relative number of FLOPs.
Here present graphical representation of Slimmable and ParaDiS switches that allows better understanding of these models. An important observation is that in most cases slimming a CNN from 1.0x (full model) to W x leads roughly W times switch W x (the same is valid for the computational complexity FLOPs). This is because most parameters reside in the weights of convolutional and fully connected layers, and the number of these weights scales with network width W as conost The same holds for ParaDiS sub-models. As such we have found very intuitive to represent slimmable switches as squares and ParaDiS switches as sets of squares, as shown on Figure 6 For each switch within this representation the surface of the blue area roughly equal to the relative number of parameters, model, and the relative number of FLOPs. For example, from this representation we may conclude that ParaDiS switch [4 0.25) x has roughly the same complexity as slimmable US switch 0.5x since their surfaces are equal. This is confirmed by the fact that, as may see from Tables and 5 below, the US switch that is closest complexity to ParaDiS switch (4 0.25) is of width 0.525 0.5. Finally, Yu et al. (2020b) were also using such representation with "squares" for slimmable networks (see Fig. in (Yu et al. 2020b)).
parameters in switch 1.0x than in
compared to the full 1.0x
B WIDE VS. NON-WIDE FOR US AND SLIMMABLE NETS
Table regroups the results of experiments using US slimmable networks for MobileNet vl or Resnet-50. Interestingly, it is MobileNetv I that benefits the most of the wide network, the results for slimmable Resnet-50 show a slight loss of performance for the biggest switches and small gain for the lowest switches.
C DETAILED RESULTS FOR IMAGENET CLASSIFICATION
Tables and 5 summarize the detailed results corresponding to those plotted on Figures and 5 respectively. We may in Table 5 that for most of ParaDiS switches do not indicate results of slimmable networks. This is simply because the complexities of slimmable switches do not correspond exactly to those of ParaDiS switches. As for results of US nets both Tables 4 and
12

---

Under review as a conference paper at ICLR 2022
(1.2x)
(1.0x)
(0.75x)
(0.5x)
(0.25x)
US Wide IPKD
72.98%
72.42%
69.34%
64.16%
55.02%
MobileNet
US IPKD
N/A
71.80%
69.50%
64.20%
55.70%
US Wide IPKD
75.85%
75.63%
74.34%
71.80%
66.41%
US IPKD
N/A
75.77%
74.52%
72.22%
67.18%
ResNet-50
Slimmable Wide IPKD
75.89%
75.68%
74.91%
72.79%
66.37%
Slimmable
N/A
76.00%
74.90%
72.10%
65.00%
Table 3: Impact of using wider configuration the training of US Nets (MobileNet v1 Ima- geNet) and US/slimmable Nets (ResNet-50- + ImageNet).
for each ParaDiS switch we indicate the accuracy of the switch having the closest complexity to the corresponding ParaDiS switch.
ParaDiS switch
1.2 x
[100
0.55 x
0.55; 0.25; 0.25|
[4 X 0.25 x
(corresponding US switch)
(1.2x)
(1.0x)
(0.725x)
(0.625x)
(0.525x)
ParaDiS Wide IPKD
73.18%
72.13%
69.10%
66.49%
64.43%
US Wide IPKD
72.98%
72.42%
68.97%
67.34%
64.85%
Individual Models
N/A
73.13%
68.13%
67.00%
61.83%
Table 4: Accuracy comparison for different switches of MobileNet vl. This table summarizes more As for US results, the accuracy of the switch having the
in details the results plotted on Figure 4 closest complexity to the corresponding ParaDiS switch is given every time.
ParaDiS switch
1.2|x
1.0
0..5
0.25; 0.25| 4 0.25|
(corresponding US switch)
(1.2x)
(1.0x)
(0.7x)
(0.625x)
(0.525x)
ParaDiS Wide IPKD-A,
76.48%
76.25%
74.56%
72.87%
71.17%
US Wide IPKD
75.85%
75.63%
74.16%
73.31% %
72.07%
Slimmable Wide IPKD Slimmable Individual Models
75.89%
75.68%
N/A
N/A
N/A
N/A
76.00%
N/A
N/A
N/A
N/A 76.32%
72.59%
72.34%
68.97%
Table 5: Accuracy comparison for different switches of ResNet-50. This table summarizes more in details the results plotted on Figure As for US results, the accuracy of the switch having the closest complexity the corresponding ParaDiS switch is given every time.
D DETAILED RESULTS FOR IMAGE SUPER-RESOLUTION
In order to investigate ParaDiS different task and with yet different architecture, study it here for image super-resolution task using WDSR architecture Yu et al. 2020a). Inspired by Yu & Huang 2019), experiment with DIV2K dataset (Timofte al.) 2017) which contains 800 training and 100 validation 2 images, on the task of bicubic x2 image super-resolution. WDSR network has batch normalization layer, instead weight normalization (Salimans & Kingma 2016) is used. We use the latest implementation of the WDSR network released in April 2020 and adapt the model and training to support Slimmable, Universally Slimmable (US) and ParaDiS configura- tions. Note that, to our best knowledge, there is no publicly available implementation of WDSR US-Nets as reported in Yuu Huang 2019). We use the same maximum width (64 channels), number of residual blocks (8) and width multiplier (4) as reported by Yu & Huang (2019). The default hyper- paramm-rrraeters (number of training epochs, learning rate, weight decay ...) are used for all experiments.
We report results in Figure and show that ParaDiS exhibits a performance on par with US-Nets and slightly below individually trained models. When distributing the ParaDiS network several devices (orange dotted curve Figure 7). the benefits of the parallclization allows our model to outperform individual models running single device. The results consistent with reported
13

---

Under review as a conference paper at ICLR 2022
34.8
34.7
34.6
34.5
US
PSNR
Slimmable
ParaDiS
ParaDiS (distributed)
34.4
0255
Single Configuration
21:25:0.25:0255::0.
34.3
[0.5,0.5]x
34.2
[0.25]x
[0.625]x
10.75]x
(0.87514 x
[1.0|x
34.1
1000
2000
3000
4000
5000
Millions of Multiply-Adds (MFLOPs)
Figure 7: Complexity in MFLOPs PSNR of different WDSR networks (US-Nets, Slimmable and ParaDiS) trained DIV2K on the task of bicubic x2 image super-resolution. The vertical dotted lines indicate the complexity of specific switches. The diamond points are individual models trained for particular switch.
results for individual models Yu al. Huang 2019). However we were unable to reproduce the results reported for US-Nets when inplace distillation is used during training. All models showed degradation of performance when using knowledge distillation.
2020a) and for US-Nets without inplace distillation Yuu &
E ABLATION STUDY
In this section present the results of ablation study for MobileNet v1 and Resnet-50 trained on ImageNet.
E.1 WIDE VS. NON-WIDE AND IPKD VS. NO KD
First we study the impact of IPKD and distilling from wide network. Figure & clearly shows the benefit IPKD from a wide model by comparing the ParaDiS models trained without Knowledge Distillation, trained with IPKD from model [1.0] x and trained with IPKI) from wide model [1. [1.2] x. interesting to note positive impact of using Knowledge Distillation on untrained switches. In our case we see big improvement in performance, thanks to IPKD, for the [0.5, 0.25, 0.25] x switch that never explicitly trained.
E.2 DISTILLATION OF THE ACTIVATIONS
We present in this section the results of experiments using MobileNet v1 and Resnet-50 with and without distilling feature activation maps from the [1.0] x model.
From Figure D one may notice that the distillation of feature activation maps does not help training different switches of ParaDiS MobileNet v1.
14

---

Under review as a conference paper at ICLR 2022
74
15:0.251 2.25..5
10.5 255
10.5:0.5.18
11.011
72
70
Accuracy (%)
68
66
64
US Wide IPKD
ParaDiS Wide IPKD
ParaDiS IPKD
ParaDiS No KD
62
200
300
400
500
600
Millions of Multiply-Addi (MFLOPs)
Figure 8: Impact of of IPKD and distilling from wide network for MobileNet vl. These results correspond to 3 rows from Table [
74
5
25.5
10.55
72
5
70
Accuracy (%)
68
66
US Wide
64
ParaDiS Wide IPKD
ParaDiS Wide IPKD-A 8=
ParaDiS Wide IPKD-A B=1
62
200
300
400
500
600
Millions of Multiply-Adds (MFLOPs)
Figure 9: Impact of distilling the activation for MobileNet vl. These results correspond to rows
from Table
15

---

Under review as a conference paper at ICLR 2022
From Figure 10
one may that the benefit of distillation of feature activation maps is clearly
visible especially for the largest switches ParaDiS ResNet-50.
78
[0.5:0.5]x
11.0x.
00.25:0.25:0.225 0...0.555255
10.5:0 0.5.0.25:0.25.k.
77
76
Accuracy (%)
75
74
73
72
Slimmable
Slimmable Wide IPKD
US Wide IPKD ParaDis Wide IPKD
71
ParaDis Wide IPKD-A B=1
70
500
1000
1500
2000
2500
3000
3500
4000
4500
Millions of Multiply-Add (MFLOPs)
Figure 10:
Impact of distilling the activation for ResNet-50. These results correspond to those from
Table 2
F SCALING PARADIS MODELS TO MORE SWITCHES
F.1 NON-TRAINED SWITCHES "FOR FREE"
First, we conjecture that if we train for example switches [1.2]x, [1.0]x, (0.5,0.5] [4 0.25]x, and [8 x 0.125] x, we would have other switches coming for free:
0.55, 0,2,,0..525 0.25]
00.5, 0.25, 0.125, 0. 1.1255 x,
[0.5, 0.125, 0.125, 0.125, 0.125]
[0.25, 0.25, 0.25, 0. 0.125, 0.125) x,
0.255, 0.25, 0.125, 0.125, 0.125, 0.125) x.
[0.25, 0.125, 0.15 0.125, 0. 02125,00 0.125, 0.125]
because they span the full width as other trained switches, and all their sub-models were trained within other switches.
Second, we have chosen ResNet-50 to verify this conjecture. Indeed, ImageNet v1 is already too optimized (thus thin) to be split into sub-models of widthes 0. 0.1 The results are shown on Fig- ure II We may see that the performance of model trained on 5 switches drops just slightly (less than % of accuracy) as compared to the one trained on switches. Moreover, the performance of non-trained switches coming "for free" seems to be descent, assuming especially that these switches are distributable over several devices. As such, this way of scaling ParaDiS to more switches seems to be viable.
16

---

Under review as a conference paper at ICLR 2022
78
(1.01 L00%
[8X0 125
12
125
76
10.5
10: [2x0 2:
74
Accuracy (%)
72
70
68
Slimmable
Slimmable Wide IPKD
US Wide IPKD
66
ParaDiS Wide IPKD-A B=1
ParaDiS Wide IPKD
Single Configuration
64
500
1000
1500
2000
2500
3000
3500
4000
4500
Millions of Multiply-Ader (MFLOPs)
Figure 11: Complexity in Millions of FLOPs accuracy of ResNet-50 trained on Imagenet with switches (orange curve): [1.2]x, x. [1.0] x, [0.5, 0.5) x, [4 x 0.25]x, x, and [8 x 0. 0.125]x, x, versus ResNet- 50 trained on switches (blue curve). The vertical dotted lines indicate the complexity of specific switches. The diamond points are individual models trained for a particular switch. The square points are switches explicitly trained, while the empty circles are never trained, only tested.
F.2 VARYING THE WIDTH SPANNED BY THE SWITCHES
This section presents detailed results when scaling ParaDiS more switches. We used MobileNet vl trained on ImageNet for the experiments. 8 different switches are explicitly trained at each epoch (compared to 4 the remainder of the experiments): [1.2] x, [1.0]x, [0.5, 0.5] [0.5] x [4 x 0.25]
[3 x 0.25) x, [2 x 0.25) x and x. Figure 12 shows the difference in performance between and switches.
17

---

Under review as a conference paper at ICLR 2022
75
70
65
Accuracy (%)
60
US Wide IPKD
ParaDiS Wide IPKD
ParaDiS Wide IPKD 8 config
Single Configuration
55
50
19.59222::
10.25.02234
45
10.25:0.255.
(9.25:0
10.5;00
19.999551
10:25
10.5.K
11.91
40
0
100
200
300
400
500
600
Millions of Multiply-Adds (MFLOPs)
Figure 12: Complexity in Millions of FLOPs vs accuracy of MobileNetvl trained on Imagenet with switches. The vertical dotted lines indicate the complexity of specific switches. The diamond points are individual models trained for a particular switch. The square points are switches explicitly trained, while the empty circles are never trained, only tested.
18


Reviews:
[{"review_id": "smlPMQ696xH", "reviewer": "Reviewer_eUCM", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nStrengths:\n\n1. It is a practical and helpful extension to execute the slimmable networks in a parallel way, which has the potential to leverage the computation power of multiple devices.\n\n2. The paper is well written and easy to follow.\n\n3. Comprehensive studies are listed to verify each component in the proposed approach.\n\nWeakness:\n\n1. Methodologically there is no fundamental differences from slimmable networks. The paper proposes 1) distilling from wider networks and 2) distilling feature maps, both of which are not new. \n\n2. There is little details on how to execute the parallel inference across multiple devices, which I think is a major issue of this paper. For instance:\n- Does it mean there are three devices holding each part of the [0.5, 0.25, 0.25] configuration? If so, there is inevitable communication among devices.\n- It would be more interesting to show the strategy to split the model across different devices, depending on their resources (memory or computation). \n\n3. No simulation results on parallel execution of ParaDis. I may still worry about the latency overhead comparing with the inference of a singe device, especially if different devices have different computation power, or there is any communication among devices.\n\n4. It seems the width configurations are not scalable and sensitive to width configurations as discussed in Section 6. From to Figure 1, is it necessary that all neurons in the first width configuration are fully connected (in terms of the performance)? \n\n\nDetailed Comments:\n\n1. It is still not clear to me the following statement: \"Indeed, since the BN statistics are computed via a forward pass during the calibration phase, the statistics of a particular sub-model are independent of the switch it belongs to. In other words, BN statistics of the last 0.25-width sub-model will be the same, whether they are computed within [0.5, 0.25, 0.25]× switch or within [0.25, 0.25, 0.25, 0.25]× switch. To summarize, the calibrated BN statistics of a given sub-model are switch-independent.\"\n\nAs the last 0.25-width receives input from different paths of previous network, what does the word \"independent/same statistics\" mean? \n\n\n2. Figure 4: what about Slimmable and Slimmable Wide IPKD for MobileNet experiments?\n\n3. What is the training cost compared with vanilla training or US training, if all switches are enumerated in each iteration?", "summary_of_the_paper": "The paper proposes ParaDiS, a slimmable network that can be executed across multiple devices, and be transferred across different devices without re-training. To train ParaDis, the authors propose to distill from a wider network, and combine activation knowledge distillation to improve the performance. While the idea of parallel execution is promising, there is little detailed discussion on that point.", "summary_of_the_review": "Marginally below the acceptance threshold.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "rCWTLKHOMTz", "reviewer": "Reviewer_fReP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n1. The experiment section is well written with clear description and thorough studies. The experimental results provide sound evidence to support claims made in the paper. \n\n2. The proposed distillation algorithm is described clearly and straightforward to re-implement. \n\n3. Described the corner cases like batch norm layers and proposed solutions to address them.\n\nWeaknesses\n1.  I hope this application of this work can be beyond the convolutional models in the image classification domain. Sharing all parameters across devices posed a serious constraints to those transformer-like models whose number of model parameters is much larger but not necessarily needs more compute flops per image.\n \n2. The proposed framework required knowledge of all devices in advance before training. And it's not scalable when the list of switches is large. \n\n3. The practice of distilling a bigger teacher to the student model is well established in the literature (like noisy student). It's better to add more related work along this line.\n\n4. Another interesting baseline will be an ensemble of smaller models, each of which will be deployed on a separate device. In this way, the whole ensemble network can still work even after the number of devices in the network changes.\n\n", "summary_of_the_paper": "The paper proposed an algorithm that distill from a larger teacher to a set of parallely distributable student models each of which only compute a fraction of channels in the network. The combined output of the student models can be used to compute the final class prediction.  In this way, student models can be deployed in a set of mobile or edge devices. ", "summary_of_the_review": "A careful study of distillation algorithm for the mobile/edge device deployment. The whole paper is well written with experimental results supporting the main contribution claimed by the authors. However, the significance of this approach is a bit limited due to the scalability issue (only works with convolutional models and  a very small set of switches.)", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "KNGEqtEJamA", "reviewer": "Reviewer_6dzy", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "=== Strengths ===\n\nFigure 1 was very helpful in giving context to different types of distributed inference. Yet, on my first reading it wasn't clear whether the image was split spatially or channel-wise. This was clear though given my previous understanding of slimmable networks.\n\nThe ablation study was useful in confirming the importance of IPKD-A on top of US networks.\n\nThe inclusion of IPKD-A seems like a strong modification from the US networks.\n\nThe paper honestly discussed its limitations and difficulties scaling up to 8 switches.\n\n=== Weaknesses ===\n\nThe major evaluation figures seem to not be highlighting the correct aspects of the method. In my understanding, this method should strive to maintain the accuracy of the single configurations as much as possible, while improving the model latency through parallelization. This is shown in the figures. Then, the comparison to US networks could either show improvements over previous work, or equality with improvements in latency or other aspects. This part seems to be lacking and currently from the evaluation itself it isn't clear why ParaDis would be preferred to US networks. US networks would also have decreased latency from their reduced width.\n\nIdeally, the evaluation should include latency results since this is the biggest advantage of parallelization. This isn't directly captured by the FLOPS.\n\nThe ImageNet results are important, but there should be at least one table confirming that ParaDis works across other architectures. I imagine that the \"information split\" that ParaDis causes could have widely different effects in different networks. Residual branches, bottlenecks, and pointwise convolutions could possibly affect this.\n\nGiven the relatively poor single configuration results on some switches, these data points may want to be run multiple times and listed with error bars. It is hard to believe that cutting the network in half improves the ResNet accuracy up to two points.\n\nI struggle to understand the statement: \"within ParaDis networks, there might be potentially much more possible switches than in slimmable or US Nets.\" In US Networks, there is an ordering on the channels so that lowest index channels are the primary ones. This leads to up to C different configurations, since each switch only adds an extra channel. It seems like this should be true for ParaDis too, where the leftmost channels of the network are considered primary. If the network has 4 channels, then the possible configurations are [1], [.5, .5], [.5, .25, .25], and [.25, .25, .25, .25], and [.25, .25, .5] for example would not be valid. In that case, ParaDis also has at most C switches in general. In reality, just as in the US paper which limits the minimum channels to .25C, there would need to be a minimum sub-model size, leading to fewer than C switches.\n\n\n=== Questions ===\n\nWhy is the [.5, .25, .25] configuration left untrained? I assume this was to demonstrate that it doesn't have to be explicitly trained to perform well, as long as its submodels are trained in other switches.\n\nOther types of networks were mentioned in the related works, e.g. early exit models. How do these other types compare to ParaDis in terms of accuracy at different latencies?\n\nIf ParaDis and US Networks perform similarly for given compute, what situations would favor ParaDis models?\n\nHow are residual branches handled during the splits in ResNet?\n\nWhat are the tradeoffs of updating all the switches vs. a subset of the switches per epoch? Did you choose to update all because in this case there were only 4 (or 3 if one of them wasn't trained)?\n\nFor the negative results for 8 switches, what is the minimum size of the channels in the split? I would be worried about the bottleneck of ResNet being split too much for example. Do these issues go away with a larger scaled width?", "summary_of_the_paper": "The authors propose ParaDis, a parallel and distributed version of universally slimmable networks. They use the fundamentals from this work, plus a few additions, to train models that can be switched between different parallel factors with only minor accuracy degradation. They evaluate using ResNet and MobileNet on ImageNet against models trained solely for a given configuration to show that their technique matches and sometimes exceeds this baseline accuracy. They finish the paper with an ablation study that examines the impact of their additions to the training procedure. ", "summary_of_the_review": "This paper provides an interesting variant on the idea of slimmable networks that allows the technique to be applied to distributed inference. The idea is well-motivated and described in detail, yet the training procedure seems slightly ad-hoc and the evaluation falls short in multiple ways. The paper needs more baseline techniques, more model architectures, and larger parallelization factors for me to fully support acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "HDG_wJHWEr", "reviewer": "Reviewer_T9P1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "* Strength: Similar to slimmable networks, this approach could train all the networks at the same time and retraining is not necessary when deploying.\n* Weakness: This approach fails to address several important problems in distributed embedded computing and the detailed advantages on hardware latency is missing.\n\n* Reasons to accept:\n  * The parallel distribution to speed up model deployment can help reduce latency.\n  * The approach to train the ParaDiS network is computation-efficient and the performance is good.\n\n* Reasons to reject:\n  * Only a few experiments have been performed on classification task, without covering detection, segmentation etc. The ablation study could not show significant advantage of distillation.\n   * Lack of specific information about how much latency is saved via parallel computing.\n\n### Questions\n* From Figure2, the second layer seems to depend on all the neurons of first layer, so how is it distributed in parallel? Is the computation done in a mixed serial-parallel style, or is the computation repeated?\n* Many modern DLA have on-chip memory for storing intermediate results to save DDR bandwidth. Parallel computing across devices will introduce extra copies from and to DDR in this case. What's the cost model to decide when to distribute and when not to ?\n\n\n", "summary_of_the_paper": "This paper proposed to train several parallel sub-networks in the slimmable networks framework, which could run in parallel on different devices subject to runtime hardware configurations to reduce latency.", "summary_of_the_review": "The paper improves upon the slimmable network framework by introducing parallel sub-networks to reduce latency. However, the improvement has not been well empirically evaluated and justified. The main discussed scenario is distributed embedded computing, but important topics in this scenario like mapping between different traits of sub-network and the capabilities of devices, has not been covered. Not to mention runtime behaviors like load balancing and allocation of shared resources like CPU and memory. The paper may need be expanded significantly to include sufficient contents on the topic.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an approach for distilling a larger teacher model into a set of students that can run in parallel at lower cost. The main strengths are that the approach appears conceptually sound and reasonably well executed. The main weaknesses are that the differences relative to previous work is fairly slim, and the experimental results are overly idealized. While the main benefit of the approach is improvement in latency, the experiments evaluate in terms of FLOPS. There was some back and forth between the authors and reviewers about these points. Authors added some additional results and seem to acknowledge the limitations, e.g., saying “Our time and hardware constraints did not allow us to perform experiments in a realistic deployment.” However, for a paper primarily concerned with reducing latency, reviewers were unconvinced that this evaluation was sufficient.
Decision: Reject