SNOW: Subscribing to Knowledge via Channel Pooling for Transfer & Lifelong Learning of Convolutional Neural Networks
Authors: Chungkuk Yoo
Bumsoo Kang
Minsik Cho
Venue:  (2020)
Keywords: channel pooling
efficient training and inferencing
lifelong learning
transfer learning
multi task

Abstract: SNOW is an efficient learning method to improve training/serving throughput as well as accuracy for transfer and lifelong learning of convolutional neural networks based on knowledge subscription. SNOW selects the top-K useful intermediate
feature maps for a target task from a pre-trained and frozen source model through a novel channel pooling scheme, and utilizes them in the task-specific delta model. The source model is responsible for generating a large number of generic feature maps. Meanwhile, the delta model selectively subscribes to those feature maps and fuses them with its local ones to deliver high accuracy for the target task. Since a source model takes part in both training and serving of all target tasks
in an inference-only mode, one source model can serve multiple delta models, enabling significant computation sharing. The sizes of such delta models are fractional of the source model, thus SNOW also provides model-size efficiency.
Our experimental results show that SNOW offers a superior balance between accuracy and training/inference speed for various image classification tasks to the existing transfer and lifelong learning practices.

Published as a conference paper at ICLR 2020
SNOW: SUBSCRIBING TO KNOWLEDGE VIA CHANNEL POOLING FOR TRANSFER & LIFELONG LEARNING OF CONVOLUTIONAL NEURAL NETWORKS
Chungkuk Yoo!
Bumsso Kangl.² Minsik Cho
IBM, Austin TX, USA "KAIST, Daejeon, South Korea ckyoo com com
bumsoo ncc ab. kaist. kr
minsi kasoucchhuus.... bmm com
ABSTRACT
SNOW is an efficient learning method to improve training/serving throughput as well as accuracy for transfer and lifelong learning of convolutional neural networks based on knowledge subscription. SNOW selects the top-K useful intermediate feature maps for target task from a pre-trained and frozen source model through a novel channel pooling scheme, and utilizes them in the task-specific delta model. The source model is responsible for generating large number of generic feature maps. Meanwhile, the delta model selectively subscribes to those feature maps and fuses them with its local ones deliver high accuracy for the target task. Since a source model takes part in both training and serving of all target tasks in inference-only mode, source model can serve multiple delta models, enabling significant computation sharing. The sizes of such delta models are fractional of the source model, thus SNOW also provides model-size efficiency. Our experimental results show that SNOW offers a superior balance between accuracy and training/infereen speed for various image classification tasks to the existing transfer and lifelong learning practices.
INTRODUCTION
Learning new tasks from old tasks over time as natural intelligence does key challenge in artificial intelligence, and transfer and lifelong learning are two popular strategies in this direction. Transfer learning delivers neural network with good predictive power by duplicating and tuning the parameters for a pre-trained source model against a dataset for new task (Dauphin et al. 2012, Donahue et al. 2014). However, transfer learning from a source task to many target tasks incurs overall significant training and space overhead due to multiple large models to customize and store (Mudrakarta et al. 2019). On the other hand, lifelong learning can enable substantial parameter sharing and deliver multiple target tasks with less training time and smaller model size, but may suffer from catastrophic forgetting or lower accuracy (McCloskey & Cohen 1989). Comprehensive efforts to tackle such challenges have been proposed but at significant computational overhead in general (Rusu et al. 2016 Guo et al. 2019 Mudrakarta et al. 2019 Li & Hoiem (0118).
In this work, we propose SNOW for efficient transfer and lifelong learning, which consists of an inference-only source model, multiple task-specific delta models, and channel pooling in-berween. Unlike transfer learning, we let multiple delta models subscribe a source model through a channel pooling layer. SNOW is fundamentally different from the prior arts in transfer and lifelong learning in the following aspects: a) the source model frozen during both training and inference, b) the top-K intermediate feature maps of the source model are learned for each delta model using channel pooling powered by Gaussian reparametrization technique (Kingma & Welling 2014). And, SNOW does require neither persistent training data nor episodic memories (Sprechmann et al.) 2018 Li & Hoiem 2018) to overcome catastrophic forgetting. Table [ compares SNOW with prior arts in transfer/lifllog learning, and Fig. [ visualizes the key structural differences between SNOW and transfer/lifelong learning where the source model for task0 is leveraged to build the target models for t-sk1-3 without persistent datasets follows:

---

Published as a conference paper at ICLR 2020
FO FE FT MP LF
PN SNOW
Overall Accuracy
poor medium good good medium good
good
Catastrophic forgetting
no no
no
no yes
no
no
Training efficiency
good good
poor
poor poor
poor good
Serving efficiency
good good poor
poor good
poor
good
Model-size efficiency
good good
poor
good good poor good
Final oulput only: re-training the final output layer of the duplicated source model Yosinski el al.
2014
b Feature extraction: re-training few output layers of the duplicated source model (Whatmough al. Fine-tuning: re-training entire duplicated source model (Dauphin et al. 2012).
2019
4 Model patch: training only the patch layers inserted into the source model MMudrakarta et al. 2019
Learning w/o forgetting: regularizing hidden activations without persistent datascts (Li & Hoiem 2018
Progressive Net: propagating combined feature maps via lateral connections (Rusu al. 2016).
Table Comparison of various transfer/lifelong schemes and SNOW.
Source model
task0
Source model
taskO
Source model
task0
transfer learning
lifelong learning
SNOW
Source model
task0
Source model
task
taskO
Channel Pooling (top-3)
task1 model
t
task 1
Shared
task1
lask2 model
lask2
model
task2
task2
task3 model
task3
task3
task3
(a) Transfer lcarning: FO, FE, FT
(h) Lifelong lcarning: MP, L.F, PN
(c) SNOW
Figure 1: SNOW vs. transfer/lifelong learning to build task1-3 from task0. The dotted box indicates model which stays frozen (or parameter update), skipping back-propagation during training.
Transfer learning in (a) tunes the parameters in copy of the source model for new task. transfer learning, updating more parameters improves the prediction accuracy but at higher computational overhead (Yosinski et al.) 2014). For example, task1 " be obtained faster (as the source model is frozen), but may show lower accuracy than task1 where the entire model is customized. Therefore, when high accuracy desired, transfer learning can involve significant training and storage overhead as separate models are trained for different target tasks.
Lifelong learning in (b) allows a common structure for all the tasks, increasing the parameter efficiency of the entire system. Although lifelong learning could be a faster methodology than conventional transfer learning for multi task delivery, it may still demand quite some training time the shared part needs to get trained for all the target tasks and/or suffer from catastrophic forgetting (unless expensive and complex persistent dataset or episodic memory is available). In (Rusu et al. 2016), constructive method for lifelong learning is proposed avoid catastrophic forgetting but at substantial computational and memory footprint overhead.
As (c) for SNOW, the source model serves as a static feature supplier (i.e., no back- )ropaapopagttion) to all the target-specific models or delta models (significantly enhancing training and serving speed). Hence, each delta model generates only a handful number of its own local features and obtain all other features for free (greatly improving parameter efficiency). However, instead of supplying all the features to the delta models, SNOW has a channel pooling layer which learns and selects the top-K features for each individual delta model (delivering excellent prediction). For an example of (c), the channel pooling layer independently selects the top-3 intermediate feature maps for t-s Unlike conventional lifelong learning, SNOW does suffer from catastrophic forgetting, as each delta model is independent of each other.
Cost benefits on SNOW architecture are threefolds, First, training time and memory footprint can be significantly reduced since the souree model runs only in inference mode (thus no back lmck-propagation Second, the overhead of channel pooling is negligible because it requires only one parameter for each feature map of the source model. Lastly, delta model needs to be
2

---

Published as a conference paper at ICLR 2020
Souree
mudel
Output layer for source task
layer
layer
layer
Channel pool
Channel pool
Channel pool
inpul
Delta models
layer
layer
layer
0.0.
Output layer for taskl
Outpur Inyer for task2
layer
layer
layer
Downsized layens [M< N) channels]
Figure 2: In SNOW architecture, the source model produces intermediate features and the delta models selectively subscribe them via channel pooling in their training and serving processes. The local features from a delta model and the subscribed features are combined through concatenation.
large enough to produce target-specific features, but can be much smaller than the source model. Fusing those features with the K features from the source model leads to significant parameter count saving compared with existing transfer learning practices.
The rest of the paper is organized as follows. Section 2e details SNOW, and experimental results are in Section 3 Review on prior arts is in Section 4 followed by the conclusion Section
SNOW ARCHITECTURE
In this section, we present SNOW which aims scalable training/serving and high accuracy without catastrophic forgetting, as efficient framework for transfer and lifelong learning. We will describe SNOW from two perspectives: a) how to connect the source model and the delta models efficiently (discussed Section 2.1) and b) how effectively subscribe the intermediate features from the source model from the delta models for higher accuracy (discussed in Section (.2). The above two challenges look orthogonal, yet highly correlated, as over-subseription of features degrades model-size efficiency while under-subscription would harm the model accuracy.
2.1 ARCHITECTURG AND LATERAL CONNECTIONS
SNOW proposes architecture where pre-trained/frozen source model and independently to-be- trained delta models for target tasks are connected via channel pooling which enables uni-directional knowledge flow or knowledge subscription from the source to the delta models as in Fig. 2 The goal of uni-directional knowledge subscription is to capture universal features at one-time computation cost which would be amortized across multiple target tasks, assuming the source model has been trained with a large and comprehensive dataset (as a typical transfer learning scenario). Mcanwhile, target task-specific features computed in the delta models with local feature extractors. In SNOW. therefore, the delta neural network can be at much smaller size than the source model as long as the source model can provide comprehensive set of feature maps. Since these two kinds of features fused and further co-processed in the delta models, we can intuitively see that the delta models are encouraged to learn set of task-specific features complementary the subscribed features. In delta model cannot extract sufficient amount of features from source net, the delta model capacity would need grow in order to generate the target-specine features sufficiently by itself. However, making the delta model too large can degrade parameter efficiency and further result in parameter over-fitting when the target dataset is small, like any other neural network (Heaton 2008).
In detail, the source model produces intermediate features for a given sample during its forwafo path (which is the same as inference). The delta model then is trained to use some of these feature maps as inputs for the next layer. The input for L; layer of the delta model is given the concatenation of the subscribed features from the source model and the output features from L,-1 layer of the delta model. In our experiments (see Section 3 for details), we used down-sized version of the source model for the delta models, but there is no restriction on how to design a delta model because concatenation is more flexible than feature superposition. More specifically, used ResNet50 (He
3

---

Published as a conference paper at ICLR 2020
Input feature map (4 chamels)
i-th forward path
i-th backward path
F+1-th forward path
i+/-th backward path
Gn
6
5
Churnet-
molt.
gradients
Normal discribution 1sparammet
mia
0J1
+0.
NA
0.2
Mig
Mia
+0,9
-0.8:+0.I NAA
0.4
0.5
0.3
0.2
0.4
0.6
0.3
0.5
0.8
0.3
0.0
0.5
0.8
0:3
0.0
AA
II
II
11
Channel pooling weights
0.5
0.5
0.8
0.2
0.4
0.3
0.2
1ig
W9
WW's
Wo
M2
We
W2
My
W1
W2
Wy
curput (3 channels)
Channol pooling
Me co Wa ca
W.
Fo We
G Co Ce-
0.
(top-3 by wa)
(top-3 by w))
The channel C2 decreases a loss so that the gradient of my is positive. It will be sclected again very likely.
The channel Cg increases a loss so that the gradient of my is negative.
The weight wz (which was ignored previously) is now selected.
Optional
BatchNorm and ReE.U
Figure 3: Forward and backward propagation paths for channel pooling.
al. 2016) trained by ImageNet (Deng et al.) 2009) as source model, and used the same ResNet50 architecture as a delta model for a new task after reducing its channel size by a factor of eight. Note that such lateral connection has been proposed in existing works (Feichtenhofer et al. 2018). but the context vastly different in a sense that is mainly to accelerate the inference of a single task during neural network architecture design, while SNOW is for efficient transfer and lifelong learning.
In our training process for SNOW, all parameters in the source model are frozen while the parameters in the delta models are updated from random initialization. Therefore, we do not need to compute back-propage propagation for the source model parameters, greatly improving training memory footprint (i.c., nothing keep for gradient computation) and throughput. And, the delta models have much fewer parameters than the source model. As result, SNOW training offers much shorter training time and demands less computing resources compared with other conventional transfer/lifelong learning techniques, e.g., FT, MP, LF, PN (see Table Furthermore, we also minimize inference time for multiple tasks by sharing the intermediate features of a single source model across all the target tasks.
The serving efficiency of SNOW is maximized when all the target tasks take the same input. By sharing a feature map from a source network across more tasks, one-time feature map generation overhead can be amortized better, increasing the overall computational efficiency of SNOW architec- ture. Such single-input-muute caappuutuutiiouuppppe scenario is highly realistic in practice: for instance, multiple neural networks take in the feed from single camera, microphone, and sensors (Mathur et al. 2017).
2.2 SELECTIVE FEATURE SUBSCRIPTION BY CHANNEL POOLING
The purpose of channel pooling is to find a small subset of intermediate features from a source model in way that the prediction accuracy from the delta model can be maximized. Therefore, each delta model will be accompanied by one channel pooling layer. In order to find the best subset, each channel pooling layer has a learnable scalar each source feature map, and SNOW selects the top-K channels to limit the computation and model-size overhead in stochastic way using normal distribution reparameterization. During inference, SNOW directly feeds the best K feature maps based on the training result without stochastic step.
Fig. 3 shows the forward and backward paths inside a channel pooling layer. At i-th forward path, first compute the following for any channel z: w2 m (0, o²). Then, select the top-3 weights (Wo. Wi, Wg) in white and perform channel-wise multiplication between the selected weights and corresponding channels the same indices so that the results be consumed by the
4

---

Published as a conference paper at ICLR 2020
corresponding delta model. The purpose of stochasticity is to provide every channel with enough chance to be selected, but we keep a small that the channels can be easily separable. For example, even if m2 mg, Wa is ignored at this iteration due to stochasticity.
In the i-th backward path, the gradients for all the mean variables are computed except m2, as We did not participate in the forward execution. Thus, the computed gradients will update (mg, M1, ma) accordingly, and affect the top-K selection in the (ē + 1)-th forward path. Assume that the gradient for mg is -0.2, as the c3 hurts specific target task, then ma becomes 0.0 (now the lowest).
In the subsequent + 1-th forward path, (wo, Wi, W2, wa) are computed in the same way. This time, the difference between my and mg is big enough keep wa from being selected. If 02 is indeed helpful to the target task, then m2 will get bigger, which would in turn help W2 be selected again in the subsequent iterations for further tuning. In this way, while the less relevant feature maps (to target task) are to be ignored with high probability over time due to the decreased channel pooling weights, the critical channels will not only survive the top-K filtering but also be thoroughly tuned to minimize the training loss. Note that we remove the stochastic part, (O0 o²) after training is over so that we directly select top-K trained weights for an inference task.
Determining K (i.e., the number of feature maps to subscribe) has a critical impact on both size and prediction accuracy of the target models. a delta model undersubscribes to the intermediate features, it would not have sufficient information for higher prediction power. If a delta model oversubscribes, it would require more neurons and computational resources to process larger feature maps, not to mention that it could introduce unwanted noises the delta model.
I I conv O(N x K)
Channel-wise attention O(N2)
Channel pooling O(N)
Model-size overhead Computation pattern convolution
matrix mult. ch.-wise scalar mult.
Table 2: Overhead of each channel subscription approach for a feature map of N x h x w size.
There are naive alternative approaches to selectively subscribe to features such as 1 x 1 convolu- tion (Rusu et al. 2016) or channel-wise attention (Woo et al. 2018) as Table 2 but they require more parameters/ comp:rattsooomtttiiooooooooo and/or offer worse accuracy than the proposed channel pooling (see Section for comparison). In detail, when downsizing a feature map of N x h x w size to K x h x w, channel pooling layer needs N parameters with K x h x w multiplications, while 1 X convolution requires N x K parameters with N x K x h x w multiplications. The channel pooling operation may look similar the channel-wise attention (Woo et al.) 2018), but its attention block keeps the channel size the same as the number of given feature maps, while the channel pooling layer prunes irrelevant features out to have smaller delta model. Finally, the size of the channel attention module is ( which is bigger than that of channel pooling O(N). ), and channel-wise attention requires expensive matrix multiplication, but channel pooling only needs channel-wis scalar multiplication.
EXPERIMENTS
We implemented SNOW in Pytorch 1.1 (Paszke et al. 2017). and compared its training and serving performance with prior arts in terms of accuracy and computational overhead in the following setup;
Source model: pre-trained ResNet50 with ImageNet available in t torchvision.
Workload: five classification tasks for Food-10 101 (Bossard et al.; 2014), Describable texture dataset (DTD) CCimpoi et al. 2014), Stanford 40 actions (Yao et al. 1. Car dataset (Krause et al.)
2013) and Caltech-UCSD birds-200-2011 (CUB) (Wah al. 2011 l (see Table for details) without any persistent dataset storage.
Comparison: SNOW and five other transfer/lifelong schemes FO, FE, FT, MP, PN in Table ] (implemented per the corresponding reference in the table note) for training the models. We
Dataset(#class) train/test
Food(101) 75,750/25,250
DTD(47)
Action(40) 4,000/5,532
Car( 196)
CUB(200) 5,994/5,794
3.760/1,888
8.144/8,041
Table 3: Datasets for experiments.
5

---

Published as a conference paper at ICLR 2020
could not compare with LF because LF exposed catastrophic forgetting the tested datasets, preventing fair comparison (see Appendix for more discussion and LF results).
Environment: IBM POWER9 3. 10GHZ (44 cores) machine with Red Hat Enterprise Linux Server 7.6, CUDAIO.1, CUDNN7.5 .5 and 4 Tesla V100 GPUs (16GB each).
3.1 TRAINING PERFORMANCE
In this section, we report the training performance of all the experimented schemes in terms of training throughput, GPU runtime memory footprint, and Top-1 test accuracy. We trained each algorithm on single GPU but with a different batch size in order to fit each algorithm's 's memory footprint into single GPU memory capacity. While we set the initial learning rate as for all other schemes under comparison, from scratch. We decayed the initial learning rate by a factor of 0.1 per every 30 epochs. For the exact batch sizes and initial learning rates, please refer to Figure 4 We used SGD optimizer with momentum of 0.9 and a weight decay factor of 10-4, We set K
applied 1.0 to SNOW, SNOW has channel pooling parts that need training
N/8 for intermediate channels
from ResNet50 and ²² 0.001, except for the Action dataset (Yao et al. 2011) (o² 10- 5). For
each algorithm in Table 1 training is done in a sequential manner for all the datasets to reflect the nature of transfer and lifelong learning.
SNOW shows the best balance between the training throughput and prediction accuracy over all the tested datasets as shown Figure Overall, SNOW-256 delivered a comparable accuracy with up to 2.4x higher throughput than FT or MP and 5.2x more than PN. This implies that SNOW enables transfer or lifelong learning be completed significantly faster under the same computing resources, yet preserving its quality.
Food (101 food classes)
DTD (47 texture classes)
Action (40 classes)
90
SNOW
Top1 test accuracy (%)
Topl test accuracy (%)
SNOW
MP
80
FE
Top1 test accuracy
80
SNOW
80
FE
PN
MP
FO
70
70
PN
70
FT
FO
FO
Memory size
FT
60
16GB
4GB
50
50
50
250
500
750 1000
1250
250
500
750
1000
1250
250
500
750
1000
1250
Throughput (images/sec)
Throughput (images/sec)
Throughput (images/sec)
(196 classes)
CUB (200 classes)
Topl validation accuracy (Car)
90
FT
90
SNOW
Top1 test accurncy (%)
Top test accuracy (%)
80
80
PN
80
SNOW
Top1 accuracy
MP
FE
MP
70
FO
FE
70
70
FE
MP
60
SNOW
SNOW batch
60
128
256
50
FO
50
40
750
1000 1250
250
500
750
1000
1250
180
Throughput (images/sec)
Throughput (images/sec)
Epoch
parameparameters
FF
SNOW-128
SNOW-256
batch sizze
256
256
128
96
128
256
initial learning rate
0.1
0.1
0.05
0.0375
0.0375
0.5
1.0
Figure 4: Training performance comparison on five different datasets on a single GPU.

---

Published as a conference paper at ICLR 2020
The reason why FT and MP show poor throughputs compared with other light- weight approaches because nearly full back-propagation is required for both cases. Especially, MP is slower than FT (despite that majority of parameters are frozen) because it has more layers deal with yet back-propagation needs to reach out the very first patched layer in the model (thus, still need to compute the gradients for the intermediate features along the way). Meanwhile, SNOW performs only for the delta models which are substantially smaller than the source model, resulting in significant reduction in training time and computing overhead, e.g., memory footprint for a given batch size. (note that the circle size in Figure represents the memory footprint size). Although PN does not require back- proppaaatnn on the source model either, its large target model size and heavy lateral connections make it the slowest scheme (Chaudhry et al. 2019).
FO and FE show marginally higher throughput than SNOW-256, but we noticed significant accuracy drops up to 31.8% with FO and 9.7% with FE from SNOW-2561 accuracy. Both approaches look similar SNOW in a way of attaching small size model and training it while the source model is frozen. However, instead of subscribing intermediate features, they use only single feature map of the third residual block (FE) or the last residual block (FO) of the source model, which prevents them from exploiting the intermediate features. From this comparison,
can see that feature subscription actually contributes to higher accuracy for a new task as we expected in Section 2 yet at a small throughput overhead with cost-efficient channel pooling.
Figure (bottom right) shows that SNOW as well as all other techniques converge around 60 epochs. Since the end-to-end training time is is proportional to the number of epochs to and inversely proportional to the throughput, SNOW is much faster than FT, MP, and PN in terms of time-to-convergence, because of 2x~~5x higher throughput for the same number of epochs.
3.2 HYPER- APPER-PARAMETER SENSITIVITY
In this section, we discussed the hyper-parament sensitivity of SNOW in terms of model accuracy with the Car dataset. We performed additional training on the Car dataset while varying the K, the delta model size, and σ², one at time to understand their impacts on the accuracy. We summarized the outcomes in Table 4 (where M is the source model size) and made the following observations with practical strategies for hyper-parameter tuning:
Parameter changes
K
delta model size R2
0²
24
20
18
37
1c
le
1c
le
8
16
Top-1 accuracy
83.10
83.79
83.39
79.02
83.79
80.36
81.27
83.79
83.35 83.45
Top-5 accuracy
96.93
96.94
96.93
96.00
96.94
95.93
95.85
96.94
96.99
96.88
Table 4: Accuracy changes over varying hyper- pperrpparmettr settings.
The accuracy is bit sensitive to the K values. As discussed in Section 2.2 both under- subscription and over-subscription can degrade predictive power: under- under-subscription means insufficient feature sets for delta model, while over-subscription may introduce unwanted noises to a delta model. Since over-subscription also incur model-size overhead, it would be desirable to start with small K values first until the peak accuracy gets reached.
It shows that the performance is sensitive the delta model size, and clearly exposes the existence of the ideal size. It is obvious that the same rule of thumb in neural network architecture design applies here too: having too few target-specific features hurts accuracy, but having too many (or too big delta net) may lead to over-fitting
Too large too small o² can lead to sub-optimal predictive power. Therefore, it is important to develop a method to find good sigma values. We're currently researching on the idea to examine the early weight distribution (i.e., for a few epochs) and then determine the sigma value to balance out exploration and stabilization.
3.3 SERVING PERFORMANCE
We measured the computation costs of an inference process by serving all six tasks concurrently (i.c.. classifiers for ImageNet and the other five in Table 3) against the same input (Zamir et al. 2018
7

---

Published as a conference paper at ICLR 2020
Serving
ImageNet, Food, DTD, Action, Cars, CUB
approach
FO
FE
FT
MP
PN
SNOW
Total model parameters (M)
26.8
28.3
144.3
26.9
257.4
30.8
Throughput (images/sec)
7060.8
5626.6
632.9
304.3
169.5
4731.0
Table 5: Computation costs for inference for the ImageNet (source model), Food, DTD, Action, Car, and CUB. We run all tasks together single Tesla V100.
Mathur et al. 2017). Overall inference results are summarized in Table 5 and indicate that SNOW has significantly less overhead than other high-accuracy approaches such as FT, MP, PN in terms of total model size and inference throughput.
FO is the smallest and shows the best inference throughput due its architectural simplicity, but please note that FO has the worst accuracy on all five datasets.
SNOW yields comparable inference throughput with FE (yet shows better accuracy as in Fig. with slightly more parameters.
FT has the 2nd biggest penalty in terms the parameter count as expected, because each task needs a full set of parameters tuned for a specific task.
MP does not have much overhead in terms of the total model size, but suffers from the poor inference performance. Such penalty comes from the expensive scatter-gather computation patterns which are required to re-use the source model parameters, in addition to the increased number of layers on the critical execution path.
PN has the biggest penalty terms of the total number of parameters expected size which leads to poor inference throughput.
In short, observe that SNOW has critical advantages in multi-task setup, it marginally increases the total model size by sharing the source model among all the individual tasks and keeps the inference performance intact, not to mention that it delivers high-quality accuracy.
3.4 CHANNEL POOLING EFFECT
In this section, first examine how the top-K selection changes during the course of training, and thencompares the channel pooling scheme in SNOW with other popular dimensional reduction and feature amplification methods in deep learning.
Pooled Channel Index
300
450
Iteration
100
200
300
400
Figure 5: Selected channel index distribution for the Car dataset during training.
Figure shows how the top-K features selected by the proposed channel pooling for the Car dataset change throughout training where the X-axis is for the channel indices, the Y-axis is for the training progress (in terms of iteration), the solid vertical lines represent the selection of the corresponding channels, and the horizontal dotted lines indicate the start of next epoch. Overall, we found that the top-K K feature selections change frequently in the beginning of the training for exploration but get stabilized after additional epochs, as discussed in Section 2.2
We applied such alternatives as feature subscription in SNOW and compared performances in terms of parameter overhead and test accuracy summarized in Table T which clearly highlights that the proposed channel pooling has the minimum overhead, while showing the overall best performance. As discussed in Section 2.2 our channel pooling requires the least number of parameters for selective
8

---

Published as a conference paper at ICLR 2020
Feature subscription method
Parm. overhead (M)
Food
DTD
Action
Car
CUB
1 Convolution
14.0
82.82
74.73
78.29
77.07
73.20
Channel-wist wise attention
35.0
84.05
74.06
79.50
82.94
75.51
Subscribing to all
17.5
82.80
73.78
78,85
76.47
72.80
Channel pooling
5.5
84.06
72.37
78.48
83.79
75,81
Table 6: Parameter overhead and Top-1 accuracy (%) of SNOW for each feature subscription method.
feature subscription among other alternative methods, including x 1 convolution, channel-wise attention (Woo al., 2018), and subscribing to all features. Notably, the channel pooling approach shows the highest accuracy for most of the datasets (Food, Car, CUB), and still achieves comparable accuracy for Action dataset.
Although the 1 x convolution uses more parameters than the channel pooling block, its accuracy 6.72% less than channel pooling while slightly higher than the case of subscribing to all. This result indicates that the x 1 convolution learns which channels are important for new task, but its performance is worse than that of channel pooling in terms all aspects, including accuracy, the number of required parameters, and computational overhead.
The channel-wise attention module (Woo et al. 2018) shows good accuracy for most of the datasets, however, its computation costs for inference are high. For the multi-task setup, its inferencing throughput is 1101.8 images/sec and the model parameter overhead is 35.0M while channel pooling achieves 4731.0 images/sec throughput with only 5.5M parameter overhead. Our channel pooling shows 4.3x higher throughput and 84.2% of parameter overhead reduction for multi-task inference than the channel-wise attention.
RELATED WORK
large body of research has been done to address such challenges by exploring the various trade-off between waaning/serving/paraeett
efficiencies and accuracy. On the transfer learning side, the draw-
backs in popular schemes such as feature extraction (Donahue et al. 2014) or fine-tuning (Dauphin et al. 2012 Cui et al.) 2018) addressed in various efforts. Mudrakarta et al. (2019) enhanced the parameter efficiency by inserting multiple patch layers into a frozen source model to capture the task-specific features at a cost of increased training and serving overhead. Knowledge flow (Liu jet al. 2019)) proposed a method that trains a learner from multiple experts by gradually distilling the layer-wise features from experts into the student at cost of increased critical execution path for both training and inference as well. FixyNN (Whatmough et al. 2019) revisited the conventional feature-extraction based transfer learning approach from the computation perspective by re-designing and fine-tuning the last few top layers, which could degrade the model accuracy substantially unless enough number of top layers are optimized. Recently, Guo al. (2019) improved the accuracy of transfer learning by adaptively selecting feature extraction blocks per sample at cost of extra policy network and low training throughput.
On the lifelong learning side, catastrophic forgetting has been researched from various angles. Weight regularizations during re-training to retain the old task performance were studied in Lee et al. (2016); Weinshall & Cohen (2018). Li & Hoiem (2018) mitigated the catastrophic forgetting without old datasets by regularizing the final hidden activations from all the old tasks. Progressive Neural Net (Rusu et al. (2166) in reinforcement learning proposed propagate the feature maps in columnized set of models for different tasks, which would greatly increase the total model size, not to mention slower inference performance due the complex per-layer inter-column adapters and expensive lateral connections (Guo et al. 2019 Chaudhry et al. 2019).
5 CONCLUSION
In this paper, we propose SNOW, an innovative way of transfer and lifelong learning by subscribing knowledge of a source model for a new task through channel pooling. SNOW architecture enables the delta model to subscribe to the intermediate features from each layer of the source model in uni- mni-directional manner. In this manner, we can effectively train a model for a new task by updating the parameters of the delta model only while keeping the source model untouched. SNOW offers

---

Published as a conference paper at ICLR 2020
excellent training/serving speed by sharing the same features of the source model across all the delta models, yet offers competitive prediction accuracy with design simple yet effective Channel Pooling. In comprehensive experiments with multiple datasets, SNOW shows the best balance between the computing performance and prediction accuracy over all the datasets among the state-of-the-art transfer and lifelong learning methods during both training and serving process.
REFERENCES
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 - mining discriminative com- ponents with random forests. In Proceedings of the European Conference Computer Vision (ECCV), 2014.
Arslan Chaudhry, Mare' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. In ICLR, 2019.
M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2014.
Yin Cui, Yang Song, Chen Sun, Andrew Howard, and Serge J. Belongie. Large scale fine-grained categorization and dopain-specifie transfer learning. CoRR, 0oss/806.061933 2018.
Grégoire Mesnil Yann Dauphin, Xavier Glorot, Salah Rifai, Yoshua Bengio, lan Goodfellow, Er- ick Lavoie, Xavier Muller, Guillaume Desjardins, David Warde-Farley Pascal Vincent, Aaron Courville, and James Bergstra. Unsupervised and transfer learning challenge: a deep learning approach. In Proceedings of ICML Workshop on Unsupervised and Transfer Learning, Proceedings of Machine Learning Research, pp. 97-110, 2012.
J. Deng, W. Dong, R. Socher, Li,. K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Dar- rell. Decaf: A deep convolutional activation feature for generic visual recognition. In Proceedings of the 31st International Conference on International Conference on Machine Learning Volume 32, ICML' 14, 2014.
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. arXiv preprint arXiv:1812.03900 2018.
Yunhui Guo, Honghui Shi, Abhishek Kumar, Kristen Grauman, Tajana Rosing, and Rogério Schmidt Feris. Spottune: Transfer learning through adaptive fine-tuning. CVPR, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Jeff Heaton. Introduction to Neural Networks Java, 2Nd Edition. 2008.
Diederik Kingma and Max Welling, Auto-encoding variational bayes. 2014.
Jonathan Krause, Michael Stark, Jia Deng, and Fei-Fei. object representations for fine-grained categorization. In 4th International IEEE Workshop on 31) Representation and Recognition (3dRR), Sydney, Australia, 2013.
Giwoong Lee, Eunho Yang, and Sung Ju Hwang. Asymmetric Multi-task Learning Based on Task Relatedness and Loss. ICML., 2016.
Zhizhong Li and Derek Hoiem. Learning without Forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40:2935-2947, 2018.
Iou-Jen Liu, Jian Peng, and Alexander Schwing. Knowledge Flow: Improve Upon Your Teachers. In ICLR, 2019.
10

---

Published as a conference paper at ICLR 2020
Akhil Mathur, Nicholas D Lane, Sourav Bhattacharya, Aidan Boran, Claudio Forlivesi, and Fahim Kawsar. Deepeye: Resource efficient local execution of multiple deep vision models using wearable commodity hardware. In Proceedings of the 15th Annual International Conference Mobile Systems, Applications, and Services, pp. 68-81. ACM, 2017.
Michael McCloskey and Neal J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. volume 24 of Psychology of Learning and Motivation, pp. 109 165. Academic Press, 1989.
Pramod Kaushik Mudrakarta, Mark Sandler, Andrey Zhmoginov, and Andrew G. Howard. K For The Price Of Parameter Efficient Multi-task And Transfer Learning. In ICLR, 2019.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In NIPS Autodiff Workshop, 2017.
Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. CoRR, 2016.
Pablo Sprechmann, Siddhant M. Jayakumar, Jack W. Rae, Alexander Pritzel, Adrià Puigdomênech Badia, Benigno Uria, Oriol Vinyals, Demis Hassabis, Razvan Pascanu, and Charles Blundell. Memory-based Parameter Adaptation. In ICLR, 2018.
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech-UCSD Birds-200-200 Dataset. Technical Report CNS-TR-2011 1--000,, California Institute of Technology. 2011.
Daphna Weinshall and Gad Cohen. Curriculum learning by transfer learning: Theory and experiments with deep networks. In ICML, 2018.
Paul N. Whatmough, Chuteng Zhou, Patrick Hansen, Shreyas K. Venkataramanaiah, Jae-sun Seo, and Matthew Mattina. Fixynn: Efficient hardware for mobile computer vision via transfer learning. In SysML, 2019.
Sanghyun Woo, Jongchan Park, Joon- Yoon-Young Lee, and In So Kweon. Cbam: Convolutional block attention module. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 3-19, 2018
Bangpeng Yao, Xiaoye Jiang, Aditya Khosla, Andy Lai Lin, Leonidas Guibas, and Li Fei-Fei. Human action recognition by learning bases of action attributes and parts. In 2 International Conference on Computer Vision, pp. 1331-1338. IEEE, 2011.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In NIPS, 2014.
Amir R. Zamir, Alexander Sax, William Shen, Leonidas J. Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning, In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.

---

Published as a conference paper at ICLR 2020
A APPENDIX
A.I CHANNEL POOLING
We present the channel pooling block in Section 2.2 with Figure 3 To better understand, we provide the details of how the channel pooling block works in operation level follows.
Algorithm 1 Channel Pooling Forward Function
channel pooling weight (NV x 1 vector)
3:: feature map from a source model (N channels)
N M pooling
procedure BIANEEPOOLINFOOWAADOG
training then
rand_ v = w + (O o²)
idx toottooKaaaa (rand_ M)
selected weiite w[idx]
else
selected idx (w, M)
end
x[idx]
I = x @ selected sclected_weight
N1: channel-wise scalar multiplication
BatchhNorm(a
//optionll
I = (x)
//optional
return x
end procedure
A.2 EXPERIMENT DETAILS TRAINING PERFORMANCE
Figure shows training performance in terms of accuracy, throughput, and memory footprint for each dataset and training method. In the following tables, we report precise numbers of the graphs in Figure 4 as well as validation accuracy curves during training for all datasets.
Training
Top-1 (top-5) accuracy Action
approach
Food
DTD
Car
CUB
FO
70.76 (90.55)
70.81 (91.65)
76.41 (95.58)
51.96 (78.68)
65.96 (89.90)
FE
82.24 (95.84)
71.45 (92.40)
73.97 (93.64)
74.05 (94.12)
67.16 (91.27)
FT
87.63 (97.32)
61.56 (85.01)
75.40 (93.02)
90.26 (98.62)
77.93 (93.37)
MP
81.97 (95.68)
72,41 (92.88)
81.51 (95.44)
81.42 (96.00)
77.22 (94.24)
PN
86.19 (96.97)
74.32 (92.34)
79.99 (94.41)
86.26 (97.71)
76.51 (93.63)
SNOW-256
84.06 (96.47)
72.37 (92.02)
78.48 (95.04)
83.79 (96.94)
75.81 (94.05)
SNOW-128
84.20 (96.52)
73.98 (93.06)
79.28 (95.25)
83.69 (97.04)
75.51 (93.87)
Table 7: Test accuracy of each approaches with datasets.
Training approach
Throughput (images/sec) Memory usage (GB) DTD
Food
Action
Car
CUB
FO
1076.08 14.12
1089.83
4,40
1106.31 44.8
1085.67
14.4
1102.50 4.63
FE
1205.84
4.05
1205.84
4.31
1230.77 4.02
1209.26
14.17
1219.63 | 4.31
FT
356.84
13.74
357.94
13.74
359.13
13.83
358.741
13.75
358.24 14.66
MP
391.36
14.57
393.60
14.68
394.25
14.69
388.66
14.67
392.96
13.72
PN
15.29 189.69
15.60 189.09
15.29 189.50
15.72 185.01
187.24
15.56
SNOW-256
947.80
4.40
959.16
4,40
949.55 4.26
965.31
4.25
970.06
4.27
SNOW-128
871.34 2.92
873.12 2.87
868.9 .7.99777 2.80
885.20 2.87
880.94
2.94
Table 8: Computational performance for training single GPU
12

---

Published as a conference paper at ICLR 2020
T validation accuracy (Food)
Top1 validation accuracy (DTD)
90
90
80
80
Top1 accuracy
Topl accuracy
70
M
70
M/
60
60
50
50
40
40
60
90
120
150
180
60
90
120
150
180
Epoch
Epoch
Top1 validation accuracy (Action)
Top1 validation accuracy (Car)
90
90
80
80
Topl accuracy
Topl accuracy
70
70
FO
FE
FT
MP
60
60
PN
SNOW
50
50
40
40
0
30
60
90
120
150
180
0
30
60
90
120
150
180
Epoch
Epoch
Topl validation accuracy (CUB)
90
80
Topl accuracy
70
60
50
40
0
30
60
90
120
150
180
Epoch
Figure 6: Validation accuracy curves during training for each dataset
13

---

Published as a conference paper at ICLR 2020
A.3 LEARNING WITHOUT FORGETTING
In our main experiment, we do not include the results from SNOW with Learning without Forgetting (LF) (Li & Hoiem 2018) due to the catastrophic forgetting issue shown in Table Simply, the accuracies from LF for the experimented datasets are quite off from other competing algorithms, making hard to put them on the same charts. We explored a few sequences to get the best outcome for LF and came up with the following in Table which led to the accuracy changes for the datasets below. We used batch size of 128, 0.005 learning rate, and A = 1 as suggested in (Li & Hoiem 2018). As you can see in the result for the training order A, once Action dataset is used, the accuracy against Car dataset drops significantly, as both are very heterogeneous (in terms of target domain and dataset size). Yet, adding CUB has somewhat limited impacts Action. Also, we observe that the accuracy of LF depends on a training order. The accuracy from the training order A and training order B is significantly different before adding Food data. Overall, the combination of datasets in our experiment seems to be a very challenging scenario for LF in terms of catastrophic forgetting.
Training Order A (starting from ImageNet)
Task
Car
Action
CUB
DTD
Food
Car
81.87
28.40
22.78
22.44
15.31
Action
77.08
68.87
62.60
56.36
CUB
70.94
46.12
32.91
DTD
71.93
39.29
Food
79.71
Training Order B (starting from ImageNet)
Task
Action
CUB
DTD
-> Car
+
Food
Action
80.63
72.31
66.74
55.5
57.36
CUB
75.91
54.33
52.19
36.00
DTD
73.26
43.27
40.78
Car
73.77
16.33
Food
79.10
Table 9: Top-1 accuracy of Learning without Forgetting (LF)
LF overall showed the training throughput of 272.69 images/sec with 13. 13.99GB memory footprint. The reason why LF is slower than FT is because LF needs to perform extra forward paths compute the loss for the old tasks.
14


Reviews:
[{"review_id": "rygKCo52KH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*** Increased to Accept from Weak Accept after author rebuttal and changes to the paper ***\n\nThis paper proposes a method, SNOW, for improving the speed of training and inference for transfer and lifelong learning. SNOW starts with a pre-trained, frozen source model, and trains delta models for target tasks which, at each layer, concatenate a small number of task-specific features with the top-K most useful subset of features in the corresponding layer in the source model. As long as the target tasks are sufficiently related to the source task, it allows for small delta models and a small additional parameter overhead in the form of one weight per source model feature map. While there are (i) some issues with the presentation of results for training efficiency, (ii) some question marks over the sensitivity of the model to hyper-parameters, and (iii) several grammatical errors / typos in the manuscript, if these can be addressed I recommend the paper for acceptance because it seems to strike a superior balance of efficiency (regarding memory usage and inference speed) and accuracy when compared to a number of baselines, and to my knowledge it is a novel approach.\n\nDetailed comments:\n* Section 2.2 - How is sigma (the exploratory noise added for feature selection during training) chosen and how sensitive is the approach to its value? It seems like it was fine-tuned, given that a different sigma is chosen for the Action dataset (several orders of magnitude difference). In practice, tuning sigma could significantly increase training time. \n* It seems like the performance of only one run was plotted per hyperparameter setting - it would be informative to see a mean and standard deviation especially since the approach seems like it could be unstable for the wrong hyperparameter settings.\n* Related to the previous point, how much do the top-K feature selections change throughout training? One would have thought that this could cause instability during training for a high sigma. If sigma is too low, you could end up with suboptimal feature selection.\n* Figure 4 graphs are a bit misleading because the throughput on the x-axis is reported per GPU and the larger models all need 2 or more GPUs. While this is mentioned in the main text, it is still optically deceptive and the results are GPU-dependent - presumably if the GPUs had a larger memory, the larger models would not seems as slow. I think it would be clearer to plot images/sec on the x-axis or to rerun the experiments just using a single GPU.\n* It is stated that  “[d]etermining K […] has a critical impact on both size and target accuracy in the target models”, where K is the number of feature maps in the source model that the delta model subscribes to in each layer. How sensitive is the accuracy exactly? Can this be quantified or discussed in more detail?\n* Furthermore, how sensitive is the performance to the number of target-model-specific features at each layer?\n* Different learning rate schedules were used for SNOW and baselines - initial lr for SNOW is 1.0, while for all other models it is 0.1. Was it checked whether the baselines improve when they are run with an initial lr of 1.0? Was this hyperparameter more heavily tuned for SNOW than for the baselines?\n* Since the source model is fixed, the applicability of the approach to lifelong learning is heavily dependent on the usefulness of the source model to subsequent tasks. If it is not, then one will have to incorporate large delta models. Furthermore, there can be no transfer between the tasks trained in the delta models.\n \nGrammatical errors / suggestions:\n* Page 1, first line: “hallmark” doesn’t make sense in this context - maybe “key objective” or “goal”?\n* Page 1, 2nd paragraph, first line: “wee” -> “we”.\n* Page 1, 2nd paragraph, line 6: “best top-K” -> either “K best” or “top K\"\n* Page 2, last paragraph, first line: “three folds” -> “threefold\"\n* Section 2.1, line 2: “pooing”->”pooling”. Same typo on Page 4, last line.\n* Page 6, line 1: “training from the scratch” -> “training from scratch\"\n* Page 6, line 9: “more 6x than” -> “6x more than\"\n* Overall, the manuscript needs to be proofread a few times.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SygBFYrscB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "After rebuttal:\nAuthors have addressed all my doubts. I recommend accepting this paper.\n\n=============================\nBefore rebuttal:\nSummary:\n\nThis paper proposes a new way to do transfer learning. Specifically, authors first train a big source ConvNet and then for each task, they train a small ConvNet in which each layer subscribes to some k channels in the corresponding layer of the source ConvNet. Authors show that this model works better than methods that fine-tune the last few layers of the source network and performs close to costlier methods like progressive networks but with lesser parameters and higher throughput. Experiments on 5 tasks verify their claim.\n\n\nMy comments:\n\nOverall, this is a very interesting paper.\n\n1. This is an interesting model to do transfer or lifelong learning but only for ConvNet architectures with image data. To avoid overstating the results, I request the authors to highlight this limitation in both the title and the abstract.\n2. Page 3, para starting with “In detail”: Is the ResNet50 for delta model pre-trained or not? I know it is not pre-trained based on future paragraphs. But it is good to clarify it here.\n3. Sharing the same source network across multiple tasks during inference time is useful only when all the tasks take the same input. This is a very restricted application. This needs to be elaborated and highlighted in the paper.\n4. I would like to see the LF results included in the paper even though it has catastrophic forgetting issues.\n5. In Figure 4, the x-axis represents training throughput or inference throughput? I guess it is training throughput. Also, are the models trained for all the tasks in parallel (as described in serving all the tasks at once section) or separately? Even though I can guess answers for these, it is better to make these explicit in the paper for the benefit of the readers.\n6. It is never a good idea to show test curves for a task. Please remove the test curves from Figure 4. Instead, use a separate validation set and show validation curves.\n7. Are the authors willing to release the code to reproduce their results?\n\nMinor comments:\n\n1. Section 1, second para, 1st line: “wee” should be “we”\n2. Table 1: Fix grammar in MP description.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "SJlLFPEjFr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper attempts to tackle transfer learning and lifelong learning problem by subscribing to knowledge via channel pooling. The channel pooling is actually selecting the subsect of the feature map according to the way that prediction accuracy from the delta model can be maximized. Experiments show effectiveness of the proposed method.\n\nPros:\nOverall, this paper is well written and easy to follow. The technique is sound and the problem studied in this paper is significant.\n\nCons:\n1.\tI do not think that the model proposed in this paper is able to tackle lifelong learning problem. The main reason is that lifelong learning basically requires only one model that will continue to learn from new tasks. After learning several new tasks, people hope this model can still perform well on the previous tasks as well as the current ones. However, in this paper, not only one model is learned. Instead, new models appear when new tasks are given, which does not meet the definition or requirement of lifelong learning. It only meets the requirement of transfer learning. The experimental results also validate my opinion since only one new task is given while in lifelong learning, continuous new tasks will come and the original model should perform well on all of them as well as on the old tasks.\n2.\tIn Figure 4, the legend in the first picture will confuse the readers. I suggest the authors put it outside all the figures. Besides, the proposed method in the last picture is not the best. What do the authors want to convey by this picture?\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method, SNOW, for improving the speed of training and inference for transfer and lifelong learning by subscribing the target delta model to the knowledge of source pretrained model via channel pooling.

Reviewers and AC agree that this paper is well written, with simple but sound technique towards an important problem and with promising empirical performance. The main critique is that the approach can only tackle transfer learning while failing in the lifelong setting. Authors provided convincing feedbacks on this key point. Details requested by the reviewers were all well addressed in the revision.

Hence I recommend acceptance.
Decision: Accept (Poster)