Sharing Less is More: Lifelong Learning in Deep Networks with Selective Layer Transfer
Authors: Seungwon Lee
Sima Behpour
ERIC EATON
Venue:  (2021)
Keywords: lifelong learning
continual learning
architecture search

Abstract: Effective lifelong learning across diverse tasks requires diverse knowledge, yet transferring irrelevant knowledge may lead to interference and catastrophic forgetting. In deep networks, transferring the appropriate granularity of knowledge is as important as the transfer mechanism, and must be driven by the relationships among tasks. We first show that the lifelong learning performance of several current deep learning architectures can be significantly improved by transfer at the appropriate layers. We then develop an expectation-maximization (EM) method to automatically select the appropriate transfer configuration and optimize the task network weights. This EM-based selective transfer is highly effective, as demonstrated on three algorithms in several lifelong object classification scenarios.

Under review as a conference paper at ICLR 2021
SHARING LESS IS MORE: LIFELONG LEARNING IN DEEP NETWORKS WITH SELECTIVE LAYER TRANSFER
Anonymous authors Paper under double-bling review
ABSTRACT
Effective lifelong learning across diverse tasks requires diverse knowledge, yet transferring irrelevant knowledge may lead to interference and catastrophic forget- ting. In deep networks, transferring the appropriate granularity of knowledge is as important as the transfer mechanism, and must driven by the relationships among tasks. We first show that the lifelong learning performance of several cur- rent deep learning architectures can significantly improved by transfer at the appropriate layers. We then develop to automatically select the appropriate transfer configuration and optimize the task network weights. This EM-based selective transfer is highly effective, demonstrated on three algorithms in several lifelong object classification scenarios.
sxppetation-mmaxiizaatii
(EM) method
INTRODUCTION
Transfer at different layers within a deep network corresponds to sharing knowledge between tasks at different levels of abstraction. In multi-task scenarios that involve diverse tasks, reusing low-layer representations may be appropriate for tasks that share feature- based similarities, while sharing high- level representations may be more appropriate for tasks that share the appropriate granularity of knowledge to transfer is an important architectural consideration for deep networks that support multiple tasks.
abstract similarities. Selecting
In scenarios where tasks share substantial similarities, many multi-task methods have found success using a static configuration of the knowledge share (Caruana 1997; Yang & Hospedales 2017] Lee et al. 2019; Liu et al., 2019; Bulat et al.] (020),, such as sharing the lower layers of deep network with upper-level task-specific heads. As tasks become increasingly diverse, the appropriate granularity for transfer may vary between tasks based on their relationships, necessitating more selective transfer. Prior work in selective sharing for deep networks has typically either (1) branched the network into tree structure (Lu et 2017; Yoon et al.) 2018 Vandenhende et al., [019 He jet al. 2018), which emphasizes the sharing of lower layers or (2) introduced new learning modules between task models (Yang & Hospedales 2017 Xiao et al. 2018 Cao et al. 2018) which increases the complexity of training. The transfer configuration could then be optimized in batch settings to maximize performance across the tasks.
However, the problem of selective transfer is further compounded in continual or lifelong learning settings, in which tasks are presented consecutively. The optimal transfer configuration may vary between tasks or it may vary over time. And indeed, we may nol want to transfer at all layers, as some task-specific layers may need be interleaved with shared knowledge in order to customize that shared knowledge to individual tasks. To verify this premise and motivate our work, we conducted a simple brute-force initial experiment: we took multi-task CNN with shared layers and lifelong learning CNN that uses factorized transfer (DF-CNN, [ce et al. 2019) and varied the set of CNN layers that employed transfer (with task-specific fully connected layers at the top). Using two data sets, we considered several transfer static configurations: transferring at all CNN layers, transfer at the top-k CNN layers, transfer at the bottom-k CNN layers, and alternating transfer/no-transfer CNN layers. The results are shown in Figure D with details given in Section 2 Clearly, we see that the optimal a posteriori transfer configuration varies between task relationships and transfer mechanisms. Restricting the transfer layers significantly improves performance over the naïve approach of transferring at all layers, with the alternating configuration performing extremely well for both multi-task and lifelong learning.

---

Under review as a conference paper at ICLR 2021
0.5
Mean Peak Per-task Accuracy
Mean Peak Perstask Accuracy
0.6
0.4
0.3
0.4
0.2
0.2
0.1
HPS all
HPS topl
HPS top2
HPS top3
HPS bettom
HPS bottom2
HPS bottom3
HPS alter.
HPS all
HPS top1
HPS tap2
HPS top3
HPS bottomi
HPS bottom?
HPS bottoma
HPS aller
(a) Multi Task CNN with HPS/ / CIFAR- 100
(b) Multi-Task CNN with HPS / Office-Home
Mean Peak Per task Accuracy
Mean Peak Per-task Accuracy
0.4
0.3
0.4
0.2
0.2
0.1
0
DF CNN all
DF CNN top1
DF-CNN top2
DF-CNN top3
DF CN-C bottoml
DF OF-CNN bottom2
DF- CNNN alter. DF- CNN bottom3
DF CN- all
DF DF-CNN top1
DF CNN top2
DF DF-CNN top3
DF CNN bottomi
DF CNN bottomz
DF- CNN alter. DF CNN bottoma
(c) DF-CNN CIFAR-100
(d) DE-CNN Office-Home
Figure 1: Accuracy of CNN models averaged ten tasks in lifelong learning setting with 95% confidence interval. This empirically shows that the optimal transfer configuration varies, and choosing the correct configuration is superior to transfer all layers.
Instead of only considering such restricted of static configurations in brute-force search, our goal is automate this process of selective transfer during learning, enabling it to customize the transfer configuration to each task. We investigate the use of architecture search during learning to dynamically adjust the transfer configuration between tasks and over time, using to learn both the parameters of the task models and the layers to transfer within the deep net. This approach, Lifelong Architecture Search via EM (LASEM), enables deep nets to transfer different sets of layers for each task, allowing more flcxibility over prior work in branching branching-bused configurations for selective transfer. It also introduces little additional computational cost the base learner in comparison to training selective transfer modules between task networks, and in contrast to the expense of brute-force search all transfer configurations. To demonstrate its effectiveness, applied LASEM to three architectures that support transfer between tasks in several lifelong learning scenarios and compared it against other lifelong learning and architecture search methods.
iititttionnmaiiizzitioonn (EM)
2 THE EFFECT OF DIFFERENT TRANSFER CONFIGURATIONS
This section further describes the initial experiments mentioned in the introduction as motivation for our proposed LASEM method. The hypothesis of our work is that lifelong deep learning can benefit from using more flexible transfer mechanism that selectively chooses the transfer architecture configuration for each task. This would permit to dynamically select, for each task model, which layers to transfer and which to keep task task-specific (enabling it to customize transferred knowledge to an individual task).
To determine the effect of different transfer configurations, we conducted set of initial experiments using two established methods:
Multi-Task CNN with hard parameter sharing (HPS): This approach shares the hidden CNN layers between all tasks, and maintains task-specific fully connected output layers. is one of the most common methods for multi-task learning of neural nets (Caruana 1997). and is widely used.
Deconvolutional factorized CNN (DF-CNN): The DF-CNN (Lee et al. 2019) adapts CNNs to a continual learning setting by sharing layer-wise knowledge across tasks. Instead of using the same
2

---

Under review as a conference paper at ICLR 2021
convolutional filters for multiple tasks, the convolutional filters are dynamically generated from insk-independent layer-dependent shared tensor through a task-specific deconvolutional operation and tensor contraction. Similar to HPS, the DF-CNN maintains task-specific fully connected topmost layers. When training the task models consecutively, gradients flow through to update the shared tensors and the task-specific parameters that transform those shared tensors to construct the task CNN. This transfer architecture enables the DF-CNN to learn and compress knowledge universal among the observed tasks into the shared tensors.
Both these methods utilize set of transfer-based CNN layers and non-transfer task-specific layers. For network with d CNN layers, there are 2d potential transfer configurations. To explore the effect of transfer at different layers, we varied the transfer configuration among several options:
All: Transfer all CNN layers. Note that the original DF-CNN used this configuration.
Top k: Transfer across task models occurs only at the & highest CNN layers, with all others remaining task-specific. We would expect this transfer configuration to benefit tasks that share high-level concepts but have low-level feature differences.
Bottom k: Transfer occurs only at the k lowest CNN layers, with all others remaining task-specific. This architecture is opposite of the Top d - k, so we would expect it to benefit tasks that share perceptual similarities but have high-level differences.
Alternating: This configuration alternates transfer and non-transfer layers, enabling the non-transfer task-specific layers to further customize the transferred knowledge to the task.
We evaluated the performance of various transfer configurations on the CIFAR-100 100 (Krizhevsky & Hinton 2009) and Office-Home (Venkateswara al.] (017)) data sets, following the lifelong learning experimental setup used in previous work (Lee et al. 2019). CIFAR- 100 involves ten consecutive tasks of ten-way image classification, where any object class occurs in only one task. Office-Home involves ten tasks of thirteen-way classification, separated into two domains: "Product" images and "Real World" images. The CNN architectures used for each data set and optimization settings follow prior work and are detailed in Appendix [A] During training, we measured the peak per-task accuracy on held-out test data, averaging results over five trials.
Our results in Figure reveal that permitting transfer at all layers does not guarantee the best perfor- mance. This observation complicates learning on novel tasks, since the best transfer configuration depends both the algorithm and the task relations in the data set. Notably, we see that the DF-CNN, which is designed for lifelong learning, can be improved beyond the original version (Lee et al. 2019) by allowing transfer only some layers. Furthermore, configuration varies between data sets and algorithms. For instance on Office-Honne sharing lower layers in the HPS multi-task CNN achieves better performance average, but transferring upper layers works better for the DF-CNN. Similarly, the Alternating configuration consistently achieves near the best performance for the DF-CNN, benefiting from permitting the non-transfer layers to customize transferred knowledge to the individual task, but it is not consistently as good for HPS.
see that the optimal transfer
3 ARCHITECTURE SEARCH FOR THE OPTIMAL TRANSFER CONFIGURATION
The experiment presented above reveals that the transfer configuration can have a significant effect on lifelong learning performance, and that the best transfer configuration varies. These observations inspire our work to develop more flexible mechanism for selective transfer lifelong learning.
We can view the transfer configuration as new hyper- parameter for each task model. Even with the constraint that all task models use the same transfer configuration, the search space grows exponentially as the neural network gets deeper (i.e., 24 configurations for d CNN layers). In more flexible settings where the transfer configuration customized to each task, this search space grows more, lincarly in the number of tasks. Although we could compound this problem further by permitting partial transfer within a layer, we focus on optimizing layer- transfer configurations.
Formally, laser- transfer configuration for task t can be specified by d-dimensional binary vector e E C = {0, 1)dd where each Ct,j is a binary indicator whether or not the jth layer involves transfer. We can compactly notate C, by set of indices of transferred layers. For example, the Alternating configuration described in Section 2 can be denoted by = [0. 1, 0, = {2, 4): Figure depicts this particular configuration for three approaches.
3

---

Under review as a conference paper at ICLR 2021
Coutpu!!ll
gutputia
Coutput
Qulput
output!!!
output
FC
EC
Conv
Conv 2
TF 2
Conv
Conv
Conv
shared tensors
Cony
Cony
Cony
Cony
Cony
Cony
Conv
Conv 2
Conv
Cony
68
Cony
Cony
Cony
Conv
Conv
Conv
Cony
input!!
input!
input!il
input!A)
input
input!
Multi-Task CNN
Tensor Factorization
DF-CNN
Figure 2: The Alternating {2, 4] transfer configuration for three different approaches using CNNs with four convolutional layers and one fully connected layer. Models are illustrated for two tasks, red and green, with shared or cranser-based layers denoted in blue.
Our goal is to determine the task-specific transfer configuration while simultaneously optimizing the log-likelihood of the task models and shared knowledge in lifelong setting. Treating e₁ latent variable of the model for task t, we can employ expectation maximization (EM) to perform this joint optimization. For each layer I, LASEM maintains shared set of transfer-based model parameters 0! and, for each new task, set of task-specific model parameters 0² using the chosen configuration C(t)) determine which combination of parameters will be used to form the specific model for that task. In brief, the E-step estimates the usefulness of the representation that each transfer configuration c, E C can learn from the given data (i.e., the likelihood of data P(ynew [enew c,)), while the M-step optimizes parameters of the task model and shared knowledge. We next detail this approach.
We first consider how to model the prior मe possible configurations of the current task's C(r)- Using a simple frequency-based -ased probability estimate with Laplace smoothing represents the prior probability of each transfer configuration
Nc. +1
P(c() = E, + +)
where C(r) denotes the configuration for task t, and ne, is the number of previous mini-batcher for which C₁ is the most probable configuration. This estimate, which we adopt in the experiments, considers each transfer configuration solely based on the current task's data. Alternative priors could instead be used, such as measuring the historic transfer configuration frequency over all tasks (which assumes substantial similarity among tasks) measuring configuration frequency over related tasks (which requires a notion of task similarity, such as via a task descriptor (Isele et al.) 2016 Sinapoy al. 2015). or as determined dynamically by the task model's relation to shared knowledge). During development, we also considered estimating the prior based on the probability of configurations averaged over training samples instead of the number of mini-batches ne; this alternative was not different statistically from Equation ii an empirical evaluation on CIFAR 100 and Office Office-Home.
In the E-step, the posterior on configurations is derived by combining the above prior and likelihood, which can be computed from the output of the task network on the current task's data (Xnews Vnew ):
P(c, neww Inew) x P(C()) = c;) P( Xnew C;)
The M-step improves the logelikeddood via the estimated probability distribution over the transfer configurations. Both 0, and O, are updated by the aggregated gradients of the log-likelihood in cases where the transfer configurations match the corresponding parameter. To combine the gradients of specific parameter vector (0, and 0,) over multiple possible configurations, we take the sum of the corresponding gradients weighted by the posterior estimate in Equation 2
0 (
A 1 P(c: X neww Vnew V log P(ynow
X Xnews (2)
W (1, .d)
(3)
000) 000 Eic,y P(c: X new Vnew V log P(ynow X Xnow, C2)
1I E (1,....
4

---

Under review as a conference paper at ICLR 2021
Algorithm 1 Lifelong Learning with EM-based Partial Layer Transfer
1: { randeminntializer(nnSSszz
Jt=
2: while vaooeerraninggtttaavllllleeee do
b Loop over entire training lifetime
3:
(Xnew, ; t) getNexfTrainingData
> Get current task data
4:
if TisNewTask(( then
b Initialize parameters for new tasks
[0() J|=1 randommnnttallieernetttzzz We < priorInitializer))
5:
6:
7:
end if
8:
Calculate likelihood P(ynew Nnow Ci) Ve; EC
9:
Calculate posterior P(c; Xnew fnow) & ((y)ev Xenw Ci) Ve, C
Eqn
10:
loop numMSStes times
> Eqn
{o 0 J + gradOptimizer Neew Unews X, P(C Xnew; Nnew), !0!0 00() 1d
11:
J1=1
12:
end loop
13: He < priorUpdater(TT, P(c: X Xnew ynew))
> Eqn
end while
The main difference in the update rules in Equation Bis the condition for the index of the summation.
Since gradient step the configurations (o!) o we can hold the likelihood fixed to take multiple M-steps per E-step by iterating Equation 3
1r_ li=1 may have little effect the likelihood,
Our approach, Lifelong Architecture Search via EM (LASEM), is specified in Algorithm The parameters of the transfer-based components are initialized only at the beginning of the lifelong learning process (line 1). while the parameters of the task-specific components and prior probability of configurations are initialized when the algorithm encounters new task (lines 4-7). Each iteration, the lifelong learner obtains a mini-batch of labeled data (X neww Inew) drawn i.i.d for some task f, training the task model online. Typically, there would be multiple consecutive task before the environment would switch a new task. The algorithm applies the E-step (lines 9, and 13) and M-step (lines 10-12) using the data mini-batch at each iteration. LASEM takes E- and M-steps for each mini-batch, so over the consecutive mini-batches per task, this process would be similar to the alternation of multiple E- and M- M-steps performed by the standard EM algorithm. We consider lifelong learning scenarios in which learner has no control over tasks, so the current task be switched to another one without convergence of the learning algorithm. Because of this setting, we do not require EM convergence, but it is possible to monitor the convergence by checking the probability weight over transfer configurations.
experienced per
LASEM uses one set of transfer-based and task-specific parameters (0, and 0,) for all transfer configurations, rather than maintaining distinet sets of parameters for each configuration. This approximation reduces the number of parameters and permits parameter updates across transfer configurations via a single gradient step.
Computational Cost The computational cost of LASEM depends heavily on the tasks, the choice of deep net, and how quickly the transfer configuration converges. We show empirically in Section that LASEM in practice introduces relatively little additional time complexity (~30--50% over the base learner's time). Empirically, the E-step takes ~15-00% 20% time of the M-step. but frequent switching of the configuration by the E-step may interfere with the M-step, consequently harming the convergence speed. Taking more M-steps per E-step (by increasing num.MSteps) can improve this stability and consequently the computational complexity by accelerating convergence.
For d-layer neural network with a time complexity of (.).. the per-iteration computational com- plexity of both the E- and M-steps are O(22 (-)). As the network depth d increases, it is well known that neural architecture search (NAS) (Pham et al. 2018 Liu et al., 2018) requires exponentially additional computation. To remedy this issue, LASEM can adopt similar solution to that of these other NAS methods by considering transfer configurations over groups of layers instead individual layers; explore this variation in Appendix E Memory requirements analyzed in Appendix F
5

---

Under review as a conference paper at ICLR 2021
1.1
Catastrophi Forgetting Ratio
CIFAR100 4% data
Office-Home 60% data
STL.10 20% data
Mean Peak Penk Pentask Accuracy
0.9
0.6
0.5
0.7
0.4
0.5
Transfer at All Layers
0.3-
Static Transfer Configurations
Best Static Transfer Configuration
Transfer All Layers LASEM
LASEM
0.3
DF-CMN
DF-CAN
DF-CNN
HPS
TF DF-ONN
HPS
TF DF-CNN
TF DF-CNN
Figure 3: (Left) Performance of LASEM applied to three methods and three lifelong scenarios. Black boxes show the range of mean accuracies that different static configurations can achieve, with the blue lines denoting mean performance of Transfer at All Layers. The red dots denote the mean performance of LASEM. Whiskers depict 95% confidence intervals. (Right) Mean catastrophic forgetting ratio after training all tasks. Less forgetting is indicated by a ratio near 1.0; see Appendix B
4 EXPERIMENTS
We evaluated LASEM following the same experimental protocol for lifelong learning as used in Section see Appendix for details. In addition to using the CIFAR 100 and Office-Home data sets, we introduce a lifelong learning version of the STL-10 data set (Coates et al. 2011). STL-10 has 5,000 training and 8,000 test images divided evenly among 10 classes, with higher resolution than CIFAR-100. We constructed 20 tasks of three-way classification using 20% and 5% of the given training data for training and validation, respectively. To increase the task variations, for each task we randomly chose three image classes, applied Gaussian noise to the images with a random mean and variance, and randomly permuted the channels. All results averaged over five trials with different random seeds. The code and data set generators are available at hhtpp:/bbt..l//CCI. LASEM
4.1 PERFORMANCE OF LASEM
We applied LASEM to three algorithms: a multi-task CNN with hard-parameter sharing (HPS) (Caruana 1997). Tensor Factorization (TF) (Yang & Hospedales) 2017 Bulat et al., 2020) and the Deconvolutional Factorized CNN (DF-CNN) (Lee et al. 2019). HPS interconnects CNNs in tree structures, with task models explicitly using the same parameters of all shared layers. In contrast, the TF and DF-CNN task models explicitly share only a fraction of tensors, and the parameters of each task model are generated via transfer.
Figure (left) compares the performance of the task-specific transfer configurations discovered by LASEM (red) using a single static transfer configuration (black boxes). These black boxes depict the performance range of the methods using various static transfer configurations (i.e., All, Top k Bottom k Alternating) for all task models, with All shown in blue. To estimate this range, we tested cight (50%) and 16 (25%) of the possible static configurations for the Cour-CNNllayer (CIFAR-100 and Office-Home) and -i-CNN-layer (STL-10) task models, respectively.
We can see that LASEM chose transfer configurations that perform toward the top of each range, especially on the DF-CNN designed for lifelong learning. LASEM clearly outperforms Transfer All Layers. Automatically selecting the transfer configuration becomes even more beneficial for methods that have a wide range of performances for different configurations. Examining the catastrophic forgetting ratio [FigureBJright with details in Appendix B reveals the importance of selecting the appropriate transfer configuration for maintaining performance on previous tasks, revealing that LASEM exhibited less forgetting than baselines in most cases, especially on the DF-CNN. Moreover, LASEM imposes little additional cost in order to determine the transfer configuration. experiments, we found that, compared to training with pre-determnned static configuration, LASEM requires only 30-50% additional wall-clock time to search over 16 configurations of a network with four layers, and only double the time to search over 64 configurations of network with six layers. In
timing

---

Under review as a conference paper at ICLR 2021
Transfer-basee Frequency (96
Tramsfer-basee Frequency (%)
80
80
60
60
40
40
20
LASEM HPS
20
LASEM HPS
LASEM TF A ALASEM DF-CNN
LASEM TF LASEM DF-CNN
layer1
layer2
layer3
layer4
layer1
layer2
layer3
layer4
(a) CIFAR- 100
(b) Office-Home Home
Figure Frequency of each layer being transfer-based according the selection of LASEM. Generally, upper layers are preferable for transfer, but there are exceptions, i.e. HPS on CIFAR-100.
Forgetting Ratio
Time (k sec)
Selective Sharing
Accuracy($5
-LASEM DF-CNN (Priar -LASEM DF-CNN
0.61
DEN
48.00 0.60
0.28 0.01
55.9
APD-Net
59.58 0.45
0,83
0.03
21.5
0.6
ProgNN
60.03 0.45
1,00
0.00
96.7
DARTS HPS
45.64 1.20
0.70 0.07
43.8
0.59
DARTS DF-CNN
56.77 1 0.49
0.35 0.04
33.2
0.58
LASEM HPS
58.44
0,81 0.08
70.2
0.6
0.8
1
LASEM TF
59. + 0.80
0,90 0.04
77.3
Probability of Top 2 Transfer
LASEM DF-CNN
59.45 1.10
0.98 0.01
83.2
Figure 5: Performance of LASEM DF- CNN compared LASEM with fixed posterior distribution over the optimal configuration (on Office-Home).
Table 1: Comparison of peak per-task accuracy, forgetting, and training time for the same epochs between basclines and LASEM Office-Home, + 95% confidence interval.
stark contrast, brute-force search over the transfer configurations requires 15x and 63 additional time per task, respectively, over the base learner (see Appendix C).
The ratio of transfer configurations chosen by LASEM are depicted in Figure see Appendix for 4 detailed results. Figuresshwws shows the proportion of time each layer was chosen to be transfer-based see that HPS tends to prefer task-specific layers, while TF and DF-CNN are more likely to use transfer layers due to their ability to adapt transferred knowledge. We can also see trends among the chosen layers, such as DF-CNN preferring transfer among higher layers.
4.2 COMPARISON TO OTHER SELECTIVE TRANSFER ALGORITHMS
We compared the performance of LASEM Office-Homa against other methods that employ some notion of selective transfer, including the Dynamically Expandable Network (DEN) (Yoon et al. 2018), the Additive Parameter Decomposition Network (APD-Net) (Yoon al. 2020)) the Progressive Neural Net (ProgNN) Ruusu et al.) 2016), and Differentiable Architecture Search (DARTS) (Liu al. 2018). DEN is a lifelong learning architecture that extends HPS by expanding, splitting, and selectively retraining the network to introduce both shared and task-specific parameters in each layer required. APD-Net has base parameters shared across tasks like HPS, but introduces task seek-specific masks and additive parameters for adaptation to each task. ProgNN learns lateral connections from earlier task models to the current task model. Both DEN and ProgNN can support complex transfer configurations due to their lack of constraints, such as assumption of tree-structured configuration. For example, a ProgNN with all zero-weigig weighted lateral connections for a level creates a task- -asc-fpcc layer, and reig-weighted current task model connections creates transfer-based layer. DARTS another general framework for neural architecture search, determining both the most suitable operation of each layer and the best architecture of stacking these layers simultaneously.
Table D summarizes the performance of these methods and our approach: mean peak per-task accuracy, catastrophic forgetting ratio and training time. APD-Net, ProgNN and LASEM DF-CNN
7

---

Under review as a conference paper at ICLR 2021
statistically indistinguishable and perform better than the other methods in terms of accuracy. However, APD-Net is weaker in retaining the knowledge of earlier tasks, as shown by its catastrophic forgetting ratio being significantly lower than ProgNN and LASEM DF-CNN. LASEM DF-CNN is ~ -44 faster than ProgNN, whose time complexity is proportional to the square of the number of tasks. DEN and DARTS have better training times, but fail to perform well. Note that LASEM shows high accuracy regardless of the base lifelong learner (c.g.. HPS, TF, or DF-CNN) while introducing relatively little additional time complexity (~30-50% over the base learner's 's time).
4.3 THE EFFECT OF NON-OPTIMAL TRANSFER CONFIGURATION ON LASEM
Besides the capability to customize the transfer configuration to each task, LASEM has a key difference from using a static transfer configuration as in Section LASEM updates both the transfer- based and task-specific parameters (0,, and 0,) by gradients backpropagated from the loss of all configurations weighted by the posterior. Consequently, gradients from non-optimal configurations might act noise or be counterproductive to the optimization process.
To determine whether this aspect had a significant effect on LASEM, we performed an ablative experiment using static probability on the transfer configurations, instead of the posterior derived from data. This makes LASEM always select the same transfer configuration for all task models, with the experiment varying the noise and controlling the amount of adverse effects from non-optimal configurations during LASEM's optimization.
We first determined the optimal transfer configuration (Top 2 in this experiment), and gave it static probability of selection, which we varied from P = 0.5 to with a uniform distribution over other configurations. Figure S compares the full LASEM DF-CNN against the ablated version. Knowing the correct transfer configuration a priori (when P = 1) certainly does improve performance, as we would expect, but the overall performance difference as P varies is relatively small. Therefore, the effect of interference from considering non- non-optimal configurations is minimal, but does exist. Using a more informed prior over the transfer configurations, such as initializing it from the posteriors from previously learned related tasks, may further improve LASEM, which we leave to future work.
5 RELATED WORK
The simplest transfer mechanism is hard parameter sharing (HPS), which directly reuses parameters (e.g., layers) between task models (Caruana 1997). HPS is beneficial when tasks share identical features, but its structural rigidity degenerates tasks become diverse. Constraints such regular-
ization (Kirkpatrick ct al. 2017 Yoon et al.) 2018 He et al. 2018), orthogonality (Suteu & Guo
2019 Riemer et al., 2019 Farajtabar et al. (019) attention mechanisms (Serra et 2018| Yoon
et al.
Abati et al.] (2000) may reduce interference among tasks, but can deter positive transfer.
Using tree-like structures (Lu al., 2017: Vandenhende al., 2019 He al. 2018 the transfer configuration for HPS in multi-task nets give flexibility, but assume that lower level representations shared, which may not be the case for diverse tasks, as shown in this paper.
Soft parameter sharing (Duong et al., 2015: Bilen & Vedaldi, 2017) builds task-specific networks with weights that are related other task models via implicit constraints. This architecture provides flcxibility to the representations that each task network can learn, so it typically outperforms HPS for more diverse tasks. Success has often been found by using task- aask-agnostic shared knowledge with a lask-specific mapping from that shared knowledge to the task models, facilitating transfer between tasks (Yang Hospedales) 2017] [ulat et al. 2020 Lee et al., 2019 Liu et al. 2019). These works focus on the mapping operation, but put less importance on what layers to transfer, as we explored.
Direct reuse of learned representations from previous tasks models (Rusu et al.] 2016 Misra et al., 2016, Cao et al.) 2018 Xiao et al. 2018) prevents forgetting, but only permits forward transfer to new tasks (not reverse transfer) and exhibits super-linear training time w.r.t the number of tasks. Progress and compress (Schwarz et al. 2018) tackles this issue by combining progressive neural nets and elastic weight consolidation (Kirkpatrick et al., 2017), but this method has a similar capacity issue of HPS for diverse tasks since neural net must handle all learned tasks.
Neural architecture search (NAS) examines both the operators and their order in a neural net to optimize performance (Elsken et al.] 2018). Our problem of selective layer-based transfer is an
8

---

Under review as a conference paper at ICLR 2021
instance of NAS. Strategies for NAS include reinforcement learning (Tan al. 2019 Chang et al. 2019), evolutionary algorithms (Fernando et al. (217)) and gradient- based learning (Alet et al. 2018). In contrast to these methods, DARTS makes optimization more feasible by using soft selection of the operators, a weighted sum of operations. Most NAS methods including DARTS train better once the architecture has stabilized, but the weights of DARTS' soft selections susceptible vanishing gradients, so it is slower to stabilize than LASEM.
CONCLUSION
We have shown that the transfer configuration can have a significant impact on lifelong learning, and that the configuration can be dynamically selected during the lifelong learning process with minimal computational cost. Choosing the optimal transfer configuration significantly improves the performance of the DF-CNN and TF over the original method. Using dynamic transfer configuration reduces the assumptions of algorithm designers in terms of task similarities, but opens potential for the selected configurations to incorporate biases from data.
Although we focused on layer-based transfer, LASEM could easily be extended to support partial transfer within a layer by imposing within-layer partitions and redefining the transfer configuration space C' to support those partitions. Discovering these partitions directly from data, or providing more flexible mechanisms for partial within-layer transfer may further improve performance.
ACKNOWLEDGMENTS
Omitted for blind review.
REFERENCES
Davide Abati, Jakub Tomezak, Tijmen Blankevoort, Simone Calderara, Rita Cucchiara, and Babak Ehteshami Bejnordi. Conditional channel gated networks for task-aware continual learning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2020.
Ferran Alet, Tomas Lozeno-Perez, and Leslie P. Kaelbling. Modular meta-learning. In Aude Billard, Anca Dragan, Jan Peters, and Jun Morimoto (eds.), Proceedings The The 2nd Conference on Robot Learming, volume 87 of Proceedings of Machine Learning Research, pp. 856-868. PMLR, 2018.
URL http::::proceeeiigg....t mlr, www.s//////aaee111... 88.
Hakan Bilen and Andrea Vedaldi. Universal representations: the missing link between faces, text, planktons, and cat breeds. CoRR, abs/1701.07225, 2017. URL http:://aarriioorggab/////0.01.0
Adrian Bulat, Jean Kossaifi, Georgios Tziniropoulos, and Maja Pantic. Incremental multi-domain learning with network latent tensor factorization. In Proceedings of the AAAI Conference on Artificial Intelligence. AAAI Press, 2020. URL httpps://www..drraamatt http://wwwiabbuut..ooooooooooooo AAAl20/incremental mwwi muli-domaiin..mmmmmm tensor factorization.pdl
Jiajiong Cao, Yingming Li, and Zhongfei Zhang. Partially shared multi-task convolutional neural network with local constraint for face attribute learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4290-4299, 2018.
Rich Caruana. Multitask learning. Machine learning, 28(1 28(1):41-75, 1997.
Michael Chang, Abhishek Gupta, Sergey Levine, and Thomas L.. Griffiths. Automatically composing representation transformations a means for generalization. In Proceedings of the Interna- tional Conference on Learning Representations, 2019. URL https::/ppenneeiie.ettffouu ?id= B1ffQnRckX
Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the International Conference on Artificial Intelligence and Statistics, pp. 215-223, 2011.

---

Under review as a conference paper at ICLR 2021
Long Duong, Trevor Cohn, Steven Bird, and Paul Cook. Low resource dependency parsing: Cross- lingual parameter sharing in a neural network parser. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference Natural Language Processing, volume 2, pp. 845-850, 2015. URL. httpps:/wwww.cclebbborg anthology/P15-22139
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. CoRR, abs/1808.0555 0038005377, 2018. URL http:hh://aariioorg/aa///888888 05377
Mchrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Onthogonal gradient descent for continual learning. CoRR, /tt//99000010000 104, 2019. URL httppp//aaxxx.orggbbb/1100....00
Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A. Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. CoRR, abs/1701.08734, 2017. URL http:////aaiiiorr////////000.000
Xiaoxi He, Zimu Zhou, and Lothar Thiele. Multi-task zipping via layer-wise neuron sharing. In Advances in Neural Information Processing Systems. pp. 6016-6026, 2018.
David Isele, Mohammad Rostami, and Eric Eaton. Using task features for zero-shot knowledge transfer in lifelong learning. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence. pp. 1626, 2016.
James Kirkpatrick, Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, Guillaume Desjardins, and Andrei A. Rusu, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, (14(13):35213335 2017.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of Technical report, University of Toronto, 2009.
features from tiny images.
Seungwon Lee, James Stokes, and Eric Eaton. Learning shared knowledge for deep lifelong learning using deconvolutional networks. In Proceedings of the Twenty-Eighth International Joint Conference Artificial Intelligence, pp. 2837-2844, 2019. URL httpss///ool.org//0222433 jjcai.2019/993
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: differentiable architecture search. CoRR, abs/1 005.09006.9 2018. URL http::/arrxivoorgaabb/ 09055
Huaping Liu, Fuchun Sun, and Bin Fang. Lifelong learning for heterogeneous multi-modat tasks. In Proceedings of the International Conference on Robotics and Automation, pp. 6158-6164. IEEE, 2019.
Yongxi Lu, Abhishek Kumar, Shuangfei Zhai, Yu Cheng, Tara Javidi, and Rogério Feris. Fully- adaptive feature sharing in multi-task networks with applications in person attribute classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.
Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-stitch networks for multi-task learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3994 4003, 2016.
Hieu Pham, Melody Y. Guan, Barret Zoph, Quoc V. Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. In Proceedings of the International Conference Machine Learning, 2018.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. In Proceedings of the International Conference on Learning Representations, 2019.
Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. CoRR, Abs/1606.04677 2016. URL
10

---

Under review as a conference paper at ICLR 2021
Jonathan Schwarz, Wojciech Czarnecki, Jelena Luketina, Agnieszka Baabska-Barwinska, Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress & compress: A scalable framework for continual learning. In Proceedings of the International Conference Machine Learning, pp. 4528-4488 -4284,57,, 2018.
Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention the task. In Proceedings of the International Conference on Machine Learning, pp.
2018.
Jivko Sinapov, Sanmit Narvekar, Matteo Leonetti, and Peter Stone. Learning inter-task transferability in the absence of target task samples. Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems, pp. 725-733, 2015.
Mihai Suteu and Yike Guo. Regularizing deep mulki-task networks using orthogonal gradients. CoRR, abs/1912.068444, 2019, URL http://awwi.ooggaab//1112..
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoe V Le. Mnasnet: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2820 2828, 2019.
Simon Vandenhende, Bert De Brabandere, and Luc Van Gool. Branched multi networks: deciding what layers to share. CoRR, 0bbs/1994.02920, 2019. URL http://axxi.oorggbbb/9900002222
Hemanth Venkateswara, Jose Fusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.
Liqiang Xiao, Honglun Zhang, Wenqing Chen, Yongkun Wang, and Yaohui Jin. Learning what to share: leaky multi-task network for text classification. In Proceedings of the 27th International Conference on Computational Linguistics, pp. 2055-2065, 2018
Yongxin Yang and Timothy Hospedales. Deep multi-tase task representation learning: a tensor factori- sation approach. In Proceedings of the International Conference on Learning Representations, 2017.
Jachong Yoon, Eunho Yang, and Sungju Hwang. Lifelong learning with dynamically expandable networks. In Proceedings of the International Conference on Learning Representations, 2018.
Jaehong Yoon, Saehoon Kim, Eunho Yang, and Sungju Hwang. Scalable and order-robust continual learning with additive parameter decomposition. In Proceedings of the International Conference Learning Representations, 2020.

---

Under review as a conference paper at ICLR 2021
Supplementary Material for ICLR 2021 Submission Sharing Less is More: Lifelong Learning in Deep Networks with Selective Layer Transfer
A EXPERIMENT DETAILS
This section provides detail on the experiments from the main paper. The experiments are based on three image recognition datasets: CIFAR-100 (Krizhevsky & Hinton 2009). Office-Home (Venkateswara et al. 2017) and STL-10 (Coates et al. 2011).
CIFAR- 100 consists of images of 100 classes. The lifelong learning tasks created following Lee et al. 2019) by separating the dataset into ten disjoint sets of ten classes, and randomly selecting 4% of the original training data generate training and validation sets in the ratio of 5.6:1 (170 training and 30 validation instances per task). The images are used after normalization.
The Office-Home dataset has images of 65 classes in four domains. Again following Lee al., lifelong learning tasks are generated by choosing ten disjoint groups of thirteen classes in two domains: Product and Real-World, There is no pre-defined training/testing split in offie-HHme,, so we randomly split the images in the ratio 6:3:3 for the training, validation, and test sets. The image sizes are not uniform, so we resized all images to 128-by-128 128 pixels and re-scaled each pixel value to the range of [0, 1].
We introduce a lifelong learning variant of the STL-10 dataset, which contains ten classes. We constructed 20 three-way classification tasks by randomly choosing the classes, applying Gaussian noise to the images (with a mean and variance randomly sampled from {- 10%, -5, 0%, 5%, 10%) of the range of pixel values) after re-scaling each pixel value to the range of [-0.5, 0.5]. and randomly swapping channels. Note that any pair of tasks differs by at least one image class, the mean and variance of the Gaussian noise, the order of channels for the swap. We sampled 25% of the given training data and split it into training and validation sets with the ratio 5.7:1 (318 training and 57 validation instances per task). All of the original STL--10 test data used for held-out evaluation of performance.
The architectural details of the task models used for each data set are described in Figure 6 We used the following values for the hyper-parametee of the algorithms, following the original papers wherever possible:
The multi-task CNN with hard parameter sharing (HPS) has no additional hyper-parameters.
Tensor factorization has scale of the weight orthogonality constraint, whose value was chosen by grid search among {0.001, 0.005, 0.01, 0.05, 0.1} following the original paper Bulltt et al. 2020).
Dataset
CIFAR-100
Office-Home 10
STL-10
Number of Tasks
10
20
Semi-heterogeneous
Type of Task
Heterogeneous Classification
Classification
Classes per Task
10
10
Amount of Training Data
4%
25%
Ratio of Training and Validation Set
5.6:1
6:1
5.7:1
Size of Image
32 32
128 x 128
96 96
Optimizer
RMS Prop
Learning Rate
1 1x 10-4
2 x 10
1 x 10-4
Epoch per Task
2000
1000
500
Ratio of M-steps to E-step (num.MSteps)
Table 2: Parameters of the lifelong learning experiments. We used the notion of task type (c.g. heterogeneous) introduced in previous work of mulli--ask learning (Yang & Hospedales] 2017). which based on the similarity of data distribution of tasks.
12

---

Under review as a conference paper at ICLR 2021
100,
sride Belll
sorie RoLL
taLU
part
mas
2X3
328322022
B0010322
30KI0004
10
Architecture of models CIFAR- experiment
31X13,
300, 226
258 stride Fel.1
Fride
strdee
stride PuLUU
BaLL
pool
(oux pool
883
3x3
2X2
2X2
(b) Architecture of task models of Office-Home experiment
Camy
Come
Rasten
fra SK3
fracupoo
4230003
0t2900122
320122611
11XU3961
213118128
1043
128
(c) Architecture of task models of STL-10 experiment
Figure 6: Details of the task model architectures used in the experiments. Text by each convolutional layer describes the filter sizes and the number of channels. All convolutional layers are zero-padded.
DF-CNN requires the size of the shared tensors and the parameters of the task-specific mappings to be specified. Following the original paper (Lee al. 2019), we chose the spatial size the shared tensors to be half the spatial size of the convolutional filters, and the spatial size of the deconvolutional filters 3 For each convolutional layer with input channels Cin and output channels Cout, the number channels in the shared tensors one-third of Cin. Cout and the number of output channels of the deconvolutional filters was two-thirds of Cin Cout-
DEN has several regularization terms and the size of the dynamic expansion. We used the regularization values the authors' published code, and set the size of the dynamic expansion to be 32 by choosing the most favorable value among {8, 12, 64).
APD-Net has two regularization terms for the sparsity of additive parameters A and catas- trophic forgetting λ2 As described in the original paper (Yoon et al. 2020), used 4e-1 and 100 as the value of λ₁ and X2, respectively.
ProgNN requires the compression ratio the lateral connections, which we set to be following the original paper (Rusu et al. 2016).
For DARTS, used the hyper-parameter settings described in the original paper (Liu et al. 2018).
A lifelong learner has access to the training data of only the current task, and it optimizes the parameters of the current task model as well as any shared knowledge, depending on the algorithm. After the pre-determined number of training epochs, the task switches to a new one regardless of the convergence of the lifelong learner, which favors learners that can rapidly adapt to each task. When the learner encounters new task, it initializes newly introduced parameters of the new task model, but re-uses the parameters of shared components, which initialize only once at the beginning of the first task. As mentioned earlier, these new task-specific parameters and shared parameters optimized according to the training data of the new task for another batch of training epochs. We used the RMSProp optimizer with the hyper-parameter values (such as learning rate and the number of
13

---

Under review as a conference paper at ICLR 2021
training epochs per task) described in 2 In experiments, conservatively have LASEM take a single M-step per E-step by setting numMSteps 1.
B CATASTROPHIC FORGETTING OF LASEM
investigated catastrophic forgetting of LASEM in addition to mean peak per-task accuracy. The catastrophic forgetting ratio is shown in Figure 7 The catastrophic forgetting ratio, proposed in Lee ctt al. (2019), measures the ability of the lifelong learning algorithm to maintain its performance on previous tasks during subsequent learning. A low ratio indicates that there is negative reverse transfer from new tasks to previously learned tasks, and so the learner experiences catastrophic forgetting. A ratio greater than can be interpreted positive backward transfer. As depicted in Figure 7 LASEM is able to retain the performance of previous tasks compared to transferring at all CNN layers and transferring specific CNN-layers for all tasks (using static transfer configuration).
Calastrophic Forgetinn Ratio
Catostrophic Forgetting Ratio
Catastrophic Forgetting
0.9
0.9
0.9
0.8
0.8
0.8
0.7
0.7
0.7
0.6
HPS all
0.6
TF
0.6
DF CNN all
..HPS best accuracy
-If best accurscy
DD CNN best accuracy
0.5
LASEM HPS
0.5
LASEM
0.5
USEM DF-ONN
6
10
2
10
Task Number
Task Number
Task Number
HIP CIFAR-100
(b) TF on CIFAR-100
(c) DF-CNN on CIFAR-100
Calastrophit Forgetling Raião
aatgorophic Forgetting Ratio
Catastrophic Forgetting Ratio
0.9
0.9
0.9
0.8
0.8
0.8
0.7
HPS all
0.7
TF all
0.7
OF-ONN all
HHS best accuraty
-TF best accuracy
-DF-CNN best accuracy
0.6
ASFM HP5
0.6
ASEM TF
0.6
- ASEM DF-CNN
6
10
10
10
Task Number
Task Number
Task Number
(d) HPS on OfficeHHome
(e) TF Office-Home
(1 DF-CNN on Office-Homee
Catastrophic Torqetiin Rabo
1.05
Catostrophic Formetting Rot
1.05
0.95
0.95
0.85
TF all
0.85
DF-CNN all
..TF best accuracy
.. best accuracy
0.8
LASEM TF
0.8
LASEM DF-CNN
10
15
20
10
15
20
Task Number
Task Number
(g) on STL-10
(h) DF-CNN STL-
Figure 7: Catastrophic forgetting ratio of transfer at all CNN layers (blue), best static transfer configuration (black) and LASEM (red), exhibiting the benefit of LASEM. Note that the y-axis range differs for each data set.
C COMPARISON TO BRUTE-FORCE CONFIGURATION SEARCH
We compared LASEM to the performance of task-wise brute-force search over transfer configurations, shown in Table 3 Here, "task-wise brute-force search over transfer configurations" refers to the method of training every transfer configuration and choosing the best configuration for each task,
14

---

Under review as a conference paper at ICLR 2021
LASEM Accuracy (%)
Transfer All
Layers
Brute-force Search
Architecture
Accuracy (%)
Rel. Time
Accuracy (%) Rel. Time
CIFAR-100 (10 Tasks)
HPS
39.3 0.1
24.7 = 0.6
0.78
40.4 0.3
6.55
TF
38.4 + 0.5
36.3 + 1.0
0.64
39.9 + 1.1
8.81
DF-CNN
42.0 + 0.6
36.3 + 1.3
0.59
42.6 + 0.7
9.45
Office-Home (10 Tasks)
HPS
58.4 + 0.9
54.9 + 0.7
0.72
59,4 + 0.2
4.72
TF
59.1 + 1.0
56.2 + 0.7
0.66
58.7 + 0.3
5.22
DF-CNN
59.5 = 1.1
49.1 = 0.6
0.61
58.8 0.3
4.04
Table 3: Comparison of test accuracy and training time for the same epochs to brute-force config- uration search, + 95% confidence interval. Training time shown above is ratio to training time of LASEM. Relative time greater than 1.0 means that training the model is slower than our approach.
thereby allowing task models to use different transfer configurations. The accuracy of brute-force search is almost indistinguishable from LASEM at 95% confidence, but it requires at least 3 additional time for training. This result shows that LASEM achieves approximately the best accuracy of dynamic transfer configuration with boost in training speed.
D LASEM- TRANSFER CONFIGURATIONS
Figure shows the most frequent transfer configurations as well as the proportion of the time each layer was chosen to be transfer-based or task specifie...... For CIFAR-100 and Hffi,,-Home, there is tendency of transferring top layers more than bottom layers. However, interesting observation is that non-tree structures, such as Alternating (2, 4| and sharing middle layers [0,1,1,0], are often chosen. This contradicts the assumption of structure made often by related research, and supports the consideration of more complex transfer configurations for diverse tasks.
The top eight most-chosen configurations of STL-10, unlike the other datasets which employed deep nets with fewer CNN layers, plateau with a peak less than 10%. This is likely due to the smaller number of STL-10 tasks, more flexible (deeper) network, and much larger number of possible transfer configurations than in the other two experiments. The tensor factorization model for STL-110 seems prefer transfer at higher layers transfer more than lower layers, while the preference for DF-CNN is more varied.
E LASEM FOR TRANSFER OVER GROUPS OF LAYERS
As discussed Section is a well-known problem in neural architecture search that the search over layer-based transfer configurations requires time exponential to depth of the network d. One common technique to compensate for this problem, as used by other methods (Pham et al.) 2018 Liu et al. 2018), search over groups of layers instead of individual layers, thereby reducing the size of the search space. LASEM easily supports this same technique by redefining the transfer configuration space {0, 1)dd 4 be binary indicators over a partition P of the set of layer indices {1,.... .d., where the cardinality [P] < Consequently, this reduces the search space from 2d 2/Pl, Most naturally, the partition P should ensure that either adjacent layers (e.g., {11, 2}, {3, 4}, {5, 6) or nearby layers (e.g. f],,3), , (2, 4), (5), {6)}) are grouped together.
We evaluated this variation of LASEM in lifelong learning scenarios using the STL-10 10 data set. Different from the aforementioned experiments using STL-10, this experiment consisted of 15 five- way classification tasks by random selection of the classes. We sampled only 10% of the given training data and split it into training and validation sets with the ratio 1:1. For this scenario, we trained DF-CNN transferring all layers and lased LASEM on DF-CNN with 9 convolutional layers. The gased LASEM used the partition {{1, 2, 3), (4, 5, 6). {7, 8, 9}}, splitting convolutional layers into three groups of three adjacent layers.
'Thi pattern of grouping nearby layers together, instead of only adjacent, may allow more flexible adaptation of transferred knowledge individual tasks, similar the Alternating configuration explored in Section
15

---

Under review as a conference paper at ICLR 2021
Tarrsfrr Configraite
70
Confegue
Contggu
16
10
100
1000
0001
1001
0000
0101
0011
0001
0100
0010
1011
0101
TTarsfer based
lask: Specific
Whhnnfenbased
Shosk-Spectff
@TTansfe-Swsed
Shatk-Speeffg ftc
100
100
100
Layeruise Trarsder the
Toanfer (6)
Tranofer
80
40
40
1eyer2
1aysr4
0-
0-
(a) HPS on CIFAR-100
(b) on CIFAR-100
(c) DF-CNN on CIFAR-100
20-
20
Trandfe
10
10
0011
0001
0101
0010 1011
1111
0101
0110
1001
0111
0011
0110
0101
0001
Translorbaa based
Tisk Specifl:
Wansferbaaa
sisck-Spccfic
TTnnfe based
WlskkSSaccff
100
100
100
Layurwee Transllrr (8)
80-
Layerwize Trarsfer
80
80
80
40
Layerwbe
40
(d) HPS on Office-Hoone Home
(e) Office-Home
(f) DF-CNN on Office-
Tanster confiurations (i
6
Tanster
2
Transfer-based
Task-Specific
Transfer-based
Task-Specific
100
100
Layerwise Transfer (%)
80
80
60
60
Layervise
20
20
layer1 layer2 layer3 layer4 layer5 layer6
layer1 layer2 layer3 layer4 layers layer6
(g) TF on STL-10
(h) DF-CNN on STL-10
Figure 8: (Top) Histogram of the most-selected configurations (i.e., the binary vectors Cr where denotes that CNN layer employs transfer). (Bottom) The fraction of the time each layer was selected to transfer-based (red) or task-specific (blue).
16

---

Under review as a conference paper at ICLR 2021
The DF-CNN achieved mean task wise best accuracy of 45.4#0.4 with training time of 7.58 10* seconds. The layer-based LASEM DF-CNN exceeded the capacity of our computing source (an Intel core 17 workstation with dual 1080 Ti GPUs). However, the group-based LASEM DF-CNN achieved a mean accuracy of 46.0 + 0.7 % in 7.02 x 10 seconds. This showcases LASEM's ability to support group-based transfer configurations in addition to layer-based configurations.
F MEMORY USAGE COMPARISON
In this section, analyze LASEM's memory requirements to show that is approximately equivalent to the other methods considered in the paper. Let the base learner require ((A) non-transfer- based task-specific storage or O(B) transfer-based task-specific storage with ( shared knowledge. LASEM shares network parameters across transfer configurations to minimize memory, so the current task model stores two parameter sets cost of O B). Earlier task models require only O(max A, B)) storage, yielding total memory requirement of O(S + (T + 1)maaz( A, B)) for T tasks when the base learner constructs one network per task. Compared this, the model with the best static transfer configuration requires ((mar( A, B)) additional storage per task, resulting in total memory requirement of O(S + maz(A, B)). Brute-force search over transfer configurations requires at least O2 max( A, B)), one for parameters of the best configuration and the other for parameters the current training, yielding O + (T + 1)max((A, (A, B)). Hence, LASEM's memory requirements approximately equivalent the alternative methods. The memory requirement may differ from the above analysis according to the base lifelong learning architecture used (such as HPS modular network), but if that base learner requires ((T) memory for T tasks, LASEM needs only O(T + 1) memory.
17


Reviews:
[{"review_id": "pMwgJk0FbMB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a lifelong learning algorithm across multiple tasks that automatically learns which layers need to be optimized using an EM learning strategy for each task. In the expectation step, the algorithm updates the next best configuration and in the M step, it optimizes the model parameters.\n\nSome details of the algorithm are not explained well. It could help a lot if the authors could provide the objective function they are trying to optimize. As examples: (1) The motivation behind Equation 3 is not clear at all. Based on Equation 3, It seems that the choice of different configurations is mini-batch dependent and not task dependent. I have a difficult time to understand why this is the case. I might have misunderstood the algorithm but providing the objective function helps a lot. (2) It is not clear why n_{c_i} in Equation 1 is the number of previous mini-batches for which C_{(t)} is the most probable configuration. It makes sense to use a soft version of this (sum of probabilities) to have it more compatible with the rest of the algorithm. Again, understanding what objective function the authors are targeting to optimize helps to understand this better too.\n\nEven though, some aspect of the proposed algorithm is not clear, the algorithm works in practice and outperform multiple suggested baselines. \n\nMinor type: Page 5: 5% percent  5%\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZJjlqIYyplh", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: authors look into life long learning setting (when task arrive one after another) and try to understand which layers of the source model need to be reused (transfered) and which should be re-learnt. Authors argue that this decision should be task specific, and come up with an algorithm (EM like) that for each task derives indices of the layers to reuse and also updates reusable and non reusable layers.\n\nOverall, this paper is really well executed. Easy to follow, enough of background and motivation is provided, and the algorithm for the most part is clear and intuitive.\nMy main complaint is that it is somewhat thin on the experiments.\n\nAdditional questions/comments:\n1) How is update 13 from Algo 1 happening? (how P(C|data) is updated? is it  working by recalculating the counts from step (1) (so it is not a smooth function that is differentiable and updated via optimization)\n2) In Algo 1: while IsMoreTrainingDataAvailable is not clear to me. I assume that you do several epochs over task specific data, don't you? Also do you monitor anyhow EM convergence?\n3) Experiments: One baseline would be useful to just randomly select c_t for each task and see how it does. EM is pretty expensive\n4) Finally, I feel that related work should really go after the into", "extra_scores": {}, "extra_text": {}}, {"review_id": "Z6LiMXoKarF", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "-Summary-\nThe paper proposes a method for selective weight sharing per layer during continual learning. The authors show observations that sharing all layers can not be optimal for lifelong learning. Hence, they adopt a layerwise transfer configuration vector which decides activated layer-sharing at specific tasks. The problem is solved by EM algorithm-based approach. \n\n-Pros-\n- Observations are reasonable and give many inspirations for solving continual learning issues and developing the existing methods.\n- The paper is well written and easy to follow.\n- The point of view connected to neural architecture search is understandable.\n- The model outperforms old baselines.\n\n-Cons-\n- The problem is task-incremental which clearly gives task oracle during training and inference. Recent continual learning works obviously tackle class-incremental learning problems that are more challenging and applicable to a broader area [1]. \n- Baselines are too weak. If the paper targets task-incremental learning problems, the authors should compare their methods with recent works, rather than with 3-5 past years' works. I recommend to include further strong baselines like [2,3,4].\n- The strengths of the methods may not be impressive on modern deeper networks, like ResNet-50. Since it requires massive computation time. The paper didn't show the results/analysis of modern deep architectures.\n- The model inevitably requires much capacity for not-shared task-specific layers. But the authors didn't include it.\n\n-Comments-\n- Why do the authors only use a fraction of datasets?\n\n\n===============\n- [1] van de Ven, Gido M., and Andreas S. Tolias. \"Three scenarios for continual learning.\" arXiv preprint arXiv:1904.07734 (2019).\n- [2] Titsias, Michalis K., et al. \"Functional Regularisation for Continual Learning with Gaussian Processes.\", ICLR 2020.\n- [3] Yoon, Jaehong, et al. \"Scalable and Order-robust Continual Learning with Additive Parameter Decomposition.\", ICLR 2020.\n- [4] Davide Abati, et al.  \"Conditional Channel Gated Networks for Task-Aware Continual Learning.\", CVPR 2020.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "4pH4DkhvrxI", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the problem of lifelong learning of a sequence of tasks by selectively transferring knowledge of some layers across tasks. The proposed approach, LASEM, uses EM to dynamically adjust transfer configuration between tasks by performing architecture search. The authors present results in benchmark datasets for continual learning. \n\nAs strengths of the paper I would remark: \n-\tThe paper is well-motivated and demonstrates experimentally why simply transferring all layers does not lead to the best transfer configuration. \n-\tThe proposed approach in section 3 is clearly explained and the main design choices are well argued. In general, the paper is well-written and easy to follow. \n-\tThe paper includes thorough results of the computational complexity of the proposed method, which could be a concern given the proposed approach of maintaining both shared and specific sets of parameters for each task. \n\nThe weaknesses I observe in this paper are: \n-\tI am concerned on the extensibility of the proposed method to a large number of tasks, which I would think would pose a challenge in the proposed setting similar to how it does in multitask learning. What would be the impact of a large number of tasks, specifically on the set of shared parameters? The experiments provide results for a limited number of tasks (up to 20).\n\nQuestions for authors: \n- Please address the question regarding the effect of number of tasks on the proposed approach.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers enjoyed reading about an interesting take on lifelong learning, encapsulating an EM methodology for selecting a transfer configuration and then optimizing the parameters. R3 made valid concerns regarding comparison with previous, recent work. R2 also would prefer to see more thorough experiments (ideally in settings where multiple tasks exist, as also commented by R4). During the rebuttal phase the authors made a good effort to run additional experiments which cover the related work aspect better. These experiments and the overall paper were discussed extensively among reviewers after the rebuttal phase.

In the discussions, the reviewers agreed that an interesting idea can be publishable even if it does not achieve SOTA results in all scenarios, as long as it brings new perspectives and shows at least comparable results. However, in the particular case of this paper, there exist remaining concerns regarding the usefulness and applicability of the method. Specifically, the paper could benefit from a more convincing demonstration about how the method can scale (e.g. R3 and R4’s comments), especially since training time and model capacity are important factors to consider for practical continual learning scenarios. Furthermore, it is not clear how the proposed method can be used in combination with other machine learning tools within a continual learning application, for example by leveraging modern deep architectures or by complementing existing adaptive knowledge approaches (as discussed by R3). 

Although the opinions of the reviewers are not fully aligned, this borderline paper seemed to lack an enthusiastic endorsement by a reviewer to compensate for the concerns discussed above and the relatively weak experimental results. Therefore I recommend rejection. 

Decision: Reject