An Evolutionary Approach to Dynamic Introduction of Tasks in Large-scale Multitask Learning Systems
Authors: Andrea Gesmundo
Jeff Dean
Venue: Submitted to ICLR 2023 (2023)

Abstract: Multitask learning assumes that models capable of learning from multiple tasks can achieve better quality and efficiency via knowledge transfer, a key feature of human learning. Though, state of the art ML models rely on high customization for each task and leverage size and data scale rather than scaling the number of tasks. Also, continual learning, that adds the temporal aspect to multitask, is often focused to the study of common pitfalls such as catastrophic forgetting instead of being studied at a large scale as a critical component to build the next generation artificial intelligence.
We propose an evolutionary method capable of generating large scale multitask models that support the dynamic addition of new tasks. The generated multitask models are sparsely activated and integrates a task-based routing that guarantees bounded compute cost and fewer added parameters per task as the model expands. The proposed method relies on a knowledge compartmentalization technique to achieve immunity against catastrophic forgetting and other common pitfalls such as gradient interference and negative transfer. We demonstrate empirically that the  proposed  method  can  jointly  solve  and  achieve  competitive  results  on  69 public image classification tasks, for example improving the state of the art on a competitive benchmark such as cifar10 by achieving a 15% relative error reduction compared to the best model trained on public data.

Under review as a conference paper at ICLR 2023
AN EVOLUTIONARY APPROACH TO DYNAMIC INTRODUCTION OF TASKS IN LARGE-SCALE MULTITASK LEARNING SYSTEMS
Anonymous authors Paper under double-blind review
ABSTRACT
Multitask learning assumes that models capable of learning from multiple tasks achieve better quality and efficiency via knowledge transfer, a key feature of human learning. Though, state of the art ML models rely on high customization for each task and leverage size and data scale rather than scaling the number of tasks. Also, continual learning, that adds the temporal aspect to multitask, often focused to the study of common pitfalls such as catastrophic forgetting instead of being studied at a large scale as a critical component to build the next generation artificial intelligence. We propose an evolutionary method capable of generating large scale multitask models that support the dynamic addition new tasks. The generated multitask models sparsely activated and integrates task-based routing that guarantees bounded compute cost and fewer added parameters per task as the model expands. knowledge compartmentalizztioo technique
The proposed method relies achieve immunity against catastrophic forgetting and other common pitfalls such gradient interference and negative transfer. We demonstrate empirically that the proposed method can jointly solve and achieve competitive results on 69 public image classification tasks, for example improving the state of the art on a competitive benchmark such as cifarl0 10 by achieving a 15% relative error reduction compared to the best model trained on public data.
INTRODUCTION
The success of machine learning continues to grow as it finds new applications in areas as diverse as language generation (Brown et al., 2020), visual art generation (Ramesh et al., 2021), chip design (Mirhoseini et al., 2020), protein folding (Senior et al., 2020) and competitive sports (Silver et al., 2016; Vinyals et al., 2019). The vast majority of machine learning models are designed and trained for a single task and specific data modality, and are often trained by starting with randomly initialized parameters, with limited knowledge transfer from pre-trained model. While this paradigm has shown great success, it uses large amount of computational resources, and does not leverage knowledge transfer from many related tasks in order to achieve higher performance and efficiency.
The work presented in this paper is based on the intuition that significant advances can be enabled by dynamic, continual learning approaches capable of achieving knowledge transfer across very large number of tasks. The method described in this paper can dynamically incorporate new tasks into a large running system, can leverage pieces of a sparse multitask ML model to achieve improved quality for new tasks, and can automatically share pieces of the model among related tasks. This method can enhance quality on each task, and also improve efficiency in terms of convergence time, amount of training examples, energy consumption and human engineering effort.
The ML problem framing proposed by this paper can be interpreted as a generalization and synthesis of the standard multitask and continual learning formalization, since an arbitrarily large set of tasks can be solved jointly. But also, over time, the set of tasks can be extended with a continuous stream of new tasks. Furthermore, it lifts the distinction between pretraining task and downstream task. As new tasks are incorporated, the system searches for how to combine the knowledge and representations already present in the system with new model capacity in order to achieve high quality

---

Under review as a conference paper at ICLR 2023
for each new task. Knowledge acquired and representations learned while solving new task are available for use by any future task or continued learning for existing tasks.
We refer to the proposed method "mutant multitask network" 22Net.. This method generates a large scale multitask network that jointly solves multiple tasks to achieve increased quality and efficiency for each. It can continuously expand the model by allowing the dynamic addition of new tasks. The more accumulated knowledge that is embedded into the system via learning on previous tasks, the higher quality the solutions are for subsequent tasks. Furthermore, new tasks can be solved with increasing efficiency in terms of reducing the newly-added parameters per task. The generated multitask model sparsely activated as integrates a task-based routing mechanism that guarantees bounded compute cost per task as the model expands. The knowledge learned from each task is compartmentalized in components that can be reused by multiple tasks. As demonstrated through experiments, this compartmentalizatioo technique avoids the common problems of multitask and continual learning models, such as catastrophic forgetting, gradient interference and negative transfer. The exploration of the space of task routes and identification of the subset of prior knowledge most relevant for each task is guided by an evolutionary algorithm designed to dynamically adjust the exploration//eplootttioo balance without need of manual tuning of meta- meta-parameter The same evolutionary logic is employed to dynamically tune the hyperparameters multitask model components.
RELATED WORK
The main novelty of the presented work is to propose and demonstrate a method that jointly provides all of the following properties: 1) ability to continually learn from an unbounded stream of tasks, 2) automate the selection and reuse of prior knowledge and representations learned for previous tasks in the solving of new tasks, 3) search the space of possible model architectures allowing the system to dynamically extend its capacity and structure without requiring random initialization, 4) automatically tune the hyperparameters of both the generated models and the evolutionary method, including the ability to learn schedules for each hyperparameter, rather than just constant values, 5) ability to optimize for any reward function, also including non-differentiable factors, 6) immunity from catastrophic forgetting, negative transfer and gradient interference, 7) ability to extend any pre-existing pre-trained model, including extending its architecture and adapting the domain on which such model have been trained to other domains automatically, 8) introduction of a flexible access control list mechanism that allows expression of variety of privacy policies, including allowing the use or influence of task-specific data to be constrained to just single task or to subset of tasks for which data or higher-level representation use should be permitted. Different lines of research have focused on distinct subsets of the many topics addressed by the proposed method. In this section we highlight a few cornerstone publications. Refer to Appendix A for an extended survey.
Different methods have been proposed to achieve dynamic architecture extensions (Chen et al., 2016; Cai et al., 2018), some also focusing on an unbounded stream of tasks (Yoon et al., 2018), or achieving immunity from catastrophic forgetting (Rusu et al., 2016; Li & Hoiem, 2018; Rosenfeld & Tsotsos, 2020). Unlike our work, these techniques rely on static heuristics and patterns to define the the structural extensions, rather than more open-ended learned search process. Neural architecture search (NAS) (Zoph& & Le, 2017) methods aim to modularize the architectural components in search spaces whose exploration can be automated with reinforcement learning or evolutionary approaches (Real et al., 2019; Maziarz et al., 2018). More efficient (but structurally constrained) parameter sharing NAS techniques (Pham et al., 2018; Liu al., 2019a) create a connection with routing methods (Fernando et al., 2017) and sparse activation techniques, that enable the decoupling of model size growth from compute cost growth (Shazeer et al., 2017; Du et al., 2021). Evolutionary methods have also been applied with success for hyperparameter tuning (Jaderberg et al., 2017).
Cross-task knowledge transfer has gained popularity, especially through transfer learning from a model pre-trained on a large amount of data for one or a few general tasks, and then fine-tuned on a small amount data for related downstream task. This approach has been shown to be very effective in wide variety of problems and modalities (Devlin et al., 2019; Dosovitskiy et al., 2021). Large scale models have recently achieved novel transfer capabilities such as few/zero shot learning (Brown et al., 2020). More complex forms of knowledge transfer such as multitask training continual learning often lead to interesting problems such as catastrophic forgetting (McCloskey & Cohen, 1989; French, 1999), negative transfer (Rosenstein, 2005; Wang al., 2019) or gradient interference
2

---

Under review as a conference paper at ICLR 2023
Layer cloning
Hyperparameter change
82.40%
55.20%
Frosen params
Trainable params
Parent model hygerparameter
layer
Leaminng 0.5
Transformer layer
Transfomerr layer
Lager
copty parmnee opernizer sane
Trunsforeer layer
Sample seighhouring value
Leyer
Layer B
Leaming rate values [- 0.2, 0.5, L.....
Posilicn enibecaling
Layer
Class loken
Chess token
Tisk input
Task inpur
Patch embedding
Patch embbedding
Child model
Parent model path
Child model
bypergarameter
Louminn
Bangla
Devanagar
pr Vavvv-TTllggee
Figure I: Graphical representation of the two mutation types used by the proposed method: layer cloning mutation (left) and hyperparameter change (center). The graph on the right represents the model generated by the preliminary experiment described in Section 4. The bottom nodes display the task names, the top nodes display the validation accuracy, and internal nodes are represented with the color of the task that has last updated the parameters of the corresponding layer.
(Chen et al., 2018; Yu et al., 2020). Research on these topics mostly focuses on approaches such as weighted combination methods (Liu et al., 2019b; Sun al., 2020b) gradient transformations (Sener& & Koltun, 2018; Kendall et al., 2018), also methods automating knowledge selection at layer level was proposed (Sun et al., 2020a).
3 EVOLUTIONARY METHOD
This section defines the proposed method capable of generating a dynamic multitask ML system. The multitask system is initialized with one root model. This model can be either pretrained or randomly initialized. During the evolutionary process, the proposed method searches for the best model for single task at time, referred to as the active task. During the active phase of task, population of models trained on the active task is evolved: the active population. The first time a task becomes active, its active population is empty. For subsequent iterations, the active population is initialized with all the models trained on the active task that have been retained from previous iterations. The active population is iteratively extended by: 1) sampling parent model (Section 3.1). 2) applying to the parent model a sampled set of mutations (Section 3.2) to produce a child model, 3) performing cycles of training and validation in order train and score the child model. Each trained model is assigned a score (Section 3.3). Early population pruning is performed by discarding the models that did not achieve better score then their parent. An active phase is composed of multiple generations in which multiple batches of child models are sampled and trained in parallel. At the end of task active phase, only its best scoring model is retained as part of the multitask system. task can become active multiple times. Details of the method are reported below (and in Algorithm 1).
3.1 PARENT MODEL SAMPLING
The first attempt to sample a parent model for the active task is done over the active population of models for that task. The models in the active population are visited in decreasing order of score, starting with the highest scoring one. Each model, m, can be accepted as parent with probability: Pparens(mr)
0.5#selectionatm..) Where (se, L) denotes the number of times the candidate model, m. has been previously selected as parent to generate a child models for task t. the current candidate parent not selected, then iteratively the model with the next best score is considered to be selected as parent with probability Pparent (-It). If a full itcration on the active population is completed without a successful parent model selection, then the same method is applied to the randomly sorted list of all remaining models: all the models currently part of the multitask system that were trained on a task different from the current active task, t. This fallback list is randomly sorted since these models have not been scored for f. As a final fallback parent uniformly sampled among all the models currently in the system. This method prioritizes the exploitation of high scoring models that had few attempts generating improved model for the active task. But also, combination with early pruning, it automatically transitions toward a more exploratory behavior in case the higher scoring models are unable generate improvement.
3

---

Under review as a conference paper at ICLR 2023
3.2 MUTATIONS
In this work we consider Deep Neural Networks (DNN) models. DNN are commonly defined by their architecture and hyperparameters. Architectures are composed of a sequence of neural network layers, each mapping an input vector into an output vector of variable dimensions. Hyperparameters specify the configuration details such as the optimizer or data preprocessing configurations. The presented method allows for two types of mutations (Figure 1):
Hyperparameter mutations can be applied to modify the configuration inherited from the parent. Each hyperparameter is associated with sorted list of valid values (Table 1). If a hyperparameter is selected for mutation, then its new value selected random among the two neighbouring values in the sorted list (Figure 1). This constraints hyperparameter mutations to only incremental changes. Notice that, every hyperparameter of child model is set to a single value. However, considering that child model continues its ancestors training with mutated hyperparameters, then the method can be regarded as capable of defining a piece-wist wise constant schedule over time for each hyperparameter.
Layer cloning mutations create copy of any parent model layer that can be trained by the child model. If a layer of the parent model is not selected for cloning, then it is shared with the child model in frozen state to guarantee immutability of pre-existing models. Child models can train only the cloned copies of the parent layers. The cloned layers are trained with possibly modified version of the parent optimizer. The configuration of the child optimizer is defined by the mutated hyperparameters. If such optimizer is of a type that stores a state (i.e. momentum), then the state is also cloned from the state saved by the ancestor that has last trained the cloned layer. Notice that, a trainable layer may be followed by frozen layers. In this case the gradients for the trainable layer are propagated through the frozen layers and applied only to the parameters of the trainable layers while frozen parameters are left unchanged. The head layer is always cloned since it always needs to be trainable. If a child model is trained on task different from the parent's 's task, then new randomly initialized head layer is created with output shape matching the number of classes of the new task.
Each possible layer cloning or hyperparameter mutation independently sampled to be applied with probability μ.. l is itself hyperparameter that is mutated by the evolutionary process. Thus demonstrating that automatic tuning is not only applied to selecting the hyperparameters of the generated models, but can also be applied to self-tune the configuration of the evolutionary algorithm.
3.3 TRAINING AND SCORING
newly sampled child model is trained on the active task for given number of epochs. The model is evaluated on the validation set after each epoch. At each intermediate evaluation, the child model is assigned a score that the evolutionary algorithm aims to maximize. The score can be defined to optimize mixture of factors such as validation quality, inference latency, training compute model size, depending on the applications requirements. The presented experiments aim to compare against the state of the art for large number of tasks without any size or compute constraint. Therefore, the validation accuracy is used directly as the score without additional factors. After training, only the parameters and optimizer state of the version of the child model achieving best score are retained.
3.4 DISCUSSION AND PROPERTIES
Notice that, none of the defined mutation actions or the evolutionary algorithm allow the creation of child models that can alter the parent model in any way. Once model has been trained, the parameters storing its knowledge cannot be modified. This method guarantees immunity from catastrophic forgetting, since the knowledge of trained model is always preserved. also provides solution to negative transfer, since it automates the selection of the knowledge that is most relevant for each new task, Furthermore, it also avoids gradient interference, that can arise when multiple gradients synchronously applied to the same set of parameters. Nonetheless, models for new tasks can use knowledge and representations from prior tasks and even extend these to improve or specialize them.
The method compartmentalizes the knowledge of each task in subset of components, allowing the implementation of different dataset privacy control policies. For example, we can introduce private tasks that can benefit from all the public knowledge embedded in the multitask system but are able to withhold the knowledge and representations derived from their private dataset from being used by other tasks. This is achieved by preventing other tasks from using cloning components trained on
4

---

Under review as a conference paper at ICLR 2023
Table 1: Hyperparameters valid values. Bold vales are defaults. This search space consists of parametrization of the configuration for the published ViT model definition library.
J E 10, 0.12, 14, 0.16, 0.18, 0.20, 0.22, 0.24, 0.26, 0.28, 0.30] Learning rate E [0.0001, 0.0002, 0.0005, 0.001, 0.002, 0.005, 0.01, 0.02, 0.05, 0.1, 0.2, 0.5] Cosine learning rate schedule warm ratio E [0.01, 0.02, 0.05, 0.1, 0.2, 0.3, 0.4]
Momentum E [0.5, 0.6, 0.7, 0.8, 0.85, 0.9, 0.95, 0.98, 0.99]
Nesterov update E [False, True]
Crop input image E [False, True]
Cropped range min E [0.01, 0.02, 0.05, 0.1, 0.2, 0.5, 1.0]
Cropped aspect ratio range min E [0.25, 0.5, 0.75, 1.0]
Flip left/right E [False, True]
Brightness delta E [0.0, 0.01, 0.02, 0.05, 0.1, 0.2]
Contrast delta E [0.0, 0.01, 0.02, 0.05, 0.1, 0.2]
Saturation delta E [0.0, 0.01, 0.02, 0.05, 0.1, 0.2]
Hue delta [0.0, 0.01, 0.02, 0.05, 0.1,0.2)
private data. This also allows to remove the knowledge learned from the private dataset at any future date by simply removing its components. This private/public distinction be generalized into an access- control-list mechanism. For example, a set of private tasks can share representations that are withheld from any other task. Privacy control capabilities empirically demonstrated in Section 4.
3.5 EXPERIMENTAL SET
This section details the instantiation of the proposed method employed the experimental analysis. The task type for the presented set of experiment is image classification. This choice allows to define a large benchmark of publicly available datasets with standardized framing. also allows us to build on top of of-t-oofthheaat models whose architecture definition and checkpoints are public: the Visual Transformer (ViT) is used as root model (Dosovitskiy et al., 2021).
create copy of any of ViT's layers: 1) Patch embed- ding: the first layer of the model maps the input image into a sequence of embedded tokens, each corresponding patch of the input image. 2) Class token: classification token is prepended to the sequence. The final hidden state corresponding to this token is used as the aggregate sequence representation for classification tasks (Devlin al., 2019). 3) Position embedding: the sequence representation is then augmented with an embedding that carries each patch positional information. 4) Transformer layers: the sequence representation generated by the input layers is iteratively trans- formed by a stack of transformer layers (Vaswani et al., 2017). 5) Model head: a final fully connected layer mapping the representation produced by the top-most transformer layer into the logits.
Architecture Layer cloning mutations
The parameters of the root model can be either randomly initialized or loaded from checkpoint. The preliminary experiment demonstrates the evolution from random initialization (see Section 4), while the large scale experiment starts from pretrained large ViT model (see Section 5).
Parameters
Hyperparameters As default hyperparameters we use those resulting from the extensive study conducted by Steiner et al. (2021): SGD momentum optimizer, cosine decay schedule, no weight decay, gradient clipping at global norm and 386 x36 image resolution. The evolutionary method can change the hyperparameters of optimizer, image preprocessing and architecture (see Table 1).
4 PRELIMINARY EXPERIMENT
This section describes a small scale preliminary experiment that introduces details of the method application and illustrates the privacy control and initialization capabilities. This experiment demon- strates the ability to generate a multitask model from random initialization and minimal architecture rather than evolving a pretrained state-of-the- model. Therefore. a randomly initialized VIT Ti/16 architecture (Steiner al., 2021) stripped of transformer layers is used root model. To allow the method to build a capable architecture, we add an extra mutation action that can insert new randomly initialized transformer layer just before the head layer.
5

---

Under review as a conference paper at ICLR 2023
Furthermore, the dataset privacy control technique (see Section 3) is demonstrated by adding private task. Three numerals recognition tasks are used as benchmark: (bangla, devanagari, telugu). Telugu is introduced as private task, that other task can access the knowledge introduced into the system by its dataset. However, its models can leverage the knowledge provided by the public tasks.
This short experiment is configured perform 2 active task iterations for each task. During each active task iteration model generations are produced. In each generation 8 child models are sampled and trained in parallel on each of the TPUv3 cores. The choice of small datasets, architecture and training budget is intended to facilitate reproducibility and fast experimental iterations. The experiment can reproduced by using the published artifacts and completes less than 5 minutes.
Figure (right) displays the resulting multitask model solving jointly the 3 tasks. We observe a high degree of cross-task knowledge and components sharing throughout the evolutionary process. Even though the root model has no transformer layers, multiple randomly initialized transformer layers are inserted and trained improving the score of each task. Note that, any point during the evolution, the components trained on the private task (red) are only used by the private task.
5 LARGE SCALE CONTINUAL LEARNING EXPERIMENT
This section reports a single large scale continual learning experiment producing multitask system jointly solving 69 visual tasks. A pretrained ViT L/16 used as root model, which has been selected for its pretraining validation accuracy on the imagenet-2 - dataset following Steiner et al. (2021).
5.1 VIT BENCHMARK
The first tasks introduced to the system are 3 tasks on which ViT was evaluated in Dosovitskiy et al. (2021). This experiment is configured to perform 5 active iterations for each task, and 4 model generations for each iteration. During each generation, child models are trained in parallel on each of the 8 TPUv3 cores. Fach model training performs 4 validation cycles. The number of train samples between validation cycles is set to min(1 epoch, 1281 16 samples) to smooth the distribution of compute across datasets of different size. 128116 equivalent to 1/10th of an epoch of the imagenet2012 12 training set. The same configuration is applied in following experiments.
This configuration results in 8 epochs for imagenet, and 80 epochs for cifar. This is roughly equivalent to the fine-tuning setup of the baseline model we compare against (Dosovitskiy et al., 2021): ): 8 epochs for imagenet and 102.4 for cifar. The proposed method can be considered cheaper since: 1) ViT fine-tuning has been repeated multiple times for the hyperparameters tuning process and 2) setting y 0.2 results in cheaper training steps, since parameters updates can be skipped for frozen layers and gradient propagation can be skipped for frozen layers at the base of the model (preliminary experiments have shown a 2.5-4% training speed-up attributable to this). In order to provide a fair comparison, root model is used the same VIT L/16 architecture, same checkpoint pretrained on the i12k dataset, same 384x 384 image resolution, and optimizer and prepossessing configuration.
Table 2 reports the top 1 test accuracy achieved. μ2Net outperforms fine-tuning with comparable training steps per task. Extending the training with 5 additional tasks iterations leads to moderate gains imagenet2012 and cifar10. Notice that, for cifar100 the accuracy decreases. This can happen since the best models are selected according the validation accuracy and, as the model gets close to convergence, small validation accuracy gain may lead to noisy perturbation of the test accuracy.
To quantify knowledge transfer, we consider the model produced for each task and examine the dataset on which each layer's ancestors were trained. On average, the layers composing the model generated
Table 2: Test accuracy achieved by 22NNtt and by fine-tuning comparable pretrained VIT model.
Model
imagenet2012
cifar100
cifar10
VIT L/16 fine-tuning (Dosovitskiy al., 2021)
85,30
93.25
99.15
p2Nett after 5 task iterations
86.38
94.75
99.35
1.2Net after 10 task iterations
86.66
94.67
99.38
1.2Net cont. after adding VTAB-full tasks
86.74
94.67
99.41
222Net cont. after adding VDD tasks
86.74
94.74
99.43
22Net cont. after adding all tasks
86.74
94.95
99.49

---

Under review as a conference paper at ICLR 2023
Table 3: Test accuracy achieved on the VTAB-full benchmark by: 1) fine-tuning with matching architecture and checkpoint (ViT L/16 i21k) reported by Steiner et al. (2021), 2) the Sup-rotation method (Gidaris al., 2018) that achieved the best result in the VTAB-full leaderboard (Zhai et al., 2019b), 3-4) μ2Nett results after 2 task iterations, 5) and after an additional iteration performed after the VDD benchmark introduction. Underlined models transfer knowledge from VDD tasks.
caltech101
cifar100
flowers102 102
SUn397
camelyon
eurosat
resise45
retinopathy
clevr-count
clevr-dist
dmlab
dspr-loc
dspr-orient
kitt-dist
snorb azim
snorb-elev
did
pets
svhn
Model
Steiner et al. (2021)
95.5 94.1 80.3 99.6 95.0 83.4 97.4 86.4 99.0 96.6 83.3 99.8 91.7 75.6 100 90.4 84.7
27.5 76.5
Zhai et al. (2019b)
94.6 84.8 75.9 94.7 91.5 70.2 97.0 85.9 98.8 94.9 79.5 99.8 92.5 76.5
100
96.5 82.3 100
98.4
p2Net cont. 1st iter.
92.6 94.6 79.8 99.6 95.3 84.5 97.3 87.5 99.1 96.3 83.7 99.8 93.2 76.9 100 96.2 83.0
32.3 94.5
p22Net cont. 2nd iter.
92.6 94.6 80.5 99.6 95.3 84.8 97.8 88.4 99.2 97.0 84.0 99.8 94.0 76.9 100 96.4 83.0
33.3 95.1
1/2Net cont. after VDD 93.0 94.7 81.0 99.6 95.3 84.8
97.8 91.1 99.1 97.0 84.0
99.8 94.0
76.9 100
96.4 82.3
33.3 95.1
for the imagenet2012 task have performed only 60.6% of the training steps on the imagenet2012 dataset, and have received 31.5% of the gradient updates from cifar 100 and 7.9% from cifar10. The layers comprising the cifar100 model have performed 42.3% of their training on imagenct2012 and 20.6% on cifar10. And layers comprising the cifar 10 model performed 46.1% of training on imagenet2012 and 35.9% on cifar100. The tasks heterogeneity improves the representations produced by the different layers, and results in generally higher performance, shown in Table 2.
The following sections 5.2 and 5.3 describe the extensions of the system performed by introducing additional benchmarks. After the introduction of each benchmark, we perform an additional iteration on imagenet and cifar tasks to analyze effects of further knowledge enrichment. As a representative example: the VDD benchmark (Section 5.3) includes a low resolution version of c0f The model that will be generated for vdd/cifar 100 will be a mutation of the current full resolution cifar 100 model. Afterward, the additional active task iteration on cifar100 will be performed, and the resulting improved cifar 100 model will be mutation of the low resolution vdd/cifar (00 model.
After final iteration, we note that 99.49 is the best cifar10 accuracy reported for model trained only on public data: to the best of our knowledge, this constitutes a 15% relative error reduction compared to the 99.40 state of the art achieved by Touvron et al. (2021). Dosovitskiy et al. (2021) achieves 99.50 with a double size ViT-Huge model trained on proprietary data. The achieved cifarl (00 accuracy is currently outperformed only by Ridnik et al. (2021) (05.10) and Foret et al. (2021) (96.08).
5.2 VUAL-FULL BENCHMARK
Next, we introduce to the system the 19 VTAB-full tasks (Zhai et al.. 2019a), plus 5 additional task variants that are not included in the standard evaluation set (Table 8). From this experiment onward, the infrastructure is scaled from 8 to 32 cores, as detailed in Appendix B. The number of task iterations reduced from 10 to 2. These changes lead to a roughly similar exploratory and training budget per task. However, the increased parallelism results in faster task iterations.
Table 3 reports the achieved results along with reference models that use limited knowledge transfer capabilities. Steiner et al. (2021) reports the quality achieved by fine-tuning a model equivalent to our root model. This outperforms j22Net on only 2 tasks, even if it has been trained multiple times to perform hyperparameter tuning. Zhai et al. (2019b) reports the results of the best model identified with large scale study. This state of the art model outperforms p2Net on 4 tasks. Again, increasing number of task iterations and additional knowledge (VDD) in the system, seem yield better quality.
Table 4: Test accuracy mean and std.dev. achieved on the VDD benchmark by system replicas.
Model:
imagenel
cifur 100
daimier
omniglot
ucf101
aircraft
Rowers
#2Net
syhm
gisrb
drd
cont.
1st iter.
89.19 98.2+0.23 97.2 99.940.04 99.940.03
84.5 83.6 64.8+3.57
75.2 09.0
2nd iter.
12 98.4 97 99.940.05 99.911.04 84.3 15.7 65.443303 76.020.49 99. 1
7

---

Under review as a conference paper at ICLR 2023
Table 5: Test accuracy achieved on the Multitask Character Classification Benchmark by j22Net continued extension with 2 active task iterations and by the model that has set the state of the art using comparable data splits: Jeevan & Sethi (2022) for digits, Kabir et al. (2020) for letters, Ajayan & James (2021) for kmnist, An et al. (2020) for mnist, Hazra et al. (2021) for ematerdb/bangla. Underlined models reuse knowledge introduced by other character classification tasks. Datasets are listed in decreasing size from the biggest emnist/digits (240k samples) to the smallest telugu (2.5k).
ematerdb/
ematerdb/
devanagari
cmaterdb//
emnist/
digits
emmist/
letters
kmnnsst
mnist
bangin
telugu
Model
State of the art
99.77
95.88
95.40
99.91
99.00
J.2Net cont. 1st iteration
99.82
93.60
98.68
99.75
98.72
98.60
96.60
97.80
j.2Net cont. 2nd iteration
99.82
93.68
98.60
99.69
99.84
99.10
98.00
99.40
5.3 VISUAL DOMAIN DECATHLON (VDD) BENCHMARK
The VDD benchmark (Bilen et al., 2017) is introduced next. The ML methodology proposed in this paper, can achieve higher efficiency by focusing the available compute on the training of a single multitask system. Though, the standard approach to measure variance relies on experiment repetitions. This section demonstrates how variance can be measured for any chosen segment of the training. In practice, 2 task iterations are performed introduce the VDD tasks starting from the state achieved after the introduction of the last benchmark as usual. But, this experiment is run on 3 parallel copies of the system, allowing to compute variance of the metrics for this set of task iterations.
The VDD benchmark is composed of 10 diverse tasks. This is also reflected in the diverse variance ranges measured (see Table 4). Variance is low for most of the tasks. However, for ucf101 and aircraft is significantly higher. The metrics that have highest correlation with standard deviation are error rate (linear proportionality in log scale) and number of training samples per class (inverse proportionality in log scale) (Figure 4). These can be considered metrics indicative of the complexity of the task. Furthermore, variance decreases with the second iteration: average standard deviation of 0.87 after I iteration and 0.61 after the second. These findings can support the intuitive hypothesis that tasks with higher complexity may benefit from more iterations to decrease variance and approach convergence. The next system extension continues from the state of one randomly selected replica
5.4 MULTITASK CHARACTER CLASSIFICATION BENCHMARK
We continue extending the system by adding a set of 8 character classification tasks. Thus offering the opportunity to study knowledge transfer across tasks with high domain correlation.
Table 5 reports the test accuracy achieved with 2 active tasks iterations. We observe that tasks with more training data (left) achieve convergence in the first iteration, this hypothesis is supported by lack of significant accuracy gains with the second iteration. While tasks with less training data (right) show a significant gain from a second training iteration. Smaller tasks use transferred in domain knowledge: bangla top model reuses components that embed knowledge introduced by emnist/letters,
Table 6: Test accuracy achieved the VTAB-IK benchmark by: 1) fine-tuning VIT L/16 i2lk (matching root model) Dosovitskiy et al. 2021),22 2) the Sup-rotation method (Gidaris et al., 2018) that achieved the best result in the VTAB-Ik leaderboard (Zhai et al., 2019b). Underlined models have at least one ancestor trained on the corresponding full form task. Doubly underlined model inherit directly from the current best model for the matching full form task.
caftech101
cifar 100
flowers 102
camelyon
resise45
retinopathy
elevr-count
clevr-dist
dspr-foc
dspr-orient
kiti-dist
snorb azim
snorb-eley
Sun397
eurosat
dmlab
did
pets
svhn
Model
Dosovitskiy et al. (2021)
90.8 84.1 74.1 99.3 92.7 61.0 80.9 82.5 95.6 85.2 75.3
70.3
56.1
41.9 74.7 64.9
79.9
30.5
41.7
Zhai et al. (2019b) 42Net cont. 1st iter.
91.7 53.7 69.5 90.8
88.1
32.8
88.5
83.4
96.0
82.0
71.1
47.3
57.2
36.6
88.3
52.1
77.1
51.6
33.7
87.1 89.4 77.6 99.2
94.5
57.6
97.5
86.0
98.6
93.4
78.0
91.2
59.9
47.6
58.4
96.2
81.9
32.1
92.5
N2Net cont. 2nd iter.
89.9 90.6 78.1
99.7
94.5
57.6
97.5 86.0
98.3
93.4
83.5
99.8
90.6
76.3
100
96.2
81.7
33.7
92.5
8

---

Under review as a conference paper at ICLR 2023
10a
47%
Activated parameters per task
33%
Added parameters per task
Percentage
10
5.2%
4.0%
3.3%
3.8%
2.3%
2.8%
2.3%
10
1.4%
VTAB-full
VDD
Chars
VTAB-IK
0
2,500
5000
7500
10000
12500
15000
17500
20000
Sampled child models
Figure 2: Activated and added parameters per task as percentage with respect to the total number of parameters the multitask system along the duration the large scale experiment (see Section 5. Vertical lines highlight the start of the introduction for each of the considered benchmark.
while devanagari transfers from omniglot and bangla, and telugu transfers from bangla and devanagari. Furthermore, the achieved quality is comparable to the state of the art published for each task.
5.5 VTAB-IK BENCHMARK
The system is further extended by adding the Ik-samples version of the VTAB tasks. Since the system contains already the knowledge learned from the full version of each task, this set allows to study how effective is the proposed method at retrieving knowledge that is already embedded in the system.
Table 6 reports results along with reference models that use limited knowledge transfer capabilities. During the first iteration, the models generated for the short form tasks can retrieve the knowledge of the corresponding full form task also without directly mutating its model, but rather mutating a model having at least one ancestor trained on the full task. For example, the model generated for Nowerss102, mutates the dtdıx model, that has 28 ancestors, of which only the 21st was trained on oxford afford_flowers10% flowers102,ull However, that is enough for flowers1021x to achieve 99.2% test accuracy.
After only one task iteration, 5 tasks achieve better test accuracy than the reference models without reusing any knowledge introduced by the corresponding full form task. Particularly interesting is the case of kitty-distıı (a.k.a kitty/closest without transferring from the matching full form task but composing the knowledge of related tasks: kitty/closest_obieee batrreess__eeeeedddsteeeeeeeeee distancce (8 ancestors), kitty/count_ stayy/count_vhhiclees..s kitty/count_ ve (3 ancestors). Thus learning estimate distance of the closest vehicle by combining the knowledge of recognizing vehicles and estimating the distance of the closest object. Also, clevr-dist achieves a strong performance by inheriting from the semantically equivalent task kitti/closest _objecttteeeeeeeeeeeeeeeeeeee_ distance fult without reusing the knowledge introduced by clevr-dist,ult
distanceeee) that achieves a strong performance
(3 ancestors) and
6 CONCLUSION
We introduced the p22eet method, aimed at achieving vate-of-the-ar quality on a large task set, with the ability to dynamically introduce new tasks into the running system. The more tasks are learned the more knowledge is embedded in the system. A ViT-L architecture (307M parameters) was evolved into a multitask system with (3'08MM parameters jointly solving 69 tasks. However, as the system grows, the sparsity in parameter activation keeps the amount of compute and the memory usage per task constant. The average added parameters per task decreases by 38% through the experiment, and the resulting multitask system activates only 2.3% of the total parameters per task (see Figure 2 and Table 7). The proposed method allows decoupling the growth of useful representations for solving new tasks from the growth of parameters/computte per task. Furthermore, experimenting with large number of tasks allowed us to identify different patterns of positive knowledge transfer and composition, achieving higher efficacy on small datasets and across related tasks. The proposed approach to mutations allows to achieve immunity against common pitfalls of multitask systems such as catastrophic forgetting, negative transfer and gradient interference, and demonstrates the key data privacy properties we want to achieve in continual learning system. Future work can continue to build toward systems that can acquire further capabilities and knowledge across multiple modalities.
All the experiments reported in this paper can be reproduced by using the public datasets and published y/2Net code, and can be extended by using the published checkpoints (see Appendix B).

---

Under review as a conference paper at ICLR 2023
REFERENCES
Arun Ajayan and A P James. Edge to quantum: hybrid quantum-spiking neural network image classifier. Neuromorphic Computing and Engineering. 1, 2021.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford. Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: visual language model for few-shot learning. ArXiv, 1tss/22044141998, 2022.
Sanghyeon An, Min Jun Lee, Sanglee Park, He Sarina Yang, and Jungmin So. An ensemble of simple convolutional neural network models for mnist digit recognition. ArXiv, abs/2008. 2020.
Paul Barham, Aakanksha Chowdhery, Jeffrey Dean, Sanjay Ghemawat, Steven Hand, Daniel Hurt, Michael Isard, Hyeontaek Lim, Ruoming Pang. Sudip Roy. Brennan Saeta, Parker Schuh, Ryan Sep- assi, Laurent El Shafey, Chandramohan A. Thekkath, and Yonghui Wu. Pathways: Asynchronous distributed dataflow for ml. ArXiv, abs/2203. 122333, 2022.
James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper- parameeee optimization. In NIPS, 1011.
Hakan Bilen, Sylvestre Rebuffi, and Tomas Jakab. Visual domain decathlon. 2017.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, T. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. ArXiv, abs/2005. 14165, 2020.
Han Cai, Tianyao Chen, Weinan Zhang, Yong Yu, and Jun Wang. Efficient architecture search by network transformation. In AAAI, 2018.
Tianqi Chen, Ian J. Goodfellow, and Jonathon Shlens. Net2net: Accelerating learning via knowledge transfer. CoRR, M3w/1511.056411 2016.
Zhao Chen, Vijay Badrinarayanan, Chen- Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In ICML, 2018.
Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, 105:1865-1883, 2017.
Mircea Cimpoi, Subhransu Maji. lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3606-3613 2014.
Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical japanese literature. ArXiv,
2018
Gregory Cohen, Saeed Afshar, Jonathan C. Tapson, and André van Schaik. Emnist: Extending mnist to handwritten letters. 2017 International Joint Conference on Neural Networks (IJCNN), pp.
2921-2928 2017.
N. Das, Jagan Mohan Reddy, Ram Sarkar, Subhadip Basu, Mahantapas Kundu, Mita Nasipuri, and Dipak Kumar Basu. A statistical-topologicca feature combination for recognition of handwritten numerals. Appl. Soft Comput., 12:2486-249, 22066,2295554 2012a
N. Das, Ram Sarkar, Subhadip Basu, Mahantapas Kundu, Mita Nasipuri, and Dipak Kumar Basu. A genetic algorithm based region sampling for selection of local features handwritten digit recognition application. Appl. Soft Comput., 1592-1666 2012b.
10

---

Under review as a conference paper at ICLR 2023
Jacob Devlin, Ming-
Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. ArXiv, abs/2010. 11929, 2021.
Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Toju Duke, Lucas Dixon, Kun Zhang, Quoe V. Le, Yonghui Wu, Z. Chen, and Claire Cui. Glam: Efficient scaling of language models with mixture-of-expert. ArXiv, abs/21 12.09905, 2021.
Kathleen S.
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. 2004.
Chrisantha Fernando, Dylan S. Banarse, Charles Blundell, Yori Zwols, David R Ha, Andrei A. Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. ArXiv, abs/1701.08733 2017.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. ArXiv, abs/2010.01412, 2021.
Robert M. French. Catastrophic forgetting in connectionist networks. Trends in Cognitive Sciences, 3:128-135, 1999.
Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2012.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. ArXiv, abs/1803.07728, 2018
Abhishek Hazra, Prakash Choudhary, Sanasam Chanu Inunganbi, and Mainak Adhikari. Bangla- meitei mayek scripts handwritten character recognition using convolutional neural network. Applied Intelligence, 11-2291-2311, 2021.
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016
Patrick Helber, Benjamin Bischke, Andreas R. Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12:2217-2226, 2019.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nip. In ICML, 2019.
Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M. Czameeki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, lain Dunning, Karen Simonyan, Chrisantha Fernando, and Koray Kavukcuoglu. Population based training of neural networks. ArXiv, abs/1711 1111109846, 2017.
Pranav Jeevan and Amit Sethi. Wavemix: Resource-effiiiet token mixing for images. ArXiv, abs/2203.03689, 2022.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross B. Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1988-1997, 2017.
11

---

Under review as a conference paper at ICLR 2023
Norman P. Jouppi, Cliff Young, Nishant Patil, David A. Patterson, Gaurav Agrawal, Raminder Singh Bajwa, Sarah Bates, Suresh Bhatia, Nanette J. Boden, Al Borchers, Rick Boyle, Pierre luc Cantin, Clifford Chao, Chris Clark, Jeremy Coriell, Mike Daley, Matt Dau, Jeffrey Dean, Ben Gelb, Tara Vazir Ghaemmaghami, Rajendra Gottipati, William Gulland, Robert B. Hagmann, C. Richard Ho, Doug Hogherg, John Hu, Rohert Hundt, Daniel Hurt, Julian Ibarz, Aaron Jaffey, Alek Jaworski, Alexander Kaplan, Harshit Khaitan, Daniel Killebrew, Andy Koch, Naveen Kumar, Steve Lacy, James Laudon, James Law, Diemthu Le, Chris Leary, Zhuyuan Liu, Kyle A. Lucke, Alan Lundin, Gordon MacKean, Adriana Maggiore, Maire Mahony, Kieran Miller, Rahul Nagarajan, Ravi Narayanaswami, Ray Ni, Kathy Nix, Thomas Norrie, Mark Omernick, Narayana Penukonda, Andy Phelps, Jonathan Ross, Matt Ross, Amir Salek, Emad Samadiani, Chris Severn, Gregory Sizikov, Matthew Snelham, Jed Souter, Dan Steinberg, Andy Swing, Mercedes Tan, Gregory Thorson, Bo Tian, Horia Toma, Frick Tuttle, Vijay Vasudevan, Richard Walter, Walter Wang, Fric Wilcox, and Doe Hyun Yoon. In-datacenter performance analysis of a tensor processing unit. 2017 ACM/IEEE 44th Annual International Symposium on Computer Architecture (ISCA), pp. 1-12, 2017.
H. M. D. Kabir, Moloud Abdar, Seyed Mohammad Jafar Jalali, Abbas Khosravi, Amir F. Atiya, Saeid Nahavandi, and Dipti Srinivasan. Spinalnet: Deep neural network with gradual input. ArXiv, abs/2007.0333 10077,337, 2020.
Kaggle and EyePacs Kaggle diabetic retinopathy detection. http://www..aagle..om/tttbbteee eetihoppatry-deeection//e
2015.
Jared Kaplan, Sam McCandlish, T. J. Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeff Wu, and Dario Amodei. Scaling laws for neural language models. ArXiv,
03s/2001.0836// 2020.
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. 2018 IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 7482-7491, 2018.
David A. Klindt, Lukas Schott, Yash Sharma, Ivan Ustyuzhaninox, Wieland Brendel, Mathhis Bethge, and Dylan M. Paiton. Towards nonlinear disentanglement in natural data with temporal sparse coding. ArXiv, abs/2007.10930, 2021.
Ef Kokiopoulou, Anja Hauth, Luciano Sbaiz, Andrea Gesmundo, Gábor Bartók, and Jesse Berent. Fast task-aware architecture inference. ArXiv, (bs//192.00781, 2019.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
Brenden M. Lake, Ruslan Salakhutdinov, and Joshua B. Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 350:1332 1338, 2015.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proc. IEEE, 86:2278-2324 1998.
Yann LeCun, Fu Jie Huang, and Léon Bottou. Learning methods for generic object recognition with invariance to pose and lighting. Proceedings of the 2004 IEEE Computer Society Conference Computer Vision and Pattern Recognition, 2004. CVPR 2004., 2:11-104 Vol.2, 2004.
Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. Learn to grow: continual structure learning framework for overcoming catastrophic forgetting. In ICML, 2019.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Transactions on Patlern Analysis and Machine Intelligence, 40:2935-2947,, 2018.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. ArXiv, 0ts/1800..9960 2019a.
Shengchao Liu, Yingyu Liang, and Anthony Gitter. Loss-balanced task weighting reduce negative transfer in multi-task learning. In AAAI, 2019b.
Krzysztof Maziarz, Andrey Khorlin, Quentin de Laroussilhe, and Andrea Gesmundo. Evolutionary- neural hybrid agents for architecture search. ArXiv, abs/181 11111981188 2018.
12

---

Under review as a conference paper at ICLR 2023
Michael McCloskey and Neal J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. Psychology of Learning and Motivation, 24:109-165, 1989.
Thomas Mensink, Jasper R. R. Uijlings, Alina Kuznetsova, Michael Gygli, and Vittorio Ferrari. Factors of influence for transfer learning across diverse appearance domains and task types. IEEE Iransactions on paltern analysis and machine intelligence, PP, 2021.
Azalia Mirhoseini, Anna Goldie, Mustafa Yazgan, Joe Wenjie Jiang, Ebrahim M. Songhori, Shen Wang, Young-Joon Lee, Eric Johnson, Omkar Pathak, Sungmin Bae, Azade Nazi, Jíwoo Pak, Andy Tong, Kavya Srinivasa, Will Hang. Emre Tuncer, Ananda Babu, Quoc V. Le, James Laudon, Richard Ho, Roger Carpenter, and Jeff Dean. Chip placement with deep reinforcement learning. ArXiv, abs/2004. 2020.
Yuval Netzer, Tao Wang, Adam Coates, A. Bissacco, Bo Wu, and A. Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pp. 722-729, 2008.
Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. 2012 IEEE Conference on Computer Vision and Puttern Recognition, pp. 3498-3505, 2012.
Hieu Pham, Melody Guan, Barret Zoph, Quoe Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. In ICML, 2018.
Colin Raffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. ArXiv, abs/1910. 106833 2020.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. ArXiv, abs/2102. 12092, 2021.
Rahul Ramesh and Pratik Chaudhari. Model zoo: A growing brain that learns continually. In ICLR, 2022.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V. Le. Regularized evolution for image classifier architecture search. In AAAI, 2019.
Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. In NIPS, 2017.
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley D. Edwards, Nicolas Manfred Otto Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. A generalist agent. ArXiv, abs/2205.06177 2022.
T. Ridnik, Gilad Sharir, Avi Ben-Cohen, Emanuel Ben-Baruch, and Asaf Noy. Ml-decoder: Scalable and versatile classification head. 2021.
Amir Rosenfeld and John K. Tsotsos. Incremental learning through deep adaptation. JEEE Transac- tions on Pattern Analysis and Machine Intelligence, 42:651-663, 2020.
Michael T. Rosenstein. To transfer or not transfer. NIPS 2005, 2005.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 1 115: 211-252, 2015.
Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick. Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. ArXiv, hbs/1600604671.. 2016.
13

---

Under review as a conference paper at ICLR 2023
Ozan Sener and Vladlen Koltun. Multi-tass learning multi-objective optimization. In NeurIPS, 2018.
Andrew W. Senior, Richard Evans, John M. Jumper, James Kirkpatrick, L. Sifre, Tim Green, Chongli Qin, Augustin Zidek, Alexander W. R. Nelson, Alex Bridgland. Hugo Penedones, Stig Petersen, Karen Simonyan, Steve Crossan, Pushmeet Kohli, David T. Jones, David Silver, Koray Kavukcuoglu, and Demis Hassabis. Improved protein structure prediction using potentials from deep learning. Nature, 577:706-710 2020.
Noam M. Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. ArXiv, abs/ 1es/1700106388, 2017.
David Silver, Aja Huang. Chris Maddison, Arthur Guez, L. Sifre, George van den Driessche, Julian Schrittwieser, loannis Antonoglou, Vedavyas Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy P. Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 529:484 489, 2016.
Jasper Snoek, H. Larochelle, and Ryan P. Adams. Practical bayesian optimization of machine learning algorithms. In NIPS, 2012.
Niranjan Srinivas, Andreas Krause, Sham M. Kakade, and Matthias W. Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. In ICML, 2010.
Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. ArXiv, abs/2106. 10270, 2021.
Ximeng Sun, Rameswar Panda, and Rogério Schmidt Feris. Adashare: Learning what to share for efficient deep multi-task learning. ArXiv, 1tt//911.11441 2020a.
Yu Sun, Shuohuan Wang. Yukun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. Ernie 2.0: A continual pre-training framework for language understanding. ArXiv, abs/1907. 2020b.
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, and Quoc V. Le. Mnasnet: Platform- aware neural architecture search for mobile. 2019 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2815 1-2111,,,,,,-11111 2019.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam M. Shazeer, Apoorv Kulshreshtha, Heng- Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du Yaguang Li, Hongrac Lee, Huaixiu Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Yanqi Zhou, Chung-Ching Chang, I. A. Krivokon, Willard James Rusch, Mare Pickett, Kathleen S. Meier- Hellstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Hartz Søraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-JJohh Josh Lee, Lora Aroyo. Ravindran Rajakumar, Alena Butryna, Matthew Lamm, V. O. Kuzmina, Joseph Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arcas, Claire Cui, Marian Croak, Ed Chi, and Quoc Le.. Lamda: Language models for dialog applications. ArXiv, abs/2201 00239, 2022.
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Herv'e J' J'egou. Going deeper with image transformers. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 32 42, 2021.
Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. ArXiv, abs/1706.03762, 2017.
Bastiaan S. Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling, Rotation equivariant enns for digital pathology. ArXiv, abs/1806 01118866036.2.. 2018.
14

---

Under review as a conference paper at ICLR 2023
Oriol Vinyals, Igor Babuschkin, Wojciech M. Czarnecki, Michaël Mathieu, Andrew Dudzik, Juny- oung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, L. Sifre. Trevor Cai, John P. Agapiou, Max Jaderberg, Alexander Sasha Vezhnevets, Rémi Lehlond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom Le Paine, Caglar Gulcehre, Ziyun Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, pp. 1 1-5, 2019.
Zirui Wang. Zihang Dai, Barnabás Péczos, and Jaime G. Carbonell. Characterizing and avoiding negative transfer. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11285-11294, 2019.
Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 3485-3492, 2010.
Huaxiu Yao, Yingbo Zhou, Mehrdad Mahdavi. Zhenhui Jessie Li, Richard Socher, and Caiming Xiong. Online structured meta-learning. ArXiv, 01122000.115555 2020.
Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. ArXiv, abs/ 00bs//708.000 1778.015577 2018.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. ArXiv, abs/2001.06778 2020.
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, André Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, Lucas Beyer, Olivier Bachem, Michael Tschannen, Marcin Michalski, Olivier Bousquet, Sylvain Gelly, and Neil Houlsby. The visual task adaptation benchmark. ArXiv, abs/1910.04867, 2019a.
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, André Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, Lucas Beyer, Olivier Bachem, Michael Tschannen, Marcin Michalski, Olivier Bousquet, Sylvain Gelly, and Neil Houlsby. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv: Computer Vision and Pattern Recognition, 2019b.
Jun Zhang, Zhi hui Zhan, Ying Lin, Ni Chen, Yue jiao Gong, Jinghui Zhong, Henry Shu hung Chung, Yun Li, and Yu hui Shi. Evolutionary computation meets machine learning: A survey. IEEE Computational Intelligence Magazine, 6:68-75 758875,17 2011.
Wen Zhang, Lingfei Deng, Lei Zhang, and Dongrui Wu. A survey on negative transfer. 2020.
Barret Zoph and Quoe V. Le. Neural architecture search with reinforcement learning. ArXiv, abs/1611.011 1005578, 2017.
15

---

Under review as a conference paper at ICLR 2023
A EXTENDED RELATED WORK SURVEY
The proposed method is designed to learn a unbounded number of tasks in a continual learning fashion. In such contexts it aims learn each task with higher quality and efficiency by automating and optimizing the knowledge transfer among any subset of tasks that can provide useful knowledge to one another. The proposed model is designed to be immune from common multitask learning pitfalls: catastrophic forgetting, gradients interference, negative transfer. Cross-task transfer-learning has gained popularity, especially through transfer learning from a model pre-trained on a large amount of data for one or a few general tasks, and then tnne-tuned on small amount of data for related downstream task. This approach has been shown to be very effective in a wide variety of problems across many modalities, including language (Devlin et al., 2019; Raffel al., 2020) and vision (Dosovitskiy et al., 2021; He et al., 2016).. The success of transfer-learning applications hinges on adequate prior knowledge selection to avoid typical negative transfer pitfalls (Rosenstein, 2005; Wang et al., 2019). Common solutions rely on data or model selection techniques, often putting emphasis on the efficiency of the exploration (Zhang et al., 2020; Mensink et al., 2021), also method aiming to automate knowledge selection at a layer level have been proposed Sun et al. (2020a). Transfer learning capabilities are critical for multitask models. ML models trained jointly on multiple tasks can be affected by gradients interference if any subset of parameters receive gradients jointly from multiple sources (Chen et al., 2018; Yu et al., 2020), and by catastrophic forgetting of prior knowledge new tasks are learned (McCloskey & Cohen, 1989; French, 1999). These knowledge loss problems can be alleviated with weighted combination of tasks (Liu et al., 2019b; Sun et al., 2020b) and gradient transformation methods (Chen et al., 2018: Sener & Koltun, 2018; Kendall et al., 2018). Stronger guarantees are provided by methods that compartmentalize task specific knowledge in dedicated parameter subsets (Rebuffi et al., 2017: Houlsby et al., 2019; Rusu et al., 2016; Rosenfeld & Tsotsos, 2020). Addressing catastrophic forgetting and identifying what subset of parameters/knoolldgee that is beneficial share with each ask is also critical for continual learning or life long learning methods (McCloskey & Cohen, 1989; French, 1999; Ramesh & Chaudhari, 2022).
The proposed method relies on an evolutionary approach to jointly search the spaces of models architectures, hyperparameters, and prior knowledge selection while optimizing for an possibly multi-factor don-differetiable reward function. The automation of hyperparameter tuning has been commonly addressed with Bayesian optimization (Srinivas et al., 2010; Bergstra et al., 2011; Snoek et al., 2012), 2), evolutionary methods have also been explored for this purpose (Jaderberg et al., 2017: Zhang et al., 2011) I). Hyperparameters tuning can be considered related to the neural architecture search (NAS), architectures can be defined by the selection of a sequence of architectural hyperpa- rameters. Initially, NAS methods have been based on reinforcement learning techniques (Zoph & Lc, 2017) but also sample efficient evolutionary approaches have also also proposed (Real al., 2019; Maziarz et al., 2018). 8). Alternative NAS methods focusing on more efficient parameter-sharing (Pham al., 2018; Liu et al., 2019a; Kokiopoulou al., 2019) or optimization for multi-factor quality/cost trade-offs (Tan et al., 2019) have also been explored.
The proposed method is capable dynamically extend the system, adding capacity novel structures in an unconstrained fashion. A few methods have been proposed to achieve dynamic architecture extensions (Chen et al., 2016: Cai et al., 2018). some also focusing on an unbounded stream of tasks (Yoon et al., 2018; Yao et al., 2020), or achieving immunity from catastrophic forgetting (Rusu et al., 2016; Li & Hoiem, 2018: Li et al., 2019; Rosenfeld & Tsotsos, 2020).
The proposed method is sparsely activated, thus the unbounded growth of knowledge and parameters is decoupled from the growth of computational cost. The growth in capabilities of state of the art models often requires growth in terms of trainable parameters (Kaplan et al., 2020). Sparse activation techniques at sub-layer level (Shazeer et al., 2017: Du al., 2021) or network route level (Fernando et al., 2017) allow to decouple model size growth from compute cost. This is achieved by integrating a routing technique that selects the appropriate subset of parameters storing the most relevant knowledge for each task, sample token/patch.
The ability of jointly solve a large amount of tasks is is commonly associated with progress toward Artificial General Intelligence (AGI). Advancements in scaling language models (Brown et al., 2020; Thoppilan et al., 2022) allowed to achieve novel discourse, reasoning and zero/few shot learning capabilities that can be applied to new tasks without/minimal additional training. Recent work aims
16

---

Under review as a conference paper at ICLR 2023
to extend these achievements beyond text modality by defining static architectures for extended subset of modalities (Alayrac et al., 2022; Reed et al., 2022). These are few examples the ML models contributing to the line of research achieving incremental milestone toward AGL. Though, each models trained from scratch with considerable resources consumption. The introduction of abstractions allowing modularize, dynamically extend and reuse these large models may contribute to accelerate the rate of innovation.
B EXPERIMENTS REPRODUCIBILITY AND DETAILS
All the experiments reported in this paper can be reproduced by using the following public resources:
The ViT model definition and checkpoints published by Steiner et al. (2021). These resources are available at https://github..oo google-reeeercchiiiion transsontrrnsorrmer and distributed under the Apache License 2.0.
Published code of the proposed method: https://... (Anonymized, refer to supplementary material).
All the used datasets are publicly available via the Tensorflow Datasets image classífication, catalog. Refer to http::/www.teessrrfll orgg/attsetsccaaallgovvviiiiiiie for detailed informa- tion regarding each dataset licence and other metadata. Table 8 reports exact dataset splits and reference foor each task.
We also publish the J.2Net checkpoint resulting from the large-scale multitask experiment reported in Section 5. This checkpoint can be used for inference on any of the 69 learned image classification tasks, for further analysis, even be extended with additional tasks or methods. For information about the checkpoint and its license refer to: https://... (Anonymized)
References for mentioned state of the public model for different tasks are sourced from hutps:/paperrsiitccdd..com as of September 2022.
The VTAB- 1k results are reported for reference and directly comparable the state of the art, as the benchmark definition specifies that VTAB-full tasks cannot be used for pre-training.
The initial experiments reported in Sections and 5.1 have been executed on TPUv3 (Jouppi et al., 2017) machine with 8 cores. While, all the following experiments have been executed on a larger scale infrastructure using 32 TPUv4 chips MegaCore mode, by using the Pathways orchestration layer (Barham et al., 2022).
Table 7 reports more details for each training segment.
Table Details for the different training segments of the large scale continual learning experiment described in Section 5.
Training
TPU
#Tasks
#Params
Activated params
segment
core-hours
#cores
type
(M)
per task (%)
VIT tasks 10 iters
4949
TPUv3
3
659
46.6%
VTAB-full iters
5927
TPUv4
26
4400
7.0%
ViT tasks +1 iter
541
32
TPUv4
26
4577
5.2%
VDD iters
1507
32
TPUv4
36
7790
3.9%
VIT tasks +1 iter
564
32
TPUv4
36
7854
3.9%
VTAB-full +1 iter
2266
32
TPUv4
36
7596
4.0%
Char. class. iters
1785
32
TPUv4
44
9368
3.3%
VTAB-1k iters
271
32
TPUv4
69
13087
2.3%
ViT tasks +1 iter
568
TPUv4
69
13090
2.3%
17

---

Under review as a conference paper at ICLR 2023
Figure Graph representing the architecture of the multitask system solving jointly image classification tasks generated by the large scale continual learning experiment described in Section 5. Each task identified with a unique color. Bottom triangular nodes represent the data input of each task. Top rectangular nodes represent the head layer of each task. Each edges sequence of the same color connecting task input to its head, a path, defines the layers sequence composing the model for each task. Each path traverses 27 round nodes representing ViT L/16 internal layers (see Section 3.5) in the following order from bottom to top: patch embedding. class token, position embedding and 24 transformer layers. Internal nodes which the parameters of the corresponding layer were trained last. Except for the gray nodes that have not received gradient updates from any of the 69 tasks and still carry the parameters of the root model that were loaded from checkpoint of a ViT L/16 pretrained on the imagenet -2k dataset (see Section 5) (Video: youtu (outu.beHHasGeeee((3))
represented with the color of the task
18

---

Under review as a conference paper at ICLR 2023
ireraff
daimnler
10
omniglot ucf101
svhn
imagenet
10
Training samples per class
insagenet
ciaaróo
10"
gtarb
Eroor rike
cifar 100
svba
flwwrss
10
10-1
ucf101
air
10-1
daimler glarb
10 fowers
omniglot
10
10P
10
10"
Standard deviation
Standard deviation
Figure 4: Correlation in log scale with the standard deviation measured during the variance analysis conducted on Visual Domain Decathlon benchmark by running the training on 3 parallel replicas of the system. We display the 2 metrics that are most correlated with the standard deviation: error rate computed on the test set (left) and training samples per class (right). The red line is fitted minimize the squared distance to the set of points.
dala
54-
8.5
82-
824
891
625
10
6/6
dip Jali
Iac didta
suamion doll.
apt.b:
1425
65-
54
033
55
60-
61
<ych votooppatti
1Pt EMERYTORE
cpt ueskoow
0.25
65-
68-
015
08
100
86
840.5
99390
Figure Distributions of the hyperparameter values used by the best models of the 69 image classification tasks at the end of the large scale continual learning experiment described in Section 5.
19

---

Under review as a conference paper at ICLR 2023
Table 8: Datasets splits and reference (part 2). For each dataset used in the experiments, this table reports: 1) dataset name indicative of the Tensorflow Datasets Catalogs identification string and linking the corresponding catalog page ("visual domain, ddecathloon''" has been abbreviated as "vdd". and "diabetic_ caaabetic_retnopaall "itinopptthydtttcttionn ___deeeeeton" as "drd"), 2) train, validation and test data splits, represented with the standard Tensorflow Datasets format ("validation" " has been abbreviated as "val"). the system.
3) corresponding scientific publication reference. Datasets are listed in the order of introduction into
Notes:
[1] The test split of the imagenet_ v2 dataset is used as validation set for imagenet2012.
[2] The test split of the cifar10_] dataset is used validation set for cifar10.
[3] The VTAB-full benchmark also includes the cifar task. Cifar 100 has been introduced to the
u2Net system as part of the initial benchmark. In the VTAB-full results tables refer the top
1 test accuracy achieved in the latest cifar100 training iteration without retraining it as part of the
VTAB-full active training iteration.
[4] The definition for the VTAB standard and additional tasks has been sourced from aidapaation/rre/maate/aaaaaakk [5] VTAB additional task, not included in the standard scoring set. These tasks were added to further
scale the system and analyze transfer across related tasks.
Splits
Name
Train
Val.
Test
Reference
imagenet2012
train train[:98%
imagenet v2ttest
val
(Russakovsky et al., 2015)
cifar100
[98in]
test
(Krizhevsky, 2009)
cifar10
train
10 [e
test
(Krizhevsky, 2009)
VTAB-full benchmark [3[4]]
caltech101
[r7in[
train[ans4:]
test
(Fei-Fei et al., 2004)
dtd
val train val
train
test
(Cimpoi et al., 2014)
oxford_flowers
test
(Nilsback & Zisserman, 2008)
seford_iiit_pe
train[:2944]
[2900[ test
(Parkhi et al., 2012)
sun397
train
test
(Xiao al., 2010)
svhn_croppen
train[36s9s]
train[ss9a:]]
test
(Netzer al., 2011)
patch_camclyon
train
val train [6000221600]
test
(Veeling et al., 2018)
curosat/rgb
train[[16200] [16200]
train [21600:]
(Helher al., 2019)
resisc45
(rain[[1s900
train 18900:25000
train [s200:
(Cheng al., 2017)
drd/bigraham-300
train
test
(Kaggle & EyePacs, 2015)
Dlevr/coun_eeylindeee cylinders(5)
train[ [63000]
val
(Johnson et al., 2017)
clevr/count_ail
train [36000]]
train 630000]
val
(Johnson al., 2017)
blevr/cosest_ohjoot_.oooo _isttance_ dmlab
train [63000]
train[ssomn:
val
(Johnson et al., 2017)
train
val
test
(Zhai et al., 2019a)
dsprites/label_1 x_possiion
train[ [309924]
train (aainsse22200
train [663s52]
(Klindt et al., 2021)
Asprites/label_orietteee orientation
train| [389924]
train 589824066955]
val train [663522
(Klindt et al., 2021)
kitt//clsess_cojccttddisttt !
train
test
(Geiger et al., 2012)
kitti/count 15
train
test
(Geiger al., 2012)
kitti/Closest_vehhieee vehicle distance
train
val
test
(Geiger al., 2012)
smallnortVlabellcateee cattroory__tggt51811
train
test[:5096]
test[50%%:
(L.eCun al., 2004)
smallnort/label_igee
h (9ghting [9]
train
test[:500%]
test[50%:]
(LeCun al., 2004)
smallnorb/label_azee azimuth
train
test[:50%
test[50%:]
(LeCun al., 2004)
smallnorb/label elevatai
train
test[:50%%]
test[50%:]
(LeCun al., 2004)
Visual domain decathlon benchmark
vdd/imagenet12
train
val[:50%%
val[50%
(Bilen al., 2017)
vdd/svhn
train
val[:500%
val[50%
(Bilen et al., 2017)
vdd/cifar 100
train
val[:50%
val[50%::
(Bilen et al., 2017)
vdd/gtsrb
train
vall:
val[50%:
(Bilen al., 2017)
vdd/daimlerpedels
train
vall:50%
val|50%:
(Bilen al., 2017)
vdd/omniglot
train
vall:50%
val[50%::
(Bilen et al., 2017)
vdd/ucf101
train
val[:500%]
val[50%
(Bilen et al., 2017)
vdd/aircraft
train
val[:50%
val[50%:]
(Bilen et al., 2017)
vdd/dtd
train
vall val[:50%
val[50%
(Bilen al., 2017)
vdd/vgg- flowers
train
vall:
val 50%%
(Bilen al., 2017)
Continues in Table
20

---

Under review as a conference paper at ICLR 2023
Table 9: Datasets splits and reference (part of 2).
Splits
Name
Train
Val.
Test
Reference
Continues from Table 8
Multitask Character Classification Benchmark
emnist/digits
train [5%:]
train[:5%]
test
(Cohen et al., 2017)
emnist/letters
train[5%] [6:]
train[:5%]
test
(Cohen et al., 2017)
kmnist
(rain[5%:
train[:5%]
test
(Clanuwat al., 2018)
mnist
train[5 [5%:]
train[:
test
(LeCun et al., 1998)
omniglot
train
smalll
small2
(Lake et al., 2015)
ematerdb/bangla
20%:: [rain]
train[:20%c]
test
(Das et al., 2012b;a)
cmaterdb/ddevnngarii
train| 20%::]
train[:20%]
test
(Das et al., 2012b;a)
cmaterdb/telugu
(rain[20%]
train[:20%[
test
(Das et al., 201 2012b;a)
VTAB 1k benchmark (4)
caltech100 101
train[:s00 -300]
train[2754:0954)
train 2754:2954]
test
(Fei-Fei et al., 2004)
cifar100
train [000]
train (40000445200
test
(Krizhevsky, 2009)
cifar10 dtd oxford flowers102
train [200
train 440000445200
test
(Krizhevsky, 2009)
train[:w0) [:000]
val[:200]
test
(Cimpoi al., 2014)
(rain[:so)
val[[2o0]]
test
(Nilsback & Zisserman, 2008)
oxford Liit_pp pet
[rain[:x0]]
train] train[2s44:3144
[294423144]
test
(Parkhi et al., 2012)
sun397
train[ [2000]
vall
test
(Xiao et al., 2010)
svhn_cropped
train[:300 [:000]
[69011666131 train[sson
test
(Netzer al., 2011)
patch_camelyon
train [:00]
[al[
test
(Veeling et al., 2018)
curosat/rgb
train [:00]
train 162002 16400]
train [2100:]]
(Helher et al., 2019)
resisc45
train [:0o]
train 19100119100)
train [5000:
(Cheng et al., 2017)
drd/htgraham-300
train[:wm [:no]
vall
test
(Kaggle & EyePacs, 2015)
clevr/count
Bevz/count_cyllnnders! [y
train [:600]
train 63000063200
val
(Johnson et al., 2017)
clevr/count_ all
train
train 63000062200)
val
(Johnson et al., 2017)
ccov/closest.oojeet.iiittee dmlab
[:00]
train [63000663200
val
(Johnson et 2017)
train[ [=00]
val[:200]
test
(Zhai et al., 2019a)
dsprites/lhee___pssiine
train [x00]
train 58088449900044
train [663552]]
(Klindt et al., 2021)
train [200]
train (19834:9900244
train [663552]
(Klindt et al., 2021)
kitti/closett_obe _ojecst_______eeeeeeeee
distance
train [:00]]
vall :200]
test
(Geiger et al., 2012)
httl/coout__ehiclesse
[sm]]
vall val[:200]
test
(Geiger al., 2012)
cutticcloses__vehicce distance
train[:wn [=00]
val[:200]
test
(Geiger et al., 2012)
smallnnorbllaeel_aatee c
train[
test[:200 :200]
test[50%:]
(LeCun al., 2004)
train [000]
test [200]
test[50%::
(LeCun et al., 2004)
smallnorb/label_
[rain[:so]]
[200[
test[50%:
(LeCun et al., 2004)
smallnorb/label eleevation
(rain[:wn)
test[:20]]
test[50%:]
(LeCun et al., 2004)
21

---

Under review as a conference paper at ICLR 2023
Algorithm Pseudocode for one active task iteration
1: Active task: t
2: Set of all the models currently in the multitask system: M
3: Active population: A + {m m E M A m trained on 1}
4: for #generations do
5:
for #child-models do
6:
Sample parent model
7:
Parent model: p none
8:
for Candidate parent model: ý [sortedscore (A), sortedrandom (M A)] do
0.5 0c.5@sctccioos((..) > x ~ Unifurrn form((, 1]) then
9:
10:
p break end if end for
11:
12:
13:
if p none then
15:
Uniform(AUM)
end if
17:
Sample child model
Set of mutations: A < (makee-raainabl--heal)
19:
for Candidate mutation: E possible-mutations())) do μ > Uni fniform(O,, 1]) then
20:
+ AU
22:
end if end for
23:
24:
Untrained child model: co apply-mutations((2 A)
25:
D Train child model
26:
Retained child model:
27:
for i E [l,... 1 #train-cycles] do
28:
C; Lrain(c min epoch, #samples-cap))
29:
if score(c) 2 max((score(()))) # none) ((soore(p)) trained on {-oo- then
c+
31:
end if end for
32:
33:
if c + none then
34:
A Au end if
35:
36:
end for
37: end for
38:
> Keep only the best model for
39: M (argmax score(m) U (m m M A trained on t}
22


Reviews:
[{"review_id": "v3qOVhD4qT_", "reviewer": "Reviewer_uZHy", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The key idea is to share common modules among tasks to improve accuracy through cross-task knowledge transfer. Specifically, the paper proposes an evolutionary algorithm that iteratively trains the model over tasks where its sub-network for each task is evolved sequentially by mutating subsets of modules or hyper-parameters. Experiments show that the method improves over iterations on several datasets, and outperforms the single-task fine-tuning baseline on ViT and VTAB benchmarks.", "summary_of_the_review": "In summary, it was an interesting paper with a novel method and extensive experiments. I think additional comparisons with existing works will be necessary.", "strength_and_weaknesses": "Strength\n\n- The method improves the baselines fine-tuned on individual tasks on some datasets (ViT and VTAB benchmarks).\n- One strength is its extensive experiments on various datasets, including the large-scale evaluation on 69 classification tasks.\n- The training procedure is visualized as a video to show the evolution of the model, which is very helpful in understanding the algorithm. The paper also provides various interesting analyses of the proposed algorithm.\n- The paper proposes a novel method that sequentially extends the model to enhance the accuracy of individual tasks by transferring knowledge over tasks.\n- The code is submitted, and the results will be reproducible.\n\nWeakness\n\n- I am not sure about the effectiveness of the search algorithm because the search space consisting of all combinations of mutations is very large, but the method searches for a few iterations with several generations and childs. Ideally, different trials should converge to a similar solution regardless of the task orders used during training. Will this be the case? I think it would be nice to include such a discussion.\n- Is there a specific reason to use only 2 active task iterations? Will the longer iterations help find better architecture and further improve the accuracy, or is it already saturated with 2 iterations?\n\n- Computational cost looks pretty high, because it requires to train models for #task * #(active task iteration) * #generation * #child times.\n\n- It would be nice if the effect of the order of active tasks can be studied. The method can be used for multi-task learning, as well as continual learning, which allows access to the previous task data during training. For the multi-task learning setting, what will be the optimal root model to start from? Also, what will be the best order of task activations?\n\n- Experimental validation seems not sufficient. There are existing methods that also allow transferring knowledge across tasks, such as PNN [A] and DEN [B], by extending architectures for a sequence of tasks. Comparison with such methods in the experiments will be necessary. In addition, I think a multi-task model with a shared backbone (e.g. multi-head model, classification model for the unified label space) is another baseline to compare with, which could be a simple but effective way to allow knowledge transfer across-task.\n\n- Current experiments are all restricted to classification tasks, which are relatively similar tasks. It would be interesting to see if the method applies to more general tasks beyond classification tasks. For example, can this method apply to dense prediction tasks, such as segmentation, depth estimation, and surface normal estimation (Taskonomy, NYUv2, Pascal-Context), where there can be clear conflict across tasks?\n\nOther minor things\n- Analysis of some hyper-parameters is missing. How is the search space decided? What is the effect of different search spaces (different ranges of hyper-parameters)? How is the number of cycles for validation decided, and what is the effect of different choices? Does it need to be long enough to ensure the model is converged? How do you select the training and validation set to use?\n- How is the order of active tasks decided? What is the effect of different orders and choices of the root task model?\n- Is Zahi et al in Table 3 share the same backbone with the proposed model, being comparable?\n- The purpose of Figure 4 is unclear to me. Is it to claim that the method improves more if the task is more difficult (higher variance)? Also, in Figure 4, training samples per class are less than 1 for some tasks. Does it mean there can be 0 sample for some classes?\n- The purpose of section 5.5 VTAB-1k benchmark (Table 6) is unclear to me.\n\n\n[A] Progressive Neural Networks, arXiv 2016\n[B] Lifelong Learning with Dynamically Expandable Network, ICLR 2018", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "J9OO7RYXZAQ", "reviewer": "Reviewer_W895", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an evolutionary approach, $\\mu$2Net, to dynamically generate a large scale multitask network for multiple tasks. The evolutionary approach comprises (1) sampling a parent model, (2) mutations, (3) training and scoring children models. Extensive experiments are performed on numerous datasets.", "summary_of_the_review": "Interesting paper but lacks sufficient novelty and comprehensive empirical evaluation.", "strength_and_weaknesses": "Strengths\n+ The paper focuses on large scale datasets and tasks which is challenging in multi-task learning and continual learning.\n+ Comprehensive Experiments\n\nWeaknesses\n- No baselines (except ViT [Dosovitskiy et al., 2021], [Steiner et al., 2021], and [Zhai et al., 2019b] in some but not all experiments) despite the presence of recent similar methods such as RCL [Xu and Zhu, 2018] and DEN [Yoon, 2018] as well as of course well-known paradigms such as multi-task learning.\n- \"The presented experiments aim to compare against the state of the art for a large number of tasks without any size or compute constraint. Therefore, the validation accuracy is used directly as the score without additional factors.\" This is a major weakness of the paper especially if the focus is on large-scale multi-task learning systems.\n- The empirical evaluation lacks any mention or analysis of model complexity (especially compared with baselines).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "HNV2l_rgy4", "reviewer": "Reviewer_uLXC", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new method for dynamic large-scale multitask learning and conducts some experiments to study its effectiveness.", "summary_of_the_review": "This paper studies dynamic large-scale multitask learning problems. However, the problem is not well-formulated and the proposed method is not clearly described. It is poorly organized. ", "strength_and_weaknesses": "Strength\n1.\tThis paper aims to provide a new and automatic method for dynamic large-scale multitask learning.\n\nWeaknesses\n1.\tThe main contribution of this paper is unclear. Although it claimed that the proposed method possesses 8 novel properties, they either somewhat overstated the ability or applicability of the proposed method or were not well-supported. The main idea of how the proposed method copes with dynamic large-scale multitasking is not clear. How the automation is achieved is also unclear.\n2.\tA more comprehensive review should be provided. How existing works deal with dynamic multitasking problems and what is the scale of the problems they face are not well presented. \n3.\tThe problem studied in this paper is not well-formulated. It is not clear what is meant by \"large-scale\" as well as how large is large in this paper.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "8i6zWE-ar7", "reviewer": "Reviewer_3FfT", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present an evolutionary algorithm capable of dynamically building a large and performing multi-task model over time. The prosed method can be considered a union of multi-task learning and continual learning, while trying to relate both to the more general concept of knowledge trasfer through experience. The authors present the algorithm and a large set of experiments to validate their claims", "summary_of_the_review": "Great results given a huge computational power.", "strength_and_weaknesses": "strenghts\n\n    - The evolutionary algorithm proposed is quite simple, straightforward and seems to work very well both in terms of performance and knowledge sharing. The idea of always freezing the parent models and only allowing weight updates through novel generations allows the method to be immune to typical continual learning issues such as catastrophic forgetting and typical multi-task issues such as gradient interference. \n    - The experimental section of the paper covers an extremly broad range of tasks and validates several claims made in the paper. Overall the section is very extensive.\n    - The results on all the benchmarks are quite impressive.\n\n\nweaknesses\n\n    - Fig. 1 leaves me with some doubts. It would seem that the private task is solved by using only a head operating on the learned layer for the green task (devanagari). This is at least what I would expect for the claims of the method to still uphold, because if the private task head can alter the weights of the Transformer layer 1 then information from the private task is flowing into the network. I would appreaciate if the authors could clarify this.\n    - Overall a lot of choices seem to lead towards the necessity for large compute power. The choice of modifying hyperparameters only by a one-hop neighbor is quite restrictive and it implies that we have to evolve/search for quite a while before stumbling on the correct hyperparams. The layer cloning and mutation probability hyperparameter is set at random by the evolutionary process, implying that the level of overall randomicity is very high and therefore large training times are needed to get stable results or be able to reproduce the claimed results (considering the authors use DNN architectures). The authors mention that \"the score can be defined to optimize a mixture of factors depending on application requirements\". It would have been nice to see what the tradeoff between training time and model size vs optimal multi-task performance is, especially considering these high levels of randomicity present in the proposed approach. (and also for others to be able to reproduce somewhat similar results on lesser compute).\n    - The parameters in Table 1, the model and the experiments seem to be only good for image data and ViT. Did the authors try to apply the same principles to other areas research areas such as NLP or simpler models in the image domain (CNNs)? I understand the latter might be due to the focus about state of the art performance, but it would show that the method can generalize to different architectures and tasks, not just transformers in vision.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject