Visually Grounded Continual Language Learning with Selective Specialization
Authors: Kyra Ahrens
Lennart Bengtson
Jae Hee Lee
Stefan Wermter
Venue: EMNLP 2023 Findings (2023)
Keywords: continual learning
lifelong learning
vision-language
language grounding
TL;DR: This study explores module specialization strategies in vision language models to strike the balance between specialization and generalization in continual learning.

Abstract: A desirable trait of an artificial agent acting in the visual world is to continually learn a sequence of language-informed tasks while striking a balance between sufficiently specializing in each task and building a generalized knowledge for transfer. Selective specialization, i.e., a careful selection of model components to specialize in each task, is a strategy to provide control over this trade-off. However, the design of selection strategies requires insights on the role of each model component in learning rather specialized or generalizable representations, which poses a gap in current research. Thus, our aim with this work is to provide an extensive analysis of selection strategies for visually grounded continual language learning. Due to the lack of suitable benchmarks for this purpose, we introduce two novel diagnostic datasets that provide enough control and flexibility for a thorough model analysis. We assess various heuristics for module specialization strategies as well as quantifiable measures for two different types of model architectures. Finally, we design conceptually simple approaches based on our analysis that outperform common continual learning baselines. Our results demonstrate the need for further efforts towards better aligning continual learning algorithms with the learning behaviors of individual model parts.

Visually Grounded Continual Language Learning with Selective
Specialization
Kyra Ahrens
Lennart Bengtson Jae Hee Lee
Stefan Wermter
University of Hamburg
[kyra. lennart bengtson, jae hee lee, stefan. wernter (unn hwwbaaambb..........
Abstract
Z
Training from scratch
Continual fine-tuning
A desirable trait of an artificial agent acting in the visual world is to continually learn se- quence of language-informed tasks while strik- ing balance between sufficiently specializing in each task and building a generalized knowl- edge for transfer. Selective specialization, i.e., a careful selection of model components to spe- cialize in each task, is strategy to provide con- trol over this trade-off. However, the design of selection strategies requires insights on the role of each model component in learning rather spe- cialized or generalizable representations, which poses gap in current research. Thus, our aim with this work is to provide extensive analy- sis selection strategies for visually grounded continual language learning. Due the lack of suitable benchmarks for this purpose, we intro- duce two novel diagnostic datasets that provide enough control and flexibility for thorough model analysis. We assess various heuristics for module specialization strategies well as quantifiable measures for two different types of model architectures. Finally, we design concep- tually simple approaches based on our analy- sis that outperform common continual learning baselines. Our results demonstrate the need for further efforts towards better aligning con- tinual learning algorithms with the learning be- haviors of individual model parts.
D
D2
Dr
Continual Task Stream
Specialization (Expert)
Generalization (Monolithic)
Selective Specialization
Figure 1: The specialization ineraalizationgggeeralliazaooo trade-off in a continual learning setting. Expert solutions train a copy of the network for each task, while monolithic networks make unrestricted updates to all parameters. In this paper, we focus on selective specialization strike balance hetween full specialization and generalization.
Ideally, the underlying neural model of an agent would sufficiently solve each isolated task (i.c., spe- cialization) while harnsssing the shared structure and subproblems underlying all tasks to create gen- eralizable representations for knowledge transfer (i.e., generalization). A strategy to provide con- trol over this trade-off is to introduce task-specific parameters a carefully selected subset of model components (i.e., selective specialization), as can be seen Fig. 1.
1 Introduction
Grounding language in visual perception is cru- cial step towards agents that effectively understand and interact with the physical world (Bisk al., 2020). realistic setting, such an agent would require the ability to continuously integrate novel experience and skills with existing knowledge in an open-ended process, a challenge commonly known Continual Learning (CL) (Chen and Liu, 2018; Parisi al., 2019; Wang al., 2023).
Understanding which model components (here referred to as modules) are suited for task- specialization requires insights on their role in solving each task, which prior works on contin- ual Vision-Language (VL) grounding fail to ex- plain. At the same time, there is lack of suitable benchmarks that allow for a fine-grained model analysis, as existing CL scenarios for language grounding based on synthetic images are too sim- plistic concerning the CL problem (e.g., only one
Code and datasets will he made available at https:// github... wwm/ky-eee..ooooo.oo..... ab/ /se seleee lect ve -eee11 - lilac.

---

distributional shift) (Greco et al., 2019) or the VL grounding problem (c.g,, single-object, trivial lan-
3. We show that carefully balancing the trade-off between generalization and specialization via selective specialization helps us design sim- ple approaches that outperform CL baselines (cf. Sec. 4.2.4), ultimately demonstrating the importance of aligning CL methods with the learning behavior of individual model compo- nents.
guage) (Skantze and Willemsen, 2022), while those based on real- world images (Srinivasan et al., 2022; Jin et al., 2020) may encourage models to take shortcuts in the rision-language reasoning process, which leaves the generalizability of statements de- rived from any model analysis questionable. To this end, we introduce the LIfelong LAnguage Compo- sitions (LILAC) benchmark suite that comprises two diagnostic VI datasets that allow for investigat- ing the continual learning behavior the models with a high degree of control and flexibility while being challenging enough to require object local- ization, spatial reasoning, concept learning, and language grounding capabilities of the continual learner.
Background
2.1 Continual Learning Setting
An overview of the learning setting is provided in Fig. 2. We consider a VL model composed of a language encoder g(-), visual feature extractor h(-), a VL fusion network fo(-), and decoder
d(-). We initialize the model at time tin and freeze g. h, and d afterwards. After initialization, the model learns an ordered sequence of tasks from their training sets Dr for E (1.2...., T) and up- dates VL fusion parameters 0 to perform well on the test sets. We assume access to the task identity of every model input during training and testing. Each data point in D, is composed of visual ob- servations and input instructions (4, Ot, (o oz where of and o, denote the visual scenes corre- sponding a correct and a wrong execution of the instruction 4 upon observing or, respectively. This setting is an extension of Natural Language Inference (Bowman et al., 2015) towards visual language grounding that can be approached using contrastive learning (Li et al., 2023), where we consider (o,, to be image-text premise and of of to be visual hypotheses.
Based on the two proposed LILAC benchmarks, we conduct fine-grained analysis two different vision-language model architectures this work that comprises the following steps: (i) We analyze whether selective specialization generally benefits from dividing the learning process into the two in- termittent stages of task adaptation, where only the specialized parameters are being updated, and knowledge consolidation, where only the shared parameters are updated with respect to the task- specific learned representations. (ii) We derive and evaluate heuristics from prior literature about introducing task-specific parameters with respect to different layer depths and specific VL network modules. (iii) We assess the suitability of param- eter importance measures from pruning research indicators of the efficacy of module selection strategies.
2.2
The Sperialiattion-GGeeelliitttii Trade-off
We conclude our work by demonstrating the superior performance of module selection strate- gies found in our analysis, when trained under the (A&C) procedure de- scribed above, over common CL baselines. Thus, we summarize the contributions made in our work as follows:
Let 6g denote the set of parameters that contains a subset of parameters specific to task t. Then we can express the learning objective finding parameter values . ......,,0 Or for network fe, that maximize the average accuracy across all test sets. From this objective arises a parameter- sharingg trade- off between specialization and generalization that can be addressed by different design choices for f (Ostapenko et al., 2021).
1. We propose two novel datasets for visually grounded continual language learning whose problem spaces have well-defined shared structure, thus inherently promoting careful selection of specialized modules (cf. Sec. 3). 2. We provide a thorough analysis of the efficacy of different selective specialization strategies for two representative dision-languee archi- tectures (cf. Sec. 4.2.1-Sec 4.2.3).
One approach is to use monolithic network with parameters to be fully shared across tasks such that 0 0, V (L....,T] T) (Kirkpatrick et al., 2017; Chaudhry et al., 2019b), which fa- cilitates knowledge transfer between tasks at the cost of forgetting. The opposite approach is to

---

Prozen ohtrr intialization fuil
Maximization objectie
No trainable parameters
Minimization objective
9
Cosine Simllartty
Cosine Similorty
Fusion Network
Decoder
Decader
Decoder d(-)
"move box the right
(w(())
d(-)
Pooling
Pooling
Pooling
Figure Example of the LILAC-2D dataset. Based the instruction and the visual premise, model learns to separate the two visual hypotheses (true target image, false target image) corresponding to right and a wrong understanding of the instruction, respectively.
Text Encoder g(-)
image Encodda
Image Encoder k(-)
Image Encoder A(-)
Iv-))
4
the big green blok the syen bow"
2.3 Intermittent Adaptation and Consolidation (A&C)
Figure 2: Overview of the model training on LILAC- 3D example. Each of vision-language fusion layers (gray box) contains modules that candi- dates for specialization.
We apply simplified version of the lifelong com- positional learning scheme introduced in Mendez and Eaton (2021) that is inspired by Piaget's the- ories on intellectual development (Piaget, 1976) and has been successfully applied CL neural architectures with specialized modules: Instead updating 0g jointly upon learning task L, we repeat- edly alternate between multiple adaptation steps (here, epochs) to update the task-specific param- eters 05 (assimilation, fast learning) and single consolidation step that updates the shared param- eters 0MS' (accommodation, slow Icarning). We show in Sec. 4.2.1 that the adaptation-consolidatie (A&C) learning scheme can consistently improve performance over joint optimization of 6, The algorithm pscudocode can be found in Sec. A.5.
train task-specific expert solutions (Aljundi al.. 2017: Rusu al., 2016) such that 0, 0g for i,j E (1....,T) with + j. which allevi- ates forgetting at high memory most and impeded transfer. van de Ven and Tolias (2019) propose to balance this trade-off by learning task-specific "multi-headed output layers. However, this ap- proach assumes uniform feature space and does not work for VL fusion networks that are trained continually.
Inspired by works modular continual learn- ing (Ostapenko al., 2021; Mendez and Eaton, 2021; Mendez al., 2022), we consider / as set M of modules, where module in this con- text can be any kind of self-contained paramet- ric function in (e.g., a convolutional layer or a batch normalization layer).2 Our aim is to find a strategy for selecting a subset S C M of mod- ules that is task-specific and parameterized by 0ş for & (1...., T). while keeping the remaining modules shared across tasks, thus parameterized by (MVs,, yielding 0: (0S, (M Notably, S 0 for monolithic (or fully shared) network and S = . for the expert solutions.
LILAC Datasets
In what follows, propose two benchmark datasets that explicitly model the compositional nature of a problem, thus maintaining high de- gree of overlap across tasks. This should encourage the models to use as few task-specific parameters as possible and facilitate the exploration of rea- sonable selection strategy for specialized modules. During training, the model receives as input three images representing objects in simulated envi- ronment, along with templated language instruc- tion. Examples of the two proposed datasets can be found in Fig. and Fig. 3.
LILAC-2D tasks. This dataset based on the minigrid Shhevalier-ooiveere ronments. For each example, three to nine objects are randomly placed in a x 7 grid. Each instruc- tion describes a desired interaction with a desig-
al., 2018) envi-
'Hence, the notion of a module in this paper is more gen- eral than that in a neural module network (Andreas et al., 2016).

---

nated target object and takes the form of "move the <color> <object> <direction>", where we choose from a set of six colors, three object types, and four directions. All three subproblems described by the instruction can be orthogonally combined, yielding 6 x 3 x 4- = 72 distinct instructions. False target images are generated by randomly choosing one of the three subproblems to be wrongly solved, e.g.. blue key is moved down, although green key was supposed be moved down. The LILAC-2D dataset comprises 500 train, 100 validation, and 100 test samples per instruction, yiclding a total of 36,000 train and 7,200 validation and test samples, respectively. For the continual learning stream, we construct T = 10 tasks comprising training sam- ples of six different instructions each and keep sam- ples from the remaining 12 instructions for model initialization at time linit-
VL Transformer Layer
FILMed Layer
Layer imommii
RKLU
Bench Norn (6m2)
Feed Forward (lnd)
(blas)
Trainable modides
Foed Forward (fm))
Sall-Anention
Aoelgny
A Layer (momm))
Corstayeer
Fully Connected
(oonud)
2D Convolution
Ree.U
Mulii Hoad Attantion
(bm1))
(etle)
Conslayer (oonvi)
a(())
b(o,)
A(or)
Figure 4: The VL. fusion networks fo, (-) used in our experiments consist of L consecutive VL transformer or FiLMed layers. Each network receives as input the encoded language input and vision input to produce joint representation fs, (o()), h((o)). Each colored box a module m and one or multiple modules can be part of selection strategy S.
LILAC-3D tasks. To further narrow the gap to real 3D images while maintaining a high degree of control and flexibility for model analysis, we additionally propose LILAC-3D, a dataset with in- creased spatial complexity and distracting informa- tion. It is based on the simulated Ravens (Zeng al., 2021) benchmark and its extension towards language instructions (Shridhar et al., 2022). Im- ages show a tabletop scenario, where between five and cight blocks and between three and four bowls are randomly placed within the range of a robot arm. Instructions describe a pick-and-place opera- tion with target block and a target bowl and take the form of "put the <size> <colorl> block in the <color2> bowl", where choose from set of two different sizes, six block colors, and six bowl colors, thus yielding total of 2 x 6 x 72 dis- tinct instructions. The sets of colors for blocks and bowls are fully disjoint to allow the objects to be clearly distinguished from one another. False target images are designed in way that either the wrong block and the right bowl or the right block and the wrong bowl are chosen for interaction. The dataset statistics well the continual stream design are similar those of the LILAC-2D dataset.
fe, (-), which are both well established in research visual language grounding both in supervised and imitation learning settings (cf. Tan and Bansal, 2019; Chen et al., 2020; Panos al., 2023; Hui et al., 2020; Lee et al., 2022). An overview of the two VL fusion networks can be found in Fig. 4, where we also indicate the modules that we con- sider for selection strategies in the experiments. We use the first block of a R (He et al., 2016) as image encoder h(.). For the VL transformer, g(-) is the concatenation of word embeddings of the in- put instruction, whereas for the FiLMed network, g()) a sequence encoding by a single-layer Gated Recurrent Unit (GRU) (Cho et al., 2014). The projection decoder d(.) is a linear fully-connected layer. The VL fusion is followed by max-pooling operation for FiLM and mean-poot mean-pooling operation for the transformer. All model configurations were found based on extensive hyperparameter tuning achieve maximum performance on each validation set in the i.i.d. training setting and are described in detail in Appendix A.
We compare to the following base- lines: Sequential fine-tuning (SFT) performs unre- stricted updates on monolithic architecture with all module parameters shared across tasks and is usually considered as lower bound for CL model performance. Experience replay (ER) (Chaudhry et al., 2019b) is an extension to SFT which stores samples in a fixed-size buffer via reservoir sam-
Baselines.
4 Evaluations
4.1 Experimental Setup
Model architectures. We conduct experiments with transformer encoder (Vaswani et al., 2017) and a fcature-wise lincar modulation (FiLM) net- work (Perez et al., 2018) as our VL fusion networks

---

pling, from which samples are drawn for rehearsal. Elastic Weight Consolidation (EWC) (Kirkpatrick et al., 2017) is another extension SFT that regu- larizes parameter updates depending on their rela- tive importance. For our experiments, we use the Online EWC version (Schwarz et al., 2018) that does not require storing separate approximation of the Fisher information matrix per task. Consider- ing potential upper bound for model performance, multi-task training (MTL) optimizes monolithic architecture on all tasks in the i.i.d. training setting and independent experts (Expert) train randomly initialized set of modules separately for each task.
Transformer
FILM
LILAC-2D
LILAC-3D
LILAC-2D
LILAC-3D
Layer
3
2
1
attn
nom1
Fm1
FFn2
nomm
attn
nome
Fm1
nomz
conv2
bnt
cont
bias
bn2
convi
bnz
conv2
weight
pnz
-1.5
-1.
-0.5
0.0
0.5
1.0
1.5
Figure 5: Difference between accuracy gains from iso- lating each module between A&C (cf. Alg. 1) and joint training of shared and task specific modules. Green values indicate the superiority of A&C over joint train- ing.
Evaluation metrics. We report the average accu- racy across all tasks under choice of parameters 0 as (CC(O) (or, ACC(Or)) for the of all task- specific parameters 01:7 {f.....,
even consistently across all VL transformer mod- ules. This indicates that despite being updated less frequently, the shared network modules learn the subproblem underlying all tasks sufficiently well while being less prone to forgetting.
(et....,,r)) (r)). Fur- thermore, we measure the accuracy gain under specialization strategy compared with using a monolithic network as
AACC(S) ACC(( ACC(O)
4.2.2 Does the performance of different module selection strategies align with insights about general model behavior from prior literature?
where 1,: = { AMS)). Both (CC()1)) and AACC(S) are measured after training on the last task T.
Specialization different layer depths. To an- alyze the effect of isolating task parameters at dif- ferent layer depths for specialization, we construct one model for each of the transformer and FiLMed layers to be specialized, respectively. The results can be found in the first column of Fig. 6.
4.2 Results
In what follows, first examine whether the A&C learning scheme is useful for training with a module selection strategy S (cf. Sec. 2.2), such that we can use A&C throughout our experiments (Sec. 4.2.1). Next, we analyze different selection strategies and compare the results with findings from prior literature (Sec. 4.2.2). We then assess whether the efficacy of selection strategies can be quantified using importance scores from pruning research (Sec. 4.2.3). Finally, we compare several selection strategies with CL baselines (Sec. 4.2.4), Each experiment is run ten times with different ran- dom seeds that affect parameter initialization and task order in the continual stream.
For the VL transformer, cialization of late layers yields a higher accuracy gain than specialization of early layers. This indi- cates that parameters of carly transformer layers benefit from being shared across tasks, as such lay- ers learn transferable representations that should be slowly learned via infrequent updates. Such findings confirm prior research transformers for natural language processing (Hao al., 2019; Ten- ney al., 2019) claiming that late layers capture most local syntactic phenomena, while early layers capture more general semantics.
observe that spe-
4.2.1
Does the A&C learning scheme benefit training with module selection strategy?
In their analysis of FiLMed networks, Perez et al. (2018) 8) claim that early layers perform low-level reasoning, such querying attributes of an ob- ject, while late layers perform high-level reasoning. such comparing two objects. As solving the LILAC-2D tasks requires the model identify and interact with a single object, whereas for solving the LILAC-3D tasks, two objects would have to
To assess the effectiveness of the A&C learning scheme, measure the difference in accuracy gain under the selection strategy S {m] for each module m M, as shown Fig. 5. Overall, we observe a positive effect of introducing A&C for the majority of specialized FiLM modules and

---

nom1
FILM
ffn2
self-attni
bn2
norm2
Transformer
Mfn ff2
0n2+
nom1 norm2
LILAC-2D
AACC
3
10
LILAC-3D
AACC
10
5
3
5
2
3
4
2
3
4
all
all
1
2
3
4
all
all
Layer
Layer
Layer
Layer
Layer
Figure 6: Evaluation of different selection strategies S trained with A&C C for specialization. 1st column: Layer- depth analysis. 2nd+3rd column: Transformer feed-forwatt and multi-head attention modules. 4th+5th column: Normalization scaling factors.
be identified and put into shared context, hy- pothesize that specialization on LILAC-2D tasks happens early and specialization on the LILAC- 3D tasks happens late in the FiL.edd network. As shown in the first column of Fig. 6, introducing task specialization to the first layer yields the high- est accuracy gain for LILAC-2D, while the highest accuracy gain for LILAC-2D can be achieved by in- troducing specialized parameters the penultimate the last layer, which confirms our hypothesis.
In recent work parameter-effiiient transfer- learning methods, Smith et al. (2023) show the efficacy of adapting self-attention blocks in a vi- sion transformer to downstream image classifica- tion tasks while keeping the remaining transformer parameters frozen. To assess whether their findings can be transferred to our CL setting, we conduct experiments with specialized self-attention mod- ules and report our results in the third column of Fig. 6. We observe substantial increase in curacy, especially for specialization in intermedi- ate transformer layers, albeit the greatest increase (~30% on LILAC-2D, ~44 on LILAC-3D) is achieved by specializing self-attention parameters across all layers. Although it is worth noting that self-attention modules account for about 73% of the parameters in each transformer layer, the results in- dicate clear benefit from including self attention parameters in specialization strategies.
Transformer feed-forward and self-attention blocks. Geva al. (2021) consider the outputs the feed-forward networks to be a composition their kay-value memories and discover that such layers learn some semantic patterns, especially in early layers. Consequently, such layers can be suitable candidates for specialization during CL. The results of our experiments the second column of Fig. 6. We make two key observations, which are roughly consistent across datasets: First, isolating the first transformer feed- forward layer alone (ffnt) is as effective as isolating the entire feed-forward block (ffn/ +ffn).. Second, a selective specialization of this layer improves performance most if applied in the early layers (particularly, isolation at the second transformer encoder layer yields the best performance through- out). These observations show that, when chosen carefully, selection strategies can achieve high ef- fectiveness with only a small amount of specialized parameters.
be found in
Normalization scaling f'actors. Bilen and Vedaldi (2017) argue that specialized instance, layer, or batch normalization scaling factors can reduce task-specific biases, allowing them to intercept distributional shifts at low additional memory cost. However, as can be seen in the penultimate column of Fig. 6, specializing batch normalization parameters of the FiLMed network hardly improves (LILAC-2D) or even degrades (LILAC-3D) performance compared with fully shared batch normalization parameters. Nevertheless, introducing task-specific layer

---

Transformer
FILM
Jung et al., 2020), Let 0 denote the parameters of network module m during training and let d denote the parameters of the m-th module after training on the t-th task.
ULAC-22D
LILAC-3D
LILAC-2D
LILAC-3D
Layer
2
The gradient-based importance score IS (Sgra((1)) of module computed the sum of the L1-norm of each parameter w of m and the accumulated absolute gradients arriving at w during training on each task:
1
1.5
0.5
1.5
Layer
2
1
4
aD;
IS 1Syna = a
w +
Layer
3
dw
1=1
WEBm
2
(2)
1
where the constant 1/(T log(((")) is used for averaging across all tasks and normalizing by the magnitude of the parameter count of m.
afth
nome
fmz
tfn
nomz
atth
pomm
tinz
tfme
poma
con br
comv?
con
biag
bmz
conve
c Cont Sollle Saad bnz
nou
0.0
0.4
1.0
The activation- importance score (m)) is computed as the total activation at module m with the same normalization constant used above:
Figure 7: 1st row: Accuracy gain of each specialized module m E M over fine-tuning monolithic network. Each field represents an experiment with S = {m]] 2nd row: Normalized gradient-based importance score (m). 3rd row: Normalized activation-based im- portance score (m).
[mar(() :- a Σ Σ
[for (g(4), h(o.))
t=1 (l.o.)DD
normalization parameters in the VL transformer can slightly improve accuracy the LILAC-2D tasks and achieves a performance gain of more than 5% the LILAC-3D tasks. We addition- ally find that specializing layer normalization parameters that follow the multihead-attention operation (norml) is generally more effective than specializing those that follow the feed-forward block (norm2), and even outperforms specializing both (norml + +norm2) on the LILAC-3D dataset.
We calculate Pearson coefficient (Pearson, 1895) between the importance scores IS (m) and (m)) of each network module m E M (cf. Fig. 7, second and third row) and the relative accuracy gain yielded from isolating this module for task specialization (S (m)) (cf. Fig. 7, top row).
The Pearson values for the gradient-based im- portance score (FiL.M/Transformer) indicate a strong positive correlation for the LILAC-2D tasks (0.91/0.90) and weak positive correla- tion for the LILAC-3D tasks (0.09/0.40), respec- tively. Conversely, there no conclusive evidence from the activation-based importance score (2D: 0.48/-0.280 0.53, 3D: 0.09/-0.33 Our results sug- gest that the magnitude of the gradients on the pa- rameters of module is a better indicator of the per- formance gain from specializing the whole module than the activation of the module during training. As shown in Fig. 7 (second row), the 2D convo- lutions in the last FiLMed layer and the attention modules in the VL. transformer have high impor- tance and thus seem to be particularly suited for specialization.
4.2.3
Can find quantifiable measures for the effectiveness of module selection strategies?
Quantifying the importance of neural connec- tions to construct specialized weight masks common practice in neural pruning research for CL (Molchanov et al., 2019). We aim to evalu- ate whether such importance scores can not only measure the suitability of specialization for single parameters but also for entire network modules. Therefore, we will compare types of measurements for calculating the impor- tance score (IS) of each module in the network: Gradient-basee (IS gaa) and activation- (IS (cf. Wang et al., 2022; Gurbuz and Dovrolis, 2022;
commonly used

---

Table 1:
ACC scores of baselines for the proposed datasets and model architectures. All selective special- ization baselines are trained with A&C learning. Results from other selection strategies as well forgetting and forward transfer measures can be found in Appendix B.
LILAC-3D
Transformer LILAC-3D
LILAC-2D
Transformer LILAC-2D
1 Task Performance
6th Task Performance
1.0
0.8
ACC
0.6
Transformer
FiLM
0.4
Baseline
2D
3D
2D
3D
Expert
85.9
88.4
76,5
78.7
MTL
88.3
95.4
87.1
80.1
Task
Task
SFT
67.0
52.1
63.2
52.4
77.7
53.1
66.9
Figure 8: Accuracy of the SFT baseline the first and the sixth task during training. LILAC-2D tasks almost completely unlearned during training of the following tasks. Conversely, after training on LILAC- 3D tasks, model performance has slower decrease and tasks are not fully unlearned despite lack of means prevent forgetting.
EWC
55.5
79.0
56.1
69.0
Syyr-
70.0
76.5
68.4
64.2
Stu-layer 74.2
76.5
66.8
68.2
Sal--iat
63.1
75.0
Sean Comm-Jast
59.4
62.5
Sall-att
81.2
81.2
Sal-ann +
88.5
attn EWC
87.1
87.9
more brittle and subject to forgetting than learning LILAC-3D tasks, as can be seen Fig. 8. Thus, a reasonable module specialization strategy signifi- cantly increases a model's robustness to forgetting. An ad advantage of selective specialization is that it can be orthogonally combined with CL methods following the paradigms of replay regularization. We find in experiments that combining the most successful specialization strategy Sall-attn with ER or EWC by performing rehearsal or regularization during the consolidation phase of A&C learning reaches an accuracy close to the network experts and even outperforms it by 1.2% margin (Sall-attn + EWC on LILAC-2D). Such results indicate that a combination of selective specialization with other CL methods is not only robust to forgetting but tually promotes transfer between shared modules. thus successfully striking the balance between spe- cialization and generalization.
4.2.4 How does the performance of different specialization strategies compare against CL baselines?
Based on the analyses conducted in Sec. 4.2.2 and Sec. 4.2.3, our aim is to determine whether selective task specialization can outperform CL baselines for the proposed LILAC datasets. Given that all possible selection strategies form power set that grows exponentially with the number of modules in a network, we choose the following strategies for baseline comparison: In line with the findings on specialization at different layer depths, we compare with specialization of the first layer (Syer)tlayer) and last layer (Syest-layer) for trans- former and FiLMed blocks, respectively. As found the selection of the first feed-forward net- work in a transformer block be particularly ef- fective, we further compare with the specialization of such layer across all blocks (San-ffii. Finally, we add the two selection strategies of specializing self-attention modules in the transformer encoder (San-aum) and the convolutions in the last FiLMed layer (Scomv-last-laye) for comparison, the corre- sponding modules yield the highest IS gra scores. The results reported in Tab.
Related Work
Continual learning. In CL scenario, an agent is exposed to a sequence of tasks with the objective of learning to solve the currently seen task while main- taining high performance on previous tasks. Ap- proaches to CL can be broadly categorized into reg- ularization (Kirkpatrick et al., 2017; Zenke et al., 2017; Li and Hoiem, 2018; Aljundi et al., 2018) that restrict parameter updates to bound plasticity, (pseudo-Jreheanse (Rebuffi et al., 2017: Chaudhry et al., 2019a; Buzzega et al., 2021), where data that are either retrieved from a memory buffer or syn- thetically generated from previous distributions are
Multiple selection strategies exhibit superior per- formance compared to CL baselines LILAC-2D tasks. However, the sole strategy that surpasses all baselines on LILACC3D is the specialization attention across all layers. A possible explana- tion is that learning LILAC-2D tasks is generally

---

periodically replayed to the model, and dynamic architectures (Rusu al., 2016; Yoon al., 2018), where models are gradually expanded response to distributional shifts.
Conclusion
Striking a balance between specialization and gen- eralization poses a challenge for agents that learn a sequence of language-conditione tasks grounded in a visual environment. In this work, we con- sider vision-language models as a composition of modules and propose two datasets that allow us to analyze and compare strategies selectively spe- cialize modules to continual tasks with respect to different layer depths and module types. We further establish gradient-based importance measure that quantifies the suitability modules for specializa- tion. Finally, we show that the module specializa- tion strategies found in our analysis outperforms common CL baselines when trained under a con- learning
A few works explore the intersection of contin- ual learning and visually grounded language learn- ing with diagnostic (Skantze and Willemsen, 2022; Greco et al., 2019) and real-world (Srinivasan et al.. 2022; Jin et al., 2020) datasets. All works conclude that common CL baselines struggle with striking a balance between forgetting and cross-task knowl- edge transfer, yet do not provide any insights on how this struggle is connected with the learning behaviors of the architecture used.
Introducing task-specific parameters. Re- search continual neural pruning assigns some model capacity each task by iteratively pruning and retraining (uub-)networks that are specialized to each task (Mallya and Lazebnik, 2018; Geng et al., 2021; Dekhovich et al., 2023; Kang et al., 2022: Hung et al., 2019; Gurbuz and Dovrolis, 2022: Jung et al., 2020; Wang et al., 2022). However, while such methods are effective in overcoming forgetting, the evolution and learning behavior of pruned subnetworks provides little interpretability regarding the role of individual parts of the network in solving CL problems.
ceptually simple adaptation-consoliiii scheme. With this work, we show the merit of designing CL methods based on a careful analy- sis of selective specialization. Beyond introducing specialization to existing model parts, we plan to leverage our insights for the design of parameter isolation CL methods that introduce additional pa- rameters into pretrained model for future work.
Limitations
Generally, we consider the results and analyses provided in this paper be merely cornerstone towards gaining more knowledge about model be- havior during learning, an insight that can be used to better understand where in model to introduce task-specific parameters. Nevertheless, we recog- nize the following limitations of our work: First, we conduct our experiments with two established vision-languee language architectures with just one model configuration each. More notably, the architectures smaller and conceptually simpler than those used for large-scale realistic datasets on vi- sual language grounding. Second, introducing task specialization to network modules naturally has a lincar growth with the number tasks. This can, however, be mitigated by choosing smaller modules for specialization with reasonable perfor- mance to strike balance between parameter size and model performance.
Another line of research is modular CL, which trains a structural combination of independent pa- rameterized components under the common as- sumption that each component solves subprob- lem of each given task (Mendez and Eaton, 2021; Mendez et al., 2022; Ostapenko et al., 2021; Ve- niat et al., 2021). Similarly to neural pruning, ap- proaches to modular CL fail to explain which role the interplay between learnable structural configu- rations and shared components takes in solving the tasks. In this topic area, the analysis provided by Csordás et al. (2021) is closely related to work, except that we analyze model behavior on the level of entire modules rather than isolated parameters.
In an effort to promote parameter-efficien trans- fer in CL with foundation models, recent works utilize task-specific plugins such as capsule net- works (Ke et al., 2021a) or adapters (Ke et al., 2021b; Zhao et al., 2022; Ermis et al., 2022) that leverage the pretrained representations as shared structure. This line of research is parallel ours rather than competing, as we analyze existing parts trainable model rather than adding additional components to pretrained networks.
Acknowledgements
The authors gratefully acknowledge support from the DFG (CML, MoReSpace, LeCAREbot). BMWK (SIDIMO, VERIKAS), and the EU Com- mission (TRAIL, TERAIS).

---

References
Zhiyuan Chen and Bing Liu. 2018. Lifelong Machine Learning, Second Edition. Synthesis Lectures on Artificial Intelligence and Machine Learning, 12(3).
Rahaf Aljundi, Francesca Babiloni, Mohamed Elho- seiny, Marcus Rohrbach, and Tinne Tuytelaars. 2018. Memory aware synapses: Learning what (not) to for- get. In Proceedings of the European Conference on Computer Vision (ECCV).
Maxime Chevalier-Boiswe Lucas Willems, and Suman Pal. 2018. Minimalistic gridworld environment for gymnasium.
Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuyte- laars. 2017. Expert Gate; Lifelong Learning With a Network of Experts. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition (CVPR).
Kyunghyun Cho, Bart Van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares. Holger Schwenk, and Yoshua Bengio. 2014. Learning Phrase Representations using RNN En- RocerrDecoder for Statistical Machine Translation. In Proceedings of the 2014 Conference Empirical Methods in Natural Language Processing (EMNLP). pages 1724-1734, Doha, Qatar. Association for Com- putational Linguistics.
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2016. Neural Module Networks. In Pro- ceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 39-48.
Róbert Csordás, Sjoerd van Steenkiste, and Jürgen Schmidhuber. 2021. Are Neural Nets Modular? In- specting Functional Modularity Through Differen- tiable Weight Masks. In International Conference on Learning Representations.
Hakan Bilen and Andrea Vedaldi. 2017. Universal rep- resentations: The missing link hetween faces, text, planktons, and cat breeds. AtXiv:1701.07277 [es, stat].
Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai Mirella Lap- ata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, Nicolas Pinto, and Joseph Turian. 2020. Experience grounds language. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), P, pages 8718 8735, Online. Association for Computational Linguisties.
Aleksandr Dekhovich, David MJJ. Tax, Marcel H.F Sluiter, and Miguel A. Bessa. 2023. Continual prune- and-sclect: class-incremental learning with special- ized subnetworks. Applied Intelligence.
Beyza Trmis, Giovanni Zappella, Martin Wistuba, Aditya Rawal, and Cedric Archambeau. 2022. Mem- ory Efficient Continual Learning with Transformers. In Advances in Neural Information Processing Sys- tems, volume 35, pages 10642. Curran Asso- ciates, Inc.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015, A large anno- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empiri- cal Methods in Natural Language Processing, pages 632-642, Lisbon, Portugal. Association for Compu- tational Linguistics.
Binzong Geng, Fajie Yuan, Qiancheng Xu, Ying Shen, Ruifeng Xu, and Min Yang. 2021. Continual learn- ing for task-oriented dialogue system with iterative network pruning, expanding and masking. In Pro- ceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing ( (Volume 2: Short Papers). pages 517-523, Online. Association for Computational Linguistics.
Pietro Buzzega, Matteo Boschini, Angelo Porrello, and Simone Calderara. 2021. Rethinking Experience Re- play: Bag of Tricks for Continual Learning. In 2020 25th International Conference on Pattern Recogni- tion (ICPR), pages 2180-2187.
Arslan Chaudhry, Mare Aurelio Ranzato, Mareus Rohrbach, and Mohamed Elhoseiny, 2019a. Efficient Lifelong Learning with A-GEM. In International Conference on Learning Representations.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021. Transformer Feed-Forward Layers Are Key-Value Memories. In Proceedings of the 2021 Conference Empirical Methods in Natural Lan- guage Processing. pages 5484-5495, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elho- seiny, Thalaiyasingam Ajanthan, Puneet K. Doka- nia, Philip H. S. Torr, and Mare' Aurelio Ranzato. 2019b. Continual learning with tiny episodic memo- ries. ArXiv:1902.1 10486 [es, stat].
Claudio Greco Barbara Plank, Raquel Fernández, and Raffaella Bernardi. 2019. Psycholinguisties Meets Continual Learning: Measuring Catastrophic Forget- ting in Visual Question Answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. pages 3601-3605, Flo- rence, Italy. Association for Computational Linguis- tics.
Yen-Chun Chen, Linjie Li. Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. UNITER: UNiversal Image-TEi TExt Representation Learning. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan- Michael Frahm, edi- tors, Computer Vision - ECCV 2020, volume 12375, pages 104-120. Springer International Publishing, Cham. Series Title: Lecture Notes in Computer Sci- ence.
Mustafa B Gurbuz and Constantine Dovrolis. 2022. NISPA: Inspireds -laasictyy Adapta- tion for Continual Learning in Sparse Networks. In

---

Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 8157-8174. PMLR.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Ag- nieszka Grabska-Bart Barwinska, Demis Hassabis, Clau- dia Clopath, Dharshan Kumaran, and Raia Hadsell. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521-3222
Yaru Hao, Li Dong, Furu Wei, and Ke Xu. 2019. Visual- izing and Understanding the Effectiveness of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNE) pages 4141- 4150, Hong Kong, China. Association for Computa- tional Linguistics.
Jae Hee Lee, Matthias Kerzel, Kyra Ahrens, Cornelius Weber, and Stefan Wermter. 2022. What is Right for Me is Not Yet Right for You: A Dataset for Grounding Relative Directions via Multi-Task Leam- ing. In Proceedings of the Thirty-Fire First International Joint Conference on Artificial Intelligence (IJCAI- 22), pages 1039-1045. International Joint Confer- ences on Artificial Intelligence Organization.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recog- nition. In Proceedings of the IEEF Conference on Computer Vision and Pattern Recognition (CVPR).
Shu' ang Li, Xuming Hu, Li Lin, Aiwei Liu, Lijie Wen, and Philip S. Yu. 2023. A Multi-Level Supervised Contrastive Learning Framework for Low- Low-Resource Natural Language Inference. IEEE/ACM Transac- rions on Audio, Speech, and Language Processing, 31:1771-1783.
David Yu-Tung Hui. Maxime Cheialier-Boisvert, Dzmitry Bahdanau, and Yoshua Bengio. 2020.
BabyAl 1.1. ArXiv:2007.1277 12770 [es].
Ching-Yi Yi Hung, Che Tu, Cheng-En Wu, Chien- Hung Chen, Yi-Ming Chan, and Chu-Song Chen. 2019. Compacting, Picking and Growing for Unfor- getting Continual Learning. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Zhizhong Li and Derek Hoiem. 2018. Learning without Forgetting. Transactions on Pattern Analysis and Machine Intelligence, 40(12).
David Lopez-Paz and Mare' Aurelio Ranzato. 2017. Gradient Episodic Memory for Continual Learning. In Advances in Neural Information Processing Sys- tems, volume 30. Curran Associates, Inc.
Xisen Jin, Junyi Du, Arka Sadhu, Ram Nevatia, and Xiang Ren. 2020. Visually grounded continual learn- ing of compositional phrases. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLPP) pages 2018-2029, Online. Association for Computational Linguistics.
Arun Mallya and Svetlana Lazebnik. 2018. PackNet: Adding Multiple Tasks to a Single Network by Iter- ative Pruning. In Proceedings of the IEFF Confer- ence on Computer Vision and Pattern Recognition (CVPR).
Sangwon Jung, Hongjoon Ahn, Sungmin Cha. and Tae- sup Moon. 2020. Continual Learning with Node- Importance based Adaptive Group Sparse Regular- ization. In Advances in Neural Information Process- ing Systems, volume 33, pages 3647-3658. Curran Associates, Inc.
Jorge A. Mendez and Eric Faton. 2021. Lifelong Leam- ing of Compositional Structures. In International Conference on Learning Representations.
Haeyong Kang, Rusty John Lloyd Mina, Sultan Rizky Hikmawan Madjid, Jaehong Yoon, Mark SasseawwaJJhnson, Sung Ju Hwang. and Chang D. Yoo. 2022. Forget-free Continual Learning with Win- ning Subnetworks. In Proceedings of the 39th Inter- national Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 10734-10750. PMLR.
Jorge A. Mendez, Harm van Seijen, and Eric Eaton. 2022. Modular Lifelong Reinforcement Learning via Neural Composition. In International Conference on Learning Representations.
Pavlo Molchanov, Arun Mallya, Stephen Tyree, Turi Frosio, and Jan Kautz. 2019. Importance Estima- tion for Neural Network Pruning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
Zixuan Ke, Bing Liu, Nianzu Ma, Hu Xu, and Lei Shu.
a.21a. Achieving Forgetting Prevention and Knowl- edge Transfer in Continual Learning. In Advances in Neural Information Processing Syxtems, volume 34.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2019. Representation Learning with Contrastive Predictive Coding, ArXiv:1807 007748 [cs, stat].
Zixuan Kc, Hu Xu, and Bing Liu. 2021b. Adapting BERT for continual learning of sequence of as- pect sentiment classification tasks. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 4755, Online. Association for Computational Linguistics.
Oleksiy Ostapenko, Pau Rodriguez, Massimo Caccia, and Laurent Charlin. 2021. Continual Learning via Local Module Composition. In Advances in Neural Information Processing Systems, volume 34, pages 30298-3031 Curran Associates, Inc.

---

Aristeidis Panos, Yuriko Kobe, Daniel Olmeda Reino, Rahaf Aljundi, and Richard E. Turner. 2023, First Session Adaptation: A Strong Replay-Free Baseline for Class-Incremental Learning. ArXiv:2303.1311 13199 [es].
Tejas Srinivasan, Ting- Yun Chang. Leticia Pinto Alva. Georgios Chochlakis, Mohammad Rostami, and Jesse Thomason. 2022. CLiMB: A Continual Learn- ing Benchmark for Vision-and-Language Tasks. In Advances in Neural Information Processing Systems, volume 35, pages 29440-29453. Curran Associates, Inc.
German I. Parisi, Ronald Kemker, Jose I.. Part, Christo- pher Kanan, and Stefan Wermter. 2019. Continual lifelong learning with neural networks: A review. Neural Networks, 113:54-71.
Hao Tan and Mohit Bansal. 2019. LXMERT: Learn- ing Cross-Modality Encoder Representations from Transformers. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNL"), pages 5099-5110, Hong Kong, China. Association for Computational Linguistics.
Karl Pearson. 1895. Note on regression and inheritance in the case of two parents. Proceedings of the Royal Society of London, 58(347-352)2240-242.
Ethan Perez, Florian Strub, Harm de Vries, Vincent Du- moulin, and Aaron Courville. 2018. FiLM:: Visual Reasoning with General Conditioning Layer. In Thirty-Second AAAI Conference on Artificial Intelli- gence.
Ian Tenney. Patrick Xia, Berlin Chen, Alex Wang. Adam Poliak, R. Thomas McCoy, Najoung Kim, Ben- jamin Van Durme, Sam Bowman, Dipanjan Das, and Ellie Pavlick. 2019. What do you learn from con- text? Probing for sentence structure in contextualized word representations. In International Conference on Learning Representations.
Jean Piaget. 1976. Piaget's Theory. In Bärbel Inhelder, Harold H. Chipman, and Charles Zwingmann, edi- tors, Piaget and His School, pages 11 11-3.. Springer Berlin Heidelberg. Berlin, Heidelberg.
Gido M. van Ven and Andreas S. Tolias. 2019. Three scenarios for continual learning. ArXiv:1904.0771 [cs, stat].
Sylvestre-Alvise Rehuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert. 2017. iCaRL: In- cremental Classifier and Representation Learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc.
Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Ko- ray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. 2016. Progressive Neural Networks. 1xXiv:1606.0467 [cs]. ArXiv: 1606.04671.
Tom Veniat, Ludovic Denoyer, and MarcAurelio Ran- zato. 2021. Efficient Continual I Learning with Mod- ular Networks and Task-Driven Priors. In Interna- tional Conference on Learning Representations.
Jonathan Schwarz, Wojeiech Czarnecki Jelena Luketina, Agnieszka Grabska-Barwinsk Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. 2018. Progress & Compress: A scalable framework for con- tinual learning. In Proceedings of the 35th Interna- tional Conference on Machine Learning, volume 8O of Proceedings of Machine Learning Research, pages 4528-45 4537. PMLR.
Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. 2023. A Comprehensive Survey of Contin- ual Learning: Theory, Method and Application. ArXiv:2302.00487 [cs].
Zifeng Wang. Zheng Zhan, Yifan Gong, Geng Yuan. Wei Niu. Tong Jian, Bin Ren, Stratis loannidis. Yanzhi Wang, and Jennifer Dy. 2022. SparCL: Sparse Continual Learning on the Edge. In Advances in Neu- ral Information Processing Systems.
Mohit Shridhar, Lucas Manuelli, and Dieter Fox. 2022, What and Where Pathways for Robotic Manipulation. In Proceedings of the 5th Conference on Robot Learning, volume 164 of Proceedings of Machine Learning Research, pages 894 906. PMLR.
CLIPort:
Jachong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. 2018. Lifelong Learning with Dynamically Expandable Networks. In International Conference on Learning Representations.
Gabriel Skantze and Bram Willemsen. 2022. CoLLIE:
Continual Learning of Language Grounding from Language-Imant Embeddings. Journal of Artificial Intelligence Research, 74:1201-1221
Andy Zeng. Pete Florence, Jonathan Tompson, Stefan Welker, Jonathan Chien, Maria Attarian, Travis Arm- strong, Ivan Krasin, Dan Duong, Vikas Sindhwani, and Johnny Lee. 2021. Transporter Networks: Rear- ranging the Visual World for Robotic Manipulation. In Proceedings of the 2020 Conference on Robot Learning, volume 155 of Proceedings of Machine Learning Research, pages 726-747. PMLR.
James Seale Smith, Junjiao Tian, Shaunak Halbe, Yen- Chang Hsu, and Zsolt Kira. 2023. Closer Look at Rehearsal-Free Continual I Learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 2419-2419.

---

Friedemann Zenke, Ben Poole, and Surya Ganguli. 2017. Continual Learning Through Synaptic Intel- ligence. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Pro- ceedings of Machine Learning Research, pages 3987- 3995. PMLR.
Yingxiu 7hao, Yinhe Zheng. Bowen Yu, Zhiliang Tian, Dongkyu Lee, Jian Sun, Yongbin Li, and Nevin Zhang. 2022. Semi-supervised lifelong language learning. In Findings of the Association for Computa- tional Linguisties: EMNLP 2022, pages 3937-3951, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

---

Appendices
ResNet-18 features of the visual observations of and o are directly max-pooled and passed through the linear projection layer.
A Experimental Settings
A.1 LILAC-2D and LILAC-3D Datasets
Vision-Lanngaage Transformer. The right part of Fig. 4 provides an overview of the transformer architecture used in our experiments. Each trans- former encoder layer follows the original design as proposed in Vaswani al. (2017) and consists of a multi-head attention operation, two normaliza- tion layers, and two fully connected feed-forward layers. Similarly to the FiL. model, visual fea- tures of o,, of and or are extracted from the first block of a ResNet-18 encoder and passed through an additional linear encoder layer to match the la- tent dimension of the word embeddings from the input instruction 4, Word embeddings of l and visual features of o, are then concatenated and fed to a multi-head attention layer, followed by resid- ual adding operation layer normalization. After- wards, the latent VL features are passed through two fully connected linear layers, i.e., the feed- forward blocks of the network, which is again fol- lowed by residual operation and layer normal- ization. After passing the total of L transformer encoder layers, the features are mean-pooled across the height and width of the language-conditioned feature maps and fed to a linear projection layer. In the same way as with the FiLM architecture, vi- sual features of of and o, from the ResNet- 18 are directly pooled and passed through the projection layer.
Examples of both datasets are provided in Fig. 2 and Fig. 3. For the LILAC-2D dataset, object col- ors (blue, green, grey, purple, red, yellow), object types are [ball, box, key), and directions are (down, the left, to the right, up). For the LILAC-3D dataset, blocks have the colors (blue. green, grey, purple, red, yellow), bowls have the colors (brown, cyan, orange, petrol, pink, white), and block sizes are (big, small].
A.2 Model Architectures
In the following, we describe further details about the two model architectures used in our experi- ments.
FILM. The FiLMed layer is illustrated in the left part of Fig. 4. We largely follow the architectural design of the FiLMed layers as suggested in Hui et al. (2020): Each FiLMed layer consists of a Con- vBlock and the linear modulation fully-connected layers y (weight) and 3 (bias) that condition the feature maps of the convolutional layers by scaling by the factor and shifting by the value B. The ConvBlock consists of two two-dimensional con- volutional layers with a kernel size of 3, a stride of 1, and a padding of 1, and two batch norm layers. Features of visual observations O,, o and o, are initially extracted by the first block of a ResNet-18 18 (this configuration yielded better performance on both datasets than using the feature extractor as pro- posed in Hui et al. (2020)). Subsequently, the visual features of the premise observation o, are passed through the first convolutional layer, followed by batch normalization, rectified linear unit (ReLU) activation, and the second convolutional layer. The language instruction 1, is passed through word em- bedding layer, followed by a single-layer GRU. The embedded instruction sequence is passed through the two fully connected layers/ 33 and 7, the output of which is used to modulate the visual features from the second convolutional layer. Finally, the modulated features passed through second batch normalization layer and ReLU activation. Af- ter passing L FiLMed blocks, the features are max- pooled across the height and width of the feature maps and finally passed through lincar projection layer. In contrast to the 'premise' observation o;
A.3 Hyperparameter Configuration
An overview of the design choices and selected parameters for the model architectures can be found in Tab. 2.
A.4 Metrics and Performance Evaluation
During training, we use the InfoNCE loss (Oord et al., 2019) Linta to maximize the cosine simi- larity between d(fe, (((oz).g(4:) and d(h(ot)) and minimize the cosine similarity between d(So, gl()) and d(h(o,)) During in- ference, the model predicts the target image by choosing the image representation of d(h(o )). d(h(o,) )) that has higher cosine similarity with the representation of d(.oo, (h((:)) gle))). i.e.
S coo(d (())) S Somntd (h(o))))))h))).n (()))))
(4)

---

Table Overview the hyperparameters selected for the model architectures with respect to the two datasets. We used the Biases (https: //wp:///w////.////// aat.iaiissseeeeee sweep with Bayes optimizer.
Transformer
FiLM
Search space
2D
3D
2D
3D
init continual batch size
4.5e-4
2e-4
4.5e-4
2c-4
[1c-6, le-3]
6e-4
7e-4
8e-4
le-3
[le-6, le-3]
128
128
128
128
init epochs
10
10
10
10
continual epochs (per t)
30
30
30
30
word embedding dim instr embedding dim
256
256
128
128
{32, 64, 128, 256, 512)
256
256
{32, 64, 128, 256, 512)
ResNet- 18 layer features
1
(1,2,3,4) (1, 10, 12
encoder layers (L)
4
attn heads
2
(1,2,4,
fin dim
64
64
{32, 64, 128, 256, 512)
EWC discount
0.9
0.9
0.9
0.9
[0,1]
EWC À (joint)
2,000
600
2,000
600
[le-2, le14]
EWC (A&C)
20,000
20,000
20,000
20,000
[1e-2, le14]
ER buffer size
3,000
3,000
3,000
3,000
The prediction accuracy A, can then be calculated the number of samples of the 1-th task for which Equation 4 holds, divided by the total number of samples from the f-th task, [D,)-
is shown in Alg. 1. OMM denotes all shared param- eters, whereas 0. are the task-specific, or isolated, parameters. We choose adaptFreq- 6 for our experiments.
Accuracy, transfer, forgetting. We provide more details on evaluation metries and some addi- tional results on catastrophic forgetting (CF), and forward transfer (FT). Let A denote the test ac- curacy of a model on the j-th task after observing the last sample of the i-th task. We largely follow the commonly used evaluation metrics as proposed in Lopez-Paz and Ranzato (2017):
Algorithm 1 Adaptaaioon-Cnnsollatttii (A&C)
Select modules S C M for task specialization Initialize all model parameters via joint training
D Amni for 1..T do Freeze AM.
for e - w.w.aaapttaioonppp....
do Update gs upon training on D;
if mod adaptFreq= then
T CF:
T
Freeze 0², unfreeze 8MSs
ACC :=
Σ
Ar.
Ar, Ar
Update OM,s upon training on D, Freeze AMMS unfreeze 05
T
T
t=1
end if
FT
A
Amiit
(5)
T-1
end for end for
1=2
where Amit, denotes the performance on the 1-th task after initialization. Note that CF can be also interpreted as negative backward transfer, as it de- scribes the influence of training on task previ- ously seen tasks.
B Additional Results
We provide some additional results the effect of introducing task specialization network modules in Fig. Fig. 10, and Fig. 11. We provide more detailed results from our baseline comparison Tab. 3. Finally, we provide two additional findings that might be interesting to the community:
A.5
Adaptation-Consslidatiin Algorithm
The pseudocode of the A&C training algorithm as modification to the lifelong compositional learning algorithm proposed by Mendez and Eaton (2021)

---

FiLM modulation parameters. We observe that introducing task-specifie parameters either the weight (t)) or bias (3) separately yields the highest accuracy gain and forward transfer as well as the lowest forgetting. However, while the FiL. model trained on the LILAC-2D tasks benefits most from weights to be specialized, the same model trained on LILAC-3D needs bias to be specialized optimize performance metrics.
LILAC-2D
LILAC-3D
MiII
fm2
fm1 ffn2
20
AACC
LILAC-2D
LILAC-3D 3D
-1
weight
bias
weight+bias
30
20
AACC
5
1
all
3
-10
6
Layer
Layer
E
LILAC-2D
LILAC-3D
norm2
norm2
nomi +nom2
-6
30
AACC
20
10
-6
0
2
3
all
Layer
Layer
Figure 9: Analysis of specializing feature-wist linear modulation parameters in different layers. Top to bot- tom: I) accuracy gain specialization and A&C com- pared with monolithic SFT baseline, 2) forward transfer, and 3) forgetting.
-4
30
10
1
2
4
all
3 Layer
all
Modules whose specialization maximized for- ward transfer. While for the FiLMed net- work the specialization of modulation parameters (weight for LILAC-2D, bias for LILAC-3D) maxi- mizes the forward transfer, for the VL transformer it is the feed-forward layers and the layer normal- ization parameters that maximize forward transfer when specialized.
Layer
Figure 10: Analysis specialization in different layers for feed-forward layers (top) and layer normalization parameters (bottom). Each plot shows (from top to bottom): 1) accuracy gain of specialization and A&C compared with monolithic SFT baseline, 2) forward transfer, and 3) forgetting.

---

LILAC-2D
LILAC-3D
LILAC-2D
LILAC-3D
20
AACC
AACC
10
10
E
20
20
CF
10
1
Layer
Layer
Layer
Layer
LILAC-2D
LILAC-3D
LILAC-2D
LILAC-3D
bnI
bnz
bnl+ôn2
conv2
Conv2
conv2+conu2
AACC
AACC
-10
-10
-6
-6
30
30
20
20
10
10
Layer
Layer
Layer
Layer
Figure 11: Analysis of specialization of layers in different depths (top left) and module specialization in different
layers for self-attention (top right), batch normalization parameters (bottom left), and convolutional layers (bottom right). Each plot shows (from top bottom): 1) accuracy gain of specialization and A&C compared with monolithic SFT baseline, 2) forward transfer, and 3) forgetting.

---

FiLM
LILAC-2D
LILAC-3D
ACC
FT
CF
ACC
FT
CF
MTL.
87.1 + 0.9%
80. 1.1%
Expert
76.5 1 1.4%
78.7 2.7%
SFT
52.1 +:: 0.3%
2.4 0.4%
35.0 1.0%
63.3 + 0.8%
2.5 4: 1.0%
17.5 + 1.7%
FR
53.1 + 0.3%
2.7 + 0.3%
33.9 0.9%
66.9 1.2%
4.8 + 1.4%
14.6 + 1.7%
EWC
56.1 +: 0.9%
2.9 0.6%
7.7 0.9%
1.4%
+ 0.6%
10.1 + 0.8%
Maystayer
68.4 + 0.8%
3.7 + 0.4%
16.4 + 0.5%
64.2 2 2.1%
-1.6 + 0.5%
11.6 + 0.9%
Myyst
66.8 + 0.8%
0.1 + 0.1%
15.2 1 0.4%
68.2 1: 1.8%
0.2 + 0.3%
7.0 + 0.9%
59.4 + 0.4%
2.9 0.5%
21.4 1.7%
62.5 0.9%
1.. + 0.6%
16.0 + 2.6%
Transformer
LILAC-2D
LILAC-3D
ACC
FT
CF
ACC
FT
CF
MTL
88.3 0.3%
95.4 + 0.3%
Expert
85.9 + 0.1%
88.4 + 0.2%
SFT
51.1 + 0.2%
2.5 + 0.6%
35.7 + 0.6%
67.0 + 0.8%
-25 + 0.8%
24.2 + 0.6%
0.4%
1 0.3%
34.6 1 0.5%
77.7 1 0.8%
0.9%
12.7 0.7%
EWC
55.5 + 0.7%
2.2 + 0.6%
14.2 + 0.4%
79.0 + 0.7%
1.8 + 0.9%
8.9 + 0.6%
Masssuer
70.0 0.4%
0.9 1 0.3%
15.3 1 0.6%
76.5 1 0.7%
0.7 1 0.4%
12.5 1 0.7%
Muattee
74.2 0.6%
1.7 + 0.3%
14.1 + 0.4%
76.5 + 1.0%
3311 + 0.6%
14.0 1.0%
Malttnat
63.1 1. 0.5%
2.3 1 0.3%
20.9 1 0.5%
75.0 1 0.7%
-1.7 1. 0.9%
14.2 1 1.0%
Maticun
81.2 0.6%
0.0 1 0.3%
7.5 0.7%
81.2 1 0.8%
0.2 = 0.5%
9.3 1 0.9%
Mall am FR
85.7 1.7%
+ 0.2%
0.8 0.2%
88.5 + 0.5%
-0.8 0.2%
+ 0.2%
Malsam EWC
87.1 1 0.4%
0.4 + 0.1%
0.6 0.2%
87.9 1 0.5%
0.5 1 0.3%
0.6 1 0.1%
Table 3: Average accuracy (ACC), forward transfer (FT), and forgetting (CF) FiLM (top) and rision-language transformer (bottom) baselines for all datasets. Results averaged across seeds. Standard error after +.


Reviews:
[{"review_id": "cue3rUX3Jf", "reviewer": "Reviewer_Lw5h", "summary": "", "questions": "-\tWhat’s the motivation of having 2D and 3D datasets? Any new challenges brought by 3D?\n-\tline 230-231,  why only 10 tasks (not 60 tasks?) need to be learned incrementally? \n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer $\\textcolor{blue}{\\textsf{Lw5h}}$'s constructive feedback on our paper and their positive assessment of the motivation, problem definition, and scope of our study.\n\n\n## 1) Writing Quality in Sec. 4.2  \n\n**Q1:** The reviewer $\\textcolor{blue}{\\textsf{Lw5h}}$ suggests improvements in the result section to enhance clarity, particularly in explaining Figure 2. Furthermore, the reviewer recommends strengthening the connections between the results and the conclusions drawn in the introduction.\n\n**A1:** We acknowledge your concern and will **provide more explicit explanations** for the interpretation of Figure 2 as well as other figures, and also **align our experimental results with the summarized conclusions** in the introduction.\n\n\n## 2) Plot Metrics along Time Scale \n\n**Q2:** The reviewer $\\textcolor{blue}{\\textsf{Lw5h}}$ suggests to add plots for analysis/results that show the metrics along a time scale.\n\n**A2:** We agree that presenting results with a time-based perspective can provide deeper insights into the dynamics of continual learning. To address this concern, we will incorporate **plots of evaluation accuracy over time**, one plot for each dataset, **contrasting both the baselines and our specialized strategies utilized in the final comparison experiments** (Sec. 4.2.4). \n\n## 3) Motivation for Dataset and Continual Stream Design\n\n**Q3.1:** Reviewer $\\textcolor{blue}{\\textsf{Lw5h}}$ asks for the motivation to propose the 3D dataset in addition to the 2D dataset.\n\n**A3.1:** While experiments with the 2D dataset provide valuable insights on the behavior of model parts during continual learning, we enrich our set of benchmarks with 3D data to **compare whether our findings still hold when increasing spatial complexity** and **introducing distracting information** such as the robot arm position. We further consider the introduction of 3D synthetic images as a step to bridge the gap to real 3D images while maintaining the control and flexibility for model analysis provided with a simulated benchmark.\n\n**Q3.2:** Reviewer $\\textcolor{blue}{\\textsf{Lw5h}}$ asks for the reason to choose 10 tasks over 60 tasks for the incremental learning setup.\n\n**A3.2:** Grouping multiple different instructions into a task (in our case, 10 tasks à 6 instructions, cf. lines 226-227) is a common practice in continual learning research. To give some examples, we refer to some of the most widely known continual learning benchmarks CIFAR-100 [1] (10 tasks à 10 classes), Permuted MNIST [2] (5 tasks à 2 digits) and CORe50 [3] (5 tasks à 20 object categories). As our extension of the contrastive learning problem in Natural Language Inference to visually grounded language is novel in our work, we aimed to **align with commonly agreed methods when designing the continual stream**.\n\n[1] Rebuffi et al. \"iCaRL: Incremental Classifier and Representation Learning\" CVPR 2017  \n[2] Kirkpatrick et al. \"Overcoming catastrophic forgetting in neural networks\" PNAS 2017  \n[3] Lomonaco & Maltoni \"CORe50: a New Dataset and Benchmark for Continuous Object Recognition\" CoRL 2017", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "-\tA well-motivated and scoped study to address the challenges with continual learning\n-\tThe definition of the problem of selection strategy is intuitive and well-explained. \n-\tTwo datasets to support research on this topic. \n", "reasons_to_reject": "-\tThe result section seems to be written in a rush. Some parts are not straightforward to follow, e.g., how to interpret Figure 2?  The connections between the results (figures) and the general conclusions in the introduction section need to be better explained. \n-\tSince this is about continual learning, I would expect some results/analysis that show some sort of metrics along the time scale (e.g., X-axis). \n", "questions_to_authors": "-\tWhat’s the motivation of having 2D and 3D datasets? Any new challenges brought by 3D?\n-\tline 230-231,  why only 10 tasks (not 60 tasks?) need to be learned incrementally? \n", "paper_topic_and_main_contributions": "This paper studies learning new skills from language instructions in a continual learning setting. It formulates the problem as a selective parameter specialization problem which identifies parts/modules for parameter sharing or specialization.  It created two datasets and conducted experiments to study the tradeoff between generalization and specialization.  Experimental results are reported. ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "CtOyTsd4tu", "reviewer": "Reviewer_F5yK", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer $\\textcolor{teal}{\\textsf{F5yK}}$ for the detailed and constructive feedback and the positive assessment of our study. \n\n## 1) Experimental Findings on Self-Attention Layer Specialization\n\n**Q1:** Reviewer $\\textcolor{teal}{\\textsf{F5yK}}$ asks to integrate the results on the self-attention layers as presented in Figure 8 in the correlation analysis. \n\n**A1:** This is already **done in the correlation analysis**, as we treat each specialized module in the transformer network (notably including self-attention layers) as a separate datapoint ($x_1$, $x_2$) where $x_1$ is the relative accuracy gain from specializing the module under the A&C learning scheme (cf. Figure 2 first row) and $x_2$ is the gradient-/activation-based importance score on this module (cf. Figure 4). However, we acknowledge that the connections between Figures 2 and 4 and drawn conclusions in the results subsections were not clear enough in the first version of this script, a remark also made by reviewer $\\textcolor{blue}{\\textsf{Lw5h}}$. Hence, we will **enhance the comprehensiveness of our evaluation** by expanding the depth of plot interpretation in Section 4.2.\n\n## 2) Generalizability to Real-World Settings\n\n**Q2:** $\\textcolor{teal}{\\textsf{F5yK}}$ raises the question whether the results from the synthetic datasets will generalize to real-world setting with more complex tasks.\n\n\n**A2:** On the one hand, prior scenarios on continual vision-language grounding that are based on synthetic/diagnostic datasets are **too simplistic** either in terms of the continual learning problem (only one distributional shift) or in terms of the vision-language reasoning problem (simplistic images with one object, cf. lines 57-62). On the other hand, analyzing model behavior based real-world images may be **strongly biased by the visual reasoning shortcuts** that have been observed in models (cf. lines 67-74).\n\nTo leverage the advantages of synthetic and real image scenarios, we provide two datasets that allow for \n1) **investigating the learning behavior of the models with a high degree of control and flexibility** while \n2) being challenging enough to require **object localization, spatial reasoning, concept learning,** and finally **language grounding** capabilities of the continual learner.\n\nWe believe that by gaining promising insights with the benchmarks proposed in this paper, we lay the groundwork for further research that can bridge the gap between synthetic and real-world scenarios.\n\n## 3) Minor Remarks on Formatting\n\nWe thank reviewer $\\textcolor{teal}{\\textsf{F5yK}}$ for notifying us for some formatting and reference issues of the Figures 2a and 4 in the manuscript and will fix these in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper studies selective specialization in vision-and-language models which is an interesting approach to continual learning.\n- It introduces two datasets with compositional structure.\n- The experiments explore various module selection strategies and show that the best strategy outperforms common baselines.\n- The paper is written clearly and the findings are connected to the wider literature.", "reasons_to_reject": "- Based on Table 1, specializing all self-attention modules seems to be by far the best strategy for transformer models. However, self-attention layers are not considered in the experiments for the previous research questions (sections 4.2.2 and 4.2.3). Since results per layer are presented in Figure 8 (Appendix B), it would be useful to include these results in the correlation analysis with the selection based on importance scores.\n- The synthetic nature of the images and instructions in the dataset makes it unclear whether results in this benchmark will generalize in real-world settings where tasks would have more complex structure.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper studies module specialization for continual learning in vision-and-language contrastive learning. To explore the trade-off between parameter specialization and knowledge transfer, the paper introduces the Lifelong Language Compositions (LILAC) benchmark that consists of two synthetic datasets with compositional instructions.   \n\nThe paper utilizes an alternating learning scheme of adaptation and consolidation phases and tests different module selection strategies for two vision-and-language architectures (FiLM and cross-modal transformer). The experiments show that gradient-based importance scores can be used successfully for selecting modules for specialization and that the examined method can outperform standard baselines of ER and EWC.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- In Figure 2a, the negative values of the colorbar are not readable.\n- In Figure 4, it is not clear what each column of the heatmap represents."}, "extra_text": {}}, {"review_id": "djrxj4nJ53", "reviewer": "Reviewer_FZU5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer $\\textcolor{orange}{\\textsf{FZU5}}$ for the constructive feedback.  \n\n## 1) Dataset Novelty and Takeaway\n\n**Q1.1:** Reviewer $\\textcolor{orange}{\\textsf{FZU5}}$ asks to elaborate on the research gap that the two proposed datasets fill in.\n\n**A1.1:** We propose two novel LILAC datasets in our work due to the lack of benchmarks that could fulfill the following criteria in our visually grounded continual language learning setting (cf. lines 53-62):\n\n1. High overlap between tasks\n2. Compositional nature of a dataset\n3. Comparable inter-task and intra-task similarity\n4. Non-trivial task (i.e., high similarity between positive and negative samples to be contrasted)\n5. Simulated environment instead of real-world images\n\nCriteria 1, 2, and 3 aim at **maximizing knowledge transfer** to bias a learner towards keeping many modules shared across tasks for generalization while *selectively* specializing a limited set of modules.  \nCriteria 4 and 5 aim at **preventing the model from doing reasoning shortcuts** that might hinder the generalizability derived from findings about model behavior (nonetheless, we do consider bridging the gap between controlled environments and real-world setting as valuable future work). \n\nAs the two proposed datasets fulfill the abovementioned criteria, they allow for a thorough analysis and provide insights insights into the continual learning (CL) behavior under different specialization strategies.\n\n**Q1.2:** Reviewer $\\textcolor{orange}{\\textsf{FZU5}}$ asks to elaborate on the takeaway and conclusions drawn from this work, particularly under the assumption that this is a methodology paper.\n\n**A1.2:** We consider this paper as an **analysis paper**. Particularly, our work has two main takeaways: \n\nCarefully balancing the trade-off between generalization and specialization\n1. provides **insights into the behavior of individual model parts** (Sections 4.2.2 and 4.2.3). This could benefit the design of future parameter isolation CL methods, e.g., by answering where in the network to insert task-specific parameters; \n2. helps us **design conceptually simple approaches that outperform common CL baselines** (Sections 4.2.1 and 4.2.4).\n\n\nAs mentioned in lines 600-612, we consider adapter methods for pretrained models as a parallel rather than competing line of research (e.g., capsule networks, adapters).\n \n## 2) Comparison with Additional Baselines\n\n**Q2:** Reviewer $\\textcolor{orange}{\\textsf{FZU5}}$ asks for 1) a comparison with a more recent CL baseline and 2) a comparison with a isolation-based method such as Progressive Neural Networks.\n\n**A2:** With reference to the two main takeaways described in **A1.2**, the **aim of this analysis paper is *not* to provide a new SOTA parameter isolation method**, but rather show the merit of careful selection of model parts to be specialized for CL. \n\nNevertheless, we follow $\\textcolor{orange}{\\textsf{FZU5}}$'s request to conduct additional experiments to compare with a strong recent pruning method. Winning SubNetworks (WSN) [1] is a recent parameter-isolation method focused on learning binary task-wise masks for each parameter (*not* module) and selectively freezing parameters used in previous tasks. As it is designed for convolutional neural networks, we perform experiments on the FiLMed network only. \n\nRunning each experiment with ten different random seeds and for two different task capacity ratios $c=0.1$ and $c=0.5$, we get the following results: WSN ($c=0.1$): $80.0 \\pm 5.7\\%$ (2D) / $78.9 \\pm 7.4\\%$ (3D), WSN ($c=0.5$): $81.4 \\pm 2.2 \\%$ (2D) / $77.8 \\pm 9.3\\%$ (3D).\n\nWSN achieves comparable performance with the self-attention specialization strategy for the Transformer backbone on the 2D and 3D datasets ($81.2\\%$). This is, however, not surprising, as we claim in our paper that pruning methods are an effective and performant method for continual learning (lines 582-584). However, as we also claim in the paper (lines 584-587), they do not add interpretability and valuable insights in the role of individual model parts in learning to solve problems continually (cf., our first takeaway).\n\nTo explain why we do not compare with Progressive Neural Networks [2] as suggested by reviewer $\\textcolor{orange}{\\textsf{FZU5}}$: This method adds a \"column\" every time the model sees a new task. In our case, this would imply to have a new copy of all modules for every task, which is in stark contrast to the objective of this paper to determine how to selectively specialize individual modules.\n\n\n## 3) Terminology and Notation\n\n**Q3.1:** Reviewer $\\textcolor{orange}{\\textsf{FZU5}}$ asks for the definition of $\\mathcal{M}$.\n\n**A3.1:** As mentioned in line 161, $\\mathcal{M}$ is a set of modules (e.g., self-attention, FFNs, batch norm) that comprise a network $f$. We will make the definition more explicit.\n\n**Q3.2:** Reviewer $\\textcolor{orange}{\\textsf{FZU5}}$ asks to clarify on the meaning of *modules* $m$, especially with respect to Neural Module Networks (NMN), and asks whether $m$ corresponds to a set of layers.\n\n**A3.2:** In our paper, a *module* $m$ is any type of self-contained parametric function within a deep neural network that can be either entirely specialized or shared across tasks. Therefore, the meaning of module is not the same meaning as in the NMN (our architectures Transformers and FiLM networks are obviously not NMNs). We agree, however, that the term *module* can confuse the redears familiar with NMNs and will improve our description in Section 2.1 in that regard.\n\nWe deliberately avoid the term *layer* in the introduction to not confuse it with, e.g., transformer encoder layers, which would consist of multiple modules like self-attention, layer norm etc. \n\n**Q3.3:** According to $\\textcolor{orange}{\\textsf{FZU5}}$, the term \"best strategy\" in Section 4.2 could be less assertive, as specialization strategies like parameter-specific network pruning were not explored in the analysis.\n\n**A3.3:** Our aim is to analyze selection strategies of *modules* (i.e., any type of self-contained parametric function within a deep neural network) and not *parameters*. While methods for parameter pruning have shown to be effective in preventing forgetting, they provide little interpretability regarding the role of individual model parts in solving CL problems (cf. 582-587). Hence, the scope of strategies for analysis is limited to modules.\n\n[1] Kang et al. \"Forget-free Continual Learning with Winning Subnetworks\" ICML 2022  \n[2] Rusu et al. \"Progressive Neural Networks\" NIPS 2016 Deep Learning Symposium", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- the paper proposes two continual learning benchmarks in visual grounding setups.\n- analysis on generalization vs. specialization in the paper.\n- the author provided code in the supplementary material, which shows their reproducibility", "reasons_to_reject": "First, the novelty is limited. Specifically\n- why do we need these two datasets? what's the motivation of creating these two datasets? what's missing in the field that these datasets fill in? assuming this is a dataset paper, the author should talk more about this.\n- what's the takeaway of the paper? assuming this is a method paper, is the message like (1) some module specialization helps on the continual learning and (2) the best specialization is on the self-attention layers? If so, I'd like to know the extra parameters required for self-attention specialization vs . LoRA or other adapter methods, otherwise I think the conclusion is not convincing enough.\n\nSecond, some baselines are not compared.\n- EWC and ER are not the best regularization-based and rehearsal-based continual learning methods. The paper could compare with at least one more recent method to show the specialization argument is also valid on that.\n- As the paper is more on the side of isolation-based methods, where the number of parameters grows with the number of tasks. It's better to compare with some method like progressive neural networks.\n\nThird, the writing is hard to follow, especially the notation.\n- Is mathcal M defined? Is it equal to the network in 4.1\n- m is defined in Sec 2 but first used in the experiment section 4.2\n- Until 4.2.4 I understood the meaning of \"module\" and it is not the same meaning as in the neural module network. Why not just call it a set of layers? Following the definition in Sec 2, the subtitle \"best strategy\" in 4.2 overclaims, the author didn't explore all the strategies, like pruning the network for a subset of parameters, etc.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper focuses on creating artificial agents that excel in a sequence of language-informed tasks within the visual world. They propose new diagnostic datasets for continual learning in visual language grounding, comparing strategies for module specialization. Experimental results demonstrate the superiority of their strategies over some common baseline methods.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "k7tbsTaSop", "reviewer": "Reviewer_LwBz", "summary": "", "questions": "- How are the proposed two datasets related to continual learning? Because we only the each item one time?\n- The problem in the  proposed dataset are not classification problem (select one image based on current image and instruction)? if it is a classification problem, why the paper talk the limitation in line 154-157.\n- what does grounding mean in this work? Just consider the visual input information?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank $\\textcolor{purple}{\\textsf{LwBz}}$ for reviewing our paper. \n\n## 1) Writing Quality\n\n**Q1:**\n> Writing needs significant improvement.\n\n**A1:** To better address this concern, we kindly request more specific information regarding the areas of the paper that require improvement in writing. Any **specific examples or suggestions** would be incredibly helpful as we work to enhance the quality of the paper.\n\n\n## 2) Contributions\n\n**Q2:**\n> This paper just propose two simple datasets and apply A&C methods on two base models to check the performance in these two datasets.\n\n**A2:** As was correctly noticed by the other reviewers ($\\textcolor{blue}{\\textsf{Lw5h}}$, $\\textcolor{orange}{\\textsf{FZU5}}$, $\\textcolor{teal}{\\textsf{F5yK}}$) and mentioned in our paper (e.g., lines 41-49), our aim is to **study different strategies for selective specialization** to **understand the role of each network module** during continual learning of language grounded in visual observations. Consequently, this paper is considered an analysis paper rather than a methodology paper.\n\n## 3) Other Questions\n\n**Q3.1:** Reviewer $\\textcolor{purple}{\\textsf{LwBz}}$ asks whether the scenario proposed with the datasets can be considered a classification problem and asks for clarification why training task-specific output heads as a specialization strategy is not applicable in our work.\n\n**A3.1:** Although the learning setting could be reformulated as a binary classification problem, we designed a scenario inspired by **contrastive supervised learning for Natural Language Inference** (cf. lines 122-125). Our model therefore learns contrasting latent representations, but does not produce any logits for multi-class classification. As for our analysis of selective specialization strategies, this **prevents a trivial solution of learning task-specific output heads**, as done in prior literature.\n\n**Q3.2:**\n> How are the proposed two datasets related to continual learning? Because we only the each item one time?\n\n**A3.2:** The proposed datasets indeed align with the continual learning (CL) paradigm, as they reflect **scenarios involving sequential and non-stationary data**. Contrary to the assumption, **each item is encountered multiple times**, as the model is trained on each task for multiple epochs (cf. Table 2). However, **training once on each data point is not considered a necessary criterion for a generic CL scenario**, but relates to the specific subfield of Online Continual Learning, which we do not address in this work.\n\n**Q3.3:**\n> What does grounding mean in this work? Just consider the visual input information?\n\n**A3.3:** *Grounding* or *visual language grounding* (line 13) in our paper refers to the challenge of a model to establish a meaningful connection between visual and language inputs. \n\nIn general, but notably also in the context of our work, grounding entails more than solely considering visual input information: The proposed LILAC 2D and 3D tasks can only be solved by **successfully understanding the premise scene** and **infering the next possible scene based on the language instruction**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Apply A&C in visual grounded setting.\n- Propose two simple datasets.", "reasons_to_reject": "- Writing needs significant improvement.\n- The proposed method is incremental. This paper just propose two simple datasets and apply A&C methods on two base models to check the performance in these two datasets.", "questions_to_authors": "- How are the proposed two datasets related to continual learning? Because we only the each item one time?\n- The problem in the  proposed dataset are not classification problem (select one image based on current image and instruction)? if it is a classification problem, why the paper talk the limitation in line 154-157.\n- what does grounding mean in this work? Just consider the visual input information?", "paper_topic_and_main_contributions": "The paper propose two datasets related to  visual grounded continual learning and  applies A&C algorithm on these two datasets to study the relation between specialization and generalization.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- line 82 to 86, quite confusing \n- line 178"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Positives of the paper include a well motivated problem and novel datasets being produced. Improvements in the writing could make the paper stronger, particularly in the results section and the notation. Reviewers also expressed desire to see results expressed in terms of time being spent learning. It is also not clear whether the results would generalize to more realistic datasets and scenarios.
Decision: Accept-Findings