Online Continual Learning for Interactive Instruction Following Agents
Authors: Byeonghwi Kim
Minhyuk Seo
Jonghyun Choi
Venue: ICLR 2024 poster (2024)
Keywords: Embodied AI
Continual Learning
Primary area: applications to robotics, autonomy, planning

Abstract: In learning an embodied agent executing daily tasks via language directives, the literature largely assumes that the agent learns all training data at the beginning. We argue that such a learning scenario is less realistic, since a robotic agent is supposed to learn the world continuously as it explores and perceives it. To take a step towards a more realistic embodied agent learning scenario, we propose two continual learning setups for embodied agents; learning new behaviors (Behavior Incremental Learning, Behavior-IL) and new environments (Environment Incremental Learning, Environment-IL) For the tasks, previous ‘data prior’ based continual learning methods maintain logits for the past tasks. However, the stored information is often insufficiently learned information and requires task boundary information, which might not always be available. Here, we propose to update them based on confidence scores without task boundary information (i.e., task-free) in a moving average fashion, named Confidence-Aware Moving Average (CAMA). In the proposed challenging Behavior-IL and Environment-IL setups, our simple CAMA outperforms prior arts in our empirical validations by noticeable margins.

Published as a conference paper at ICLR 2024
ONLINE CONTINUAL LEARNING FOR INTERACTIVE INSTRUCTION FOLLOWING AGENTS
Byeonghwi Kim
Minhyuk Seo
Jonghyun Choi
Yonsei University Seoul National University
(byeonghwikim, abd0508) @yonsei..... ac. kr, wwwghyunchcii@mmte...omm
ABSTRACT
In learning an embodied agent executing daily tasks via language directives, the literature largely assumes that the agent learns all training data at the beginning. We argue that such learning scenario is less realistic since a robotic agent is supposed to Iearn the world continuously take a step towards a more realistic embodied agent learning scenario, we pro- pose two continual learning setups for embodied agents; learning new behav- iors (Behavior Incremental Learning, Behavior-IL) II) and new environments (En- vironment Incremental Learning, Environment-Ie -IL) For the tasks, previous 'data prior' based continual learning methods maintain logits for the past tasks. How- ever, the stored information often insufficiently learned information and re- quires task boundary information, which might not always be available. Here. we propose update them based on confidence scores without task boundary information during training (i.e., task-free) in a moving average fashion, named Confidence-Aweer Aware Moving Average (CAMA). In the proposed Behavior-II. and Environment-lL. -L setups, our simple CAMA outperforms prior state of the art in our empirical validations by noticeable margins. The project page including codes is /ttp:://githbb com/ snumpr ab/ -l-aclllllll alred
it explores and perceives it. To
INTRODUCTION
Recent advances in computer vision, natural language processing, and embodied Al have led to vari-
ous benchmarks for robotic agents, encompassing navigation (Savva et al. 2019:
Deitke al. 2020
Anderson et al., 2018 Krantz et al. 2020), object interaction (Zhu et al. 2017 Misra et al.
2017
Weihs al.] 2021: Ehsani et al., 2021), and interactive reasoning (Das et al. 2018 Gordon al,
2018). To create more realistic agents, challenging benchmarks (Shridhar et al. 2020 Padmakumar al. 2022 require all of these tasks to complete complex tasks based language directives.
However, most embodied AI literature assumes that all training data are available from the outset but it may be unrealistic agents may encounter novel behaviors environments after deployment. To learn new behaviors and environments, continual learning might be necessary for post-deployment.
To learn new tasks, one may finetune the agents. But the finetuned agents would suffer from catas- trophic forgetting that loses previously learned knowledge (McCloskey & Cohen 1989: Ratchif 1990.. To mitigate such forgetting, (Powers et al., 2022) introduced a continual reinforcement learning framework that incrementally updates agents for new tasks and evaluates their knowledge of current and past tasks. However, this operates a simplified task setup of (Shridhar et al. 2020), excluding natural language understanding and object localization.
Taking a step forward to bring the instruction following task to real-world scenarios, we propose two continual learning scenarios for embodied agents: Behavior Incremental Learning (Behavior-IL) and Environment Incremental Learning (Environment-IL II.) as depicted in Figure [ Behavior-IL. the robot learns behaviors incrementally. For example, it may initially learn object movement and subsequently acquire the skill of object heating. In Environment-IL. instead of being limited to specific scenes such as bathrooms, the robot progressively learns perform behaviors in diverse environments such as kitchens and bedrooms.
Equual contribution. 'Corresponding author. Most of the work is done while JC with Yonsei University.

---

Published as a conference paper at ICLR 2024
Behavior Incremental
Each
denotes an expert demonstration.
Behavior 1: EXAMINE
Behavior 2: HEAT
Behavior N: MOVASLE
Darine
Orlie Steem
Oniine Sawam
Environment Incremental
Each
denates an expert demonstration,
Environment 1: BEHROOMS
Environment BamioooM
Environment M: KrTCHESS
Oniine
Orlice
Online
Figure 1: Proposed two incremental learning setups. In the 'Behavior Incremental' setup, the agent is ex- pected to learn new behaviors while preserving previously learned knowledge. In the 'Environment Incremen- tal' setup, the agent is expected to learn tasks in new environments with the preservation of previously Icarned knowledge. Note that each image in the figure denotes an expert demonstration (i.e,, a sequence of frames with natural language instructions followed by a corresponding sequence of actions and object class labels).
In the continual learning literature, significant progress (Mai al.) 2022 Biesialska al.) 2020)
has been made in addressing continual learning by storing models learned in the previous task of extracting information about past data, requiring a substantial storage cost (Zhou et al.) 2022a To address this, [Buzzega et al. (2020); Boschini al. (2022a) propose store logits of past models for knowledge distillation, reducing storage costs while maintaining learning efficacy. However, the stored logits may be the underfitted insufficiently learned solution as the model has not sufficiently trained in the early stage of learning, hindering the effective use of prior knowledge. Moreover, such an update often exploits task boundary information that might not always be available, especially in the cases of streamed data without explicit task boundaries (Shanahan et al. 2021 Koh et al. 2023
To develop continuously updating embodied agents, propose to update logits by combining the previously stored logits and the newly obtained ones the moving average, call "Confidence-Aware Moving Average' (CAMA). In particular, dynamically determine the moving average coefficients based on the classification confidence scores inferred by the agents as indicators of the 'quality' of the newly obtained logits (i.e., how much they contain accurate knowledge of the corresponding tasks), as empirically observed that high confidence tends to have high accuracy in Figure
Contributions.
We summarize our contributions as follows:
We propose behavior incremental (Behavior-II.) and environment incremental (Eavironment-Il)) setups for online continual learning for interactive instruction following agents.
We propose Confidence-Award Moving Average (CAMA) that dynamically determines coeffi-
cients for logit update to prevent logits from being outdated for effective knowledge distillation.
Our proposed method outperforms comparable methods most metrics with noticeable margins.
2 RELATED WORK
Continual learning setups.
Continual Learning (CL) typically categorized into two main sce-
narios: offline (Rebuffi et al. 2017 Kirkpatrick et al. 2017 Chaudhry et al.) 2018 Wu et al.) 2019) and online (Koh al. 2022 2019a), based on the frequency with which the model accesses task data. In the offline setup, data from the current task are used for training multiple times, but this often requires significant memory
Buzzega et al. 20200 Mai
2022 Koh et al. 2023 Aljundi al.
2

---

Published as a conference paper at ICLR 2024
capacity to store all task data (Hayes & Kanan 2022). On the other hand, online CL involves pro- cessing individual or small batches of samples, each of which was used only once for training (Koh)
let al. 2022; Aljunnd et al. 2019b). Considering memory constraints and the continuous arrival of limited data points over time in practical scenarios, we focus on the online CL setup.
Task-free continual learning setups. Approaches to continual learning can be categorized into task-free methods (Aljundi al. 2019a; Koh et al. 2023] Ye & Bors, 2022] Koh et al. 2022) and task-aware methods (Kirkpatrick et al. [017 Li & Hoiem 2017] Wu et al., 2019; Boschini et al.,
2022a) based on the use of task boundary information during training (Aljundi al. (2018b). Sev- eral task-aware methods (Li & Hoiem 2017 [u et al. (019) that exploit task boundary information during training distill the knowledge of past tasks from the previously learned model saved in mem- ory. However, in real-world scenarios, it is often impractical to know the task identity of streamed input data (Aljundi et al. al. 2018b). Therefore, even if tasks are discretely defined (e.g., blurry (Prabhu ct al. 2020; Bang et al. 2. disjoint (Parisi et al., 2019), the task-free assumption indicates that no task-specific information or identifier is used during training.
Knowledge distillation in online continual learning, Continual learning has made significant progress, employing methods such as replay-based (Wu et al.
2019 Rolnick et al. 2019: Prabhu et al. 2020; Bang et al. 202T; Koh et al. 2022), distillation (Li & Hoiem, 2017 Buzzega el al.
2020
Koh et al. | (023), and regularization Zenke al. 2017 Kirkpatrick et al.] Lesort al.
2019)
In particular, distillation based approaches (Li & Hoiem, 2017; Koh et al. [203]] Boschini
jet al.
2022b) have been extensively investigated to leverage prior knowledge, but often require substantial memory and additional computation. Memory requirements make them unsuitable for settings with limited memory in edge devices (Zhou et al., 2022a).
To address these issues, (Buzzega et al., 2020 Michieli & Zanuttigh 2021 Boschini al. 2022a) propose using logits instead of storing previous models, saving memory and inference overhead. However, the method of storing logits in memory without updating may hinder the current model in distilling outdated past information from stored logits, since the stored logits may represent in- complete learning for past tasks. To tackle this, a recent approach X-DER (Boschini al. 2022a) updates logits through weighted summation with logits maintained in memory and those from the current model, preventing the previously stored logit from becoming outdated, the model updates.
However, Boschini et al. (2022a) requires task boundary information during the training process for logit update, making it unsuitable for setups where data arrive in continuous stream without task boundaries. In contrast, our approach updates logits based on the agent's confidence without task boundaries, making suitable for more general setups where we do not have such information.
Lifelong learning for robotic agents. Going beyond relatively straightforward task setups such as image classification, a substantial amount of literature has emerged to construct a more realistic agent capable of incremental learning novel tasks (Lesort et al. 2020) in various aspects including
learning strategies (e.g., reinforcement learning (Khetarpal et al. 2018 Wolczyk et al. 2021 Xie &
Finn 2022). imitation lcarning (Mendez et al. 20188] Gao et al. 2021). etc.) and task formulations
(e.g., manipulation (Yang et al. 2022 Liu al. 2023), walking (Zhou al., (022b), etc.). Typically, most of this research has focused mainly on relatively fine- fine-grained manipulation tasks, while the
navigation aspect (Krantz et 2020 Deitke al., 2020) has received comparatively less attention.
Concurrently, there is recent literature that delves into the dual aspect of navigation and interac- tion (Powers al. 2022] Wang et al. 2023) in 3D interactive environments (Kolve al.) 2017, Fan et al.) (022) to perform more demanding tasks. In this context, agents are required to become proficient in both navigating and interacting with task-related objects. Here, the tasks in our pro- posed continual learning setups are similar to (Powers et al.) (022) that simplifies the task setup of (Shridhar et al. 2020). While (Powers et al., 2022) excludes natural language understanding and object localization, we include them to train agents to complete the desired tasks in the challenging full-fledged interactive instruction following setup, along with navigation and object interaction.
We review more relevant literature and provide extended related work in Sec. A for space's 's sake.
3 CL-ALFRED: CONTINUAL LEARNING SETUPS FOR EMBODIED AGENTS
Continual learning enables agents to adapt to new behaviors and diverse environments after deploy- ment, mitigating the risk of forgetting previously acquired knowledge. To foster active research
3

---

Published as a conference paper at ICLR 2024
on mitigating catastrophic forgetting, recent literature (Powers et al., 2022) proposes benchmark that continuously learns new types of household tasks, but lacks natural language understanding and object localization, potentially limiting the deployability of agents.
To comprchensively address the combined challenges of continuous learning of an agent with natural language understanding and object localization, we use full- full-fledged interactive instruction following tasks and propose two incremental learning setups, Behavior Incremental Learning (Behavior-IL.) and Environment Incremental Learning (Environment-IL) We outline our task formulation and detail these incremental learning setups in the following sections.
3.1 TASK FORMULATION
As the ALFRED dataset (Shridhar et al. 2020) requires a comprehensive understanding of natural language and visual environments for inte llgent agents, we build our continual benchmark top of it. The agent is first spawned at random location in an environment and then given natural language instructions, I, that describe how accomplish task. For each time step the agent takes input visual observation interaction is required. Here, we learn policy parameterized by 0, ma + y, with input zt. i.e. (Ur, !) and output yt, i.e., (yo,t. Mm.s). The goal for the policy Te is to predict sequence of actions and object masks to complete the task. Kindly refer to Shridhar et (2020) for more details.
and prediets an action Va.: and a mask Vm, of an object class Ye,t if object
Most previous methods (Singh et al., 2021 Pashevich et al. 2021: Min et al. 2022, Kim et al.
2023) for object localization utilize a two-stage approach, separating into object class prediction and mask generation to enhance object localization. Since mask generation is handled by separate mask generators, however, continual updates of these networks are also required. Unfortunately, continuously updated instance segmentation models (Joseph al. [021] Cermelli et (002)) often noticeably underperform jointly trained models. Here, we only address class prediction, assuming the availability of object masks, leaving the continual updating of mask generators for future work.
3.2 CONTINUAL LEARNING SETUPS
There is significant progress in developing agents that can perform the desired tasks through lan- guage directives (Krantz et al. 2020 [Shridhar et al. Padmaakmarr et al.) 2022). It often con- fronted with new behaviors and environments after being deployed and are required to learn them while maintaining previously learned knowledge. However, prior methods either presuppose the availability of pre-collected datasets or utilize simplified task configurations (Powers et al. 2022).
To address this limitation, we introduce two continual learning setups: 1) Behavior Incremen- tal Learning (Behavior-IL) to incrementally Iearn what to do and 2) Environment Incremental Learning (Environment-II.) to incrementally learn where to do, as in Figure T In addition, we focus on online CL, which assumes more realistic scenario where novel data encountered in a streaming manner (Aljundi et al., 2019a] Koh et al.] [023 2022) rather than assuming an offline
CL in which novel data are provided in chunks of tasks (Wu et 2019 Saha et al. 2021). More
details about the continual setup can be found in Sec. B.
3.2. BEHAVIOR INCREMENTAL LEARNING
Behaviors described by instructions may exhibit considerable diversity as novel behaviors may arise over time. To address this scenario, we propose the Behavior-IL setup that facilitates the agent's incremental learning of new behaviors while retaining the knowledge obtained from previous tasks.
Formally, for set of behaviors, T, an agent sequentially receives N, training episodes, {s(² 1/ for each type of behavior, Tj E T. When receiving the final episode (i.e., sR,)) for the current behavior type, Tj, the agent starts to sequentially receive episodes, {s77+1) 11
N541 for the next behavior type,
TJ+1- The episode stream ends with the last training episode, 7 NNi: of the last task type, T/TI-
Here, we adopt seven different types of behavior from (Shridhar et al. 2020): EXAMINE, PICK&PLACE, HEAT, COOL, CLEAN, PICK2&PLACE, and MOVABLE. To ensure the adaptability of agents and avoid favoring particular behavior sequences, we train and evaluate agents using mul- tiple randomly ordered behavior sequences. Refer to Sec. B.2 for more details about the sequences.
4

---

Published as a conference paper at ICLR 2024
Classes Present in Current Batch:
Current Logits
Confidence-Aware Coefficient Determination
Logit Update
Carfidence Scorea
Confidence Score Queuns
Q1
0.9
0.9
0.9
0.7
71
Previous Logits
Current Logits
1/1
Q2
0.9
9/3
0.8
1
7
7
Q3
0.8
0.9
0.7
0.6
imaan
1/4
0.7
y6
0.9
Q4
0.9
0.8
0.7
0.8
0.8
&
Y4
Updated Logits
Figure 2: Proposed Confidence-Aware Moving Average (CAMA). 'Current Logits' denotes the model's logits obtained from the imput samples from the current stream and episodic memory. 'Previous Logits' denotes logits stored in episodic memory hefore update. Q; denotes queue that stores ground truth confidence scores acquired from the current logits, V1 02,-.., for the 1d class. To obtain %, maintain the recent N confidence scores for the 1t class and calculate the mean value of the scores followed hy a clip function. Finally, we use all 7 to daissy weight-sum previously stored logits (i.e. 'Previous Logits') ) and newly obtained logits from the current stream (i.e. 'Current Logits").
3.2.2 ENVIRONMENT INCREMENTAL LEARNING
The Environment-IL. setup allows agents to learn the environment incrementally. In the real world, agents often need to perform actions not only in the environment in which they were initially trained but also in new and different environments that are presented. For example, the agent may first learn various actions in kitchen and then subsequently learn the actions in a bathroom.
Formally, for set of environments, ε, an agent sequentially receives Ms training episodes, {s 1M for each environment type, ej E E. When receiving the final episode (i.e., 8it for Me+1
the current environment type, ex, the agent begins to sequentially receive episodes, {st 0k+1 Ji=1 for the environment type, 6k+1- This is repeated until it reaches the last environment type, E|g)-
In this setup, we adopt four different types of environments supported by (Kolve et al. KITCHENS, LIVINGROOMS, BEDROOMS, and BATHROOMS. Like the Behavior-IL setup, we con- duct training and evaluation using multiple sequences of randomly ordered environments. We also provide more details of the multiple environment sequences in Sec. B.3
2017):
However, we observe an imbalance in the training and evaluation episodes between different types of environment (Shridhar al.) 2020), where majority of them originate from specific environment type in many instances. The imbalance can potentially lead to biased learning of model towards the dominant (i.e., majority) environment type (Chakraborty Chakraborty 2020) [Zhao et al., (021a)) To address the issue, we balance them by simply subsampling the training and evaluation episodes for each environment to match the number episodes across environment types equally.
APPROACH
To mitigate catastrophic forgetting, recent ap-
1.0
1.0
proaches (Li & Hoiem| 2017] Koh et al.] 2023
Connidance
0.8
0.8
Boschini et al. 2022b) knowledge distillation
Accura
with the trained model until past tasks, but this of-
entails significant memory (Zhou al. (022a) and computational overhead caused by additional model inference (Prabhu al., (023). Due to the limitations in memory and computation on edge devices (Lee et al.) 2022), logit distillation meth- ods (Buzzega et al. 2020; Boschini et al.) (022aa) have been proposed as alternatives to those that store entire models for distillation (Li & Hoiem] 2017 Koh et al. 2023), offering improved mem- ory and computation efficiency. Despite such improved efficiency, some of the logit distillation
0.2
Confidence Accuracy
0.2
10k
12K
Sampies
Figure 3: Confidence and accuracy of logits used for logit update CAMA. 'Accuracy' denotes mean of the frame wise accuracies measured from the newly obtained logits (here, x() in Equation | 'Confidence' denotes the dynamically determined coefficients (here, % for the update in Equation 2
5

---

Published as a conference paper at ICLR 2024
methods (Buzzega al. 2020) often face outdated logit problem, as the memory-storee logits are not updated to preserve information on previous tasks.
To address this issue, recent approach (Boschini et al., 2022a) attempts to partially update logits stored in the past using the current model. It uses task boundary information (e.g., input's 's task iden- tity) during training, but it may not always be available, especially in task-free CL setups including our proposed ones. Towards a general logit-update framework devoid of such information, we up- date the stored logits using the agent's confidence scores indicating how the newly obtained logits for update contain accurate knowledge of the corresponding tasks, as observed in Figure
4.1 CONFIDENCE-AWARE MOVING AVERAGE
As illustrated in Figure 2 the overall process of Confidence-Awar Moving Average (CAMA) can be summarized as follows: Initially, exploiting the model's 's confidence scores of ground-truth labels, we evaluate the extent to which the model has acquired proficiency in the current samples. Subse- quently, during the computation of the updated logits based on the previous and current logits, we allocate higher weight to the current logits when exhibiting higher confidence scores, and con- versely, assign a higher weight to the previous logits demonstrating lower confidence scores. For better understanding, we outline the high-level flow of our CAMA in Algorithm
I in the appendix.
Following the common practice KKrkpatrick et al., 2017 Rolnick et al., 2019 Buzzega et al,) 2020), we construct an input batch, [x:="), by combining data from both the training data stream (x, Ba. Me) ~ DD and the episodic memory (a', V/ yer *old,a *old,c) ~ .,, where each a E and C indicates an action and object class label from the action and object class sets, and C, present in the input batch, [x; Here, represents the input (i.e., images and language directives), Ya and Ye denote the corresponding action and object class labels, and z'id, and =otd, refers to the corresponding stored logits. za, Zes =/g and z/c denote the current model's logits for the input batch.
To prevent the logits maintained in the episodic memory from becoming outdated, we obtain the updated logits, =newaa and 2 =''ewee> by weighted-summingg z'Id,a and ='dd.c with z'- and z'c using coefficient vectors, Ya and %e. using Hadamard product, denoted by @, in Equation
know = (1 - Va) O Larda +a C Z/as Enewe. = =(1 - Tc) O know,o + Y ① 2
(1)
To obtain Ya and Ye. first maintain the most recent N confidence scores for each action and object class label for æ. Then, to approximate the agent's proficiency in learning tasks over time, we compute the average of the scores associated with each action (i) and object class (j) label, denoted by 8 and gc. We then set each element of Ta and Ye, denoted by Ya,s and Va.j. to 8 and 83 followed by CLIP function in Equation [
To = Ota CLIP ( 1,,1,, Te. - aL,IIP 115.LIP(55-111--.000
where CLIP min, max) denotes the clip function that limits the value of x from min max. Here, the constants a, < 1 and a, < 1 are introduced to prevent Ja,5 and %e. from reaching a value of as this indicates complete replacement of the prior logits with the current logits, which implies that the updated logits would entirely forget the previously learned knowledge. The inclusion of these constants ensures that some information from the past is retained and not entirely overridden by the current logits during the update process. In addition, we subtract |A|-1 and [CI-1 enhances the effective utilization of confidence scores comparison to a 'random" selection, which would otherwise be realized by uniform distribution (Koh et al. 2022
4.2 MODEL. TRAINING
Given expert demonstrations, x as input, we train our agent, np, by minimizing the objective below:
min (eeneeeeDD 8
EB v)] + Ee - (z)
(3)
where denotes the ground-truth labels corresponding to x and 2 the logits maintained in the episodic memory. We provide more details of the training loss, L, in Sec. D.3 in the appendix.

---

Published as a conference paper at ICLR 2024
5 EXPERIMENTS
Evaluation metries. For evaluation of task completion ability, we follow the same evaluation protocol of (Shridhar al., 2020). The primary metric is the success rate (SR) which measures the ratio of the succeeded episodes among the total ones. The second metric is the goal-condition success rate (GC) which measures the ratio of the satisfied goal conditions among the total ones. Furthermore, we evaluated all agents in two splits of environments: seen and unseen environments which are/are not used to train agents. We provide more details of the evaluation protocol in Sec. [
To evaluate the last and intermediate performance of continual learning agents, we measure two vari- ations of a metric, A: Atast and Aavg- Atast (here, SRist and GCitst indicates the metric achieved by the agent that finishes its training for the final task. Aavg (here, SRaeg and GCavy) indicates the average of the metries achieved by the agents that finish their training for each incremental task.
All the models are trained sequentially over a sequence of behaviors (Behavior-IL, and environments (Environment-IL) and then evaluated over the behaviors and environments that the models have learned so far. For evaluation, we use episodes different from those used for training. The same trained models are evaluated in both seen and unseen environments. For seen and wnseen, we denote by seen the evaluation with the episodes generated from scenes used in training, while we denote by unseen the evaluation with the episodes generated from scenes not used training.
Baselines. We compare our CAMA with competitive prior arts in continual learning literature: CLIB (Koh et al. 2022). DER++ (Buzzega et al. 2020), ER (Rolnick et al. 2019). MIR (Aljundi al. 2019a), EWC (Kirkpatrick et al. | 2017). and X-DER (Boschini al. [2022a). In addition, we also compare our CAMA with two models: 'Joint' and 'Finetuning.. "Joint' denotes that the agent trained with all task data jointly, which works as an upper bound. 'Finetuning' denotes that the agent is fine-tuned for the new tasks or scene types, which can serve as one of the trivial solutions for continual setups. We provide further explanation for each baseline in Sec. D.2 We further detail the model architecture and training used for the methods above in Sec. D.3 for space's sake.
Implementation details. It is a common practice in continual learning literature (Bang et al. 2021 Koh et al. 2022. 2023) to set the size of episodic memory to less than 5%. To align with previous works in continual learning, we set the size of the episodic memory to M = 500 for expert demonstrations, which is approximately 2.38% of the training episodes in the ALFRED bench- mark (Shridhar et al. 2020). For our CAMA, empirically set O. 0.99 and a 0.99. We provide more implementation details such as hyperparameters in Sec. D.4 for space's sake.
5.1 COMPARISON WITH STATE OF THE ART
We present the quantitative results of our CAMA in Table 12 As mentioned in Sec. 3.2 we train and evaluate our CAMA and baselines for three random seeds and report the results with their average and standard deviation avoid favoring particular behavior and environment sequences. We provide quantitative analyses in various aspects as follows.
Joint training vs. Finetuning. Before investigating the effectiveness of our CAMA, we first in- vestigate how challenging the proposed Behavior-IL and Environment-IL. setups are. We observe significant performance drops in 'Finetuning" compared to 'Joint" with 51.0% and 47.5% relative drops. This implies that simply finetuning agents to novel behaviors and environments cannot effec- tively address the forgetting caused by distribution shifts between behaviors and enviromments.
Ours vs. Regularization-base model. We observe that our CAMA achieves better performance than the regularization-base approach (i.e., EWC++) with noticeable margins in both seen and unseen environments for all metrics and setups, indicating that regularizing changes in importance parameters may not effectively prevent forgetting than distilling knowledge from updated logits.
Ours vs. Rehearsal- models. We observe that our CAMA outperforms all rehearsal-based approaches (i.e., ER, MIR, and CLIB) for all metrics in both seen and unseen environments in both Behavior-IL IL and Environment-Il IIL setups. We believe that this implies that solely depending on sample replay amidst rapid data distribution shifts can result in insufficient task forgetting mitigation and hinder the agent's 's ability to adapt to novel tasks, ultimately impeding effective task completion.
Ours vs. Distillation-basse models. We compare our CAMA with the distillation bastd ap- proaches (i.e., DER and X-DER) to investigate the effectiveness of our logit-update approach. First,
7

---

Published as a conference paper at ICLR 2024
Validation Seen
Model
Bebavior-IL
Environment-IL
SR....
GCrast
SR.u
GC..g
SR
GC
SRoug
GCany
Finetuning
9.51
1,09
21.33 1: 0.61
17.07
= 0.88
26.11 1: 0.95
8.72 1: 2.12
15.38 1,29
16.25 1 3,95
21.40 1: 4.65
EWC++
20.37 5.19
5.92
22.21
4.34
30.97
4.28
26.79
2.24
36.79
1.83
31.01
1: 2.76
40.56
2:22
ER
26.71 +: 1.49
36.59
1.36
27.67
2,08
36.20
1.96
30.28
1.07
39.15
+ 0.83
34.72
4: 1.56
44.00
1.52
MIR
30.27 1.33
40.14
2.00
28.12
1.78
36.76
1: 1.73
27.50
1.48
36.31
1.43
31.81
0.8.
00.94
0.95
23.85 2,02
34.25
1.81
23.94
2.36
32.65
2:22
25.47
1.42
34.63
1.55
32.51
2.40
41.25
DER+
29.15 4: 1.29
39.39
1: 1.16
27.49
2:277
36.
1: 1.92
28.25
1: 1.18
36.18
1: 0.60
28.68
2:5.4
38.01
3.16
X-DER
28.76
1.25
38.45
1.18
21.21
2.53
35.88
2.22
28.30
0.87
37.02
0.61
20.32
2.36
30.13
2.00
CAMA win D.C.
30.54 + 1.27
39.32 + 1.50
29.89
2.32
16.16.1
+ 2.71
31.58
0.87
39.29
1.03
34.49 + 2.29
43.28
2.17
CAMA (Ours)
30.71
0,78
40.85
0,73
29.67
2.06
38.17
2.34
29.48
0.27
38.13 = 0.85
35,09 = 1.92
44,02
2.21
Joint
00:47 = 0.33
63.77 +: 0.78
36:25 0.89
02.13 = 0.84
Table 1: Comparison with state-of-the-art methods (validation seen). The highest value per metric is in
bold. We report the averages and standard deviations of multiple runs for random seeds as in Sec. 3.2
Validation Unseen
Model
Behavlor-IL
Enviromment-ll
SRioet
GCoast
SRaus
GCoug
SRroet
GCacet
SRang
GCany
Finctuning
1.18
1.09
12.09 1.65
3.03
1:29
13.95
1.01
2.01
0.86
11:20 1.92
2.90 2.16
13.53 +: 4.33
EWC+
8,50 = 2.15
21.63 + 3.60
8.33
1.33
20.71 # 1.99
11.61 +: 1:29
28.17 = 0.83
12.37 :: 1.30
29.90
1.30
ER
9.43 + 1.25
24.22
+ 1.54
9.47 + 1.51
22.79 + 1.30
11.44
+ 1:36
20.11 0.96
14.25 + 1.47
31.98 + 1.64
MIR
11.01
1.16
25.31
1.14
10.88
1.51
21.20
1.45
12:01
0.61
29.67
12.58
29.11
CLIB
8:26 = 1.03
22.00
+ 1.31
8.50
= 0.66
21.03
: 1.15
10.46
+: 1.18
27.40
0.78
11.95
2: 1.51
29.93
2.05
DER++
13.16 = 5.36
28.70
1: 5.63
10.60
4.01
21.91
2.69
10.29
1.05
26,90
1.31
10.25
2 1.72
26.83 1.97
X-DER
12.59
1.92
2.05
12.04
1.56
25.50
1.48
10.75
1.15
28.37
1.05
11.14
1.38
28.56
1.44
CAMA w/o D.C.
14,06
1.20
28.33
1: 1,58
12.52
1.46
26.02
1: 1,38
13.57 1: 1.25
29.54
1.41
12.78
1. 0.57
29.76
1 0.84
CAMA (Ours)
13.61
0.94
28.75
14.19
1.38
27.30
1,38
14.60
0.43
30.99
0.75
15.67
0.77
33.40
1.45
Joint
24.00 + 0.93
38.24 + 1.55
19.73 + 2.31
30.02 + 0.53
Table 2: Comparison with state-of-the-art methods (validation unseen). The highest value per metric is in bold. We report averages and standard deviations of multiple runs for random seeds in Sec. 3.2
we observe noticeable performance drops in DER, which does not update logits, compared to our CAMA for all metrics in seen and unseen environments in both Behavior-IL and Environment-IL. setups, which highlights the importance of updating logits to prevent them from being outdated.
In addition, we observe that our CAMA outperforms X-DER, which partially updates logits only for novel classes, with noticeable margins for all metrics and environments in both the Behavior-IL IL and Environment-IL setups, highlighting the efficacy of our CAMA. We note that while X-DER updates logits based on task boundary information during training, our CAMA does not assume the availability of such information (i.e., task-free), which highlights the generality of our CAMA.
5.2 THE EFFECTIVENESS OF DYNAMICALLY DETERMINED COEFFICIENTS
We investigate the effect of dynamically determined coefficients of our CAMA by fixing them with constant value and provide the results ('CAMA' vs. 'CAMA w/o D.C.' in T 'CAMA w/o D.C.' assumes that the agent is always 100% confident in what it learns. Consequently, we directly set 7/2 and Yo Oa and or by omitting the process of dynamically determined coefficients.
We observe that the ablation of dynamically determined coefficients consistently yields performance drops in all metrics in seen and unseen environments in both Behavior-IL and Environment-IL. se- tups, indicating the importance of the process of finding such coefficients. This could be attributed to the fact that while logit updating with a constant coefficient helps mitigate the obsolescence of the logits to some extent, it also combines them with logits from the current model that lacks sufficient training for novel tasks, particularly during the initial phase of learning these tasks. Consequently, this can lead to performance degradation due to incomplete knowledge of these new tasks.
5.3 QUALITATIVE ANALYSIS
We provide qualitative examples of our CAMA in each Behavior-IL and Environment-IL setup by comparison with the naïve (i.e., Finetuning) and prior best-performing (DER++) methods. For space's 's sake, we provide the qualitative example in the Environment-Il II. setup in Sec. D.5
8

---

Published as conference paper at ICLR 2024
Instructions:
Irrelevant Action
Inseracted Object
TJ-1
HEAT
PICK2&PI.ACT
Turn around to your right move forward then turn left, head to the coffee maker. Pick up the mug in front coffee maker on counter. Turn to your left head to microwave above stove. Open the microwave then put and out the mug then close the microwave, Turn to your right and head to coffee maker. Put the mug on the coffee maker.
MiNg
Finetuning
DER++
Right
Left
Abead
Pickup
Right
Ahead
Right
CAMA (Ours)
Mrg
Mirowarr
Microware
Microwave
Microware
Migg
Pickup
Open
Put
ToggleOn
ToggleOff
Pickup
Put
Figure 4: Qualitative analysis of the proposed method (Behavior-IL). The agent, having already acquired knowledge of the behavior 73-1 HEAT, proceeds to learn the new behavior TΙ = PICK2&PLACE. Subse- quently. evaluate the agent's ability of the prior behavior 7j-1 to determine if any forgetting has occurred. Irrelevant Action denotes action that results in incorrect navigation. 'Finetuning' fails find target ohject, 'Mug'' and eventually fails at task. DER++ succeeds in navigating to and picking up the mug but fails to reach a microwave above the agent, also leading to task failure. On the contrary, our CAMA further succeeds in reaching the microwave, heating the mug, and putting it back on the coffee machine, leading to task success.
The Behavior-IL setup. Figure the agent evaluated for the previous behavior, Tj-1 HEAT while learning the current behavior, тj PICK2& Here, the agent is required to heat a mug and put on the coffee machine. The agent can sequentially complete the task by 1) picking up mug. 2) heating it using microwave, and 3) putting it the coffee machine.
Finetuning' first explores the environment to find a mug. However, it fails to recogmize the mug and therefore keeps wandering in the environment, eventually leading to task failure. Meanwhile, DDE++'' succeeds in finding the mug and picking it up but forgets how to reach a microwave above the agent to heat object. The agent also keeps wandering the environment and eventually, it fails at the task. In contrast, our CAMA succeeds in navigating to and picking up the mug, After grabbing the mug, agent finds, reaches the microwave above, and successfully heats the mug. Finally, our agent then put the heated mug on the coffee machine, as described in the instructions, which implies that our CAMA enables the agent to maintain the knowledge of the previous behaviors.
CONCLUSION
We propose two continual learning setups that learn new behaviors (Behavior Incremental Learning, Behavior-II.) and environments (Environment Incremental Learning, Environment-ll) continually. Prior methods employ the storage of model logits from previous tasks but they are updated either only once upon obtaining new logits, potentially resulting in learning with outdated data or uti- lizing logits from a model that has incompletely learned the new tasks.
To effectively update the logits, propose Confidence-Aware Moving Average (CAMA), simple yet effective approach that dynamically determines moving average coefficients based on the agent's confidence scores. We observe that the CAMA outperforms all prior arts by noticeable margins.
Limitation and future work. While the disjoint setup operates under the assumption that tasks in streaming data are aoerolerlgpping (Parisi et al., (019), posing stringent test for catastrophic always be the case in real-world scenarios. To address this aspect, extending our proposed setups to feature overlapped tasks in streaming data, such
forgetting, such non-overlapping scenarios might
blurry setups (Prabhu et al. 2020 Bang al. 202T) Gaussian scheduled regimes (Shanahan
et al. 2021 Wang et al. 2022 represents promising avenue for future research.

---

Published as a conference paper at ICLR 2024
ETHICS STATEMENT
This work introduces continual learning setups for interactive instruction following agents and logit-update approach to enhance the effectiveness of knowledge distillation. While the authors do not aim for this, the increasing adoption of deep learning models in real-world contexts with stream- ing data could potentially raise concerns such as privacy and model robustness. There is possibility of these deployed deep learning models inadvertently introducing biases or discrimination, unre- solved issues like model bias persist within deep learning. We are committed to implementing all feasible precautions to avert such consequences, as they are unequivocally contrary to our intentions.
REPRODUCIBILITY STATEMENT
We take reproducibility deep learning very seriously and highlight some the contents of the manuscript that might help to reproduce work. We release the data splits of the proposed bench- marks in Sec. 3 our implementation of the proposed method in Sec. 4 and the baselines used in our experiments in Sec. in https::ss....... /ttp//gttubb../ com/ snumpr Lab cl-cllllllll al-alfred
ACKNOWLEDGMENT
This work was partly supported by the NRF grant (No. 210.2022RIA224402220 15%) and IITP grants (No. NN0.2020-0-1161 (10% Yonsei AI), No.2021-0-01343 (5%, SNU AI), No.2022-0-00077 (10%), No.2022-0-00113 (20%), No.2022-0-00959 (15%), No. N0220-2-0000871 (15%), 20.2021-0-02068 (5%, AI Innov. Hub), No.2022-0-00951
(5%)) funded by the Korea government (MSIT).
REFERENCES
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In ECCV, 2018a.
Rahaf Aljundi. Klaas Kelchtermans, and Tinne Tuytelaars.
Task-free continual learning.
arXiv:1812.03356. 2018b.
Rahaf Aljundi, Lucas Caccia, Eugene Belilovsky, Massimo Caccia, Min Lin, Laurent Charlin, and Tinne Tuytelaars. Online continual learning with maximally interfered retrieval. In NeurIPS, 2019a.
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. In NeurlPS, 2019b.
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. Vision-and- language navigation: Interpreting visually-grounded navigation instructions in real environments. In CVPR, 2018.
Jihwan Bang, Heesu Kim, YoungJoon Yoo, Jung-Woo Ha, and Jonghyun Choi. Rainbow memory: Continual learning with memory of diverse samples. In CVPR, 2021.
Suvvansh Bhambri, Byeonghwi Kim, and Jonghyun Choi. Multi-level compositional reasoning for interactive instruction following. In AAAI, 2023.
Magdalena Biesialska, Katarzyna Biesialska, and Marta R Costa-Jussa. Continual lifelong learning in natural language processing: A survey, arXiv:2012.0982. 2020.
Matteo Boschini, Lorenzo Bonicelli, Pietro Buzzega, Angelo Porrello, and Simone Calderara. Class- incremental continual learning into the extended der-verse. In IEEE TPAMI, 2022a.
Matteo Boschini, Lorenzo Bonicelli, Angelo Porrello. Giovanni Bellitto, Matteo Pennisi, Simone Palazzo, Concetto Spampinato, and Simone Calderara. Transfer without forgetting. In ECCV, 2022b.
10

---

Published as a conference paper at ICLR 2024
Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: a strong, simple baseline. In NeurIPS, 2020.
Fabio Cermelli, Antonino Geraci, Dario Fontanel, and Barbara Caputo. Modeling missing annota- tions for incremental learning in object detection. In CVPR, 2022.
Tanujit Chakraborty and Ashis Kumar Chakraborty. Superensemble classifier for improving predic- tions in imbalanced datasets. In Communications in Statistics: Case Studies, Data Analysis and Applications, 2020.
Arslan Chaudhry, Puneet K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In ECCV, 2018.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. ICML, 2020.
Ekin Dogus Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoe V. Le. Autoaugment: Learning augmentation policies from data. In CVPR, 2019.
Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Em-
bodied question answering. CVPR, 2018.
Matt Deitke, Winson Han, Alvaro Herrasti, Aniruddha Kembhavi, Eric Kolve, Roozbeh Mot- taghi, Jordi Salvador, Dustin Schwenk, Eli VanderBilt, Matthew Wallingford, Luca Weihs, Mark Yatskar, and Ali Farhadi. Robothor: An open simulation-to-real embodied ai platform. CVPR, 2020.
Kiana Ehsani, Winson Han, Alvaro Herrasti, Eli VanderBilt, Luca Weihs, Eric Kolve, Aniruddha Kembhavi, and Roozbeh Mottaghi. Manipulathor: framework for visual object manipulation. In CVPR, 2021.
Linxi Fan, Guanzhi Wang. Yunfan Jiang, Ajay Mandlekar, Yuncong Yang. Haoyi Zhu, Andrew Tang, De-An Huang. Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. In NeurIPS, 2022.
Chongkai Gao, Haichuan Gao, Shangqi Guo, Tianren Zhang, and Feng Chen. Cril: Continual robot imitation learning via generative and prediction model. In IROS, 2021.
Xiaofeng Gao, Qiaozi Gao, Ran Gong, Kaixiang Lin, Govind Thattai, and Gaurav S Sukhatme. Dialfred: Dialogue-enabled agents for embodied instruction following. In RA-L, 2022.
Daniel Gordon, Aniruddha Kembhavi, Mohammad Rastegari, Joseph Redmon, Dieter Fox, and Ali Farhadi. Iqa: Visual question answering in interactive environments. In CVPR, 2018.
Tyler L Hayes and Christopher Kanan. arXiv:2203. 10681, 2022.
Online continual learning for embedded devices.
Jiangpeng He, Luotao Lin, Jack Ma, Heather Eicher-Miller, and Fengqing Zhu. Long-tailed continual learning for visual food recognition. arXiv preprint arXiv:2307.00/88 2023.
Md Sazzad Hossain, Pritom Saha, Townim Faisal Chowdhury, Shafin Rahman, Fuad Rahman, and Nabeel Mohammed. Rethinking task-incremental learning baselines. In ICPR, 2022.
Vihan Jain, Gabriel Magalhaes, Alexander Ku, Ashish Vaswani, Eugene Ie, and Jason Baldridge. Stay on the path: Instruction fidelity in vision-and-language navigation. In ACL, 2019.
KJ Joseph, Jathushan Rajasegaran, Salman Khan, Fahad Shahbaz Khan, and Vineeth N Balasubra- manian. Incremental object detection via meta-learning. IEEE TPAMI, 2021.
Khimya Khetarpal, Shagun Sodhani. Sarath Chandar, and Doina Precup. Environments for lifelong reinforcement learning. arXiv:1811.
2018.
Byeonghwi Kim, Suvaansh Bhambri, Kunal Pratap Singh, Roozbeh Mottaghi, and Jonghyun Choi. Agent with the big picture: Perceiving surroundings for interactive instruction following. In Embodied Al Workshop @ CVPR, 2021.
11

---

Published as a conference paper at ICLR 2024
Byeonghwi Kim, Jinyeon Kim, Yuyeong Kim, Cheolhong Min, and Jonghyun Choi. Context-aware planning and environment-awee memory for instruction following embodied agents. In ICCV, 2023.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Seaaska--Brwwinge Demis Hass- abis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. In PNAS, 2017.
Hyunseo Koh, Dahyun Kim, Jung- Woo Ha, and Jonghyun Choi. Online continual learning class incremental blurry task configuration with anytime inference. In ICLR, 2022.
Hyunseo Koh, Minhyuk Seo, Jihwan Bang, Hwanjun Song, Deokki Hong, Seulki Park, Jung- Woo Ha, and Jonghyun Choi. Online boundary-free free continual learning by scheduled data prior. In ICLR, 2023.
Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. Ai2-thor: An interactive 3d environment for visual ai. 1rXiv::712.088:
2017.
Jacob Krantz, Erik Wijmans, Arjun Majumdar, Dhruv Batra, and Stefan Lee. Beyond the nav-graph: Vision-and-language navigation in continuous environments. In ECCV, 2020.
Soobee Lee, Minindu Weerakoon, Jonghyun Choi, Minjia Zhang, Di Wang, and Myeongjae Jeon. Carm: hierarchical episodic memory for continual learning. In DAC, 2022.
Timothée Lesort, Andrei Stoian, and David Filliat. Regularization shortcomings for continual learn- ing. arXiv: 19t.iv//9/.000 /9/.99922.999.// 2019.
Timothée Lesort, Vincenzo Lomonaco, Andrei Stoian, Davide Maltoni, David Filliat, and Natalia Díaz-Rodríguez. Continual learning for robotics: Definition, framework, learning strategies, op- portunities and challenges. In Information 2020.
Zhiyuan Li and Sanjeev Arora. arXiv: 1910.07444 2019.
An exponential learning rate schedule for deep learning.
Zhizhong Li and Derek Hoiem. Learning without forgetting. In IEEE TPAMI, 2017.
Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning. arXiv:23306003311 2023.
Xialei Liu, Yu-Song Hu, Xu-Sheng Cao, Andrew D Bagdanov, Ke Li, and Ming-Ming Cheng. Long- tailed class incremental learning. In ECCV, 2022.
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. In NeurIPS, 2017.
Ilya Loshchilov and Frank Hutter. 1rX.::1608.03983. 2016.
Sgdr:
Stochastic gradient descent with warm restarts.
Chih-Ya Yao Ma, Jiasen Lu, Zuxuan Wu, Ghassan AlRegib, Zsolt Kira, Richard Socher, and Caiming Xiong. Self-monitoring navigation agent via auxiliary progress estimation. In ICLR, 2019.
Zheda Mai, Ruiwen Li, Jihwan Jeong, David Quispe, Hyunwoo Kim, and Scott Sanner. Online continual learning in image classification: An empirical survey. In Neurocomputing, 2022.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, 1989.
Jorge Mendez, Shashank Shivkumar, and Eric Faton. Lifelong inverse reinforcement learning. In NeurlPS, 2018.
Umberto Michieli and Pietro Zanuttigh. Knowledge distillation for incremental learning in semantic segmentation. In CVIU, 2021.
12

---

Published as a conference paper at ICLR 2024
So Yeon Min, Devendra Singh Chaplot, Pradeep Ravikumar, Yonatan Bisk, and Ruslan Salakhutdi- nov. Film: Following instructions in language with modular methods. In ICLR, 2022.
Dipendra Misra, John Langford, and Yoav Artzi. Mapping instructions and visual observations to actions with reinforcement learning. In EMNLP, 2017.
Van-Quang Nguyen, Masanori Suganuma, and Takayuki Okatani. Look wide and interpret twice: Improving performance interactive instruction-following tasks. In IJCAI, 2021.
Aishwarya Padmakumar, Jesse Thomason, Ayush Shrivastava, Patrick Lange, Anjali Narayan- Spandana Gella, Robinson Piramuthu, Gokhan Tur, and Dilek Hakkani-Tur Teach: Task-driven embodied agents that chat. AAAI, 2022.
German Parisi, Ronald Kemker, Jose Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. In Neural networks, 2019.
Jaeyoo Park, Minsoo Kang, and Bohyung Han. Class-incremental learning for action recognition in videos. In ICCV, 2021.
Alexander Pashevich, Cordelia Schmid, and Chen Sun. language navigation. In ICCV, 2021.
Episodic transformer for vision-and-
Sam Powers, Eliot Xing, Eric Kolve, Roozbeh Mottaghi, and Abhinav Gupta. Cora: Benchmarks, baselines, and metrics platform for continual reinforcement learning agents. In CoLLAs, 2022.
Ameya Prabhu, Philip HS Torr, and Puneet K Dokania. Gdumb: A simple approach that questions our progress continual learning. ECCV, 2020.
Ameya Prabhu, Hasan Abed Al Kader Hammoud, Puneet K Dokania, Philip HS Torr, Ser-Nam Lim, Bernard Ghanem, and Adel Bibi. Computationally budgeted continual learning: What does matter? In CVPR, 2023.
Roger Ratcliff.
Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. In Psychological review, 1990.
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl:
Incremental classifier and representation learning. In CVPR, 2017.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning. In NeurlPS, 2019.
Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient projection memory for continual learning. arXiv:2103.09762. 2021.
Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. Habitat: A Platform for Embodied AI Research. In ICCV, 2019.
Murray Shanahan, Christos Kaplanis, and Jovana Mitrovié. Encoders and ensembles for task-free continual learning. arXiv:2105. 13327, 2021.
Dongsub Shim, Zheda Mai, Jihwan Jeong, Scott Sanner, Hyunwoo Kim, and Jongseong Jang. On- line class-incremental continual learning with adversarial shapley value. In AAAI, 2021.
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In CVPR, 2020.
Kunal Pratap Singh, Suvaansh Bhambri, Byeonghwi Kim, Roozbeh Mottaghi, and Jonghyun Choi. Factorizing perception and policy for interactive instruction following. In ICCV, 2021.
Andrés Villa, Kumail Alhamoud, Victor Escorcia, Fabian Caba, Juan León Alcázar, and Bernard Ghanem. velimb: A novel video class incremental learning benchmark. In CVPR, 2022.
13

---

Published as a conference paper at ICLR 2024
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv:2305.16221, 2023.
Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vin- cent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In CVPR, 2022.
Luca Weihs, Matt Deitke, Aniruddha Kembhavi, and Roozbeh Mottaghi. Visual room rearrange- ment. In CVPR, 2021.
Maciej Wolczyk, Michat Zajac, Razvan Pascanu, Łukasz Kuciñski, and Piotr Mitoš. Continual world: A robotic benchmark for continual reinforcement learning. In NeurlPS, 2021.
Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu.
Large scale incremental learning. In CVPR, 2019.
Annie Xie and Chelsea Finn. Lifelong robotic reinforcement learning by retaining experiences. In CoLLAs, 2022.
Fan Yang, Chao Yang, Huaping Liu, and Fuchun Sun. Evaluations of the gap between supervised and reinforcement lifelong learning on robotic manipulation tasks. In CoRL, 2022.
Fei Ye and Adrian G Bors, Task-free continual learning via online discrepancy distance learning. In NeurIPS, 2022.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In ICMI, 2017.
Bowen Zhao, Chen Chen, Qi Ju, and ShuTao Xia. Energy aligning for biased models. 0rXiv:2/06.03343. 2021a.
Hanbin Zhao, Xin Qin, Shihao Su, Yongjian Fu, Zibo Lin, and Xi Li. When video classification meets incremental classes. In ACM MM, 2021b.
Da- Wei Zhou, Qi-Wei Wang, Han-Jia Ye, and De-Chuan Zhan. A model or 603 exemplars: Towards memory-efficient tlass-incrementae learning. arXiv: /3218, 2022a.
Da-Wei
Wenxuan Zhou, Steven Bohez, Jan Humplik, Nicolas Heess, Abbas Abdolmaleki, Dushyant Rao, Markus Wulfmeier, and Tuomas Haarnoja. Forgetting and imbalance in robot lifelong learning with off- policy data. In CoLLAs, 2022b.
Yuke Zhu, Daniel Gordon, Erie Kolve, Dieter Fox, Li Fei-Fei, Abhinav Gupta, Roozbeh Mottaghi, and Ali Farhadi. Visual semantic planning using deep successor representations. In ICCV, 2017.
14

---

Published as a conference paper at ICLR 2024
APPENDIX
A EXTENDED RELATED WORK
Continual learning setup. We can categorize continual learning setups into Task Incremental (task-IL) and Class Incremental (class-IL) depending whether task ID is given during inference.
In task-II., task-ID is provided during inference (Aljundi et al. 201 8a Lopez-Paz Paz & Ranzato 2017
Hossain et al. 2022), , while in class-II., task II) is not given (Koh al. 2022 Buzzega al. 2020
Koh et al., 2023 Bang et al. 202I). As task-ID is not provided during inference in the real world, class-IL is closer to the real setup and is more challenging because requires classification across
all classes. We focused on class-II. to deal with more realistic situations.
Continual learning methods. As the need for continual learning is increasingly highlighted across various fields, researchers have proposed a wide array of continual learning methods to pre- vent catastrophic forgetting. (1) Replay-
Methods (Prabhu et al., 2020 Koh et al. 2022 Bang et al. [021] Wu et al., 2019] RRlnick et al. (219)) store some stream data in episodic memory and replaying memory data during the future learning process to prevent forgetting about previous tasks.
(2) Distillation- Based Methods (Buzzega ct al. 2020; Koh et al. 2023 Li & Hoiem 2017) retain
knowledge about past data by storing logits (Buzzega et al. 2020) or models (Koh et al. 2023
Li & Hoiem| 2017 to distill knowledge. (3) Regularization- Based Method (Zenke al. 2017
Kirkpatrick et al.) 2017: Lesort et al.] 2019) prevents the overwriting of important parameters by imposing a penalty on changes to these crucial parameters.
Recently, continual learning has also been actively investigated in more challenging task setups such video domains (Zhao al. 2021b Villa al. 2022: Park al., 202T]. The focus of their work is generally on classification problems such as action recognition in a video by observing full video frames as given. In contrast, rather than receiving predetermined frames at once, the next observations (frames) of agents are determined by the actions that the agents take, which then require the agents to plan subsequent actions to complete tasks based on the next observations.
Embodied AI. Embodied AI (EAI) has garnered substantial attention, and notable advancements have been made in various tasks (Anderson et al. 2018 Krantz et al. 2020 Jain et al. 2019 Savva
et al. 2019 Deitke et al. 2020 Weihs et al.) 2021 Ehsani et al., 2021 Shridhar et al.. 2020 Pad- maakmar et al., 2022; Gao et al. 2022). For instance, visual navigation tasks necessitate that the
agent use visual observation to reach designated locations (Savva al.) 2019) or objects (Savva
ct al.
2019; Deitke et al., 2020). Meanwhile, vision-language navigation (VLN) (Anderson et al.
2018 Jain et al. 2019 Krantz et al.; 2020) augments visual observation with natural language de-
scriptions, enabling the agent to plan a sequence of actions based on a comprehensive understanding of multiple modalities to successfully reach the target locations.
Furthermore, the scope of FAI tasks has expanded through the inclusion of object interaction. (Weihs al.. 2021 necessitates the agent relocate objects to their original state by manipulating them, while (Ehsani et al. 2021) requires the agent to move objects to designated locations using six degrees of freedom (6-DoF) manipulation. (Shridhar al. 2020) presents natural language de- scriptions, which the agent must comprehend to plan sequence of actions and utilize predictive 2D object classes locate objects for interaction. Meanwhile, (Padmakumar et al., 2022 Gao et al,, 2022) provide natural language dialogues. in which the agent must engage in reasoning to determine the appropriate course of action and complete the tasks at hand.
However, the agents evaluated in those benchmarks are typically trained using pre-existing datasets. Given that the data collection process can be both expensive and time-consuming. it may not always be feasible to pre-collect the requisite dataset, implying the need for continual learning for the agents.
B ADDITIONAL CL-ALFRED BENCHMARK DETAILS
B. CONTINUAL LEARNING SETUPS
In alignment with the common practice in the prior arts (Aljundi et al.) 2019a Buzzega et al. 2020]] Shim et al. (02T). we assume that both setups follow an online and disjoint paradigm. In an online setup, individual samples (here, expert demonstrations) presented sequentially rather than being
15

---

Published as a conference paper at ICLR 2024
available simultaneously. While portion of the data is retained in episodic memory, the streaming data is accessible for learning only once. In a disjoint setup. each task (here, each behavior and environment type) contains distinct and unrelated information from the others. In this setup, the agent embarks on learning new tasks, it does not receive any samples from previously encountered tasks. Importantly, we do not rely on predefined task boundaries for training and evaluation.
B.2 BEHAVIOR INCREMENTAL. LEARNING
In the ALFRED benchmark, episodes comprise seven distinct behavior types: EXAMINE, HEAT, PICK&PLACE. COOL, CLEAN, Bick2&PLACE, and MOVABLE. Each behavior type presents dis- tinct goal conditions that the agents must fulfill by learning how to achieve them. In the Behavior-Il. setup, we employ the original validation split of the ALFRED benchmark for validation and five randomly ordered sequences of behavior types for training as follows.
EXAMINE + HEAT - PICK2&PLACE - COOL - PICK&PLACE + CLEAN - MOVABLE
2. PICK&PLACE PICK2&PILACE CLEAN HEAT EXAMINE MOVABLE COOL.
3. PICK&PLACE EXAMINE MOVABLE CLEAN - PICK2&PLACE COOL. - HEAT
MOVABLE - PICK2&PLACE - EXAMINE - PICK&PLACE - HEAT - COOL- - CLEAN
5. CLEAN PICK&PLACE + MOVABLE HEAT COOL PICK2&PLACE EXAMINE
B.3 ENVIRONMENT INCREMENTAL LEARNING
Episodes in the ALFRED benchmark are generated from four distinct environment types supported by A12-THOR (Kolve al.) 2017): KITCHENS, LIVINGROOMS, BEDROOMS, and BATHROOMS. Each environment type has 30 variations of the environment type that enable agents to learn behav- iors diverse room layouts and visual appearances. Similar to the Behavior-IL setup, for training, Environment-II IL also employs five randomly ordered sequences of environment types follows.
BEDROOMS + BATHROOMS + LIVINGROOMS + KITCHENS
2. BATHROOMS BEDROOMS + KITCHENS LIVINGROOMS
3. BEDROOMS LIVINGROOMS BATHROOMS - KITCHENS
4. BEDROOMS + BATHROOMS - KITCHENS - LIVINGROOMS
BATHROOMS >> KITCHENS > BEDROOMS > LIVINGROOMS
As discussed in Sec. 3.2.2 we observe imbalance in the 'train' and 'validation' splits of the original ALFRED across the environment types: for each KITCHENS, LIVINGROOMS, BEDROOMS, and BATHROOMS, 11, 056, 3, 456, 3. 370, and 3, 141 episodes for the 'train' split and 432, 129, 106, and 153 for the 'validation" seen split, and 468, 146, 120, and 87 for the 'validation' unseen split.
To balance them, subsample the train and validation episodes per environment type follows. For the 'train' split, we subsample 3, 141 episodes, leading to 12, 564 episodes in total. For the 'validation' seen split, subsample 106 episodes, leading to 424 episodes in total. Finally, for the 'validation' unseen split, we subsample 87 episodes, leading to 348 episodes in total.
C EXTENDED APPROACH
As described in Sec. 4.1 the high-level procedure of our CAMA is described in Algorithm [ When new samples, denoted as x (representing expert demonstrations), are received, we retrieve samples, denoted by x'. from episodic memory. Subsequently, we obtain the respective logits, denoted by and z'. from the model, denoted by Ta. With these logits, we compute the gradient of the joint loss, which combines cross entropy and knowledge distillation, to update the model parameters 0.
After updating 0, maintain record of the N most recent confidence scores in separate queues, denoted by Q. for each action and object class for x. Once these recent scores are maintained, we dynamically calculate the coefficients, %a and Te, weight-sum the previous and current logits, denoted by z'old and z', leading to the updated logits, denoted by 2' z'new, which are then stored in episodic memory. For more details the %a and % calculations, kindly refer Sec. 4.1
16

---

Published as a conference paper at ICLR 2024
Algorithm Pseudo code for CAMA
Input Model To parameterized by 8, Memory M, Training data stream D, Learning rate 14, Confidence score queues Q, Scalar 3, N, Appeared action set A, Appeared object class set C I/ Samples arrive from the stream If Acquire triplets from the memory I/ Obtain logits from the model Compute the total loss
for (z, v) E "D do
Sample (x'.y', Édda) < RandomSample(.M)
2, x' <- fo[[x; z'))
L = CrossEntropyLoss(( z/1, v/) + 3112' 2/aa 0 pr. VoL
Q < MaintaineeentCCofiddnneeee (Q, 2,
Maintain N recent confidence scores
Y CalculaaeAdaapiiveattt....
I/ Update old logits by CAMA (Sec. Update _M('', 1/' Losaa + M(x'.// 3/, <(ew)) // Update logits for samples retrieved from memory // Update Memory
new 7Faold + 7)²²
M t ReservoirSampler (M, (z, 2))
end for
D EXTENDED EXPERIMENT RESULTS
D.1 EVALUATION METRICS
For training and evaluation, the ALFRED dataset (Shridhar et al. 2020) consists of three splits; 'train,' "validation," and "test.' Agents are trained with the 'train" split and validate their approaches in the 'validation' split with the ground-truth information of the tasks in those splits. The agents are then evaluated in the 'validation" and 'test' split, but they do not have any access to the ground-truth information of the tasks. As the ground-trun truth labels of the 'test' split are not publicly available, we evaluate our agent and baselines and report the results in the validation split.
The validation split also consists of two folds: seen and unseen environments in which agents are/are not trained. Seen environments allow evaluating the task completion ability of agents in the same visual domain as training environments. Unseen environments further allow evaluating agents' task completion ability in different visual domains from training environments, which is considered more challenging than seen environments.
For the evaluation of task completion ability, the primary metric is the success rate (SR) which measures the ratio of completed tasks, indicating the task completion ability of the agent. Another metric the coul-condition success rate (GC) which measures the ratio of satisfied goal conditions, indicating the partial task completion ability of the agent. We evaluate all agents' performance in terms of SR and GC both seen and unseen environments and the main metric unseen SR.
Shridhar et al. 2020) also penalizes SR and GC with path-length -eii--eggtt-wwwwteedddd (PLW) scores propor- tional to trajectory lengths. Considering the model (Kim et al.) 202T) used in our experiments lags significantly behind human performance in terms of task completion ability (i.e., unseen success rates), however, we focus mainly on task completion ability and leave the examination of efficiency aspects for future investigations.
D.2 BASELINES
CLIB Koh et al. 2022 is a method that maintains an optimal episodic memory based on the im- portance of each sample. DER++ (Buzzega et al.) 2020) aims distill information about past data by storing not only images and labels but also logits, comparing them with the logits of the current model. ER (Rolnick et al. 2019) constructs the training batch with half of the current task stream and the other half of the data in memory, to remember past data while learning about new task. MIR (Aljundi et al. 2019a) uses samples that received the most interference from previous learning, rather than randomly y retrieving from memory when composing the training batch. EWC++ (Kirk-) patrick et al. 2017) prevents forgetting from parameter overwriting by penalizing changes of impor- tant parameters. X-DER (Boschini al. 2022a) rewrites logits for the portions corresponding to the classes of past tasks to incorporate newly acquired experience information about past tasks while learning new tasks. Following the prior methods (Singh et al. [022] Pashevich al. 2021 Nguyen] ct al., 202I) that keep visual encoders frozen, our agent's 's visual encoder also remains frozen, and thus omit the contrastive learning part (Chen al. 2020) in X-DER.
17

---

Published as a conference paper at ICLR 2024
D.3 MODEL ARCHITECTURE AND TRAINING
For model architecture, we adopt recently proposed learning-based agent (Kim et al., 2021 that perceives the surrounding views and predicts sequence of actions and object masks using factorized branches (Singh et al. (202T). Following the common practice of prior arts (Shridhar et al. 2020)
Singh et al., |021] Pashevich al., 2021; Nguyen al., al., [021] Kim et al., 2021), we train our agent with imitation learning, specifically behavior cloning. We detail the architecture and training below.
Model Architecture. The architecture of our CAMA and the baselines is based on recent ap- proach (Singh et al.) (021) that uses separate modules for effective action prediction and object localization to better address different semantic understandings from each other. Specifically, let V1 = (xx) be the agent that maps the input #t = (v;, Ma,t-1) to the output Yt (Ya,t, Vc,t)- For the input, each and I denotes the RGB images (i.e., surrounding views) and step-by-step instructions. For the output, each y/ and Ye denotes the action and object class to be interacted with.
As mentioned above, the agent f is comprised of two separate modules: the action prediction mod- ule a; = Saction (Vr, I, :o.t-1) and the object localization module C = Scloss (Ut, Wo..--). Both modules encode the instructions with attended language feature, i. To capture the correspondence between visual and textual information, conduct point-wise convolution for v, by filters generated from i, resulting in the attended visual feature 6, The decoder of each module updates its hidden state based on the attended visual and textual features, ti and i, with the previous action Va,t-1. resulting in he and hg for the action pre- diction module and the object localization module. Finally, fully connected layers in faction take as input f. Va,t-1, and hy and predict the current action Ya.e- Similarly, fully connected layers in faluse take input hf and predict the current object class fe,s with which to interact.
telf-attention-beet Bi-LSTM network, resulting in the
For more details of the modules, kindly refer to (Singh et al. 2021).
Training. Following (Shridhar et al. 2020), we adopt imitation learning for training, specifically behavior cloning, that mimics an expert's behaviors in reacher-foreing manner. Formally, let and a be a sequence of predicted actions and the corresponding ground-trute -tuth actions. Similarly, let e and e" be sequence of predicted object classes be interacted with and the corresponding labels. Then each loss of the action and object class prediction is obtained by a cross-entropy loss as below:
T [a interaction] of log ct
Laction (a.a")
Σ at log az, Lelass(c, =
Σ
where T denotes the length of episode that the agent conducts and [ interaction] is an indicator function that activates when an action of is an object interaction action.
In addition, (Ma et al. 2019) adopts progress monitoring. Formally, let p and p" be sequence of predicted progress values and the corresponding ground-truth progress values. Then the progress loss is obtained by mean square error (MSE) loss as below:
T
Eprogress (p.p") T
Σ (pt ..)
(5)
Using them, the agent jointly minimizes the joint loss follows:
v(y.v") = >a Lactioon vo)) + Ac vo.) + XpCorogrees(Upx U2y)
(6)
where indicates the output of a model including an action sequence, class sequence, ye, and a progress value sequence, 3p- for an auxiliary task. ya ve, and y denote the corresponding ground-truth labels. The loss terms are summed by the balancing coefficients Xa. des and Xp.
D.4 IMPLEMENTATION DETAILS
For visual observation, inspired by (Nguyen et al. 2021 Kim et al. 2021: Bhambri et al. 2023), allow the agent perceive surrounding views (in this case, 5 views from the front, left, right, up, and down directions). For language instructions, the agents receive by-peb instructions that describe how accomplish the goal in detail.
18

---

Published as a conference paper at ICLR 2024
Instructions:
Irrelevant Action
Interacted Object
e
BEDROOMS
C
BATHROOMS
Head to the right side of the desk. Take the CD from the desk. Take step to the right. Turn on the lamp.
Finetuning
DER
Left
Abead
Right
Abead
Left
Pickup
Left
CAMA (Ours) (Ours)
Abead
Right
Left
Pickup
Right
ToggleOn
The agent that has already ac- quired knowledge of the environment 6k-1 BEDROOMS proceeds to learn the new environment é& = BATHROOMS. We then assess the agent's capability in the prior environment e 1 determine whether any forgetting occurred. Irrelevant Action denotes action that results incorrect navigation. 'Finctuning' fails find target object, 'CD," eventually leading to failure. DER+ can navigate to and pick up CD, but fails to turn on the lamp. On the contrary, our CAMA can also turn on the lamp and complete the task.
Figure 5: Qualitative analysis of the proposed method (Enviromment-IL).
For the training loss described in Section D.3 we set the balancing coefficients A 1.0, A = 1.0, and >p 1.0 for our CAMA and the baselines. Following (Singh et al. (021). augment visual observations by adopting two strategies: Auto/Augent (Cubuk et al. (019)) and RGB-channe swapping. For computational efficiency, we cache types augmented episodes per episode and choose one of them whenever we augment it.
To update the parameters our CAMA and the baselines, we use the Adam optimizer with an initial learning rate of 0.001 and batch size of 32 per streamed sample. We utilize the ExponentalLR (Li Arora 2019) and ResetL.R (Loshchilov & Hutter 2016) schedulers with y = 0.95 and m T0 for our CAMA and the baselines except CLIB with y 0.9999.
D.5 QUALITATIVE ANALYSIS
We provide a qualitative example our CAMA in the Environment-IL. setup by comparison with the naïve (i.e., Finetuning) and prior best-performing (DER++) methods.
The Environment-Il setup. Figure 5 the agent evaluated for the previous environment, ek-1 BEDROOMS, while learning the current environment, eμ = BATHROOMS. The agent is required to examine CD under the light of lamp. To complete the task, the agent needs 1) pick up CD and 2) turn lamp while holding the CD.
Similarly the Behavior-IL setup, 'Finetuning" cannot find a CD and therefore navigates to other objects irrelevant to the task, which eventually leads to task failure. On the other hand, 'DER++' successfully picks up a CD and reaches the lamp in the close vicinity. However, the agent forgets to turn on the lamp and therefore starts navigate other irrelevant objects, which also leads to task failure. In contrast, our CAMA can pick up the CD and navigate to the lamp as "DE++*' does. Our agent then turns the light examine the held CD and succeeds in the task.
D.6 IMBALANCED SCENARIOS IN THE ELVIRONMENT-II. SETUP
To explore data-in data-imbalance scenario (Liu al. 2022 He et al. 2023 we remove the subsampling process in Sec. 3.2.2 for the Environment-IL II setup and construct imbalanced dataset that name the imbalanced Environment-ll setup. the imbalanced Environment-Il setup, we compare our CAMA with the baselines and summarize the result in Table
We observe that even with such an imbalance, CAMA still outperforms the baselines by noticeable margins, highlighting the efficacy of the proposed approach. In particular, observe significant improvements SRang and GCavg both valid seen and unseen splits, implying that CAMA achieves promising performance in both partial and full task completion.
19

---

Published as a conference paper at ICLR 2024
Imbalanced Euvironment-IL
Model
Valid Seen
Valid Unseem
SR...
GCt.st
SR...
GC..g
SRue
GC
SR.uy
GCany
EWC
27.58 1: 2.03
38.38
2.40
35.44 1: 1.42
45.14 1: 1.58
9.76 1: 0.70
22.94 2 0.61
12.32 1: 1.03
28.06
1.31
38.87
0:23
13:13
1.:3::0.6
0.552
MIR
27.58 1: 2.03
38.58
2.40
35.44
1.42
45.14
1.58
9.78 1 0.70
22.94
1. 0.61
12:32
1: 1.03
28.06
1.31
CLJB
24.70 + 2.20
35.21
+ 2.78
35.32
1.77
44.55
+ 1.78
8.24
+ 1.14
21.96
1.11
10.97
1.50
28.03
1.11
31.54
1.42
42:90
1.61
33.01
1.97
43.89
12:20
27.28
11.90
1.6
X-DER
31.10 +: 1.10
42.83
1.13
32.96
1.70
43.81
+ 1.64
12.03
+ 0.51
27.44
0.97
12.49
+ 0.90
20.07
1.30
CAMA win D.C.
35,40
1.34
46.55
1.40
39.54
1.30
49.27
1.33
14.19
1.22
29.11
1.42
16.02
1.20
33.16
1.25
CAMA (Ours)
32.32
= 1.62
13.72
1.81
38.67
= 2.07
19.09
1.90
14.53
0.40
28.73 = 0.92
17.24
1.78
33.36
1.47
Table 3: Comparison with state-of-the-art methods in the imbalanced Environment-I IL setup. The highest value per metric is in bold. We report the means and standard errors of multiplc runs for random scods.
600
1200
Examine
Cool
Frequency
1000
Frequency
Frequency
400
300
400
200
200
100
20
40
80
40
80
20
40
80
Figure 6: The frequency of objects used for each behavior in the Behavior-IL setup, Each x-axis and y-axis denotes the index of object and the object's frequency appearing in the corresponding behavior. While the hehaviors, HEAT and CooL,, have several shared objects (e.g., apples, tomatoes, etc.) during task completion, the behavior, EXAMINE, rarely have them with HEAT and CooL..
E DISCUSSION
E.1 CONFIDENCE SCORES AS A GOOD INDICATOR OF NEW LOGITS' QUALITY
We use the averaged class-wise confidence score an indicator to estimate how much new logits are informative. This because using the confidence scores for the ground truths, which we use for logit update, allows us to estimate how well the model has learned respective classes.
For example, the model p predicts p(i) the class i well, i.e., it may contain ample information about the class i. Conversely, if the model predicts p(t) = 0, it implies that the model has learned the class poorly. i.e., it may contain little information about the class i (Koh al. 2023
for the class i, it implies that the model has learned
We can use the most single recent confidence score as the 'indicator,' but such a single confidence score could be noisy during training for various reasons such as the degree of augmentation and the difficulty of a sample. To alleviate this issue, we use the N recent confidence scores as an indicator of the quality of the new logits.
E.2 DEPENDENCY PERFORMANCE ON A TASK ORDER IN INCREMENTAL SETUPS
We agree that the performance improvements seem relatively marginal, possibly due to the large standard error of the means. We believe this is because, in an incremental setup in embodied tasks, some tasks may share relatively many action and object classes, while others may share fewer classes. Previously learning such shared action and object classes may help better learning the cur- rent task (i.e., forward transfer) and this implies that model's performance may depend on the order of the tasks (i.e., how much the model learns the shared action and object classes ahead).
For example, while learning cool object, learning some actions (e.g., opening/closing fridge) and object classes (e.g., apples, tomatoes, etc.) may help next learn to heat an object as such actions and object classes also be used for heating (e.g., heat an 'apple,' 'tomato,' etc. by 'open- ing/closing' a microwave). We empirically observe that for the behavior, 'Heat,' our agent achieves 3. 3.70% Valid Unseen SR after learning the behavior, 'Cool,' while it achieves zero Valid Unseen SR after learning the behavior, "Examine," which does have fewer shared object classes as illustrated in Figure implying the dependency of performance on task order.
20


Reviews:
[{"review_id": "EiDKAdSa4O", "reviewer": "Reviewer_oYjZ", "summary": "This paper studies continual learning for instruction-following embodied agents. The authors argue that continual learning is a more realistic setting for embodied AI. To evaluate embodied agents in a continual learning setting, the authors first propose two new continual learning setups for embodied agents:  Behavior Incremental Learning (Behavior-IL) and Environment Incremental Learning (Environment-IL) for new behavior and new environment learning, separately. Then, the authors proposed a new approach for weighting the new logits and old logits. Specifically, a confidence-aware moving average approach which updates logits based on confidence scores of class labels is proposed. The experimental results show that the proposed approach outperforms competitive baselines on the two proposed setups.", "questions": "1. Please elaborate why the confidence scores are good indicators of the ‘quality’ of the new logits. Giving simple examples that demonstrate the effectiveness could be very helpful.   \n\n2. This paper uses imitation learning to train the agents. Is the proposed approach extendable to a reinforcement learning setting?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality and Significance:    \nThe paper is well-motivated. Continual learning for embodied agents is a more realistic yet less explored area in our community. In addition, the two new continual learning setup seems reasonable and could be valuable for evaluating continual learning agents. \n\n\nQuality:   \nThe paper is technically sound. Most of the claims are supported by experimental results. \n\n\nClarity:    \nThe paper is generally well-organized and implementation details are presented. However, some technical details may need further clarification. Please see the Weakness section for more details.", "weaknesses": "1. While the empirical results seem promising, the reviewer is not fully convinced by the proposed confident-aware moving average approach. Specifically, it is unclear why the average of most recent N confidence scores could be a good indicator of the ‘quality’ of the newly obtained logits. More concretely, the behavior $t_i$ of the current episode is different from the behavior $t_j$ that is associated with the sampled past experience $x’$. Why would confidence scores of current episodes with behavior $t_i$ provide information on a different behavior $t_j$.   \n\n\n2. In equation (1), subscript $a, c$ are missing in $\\gamma_a$ and $\\gamma_c$", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bxUcScYFLT", "reviewer": "Reviewer_UTbC", "summary": "This work looks at the continuous learning problem for embodied agents that follow verbal task instructions. The outlined challenge is very relevant in the community as such agents need to constantly adapt to new tasks and environment challenges. In this work, the authors address Behavior-IL and Environment-IL through a newly proposed method, Confidence-Aware Moving Average, that updates an experience replay of stored logits based on model confidence. The proposed method is evaluated over a set of tasks and behaviors in the ALFRED simulation environment against state-of-the-art baselines.", "questions": "1) Is there a concern about over/under-confidence with such an approach? Presumably, such a model is not calibrated since the model is continually out-of-distribution. Given that modern deep neural networks are particularly susceptible to confidence issues, an analysis on this aspect would be useful.\n2) Does performance depend on the order of the tasks/environments? I see reference to random ordered sequences in Sec. B.3, but it's not clear to me how this ties into the training/evaluation methodology.\n3) Is there any insight on how the size of the memory buffer relates to the number of learning tasks? Is 500 an arbitrary number or was this empirically chosen?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The proposed work addresses a relevant challenge for the deployment of embodied agents in challenging home environments, particularly addressing the problem of adapting previously learned policies to new environments and tasks.\n* The idea of using confidence to update the stored logits is interesting, and potentially quite powerful (if there is any way to know that the confidence is reasonable and/or calibrated, see below).", "weaknesses": "* I think some of the assumptions this paper makes are not realistic. E.g., the paper indicates that this approach is task-free and that task-identifiers are not available, but then appears to assume that the data is balanced. In a real-world continual learning setup there is no guarantee that the tasks/data the agent encounters will be balanced, and so if data balance is enforced via sub-sampling I think this is implicitly leaking information regarding task identifiers (as they would be needed to balance an unbalanced dataset in practice). An analysis on how class/environment/behavior imbalance affects the model would be beneficial here.\n* The performance improvements seem relatively marginal for many of the evaluation metrics. Were any statistical significance results run over these values?\n* The training methodology is not entirely clear to me, and I think some of it should be pulled into the main paper as it reduces the clarity of the empirical results. Are all the models trained sequentially over a set of tasks/environments and then evaluated over the same set (in the seen case)? Do the seen/unseen evaluations use the same underlying trained model?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HyGMXgQB7q", "reviewer": "Reviewer_z9k5", "summary": "The paper studies the online continual learning problem without task boundary information in an imitation learning context. It adapts a distillation based approach that was developed for the setting with task boundaries. Prior methods store past model logits in memory and update these with a weighting scheme to prevent these logits of becoming outdated using task boundary information. Here the weights are determined dynamically via the confidence of the past N predictions of the model, not relying on task boundary information. The method shows improved task performance compared to a variety of continual learning baselines in a simulated household environment. The evaluation is split into a setting where the agents has to continually learn new behaviors and a setting where the agent has to continually adapt to new environments.", "questions": "- The performance of the \"finetune\" baseline with respect to the $SR_{last}$ and $GC_{last}$ metrics. If I understand the baseline correctly is that the currently trained policy is finetuned on the data of the next task without any memory. Should the performance on the final task not be . Why is the performance on the . \n\n- Which episodes are stored in the memory M? Is it simply a queue? If yes then given that its capacity is such a small percentage of the total number of episodes (as said in the text that is the standard) and given the number of tasks for each of the evaluation episodes will most episodes not be about the same same task such that the logits there should be replaced with the current logits of the model?\n\n- In the CAMA w/o D.C. baseline, was the fixed coefficient tuned for optimal performance of the baseline?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The way the method and experimental setup is presented, it can be easily followed and understood. \n\n- The method is tested against a variety of baselines representing different approaches to the continual learning problem.\n\n- The method outperforms all baselines.\n\n- Although the novelty is on the smaller side it becomes apparent how the work relates to prior approaches and what limitation is tackled. It is also shown that tackling exactly that limitation leads to better performance.", "weaknesses": "- Depending on the availability of compute it would be beneficial to increase the number of seeds (3->5) to reduce the standard errors of the results. At the moment the standard errors are often larger than the gaps in mean performance. \n\n- Effect of the hyperparameter N (number of past confidences averaged over). I think it would be interesting and important to see how sensitive the method is to that hyperparameter. If it works only well for a specific N, a lot of hyperparameter tuning will be needed to find that N for a given environment. If performance is robust with respect to N it would make the method more easily applicable. \n\n- The evaluation is split between environment changes and behavioral changes. I understand the point of separating these two factors, but why was the crossing of the factors not studied? What happens if two factors change at the same time?\n\n- As shown in Figure 3, when a new tasks arrives confidence drops. However, there is also evidence that in OOD cases a model can be confidently wrong (Do Deep Generative Models Know What They Don't Know?, Nalisnick et. al.). How detrimental would that be for performance?\n\nFormalities:\n\n- Abstract, last sentence: \"outperforms prior arts\", was that supposed to be \"outperforms prior state of the art\" ?  \n- Figure 4: In the second paragraph of 5.3 that references the figure 4, \"HEAT\" is claimed to be the newly learned task and Pick2&Place the old one, while in the caption of the figure it is the other way around.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Xd1zqYMD2t", "reviewer": "Reviewer_q27B", "summary": "This paper first introduced two new settings to coninual learning embodied AI: behavior incremental learning and environmental incremental learning. The paper then introduces a method to tackle both, in a task boundariless setting, called Confidence-Aware Moving Average. The method is employed in Alfworld, and the task at each step is to predict action and object class.", "questions": "- equation (1), the $\\gamma$  in front of $z$ is it missing the subscript $a$ and $c$? or is it a fixed value?\n- for equation (2), how are the confidence scores calculated, mathematically\n- I am not too familiar with the benchmark, but when is the object class used when making decisions? if you need to heat a mug, does your agent also need to recognize the object in front of it is a mug too?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The continual learning setting is a relevant topic to be explored in the embodied AI field\n- the writing is clear and the method is well motivated and easy to understand\n- the number of baselines compared is very comprehensive\n- the method is simple yet effective", "weaknesses": "- conceptually, the task boundariless scenario in embodied instruction following AI is very limited, since the user will know when to switch to a new task\n- introduction to a new hyperparameter for each of the prediction ($\\alpha$'s)\n- another baseline I could think of is to use the loss as the weight to dynamically change the old logits. What are some intuition on why confidence is better than loss to use to dynamically weight?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)