CAREL: Instruction-guided reinforcement learning with cross-modal auxiliary objectives
Authors: Armin Saghafian
Amirmohammad Izadi
Negin Hashemi Dijujin
Mahdieh Soleymani Baghshah
Venue: Accepted by TMLR ()

Abstract: Grounding the instruction in the environment is a key step in solving language-guided goal-reaching reinforcement learning problems. In automated reinforcement learning, a key concern is to enhance the model's ability to generalize across various tasks and environments. In goal-reaching scenarios, the agent must comprehend the different parts of the instructions within the environmental context in order to complete the overall task successfully. In this work, we propose \textbf{CAREL} (\textit{\textbf{C}ross-modal \textbf{A}uxiliary \textbf{RE}inforcement \textbf{L}earning}) as a new framework to solve this problem using auxiliary loss functions inspired by video-text retrieval literature and a novel method called instruction tracking, which automatically keeps track of progress in an environment. The results of our experiments suggest superior sample efficiency and systematic generalization for this framework in multi-modal reinforcement learning problems.

Published in Transactions on Machine Learning Research (08/2025)
CAREL: Instruction-guided reinforcement learning with cross-modal auxiliary objectives
Armin Saghafian, Sharif University of Technology
orminn.eghhafinnnggmlll.om
Amirmohammad Izadi, Sharif University of Technology
amirmahammaad izudi0 iecaaddii@@@shriir @sharif, edu
Negin Hashemi Dijujin, , Sharif Universily of Technology
n.hashemi99 @sharif. edua
Mahdieh Soleymani Baghshah, Sharif University of Technology
soleymanid shorif. edu
Reviewed on OpenReview: https: /ttp::///penreeveuu net/ per/form?? id= =JUEYr5X1X
Abstract
Grounding the instruction in the environment is a key step in solving language-guided goal-reaching forcemettmntt learning problems. In automated reinforcement learning, a key concern is to enhance the model's ability to generalize across various tasks and environments. In goal-reaching scenarios, the agent must comprehend the different parts the instructions within the environmental context order to complete the overall task successfully. In this work, we propose CAREL (Cross-modal Auriliary REinforcement Learning) as new framework solve this problem using auxiliary loss functions inspired by video-text retrieval literature and novel method called instruction tracking, which automatically keeps track of progress in an environment. The results of our experiments suggest superior sample efficiency and systematic generalization for this framework in multi-modal reinforcement learning problems. Our code base is available here
1 Introduction
Numerous studies have examined the use of language goals or instructions within the context of reinforcement
learning (RL) Röder et al. (2021); Geffner (2022): Luketina et al. (2019). Language goals typically provide a higher-level and more abstract reprosentation than goals derived from the state space Rolf Asada
(2014h. While state-based goals often specify the agent's final expected goal representation Liu al. (2022): Eysenbach et al. (2022), language goals offer more information about the desired sequence of actions and the necossary subtasks Liu et al. (2022). Therefore, it is important to develop approaches that extract concise information from states or observations and effectively align it with textual information, process referred to as grounding Röder et al. (2021)
Previous research has attempted to ground instructions in observations or states using methods such as reward shaping Goyal et al. (2019) Mirchandani et al. (2021) or goal-conditioned policy/valit functions Zhong et al. (2019); Hejna III
2021 Akakzia et al. (202 Deng et al. (2020), with the latter being a key focus of many studies. Their approaches incorporate various architectural or algorithmic inductive biases, such as pross-attention Hanjie et al. (2021), hierarchical policies Jiang et al. (2019) Andreas et al. (2017), and
et al. (2018). Typically, these works involve feeding instructions and obscrvations into policy or value networks, extracting internal reprosentations of tokens and observations at each time step, and propagating them through the network. Previous studies have explored auxiliary loss functions to improve these internal representations in RL (Stooke et al. (2021): Wang et al. (2023): Zheng et al. (2023). and have emphasized the importance of self-supervised) wwwpaad/uuuuuusisseee learning objectives Levine (2022) RL. However, these loss functions lack the alignment property between different input modalities, such as visual/symbol states and text pewaaanddddsseepptiionn Recent studies
feature-wise modulation Madan al. [2221] Chevalier-

---

Published in Transactions on Machine Learning Research (08/2025)
have suggested contrastive loss functions align text and vision modalitics in unsupervised manner Ma et al.] (2022) Yao et al. (2021) ) Radford et al. (2021) ys Yu et. al. (2022); Li et al. (2022). Most of these studies fall under the rideo-text retrieval literature Zhu et al. (2023); Ma et al. 20222) where the language tokens and video frames align at different gramularities. Since these methods require a corresponding textual input along with the video, the idea has not yet been employed in language-informed reinforcement learning, where the sequence of observation might not always match the textual modality (due to action failures or inefficacy of trials). One can leverage the success signal or reward to detect the successful episodes and consider them aligned to the textual modality containing instructions or environment descriptions. Doing so, the application of the above mentioned auxiliary loss functions makes sense.
Contrastive loss serves as a fundamental mechanism in representation learning, particularly in scenarios where distinguishing between similar and dissimilar objects is essential. As outlined in Radford et al. (2021) this loss function operates by minimizing the distance between corresponding items two modalities, such as sequences of observations and their corresponding instructions, while simultancously maximizing the distance between unrelated or dissimilar items. This loss enhances represenlations via utilizing the weak supervision that is available pairs of matched observation sequence and instruction from the successful trajectorics and unmatched pairs from unsuccossful ones.
In this study, we propose a new framework, called CAREL (Cross-modal A Auziliary R Eiiforremenn Learning), for the adoption of auxiliary grounding objectives from the video-text retrieval literature Zhu et al. (2023), particularly X-CLIP Ma et al. (2022), to enhance the learned representations within these networks and improve cross-modal grounding at different granularities. By leveraging this grounding objective, we aim to improve the grounding between language instructions and observed states by transferring the multi- grained aligument property of video-text retrieval methods to inttruction-follownng agents. We also propose novel method to mask the accomplished parts of the instruction via the auxiliary score signals calculated for the cross-modal loss while the episode progresses. This helps the agent to focus on the remaining parts of the task without repeating proviously done sub-tasks or being distracted by past goal-relovant entities in the instruction. Our experiments on the MiniGrid and Baby AI enviromments Chevalier- Boisvert et al.) (2018) showcase the idea's effectiveness in improving the systematic generalization and sample efficiency of instruction-folloinng following agents. The primary contributions of our work are outlined as follows:
We designed an auxiliary loss function to improve cross-modal alignment between language instruc- tions and environmental observations.
We introduced novel instruction tracking mechanism to help the agent focus on the remaining tasks by preventing the repetition of completed sub-tasks.
We enhanced overall performance and sample efficiency in two benchmarks.
2 Methods
Tanguage--nformedd Reinforcement Learning (LIRL. builds upon traditional RL by integrating natural lan- guage instruction as a structured input to enrich the learning process. In our proposed method, we aim to leverage the instruction more effectively, going beyond merely conditioning the model on it. We introduce a novel loss function that enhances representation learning by incorporating a contrastive loss between the obscrvation and instruction modalitics, thereby achicving more meaningful and grounded represcntations for observations. The enhanced representation not only streamlines the reinforcement learning process but also enables tracking of the instruction and dynamic conditioning of the model on the remaining components of the instruction throughout the trajectories.
In this study, we first incorporate an auxiliary loss inspired by the X-CLIP model Ma el al. 2022 to enhance the grounding between instruction and observations in instruction- followiwnnn RL agents. This auxiliary loss serves as a supplementary objective, augmenting the primary RL task with a multi-grained alignment prop- erty, which introduces an additional learning signal to guide the model's learning process. This design choice was motivated by the ability of contrastive loss to effectively create meaningful embeddings through aligning
2

---

Published in Transactions on Machine Learning Research (08/2025)
Policy/Value Heads
Auzillary Loss
Lang. Global Aggregator
E-1 Score
Memory
Value
Put the rnd bail next to the purple bex.
Instruction Tracking via {8
A0SM
Oba. Globel Aggregator
O-f Value Vec.
O-1 Score
Obs. Encoder
O-W Value Vec.
Figure 1: Overall view of CAREL. In this figure, we showcase CAREL over a candidate baseline model from -oisverr- al. 2018)) (Ieft) The blue box handles the (masked) instruction and its lo- cal/global representations, while the pink box contains the components related to observation. (Righh) The purple box shows the calculation steps for the X-CLIP loss and tracks scores for instruction masking.
observations with the intended instruction, ultimately enhancing the overall performance of the RI. system. Additionally, we leverage the alignment scores calculated within the X-CLIP loss to track the accomplished sub-tasks and mask their information from the instruction. This masking aims to filter out the distractor parts of the instruction and focus the remaining parts, improving the overall sample efficiency of the agents. We call this technique instruction tracking. Both the proposed auxiliary loss and the instruction tracking ability can act as frameworks and don't alter the structure of the base model. In the remainder of this section, we explain the auxiliary loss and the instroction tracking separately.
2.1 Auxiliary Loss
The auxiliary loss in CAREL is applied only to successful trajectories to ensure that the loss signals align with goal-relevant behavior. During training, a batch of trajectorics is collected, cach containing observations, actions, rewards, and corresponding instructions. The aggregated rewards for each trajectory are calculated determine success. fraction of the maximum achievable reward used as the acceptance threshold, ensuring that only sufficiently high-performing trajectories are classified successful.
This separation is done only for the auxiliary loss, and the base model's RL loop is run over all interactions, whether successful or unsuccessful. Hence, it differs from offline RI in which only certain episodes are selected for the whole training process Levine et al. (2020).
Each successful episode contains a sequence observation-action pairs ep = ([O2, a.]..., [ol....,on, an)) meeting the instructed criteria and an accompanying instruction instr (Iı, Jm.) with m tokens. Since the X-CLIP loss requires local and global encoders for each modality, we must choose such representations from the model or incorporate additional modules extract them. To explore the exclusive impact of the auxiliary loss and minimize any changes to the architecture, which are crucial components of the model itsolf. We utilize these encoders to extract local representations for each
the model's existing observation and instruction encoders,
[O,, ar] denoted as H, E [Rdx1 / -= 1,..., n in which each action is embedded similar lo positional embedding in Transformers Vaswani et al. (2017) and is added to the observation representation. Each instruction token I, is encoded as v; E बे
I,...,m. m. The global representations can be chosen from the model itself or added to the model by aggregation techniques such mcan-pooling or attention. We denote the global representations for observations and the instruction by &. and i, respectively. The
3

---

Published in Transactions on Machine Learning Research (08/2025)
auxiliary loss function is then calculated according to Ma et al. 2022 as below. We restate the formulas in our context to make this paper self-contained.
To utilize contrastive loss, we first need to calculate the similarity score for each episode (ep), a sequence of observations, and an instruction (instr) pair denoted as s(ep, instr). To do this, we calculate four separate values; pppisdde-nnsruction (Sg-1), as well as Episode-Word (Sg-w), DDservation-Instuucee Observation-Wel -orrrttooo-Wodddd Word (So-w) similarity values. SpisoodeInssructionn score can be calculated using this formula:
(So-r) and
Sg-1 = ,"6
(1)
with ā, 8 E Rddx1 SE-F E IR. Other values are calculated similarly:
Sg-w = (V+)*,
(2)
- Xū,
(3)
So-w XVT.
(4)
where X = ; E Rad is local representation for observations, V = i...i,mm E Ruuxd in local reprosentations for instruction tokens, and Sg-w E I So-1 E Raxl and So-w E R2 provides aro-granular similarities between the language instruction and the episode of observation. These values are then aggregated with appropriate attention weights via a technique called Attention Over Similarity Matrix (AOSM). Episode-Word (w) and Observation-Instruciii (8)-) scores are calculated from the values follows:
So l1, [
(5)
gg-w [
(6)
where:
exp(xl.]/T) Soflmar(x(l) = E, exp(x<il/7)*
(7)
in which, 7 controls the softmax lemperalur ure. For the Observation- Word score, bi-level atlention is per- formed, resulting two fine-gra the-grained similarity vectors. These vectors are then converted to scores similar to the previous part:
Snstt [i, 1] Softmac(SS-wwlt.>) So w [i,b,
i (1,..,),
(8)
So [1, So/lmar(So-w(..)) T So w(. i m),
(9)
where Systtr E _gxx show the similarity value between the instruction and n observations in the episode and S% E R xm represents the similarity value between the episode and m words in the instruction.
The second attention operation performed on these vectors calculate the Observation-Won Word similarity score (S-w))
Softmaz Soww = ( (fffaa/ss............oooooooo..d 1)) S I 1))
(10)
The final similarity score between episode and an instruction is computed using the previously calculated scores:
sep,, instr) = (Sg-1 + S + + S/w)
(11)
This method takes into consideration both fine-grained and coarso-grained contrasts. Considering N episode- instruction pairs in batch of successful trials, the auxiliary loss is calculated as below:
4

---

Published in Transactions on Machine Learning Research (08/2025)
Louz 1
exp(s(eps; instr:))
log exp(s(epp, (nstr:)))
(log
(12)
Ej=1 Ej= exp(s(ept, instr,))
Σj=1 exp(s(epj, instr.))
The lotal objective is calculated by adding this loss to the primary RL loss, LRz. with a coellicient of Ac.
Liotal LRL. + Ac.Laua
(13)
The overall architecture of a base model Chevalier- Boisverr et al. (2018) and the calculation of the auxiliary loss are depicted in Figure If the shape of the output representations from the observation and instruction encoders does not align, we employ linear transformation layers to bring them into the same feature space.
This transformation is crucial it facilitates the calculation of similarity between these representations within our loss function.
Algorithm CAREL framework
Initialize baseline (a s, and env
2: Get instruction 7 from the environment
3: for each batch of rollouts do
4: for each rollout in batch do while episode not done do
ag ~ To st, Execute at, and Recieve rt, 8141 Baseline stores (st, I, a,, 8041) for it's own loss
St <- 8t+1
10:
if episode terminated and successful then
11:
Save the whole episodes: Baucress f- epi
12:
end if end while
13:
14:
end for
15: Update 0 based baseline's RL loss
16:
Update 0 using Bsuccess to compute and apply the auxiliary loss (Caur) based on Eq.
12
17:
end for
2.2
Instruction Tracking
We can consider the similarities from eqs. [ to as measure of matching between the instruntion and the episode different granularities. Once calculated at each time step of the episode, this matching can signal the agent about the status of the sub-task accomplishments. The agent can then be guided toward the residual goal by masking those sub-tasks from the instruction. More precisely, at time step of the current episode, the agent has seen partial episode ep(*) = ([O1, [O., a:)) that in fairly trained model should align with initial stages of the instruction. The instruction itself can be parsed into a set of related sub-tasks C = [c} via mnlebbaed heuristics, and there can be constraints on their interrelations. However, if prior knowledge of the instruction is unavailable, we can leverage Natural Language Processing tools such as SpaCy or Language Models to carry out this process. For example, an instruction of the form "Do X, then do Y, then do Z includes three sub-tasks (X X-Y-- Z). Other examples could involve different forms of directed graphs where a specific sub-tard task is acceptable only if its parents have been satisfied before during the episode. The set of acceptable sub-tasks at time step is denoted by Cr, which contains the root nodes in the dependency graphs at the start of the episode.
Y, and Z which have a sequential order constraint
In order to track the accomplished sub-task, we assess the similarity between C members and the partial episode. This can be done by tracking Sg-w or So-w, which provides fine- ghe-ggr similarities across the language modality. In the case of Sg-w, w, the similarity per token in is averaged to get. final scalar
5

---

Published in Transactions on Machine Learning Research (08/2025)
similarity. For So-w, the maximum similarity between the observations and each word is considered for averaging across tokens. Another option is to calculate learned representation for the whole instead of averaging and to track the instructions based on their similarity with the partial episode, aiming at preserving the contextual information in the representation
the sub-task. The final calculated similarity of each acceptaale sub-task c; denoted by S is tracked al each time step. Once this similarity rises significantly, the matching detected, and e removed from the instruction participating in the language-conditione model. More precisely, remove e from the instruction when the following condition is satisfied:
(a E C A (S, k
S Σ
(14)
Here, h > is a hyperparameter that specifies the significance of the matching score's spike. While the auxiliary loss described in the previous subsection is applied on the episode level, instruction tracking happens at every time step the episode over the partial episode and the masked instruction. This process is represented in Figure
These two techniques can be applied jointly, as the auxiliary loss improves the similarity scores through time, and the improved similarities enhance the instruction tracking. To prevent false positives during tracking the initial epochs of training, one constrain the probability of masking and relax this constraint gradually as the learning progresses.
Algorithm 2 Instruction Tracking (IT) framework
1: Input: Environment env and instruction 1
2: Split instruction / into sub-tasks by rule-based heuristics (C = {GH)
3: for in max sleps do
4: Or = env.step()
Compute cach sub-task's (c:) similarity with this step's obscrvation (o:)
St mean((a..j)
7: Keep moving average of So; over time in S.
if St, Se, x k then
with a probability y:
10:
I + Omit detected sub-task (c:) from the instruction
11:
+ Omit detected sub-task (c)) from the C
12:
end if
13: Give the new 1 to the policy
14: end for
2.2.1 Instruction Tracking Implementation Details
Splitting into subtasks
To implement Instruction Tracking, the subtasks need to be extracted from the initial instruction. In our case, the environments provide a clear and consistent instruction format, which enables us to split sentences into individual words by using string matching to identify conjunctions for subtask separation. We tokenize the instructions and determine the positions of conjunctions. This process produces a list of subtasks, each paired with its corresponding conjunction.
For example, the "GoToSeq" environment uses the following structure:
"go to a/thc fcolor} (type}" + "and go to a/the {color} {type}" + ", then go to a/the {color) (type}" + "and go to a/thn the {color} (type)"
Given this format, we tokenize the sentence and match the words "and" and "then" to identify and separate the subtasks.
Example:

---

Published in Transactions on Machine Learning Research (08/2025)
"Go to the red box and go green ball, then go to the blue key."
We use the conjunctions "and" and "then" to split the instruction into the following subtasks:
to the red box" and "go to a green ball" then "go to the blue key."
Masking Process
To exclude a completed subtask, we mask both the subtask tokens and their associated conjunction by replacing these tokens with <mask> token when reconstructing the instruction. This idea, inspired by soquential thinking, helps the model avoid repeating tasks by letting it focus on the remaining tasks when portion of the tokens is masked.
If the condition in 14 is met for any of the subtasks for example, the first one we mask that subtask with a probability. The final instruction will then look like this:
"<mask> <mask>
<mask>
<mask>
<mask> <mask> go to green ball, then go to the blue key."
This masked instruction is then passed to the model.
To extend this for other environments without a clear format, NLP tools such as SpaCy or Large Language Models like GPT-4o can be used to break the instructions down to subtasks and subsequently to mask the completed ones.
Masking Probability.
When the threshold in Equation 14 is reached, we only apply the masking with certain probability. This number starts very low in the initial stages of training and rises over time following tanh function. We implement this to avoid falsely masking uncompleted subtasks at the early stages of training, when the encoders haven't been properly trained and don't 't produce strong representations.
The exact function for the probability as follows:
prob = tanh (current frame
max (max__ frames
where max frames determines the total number of training steps.
3 Experiments
In our experiments, we conducted a comparative analysis to assess the impact of X-CLIP Ma el al. (2022)
auxiliary loss on generalization and sample efficiency of instructionnffolloiing agents. We showcase the success CAREL along with the instruction tracking technique in our experiments[i For this purpose, al. (2018) (the proposed model along with the
we employ a bascline called Baby CChevalier-Boisver Baby benchmark), for which we explain the experimental setup and results in the following paragraphs.
Additionally, to compare results with recent works, consider SHELM Paischer et al. 20023 which
uses CLIP embedding to detect objects in the observation, and LISA Garg et al. 2022 imitation learning for a sample efficient training process.
which leverages
3.1 CAREL Results
We employ the Baby AI environment. Chevalier-Boisver et al. (0018) lightweight, but logically complex benchmark with procedurally generated difficulty levels, which enables in-depth exploration of grounded language learning in the goal-conditioned RL context. We use BabyAl's bascline model as the base model and minimally modify its current structure. Word-level representations are calculated using a simple token
For the experiments reported this paper, have used one NVIDIA 3090 GPU and one TITAN RTX GPU over two weeks.

---

Published in Transactions on Machine Learning Research (08/2025)
10
G:Rddegg5 12
OpenDeondrdertii
Picsppo:
Addul cares (enw)
Srgui
2.03
0.35
2.55
0.25
0.50
L05 Pyymo
(c)
NVNeettocaiS0N
Synth55R2
3.0
sakyyi EXbet (nvces
Kidas <548: Oreeng
0.0
Rakkyd
9.50
Tomes
(d)
(e)
Figure 2: Test SRs indicating the overall effect of Vanilla CAREL on BabyAL. (The results are smoothed before plotting.)
embedding layer. Then, GRU encoder calculates the global instruction representation. Similarly, we use the model's default observation encoder, convolutional neural network with three two-dimensional convolution layers. All observations pass through this encoder to calculate local representations. Mean- pooling/ Attention over these local representations is applied as the aggregation method to calculate the global observation representation.
The evaluation framework for this work is based on systematic generalization to assess the language grounding property the model. We report the agent's success rate (SR) over a set unseen tasks at cach BabyAI level, separated by pairs of color and type of target objects or specific orders of objects in the instruction. This metric is recorded during validation checkpoints throughout training.
Figure 2 iillsstrates the improved sample efficiency brought about by CAREL auxiliary loss (without instrue- tion tracking and action embedding to minimize the modifications to the baseline model, hence called Vanilla CAREL). All results are reported over two random seeds. The results indicate improved sample efficiency of CAREL methods across all levels, especially those with step-by-step solutions that require the alignment be- tween the instruction parts and episode interactions more explicitly, namely GoToSeq and OpenDoorsOrdes which contain a sequence of Open/GoTo subtasks described in the instruction. The generalization signiti- cantly improved in more complex tasks, i.c., Synth.
3.1.1 Sample Efficiency
Figures
2 and 3 present a comparison between the sample efficiency of our method and other baselines.
addition to this, we have conducted a quantitative study. Table ! highlight the superiority of our method's sample efficiency,
we achieve better performance with the same number of samples. The reported values roughly represent the halfway point of the samples required for model convergence, except for the Synth environment, which did not converge due to its complexity.
3.1.2 CAREL + Instruction Tracking
For instruction tracking, we use only the Sg-w vector and average over tokens each sub-task to track the score over time. To detect sub-task matching from the score signal, we set à 2 Equation [4] All the other settings are kept the same as in vanilla CAREL, except that we also add action embeddings to local observation representations, described in the Auxiliary Loss section. We mask acceptable sub-tasks with
8

---

Published in Transactions on Machine Learning Research (08/2025)
Method
Task
Samples
Success Rate
Babyai
10M
41%
GoToSegS5R2
Babyai CAREL
10M
73%
Babyai
3M
24%
OpenDoorsOrderN4
Babyai CAREL
3M
79%
Babyai
5M
35%
PickupLoc
Babyai + CAREL
5M
53%
Babyai
7.5M
34%
PutNextLocalS0N4
Babyai CAREL
7.5M
63%
Babyai
20M
20%
SynthS5R2
Babyai CAREL
20M
28%
Table Impact of CAREL on sample efficiency.
certain possibility that follows a hyperbolic tangent function in terms of training steps. This is meant minimize the amount of masking at the start of the learning process when the model has not yet learned good embedding for instructions and observations, and increase it over time.
Go ToSegS5R2
OpenDoorsOrderrN4
1.0
1.0
Babyal e CAREL
Babyal CAREL
Bebyni CAREL
Babyai CAREL
0.0
Babyai
0.8
Success Rate (SR)
0.8
0.6
Sucress
0.4
0.4
0.2
0.0
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2,00
0
6
Frames
Frames
(a)
(b)
Figure
Test SRs indicating the effect of CAREL + iinstruction tracking on the baseline models. (The results smoothed before plotting.)
The results of the full CAREL method (with instruction tracking and action embedding) are reported on the GoToSeq and DpenDoorsOrder environments. We break down the instructions in this environment. with a rule- based parsing to increase the level of detail in the instructions. The instruction, stated initially as "open the [color1] door, then open the [color2] door", converted to "go to the [colori] door, then open the [colori] door, then go [color2] door, then open the [color2] door' and This introduces the challenge of sequential sub-tasks into Baby tasks. The results in Figure B indicate that instruction tracking improves CAREL. This improvement is more apparent in the case of environments with multistep, complex tasks that include more intermediate sub-tasks, such as GoToSeq.
3.2 Comparison with Recent Models
To compare our model against recent baselines, we apply and evaluate our models using two baselines, SHELM Paischer et al. (2023) and LISA, Garg al. (2022). We evaluate the effectiveness our framework by applying it on top of SHELM while comparing the performance of LISA with the BabyAI + CAREL model.

---

Published in Transactions on Machine Learning Research (08/2025)
To evaluate the capability of our framework on RGB environments, apply and test it SHELM Paischer et al. (003)). SHELM leverages the knowledge hidden in pre-trained models such CLIP and Transformer- XL. It also uses CLIP to extract textual tokens related to every observation. Then, these tokens are passed through the frozen Transformer-XI network to form a memory tokens throughout the episode. This hidden memory then concatenated to a CNN representation of observation and passed to actor/critic heads. We must modify SHELM's structure as doesn't use the environment's instructions, which are crucial success in multi-goal setting. To do so, we utilize BERT's tokenizer to embed the instructions and pass them through Multihead-Attentinn layer with four heads. The resulting embedding is concatenated to the hidden layer alongside the outputs of the CNN model and Transformer-XT, which are then passed to the actor-critic head. We consider the encoder output for observations as the local representations and add another Multi-head Atlention layer followed by a mean-pooling over them to calculate the corresponding global representations. Table 2 shows the sample efficiency of our approach by reporting the success rates for each model. The number of frames chosen as the midpoint of convergence. Additionally, Figure demonstrates faster convergence of our framework compared to the SHELM baseline.
We conducted further experiments to evaluate model against imitation learning, which is known for its fast training and sample efficiency. Table 3 details the comparison in samples and success rate between our models. We present the results for the LISA and BabyAI + CAREL models on the mumber of frames required for their convergence. Additionally, we provide the results for BabyAI under the same number of frames as BabyAl + CAREL to demonstrate our sample efficiency. The results for LISA represent the highest performance that this model can achieve whereas our model achieves significantly higher performance upon convergence.
Please refer to the Appendix for more results on time consumption A. coefficient tuming, and ablation studies on Instruction Tracking A.2
Nirdo Goltobbjore-2---
ViriCrid Weiiciis-ootooerrde 6a6-M2-0
SHEUP CARE.
CA-
2034
S1EU
03
0.2
(a)
(b)
(c)
Figure 4: Sample efficiency comparison after applying the CAREL framework to the SHELM baseline. (The results are smoothed before plotting.)
Model
Task
Frames
Success Rate
SHELM
5M
20% + 3%
PutNear
SHELM + CAREL
5M
27% + 2%
SHELM
1.5M
50% + 3%
GoToDoor
SHELM + CAREL
1.5M
83% + 1%
SHELM
1.5M
77% + 1%
GoToObject
SHELM + CAREL
1.5M
78% 2%
Table 2: Sample efficiency and performance comparison of CAREL on SHELM baseline.
10

---

Published in Transactions on Machine Learning Research (08/2025)
Model
Task
Frames
Success Rate
AT
14M
76% 3%
LISA
GoToSegS5R2
7M (100k trajectories)
77% = 2%
Baby AI + CAREL
14M
93% + 2%
BabyAl
4M
62% + 5%
LISA
OponDDorsOAderr4
1.5M (100k trajectories)
60% 3%
BabyAI + CAREL
4M
97% 2%
Table 3: Sample efficiency and performance comparison with LISA. (LISA converges on 100k trajectories, and an increase in the number of frames will not improve performance.)
4 Limitations
CAREL framework is designed to improve the model's understanding of common concepts in observation and instruction via representation learning. As a result, it can only be applied to environments that provide clear textual instruction to the agent. The framework does not impose any constraint on the environment or the MDP beyond providing instructions.
The primary requirement for applying CAREL is that the base model must generate embeddings for visual observations and textual instructions via separate encoders (e.g. dual-encoder models) or another mechanism.
5 Conclusion
This paper proposes the CAREL framework which adopts auxiliary cross-modal contrastive loss functions to the multi-modal RL setting, especially instruction-folloinng agents. The aim is to improve the multi-grained alignment between different modalities, leading to superior grounding in the context of learning agents. We apply this method to existing instruction-folooiing agents. The results indicate the sample efficiency and generalization boost from the proposed framework. As for the future directions of this study, we suggest further experiments on more complex environments and other multi-modal sequential decision-making agents. Also, the instruction tracking idea seems be promising direction for further investigation.
References
Ahmed Akakzia, Cédric Colas, Pierre-Yve Oudeyer, Mohamed Chetouani, and Olivier Sigaud. Grounding language to autonomoosly--cqquree skills via goal generation. arXiv preprint arXic:2006. 07185, 2020.
Jacob Andreas, Dan Klein, and Sergey Levine. Modular multitask reinforcement learning with policy sketches. In International conference on machine learning, pp. 166 175. PMLR, 2017.
Maxime Chevalier-Boisvert Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Hun Nguyen, and Yoshua Bengio. Babyai: A platform to study the sample efficiency of grounded language learning. ar preprint ar 1a::1800.08200 0.08272, 2018.
Zhiwei Deng, Karthik Narasimhan, and Olga Russakovsky. Evolving graphical planner: Contextual global planning for vision-and-language navigation. Advances in Newral Information Processing Systems, 33: 20660 20672, 2020.
Benjamin Eysenbach, Tianjun Zhang, Sergey Levine, and Russ R Salakhutdinov. Contrastive learning goal-conditioned reinforcement learning. Advances in Neural Information Processing Systems, 35:35603 35620, 2022.
Divyansh Garg, Skanda Vaidyanath, Kuno Kim, Jiaming Song, and Stefano Ermon. Lisa: Learning inter- pretable skill abstractions from language, 2022. URL htttp::/aarriv.orr/aaa/////////a
11

---

Published in Transactions on Machine Learning Research (08/2025)
Hector Geffner. Target languages (vs. inductive biases) for learning to act and plan. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 36, pp. 12326-12334 2022.
Prasoon Goyal, Scott Niekum, and Raymond J Mooney. Using natural language for reward shaping in reinforcement learning. arXiv preprint arXiv:1903 02020, 2019.
Austin W Hanjie, Victor Y Zhong, and Karthik Narasimhan. Grounding language to entities and dynamics for generalization in reinforcement learning. In International Conference on Machine Learning, pp. 4051- 4062. PMLR, 2021.
Donald Joseph Hejna III, Pieter Abbeel, and Lerrel Pinto. Improving long-horizon imitation through lan- guage prediction. 2021.
Yiding Jiang, Shixiang Shane Gu, Kevin P Murphy, and Chelsea Finn. Language as an abstraction for hierarchical deep reinforcement learning. Advances in Neural Information Processing Systems, 32, 2019.
Pawel Ladosz, Lilian Weng, Minwoo Kim, and Hyondong Oh. Exploration deep reinforcement learning: A survey. Information Fusion, 85:1 22, September 2022. ISSN 1566-2535. doi: 10.10166). in0p::...nnnuus.222........ 2022.03.003. URL
inffus....... 2022.03.000
Sergey Levine. Understanding the world through action. In Conference on Robot Learning, pp. 1757-1757. PMLR, 2022.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open probloms. arXiv preprint WXXu:2005.01643 2020.
Juncheng Li, Xin He, Longhui Wei, Long Qian, Linchao Zhu, Lingxi Xie, Yueting Zhuang, Qi Tian, and Sil- iang Tang. Wiie-grrinee semantically aligned vision- language pre-training. Advances in neural information processing systems, 35:7290-753 -303, 2022.
Minghuan Liu, Menghui Zhu, and Weinan Zhang, Goal-conditioned reinforcement learning: Problems and solutions. arxiv preprint arXiv:2201.00 082999, 2022.
Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward Grefenstette, Shimon Whiteson, and Tim Rocktäschel. A survey of reinforcement learning informed by natural language. arXiv preprint Xiv: 1906. 039666 2019.
Yiwei Ma, Guohai Xu, Xiaoshuai Sun, Ming Yan, Ji Zhang, and Rongrong Ji. X-elip: End-to-end multi- grained contrastive learning for video-text retrieval. In Proceedings of the 30th ACM International Con- ference Multimedia, pp. 638-647, 647, 2022.
Kanika Madan, Nan Rosemary Ke, Anirudh Goyal, Bernhard Schölkopf, and Yoshua. Bengio. Fast and slow learning of recurrent independent mechanisms. arXiv preprint arXiv:2105.087100 2021.
Suvir Mirchandani, Siddharth Karamchoti, and Dorsa Sadigh. Ella: Exploration through learned language abstraction. Advances in Neurul Information Processing Systems, 34:29529 29540, 2021.
Fabian Paischer, Thomas Adler, Markus Hofmarcher, and Sepp Hochreiter. Semantic helm: An interpretable memory for reinforcement learning. Xi preprint arXiv:2306.0931 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PMLR, 2021.
Frank Röder, Ozan Özdemir, Phuong DH Nguyen, Stefan Wermter, and Manfred Eppc. The embodied crossmodal self forms language and interaction: a computational cognitive review, Frontiers in psychology, 12:716671, 2021.
12

---

Published in Transactions on Machine Learning Research (08/2025)
Matthias Rolf and Minoru Asada. Where do goals come from? generic approach to autonomous goal-system development. arXiv preprint Xrv: 10.5557, 2014.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy oplimiza- tion algorithms. arXiv preprint arXiv: 06347, 2017.
Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement learning. In International Conference on Machine Learning, pp. 9870 9879. PMLR, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Haoyu Wang, Xinrui Yang, Yuhang Wang, and Tan Xuguang. Constrained contrastive reinforcement learn- ing. In Asian Conference on Machine Learning, pp.
PMLR, 2023.
Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fine-graiied interactive language-image pre-training, arXiv preprint arXiv:111.00788, 2021.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01911 2022.
Yang Yu. Towards sample efficient reinforcement learning. In Proceedings of the Twenty- Seventh Inter-
national Joint Conference on Artificial Intelligence, IJCAI-18, pp. 5739 5743. International Joint Con- ferences on Artificial Intelligence Organization, 2018. doi: 10.24963 10.24963/ijcai.22 /it./aa.....///2222 1/40i..rr/10.249633////33
URI. https:
Ruijie Zheng, Xiyao Wang, Yanchao Sun, Shuang Ma, Jieyu Zhao, Huazhe Xu, Hal Daumé III, and Furong Huang. Taco: Temporal latent action-driven contrastive loss for visual reinforcement learning. arXiv preprint XrXi::2306. 18229, 2023.
Victor Zhong, Tim Rocktäschel, and Edward Grefenstette. Rtfm: Generalising to novel environment dy- namics via reading, arXiv preprint Xrx::11910008210 2019.
Cunjuan Zhu, Qi Jia, Wei Chen, Yanming Guo, and Yu Liu. Deep learning for video-text retrieval: review. International Journal of Multimedia Information Retrieval, 12(1):3, 2023.
A Appendix
Implementation Details
Base Models. The BabyAT environment serves as standard benchmark for instruction-follloinn rein- forcement lcarning. Its base model is trained using the PPO algorithm [chulman et al. (2017) and Adam optimizer with parameters
0.9 and 82 0.999. The learning rate is 7e 4, and the batch size is as CAREL-specific hyperparameters. To minimize the
256. We set Ac 0.01 and the temperature changes to the bascline model updates, we backpropagate the gradients in an outer loop of PPO loss to be able to capture episode-level similarities. This gradient update with different frequencies has been tried in the previous literature Madan et al. (2021).
The actor-critic model from the SHELM model was also used as bascline. We train the learnable parts of the model using the PPO algorithm and Adam optimizer with the same hyperparameters. The learning rate is le 4, and the batch size is set to 16.
13

---

Published in Transactions on Machine Learning Research (08/2025)
A.1 Time Consumption
To evaluate the CAREL framework's time consumption, we designed an experiment to measure the frames per second (FPS) achieved by our method and the bascline across various Baby benchmark environments. We calculated the average FPS over 100,000 frames detailed in Table The FPS results provide indication of our framework's computational efficiency and time consumption compared to the baseline. Based on the results obtained across different tasks in the BabyAI environment, our method processes approximatcly 420 frames (observations) per second on average, compared to the 668 FPS achieved by the Baby baseline. This translates to an approximate runtime of 13 hours for processing 20 million frames (as commonly presented the tables as the maximum step count), compared to hours for the Baby Al baseline. However, this does not provide a substantial challenge, as the main issue Reinforcement Learning is the number of interactions with the environment. needed for training.
Many works on RL Yu 2018 Ladosz et al. 2022 consider the number of interactions between the agent and the environment- referred to as sample efficiency as the primary challenge, rather than the overall training time. This is because interactions with the environment are inherently constrained by the environment' ss properties and cannot be accelerated by increasing computational resources.
Environment
Average-FPS (Babyai)
Average-FPS (CAREL)
GoToSegS5R2
629.60
512.19
OpenDoorsOrdez24
690.80
641.50
PickupLoc
679.86
265.33
PutNextLLcalSGNA
688.75
287.51
SynthS5R2
655.66
394.32
Average
668.93
420.17
Table 4: Comparison of Average Frames Per Second (FPS) Environments.
Between Babyai and CAREL Across Multiple
A.2 Ablations
A.2.1 Auxiliary Loss Coefficient
We evaluated the impact of auxiliary loss coefficient (X) on performance of CAREL framework. As shown in Fig 5 values above 0.01 or below 0.001 result in sudden decline in performance with 0.01 achieving the highest performance.
GoToSeqS5R2
CAREL (coef=1)
CAREL (coof-0.1)
0.8
CAREL (coef=0.05)
Success Rate (SR)
(cocf-0.01)
CAREL (coef=0.005)
0.6
CAREL (coef=0.001)
(coef=0.000
CAREL (coof=0.0001)
Babyai
0.0
Frames
Figure 5: Experiment on the impact of the cocfficient for our auxiliary loss.
14

---

Published in Transactions on Machine Learning Research (08/2025)
A.2.2 Action Embeddings in Instruction Tracking
Instruction Tracking (TT) can actions taken up this point. to predict the completion of subtask; therefore, adding the action embeddings to the observations helps the model to better detect the completion of subtasks that are utilized in the IT mechanism. To further investigate this, we conducted a CA experiment on the GoToSeqS5R2 environment, comparing performance with and without action embeddings. The results are depicted in Figure
GoToSeg55R2
Babyal CAREL + IT
Babyal
Success Rate (SR)
0.6
0.4
0.000
0.25
0.50
0.75
1.25
1.50
1.75
2,00
Frames
Figure 6: Effect of adding action embeddings to the observations in Instruction Tracking.
A.2.3 Instruction Tracking Similarity Score
When calculating the threshold 14 we can only use the Sæ-w w or So-w because we need the similarity of each word separately be able calculate final score for each subtask. Between the two, we only used Sg-w in our experiments for computational efficiency. The experiment in Figure / indicates that the Sg-w works as well as So-w. We employ SE-W because il is more computationally efficient due to its lower dimension.
GoToSegS5R2
1.0
Babyai+ CAREL IT (.WW)
Babyai CAREL IT IS, OW)
0.8
Success Rate (SR)
0.6
0.4
0.2
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2,00
Frames
Figure Results for using different embeddings for calculating the similarity score Instruction Tracking
A.2.4 Impact of CAREL in Instruction Tracking
This experiment shows the effect of CAREL on the improvement achioved by applying Instruction Tracking. As shown Figure g applying Instruction Tracking to our model (baseline CAREL) results in a slightly higher improvement compared to its application the original baseline.
15

---

Published in Transactions on Machine Learning Research (08/2025)
GoToSeg55R2
Babyal CAREL
Babyai
0,8
Babyal
Success Rate (SR)
Babyai
0.6
0.4
0..00
0.25
0.50
0.75
1.00 Frames
1.25
1:50
1.75
2,00
Figure 8: Ablation study the impact of combining CAREL with Instruction Tracking.
A.3 Environments
MiniGrid Environment. MiniGrid designed for research in reinforcement learning and decision-making tasks. It provides grid-based world where agents interact with their surroundings by perceiving symbolic or RGB observations and executing discrete actions. The environment is fully customizable, allowing re- searchers to define unique grid layouts, object types, and interaction rules. MiniGrid supports tasks such navigation, object manipulation, and goal-directed behaviors, offering varying levels of complexity suit different experimental needs.
AI Environment. BabyAl is an environment based on the MiniGrid environment. It is specifically crafted to assess the performance of agents in following natural language instructions. Baby provides a rango of procodurally generated tasks that require agents to achieve goals by acting on textual commands. These lasks are diverse, challenging agents in areas like navigation and interaction with objects.
The complexity of tasks in BabyAl varies significantly. Basic tasks involve simple objectives such moving to specific location (e.g., "Go to the green squarc') or retrieving an object (c.g., 'Pick up the red key'). More intricale lasks demand multiple interactions, such as 'go to red ball then go to blue box'.
A.4 Preliminary
Reinforcement Learning. Our experiments were conducted in two environments modeled partially observable Markov decision processes (POMDPs), represented by the tuple (S, O, 2, P, R). In this framework s E S represents the states of the environment, a E A denotes the actions, E Ω refers to observations drawn from the observation model O(ols,a) a), P(s'ls, a) defines the transition dynamics, reward function, and y is the discount factor. The objective during training is to determine a policy æ that maximizes the expected total discounted reward, expressed as:
the
max E,
Σ R((s az)
16


Reviews:
[{"review_id": "zeJY9gDDiA", "reviewer": "Reviewer_Fzp8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- One of the main methods is mainly adopted from a well recognized research work.\n- Method description is well written and clear.\n- Method is simple and model agnostic (it does ask for a language encoder and visual encoder to be already presented in a model)\n- The method brings substantial sample efficiency as per the presented experiments.\n\nWeakness:\n- More experiments on settings other than BabyAI should be considered to prove the effectiveness of the method. Since it is mostly adopted, I believe diverse set of experiment should help enhance the evidence of the method working in language guide RL setting, hence the adoption itself can be valuable.\n- There should be more analysis on if the added loss brings about better cross modal grounding rather than only claiming it to be true.\n- \"instruction tracking help the agent focus on the remaining tasks by preventing the repetition of completed sub-tasks.\" is also not evidently proved or discussed in experiments.\n- Some experimental choices are confusing (see requested changes).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work explores enhancing existing language guided RL tasks with an multi-grained contrastive learning loss adopted from XCLIP deemed CAREL (Cross-modal Auxiliary Reinforcement Learning) framework. Authors claimed the loss should bring better cross modal grounding. In addition, an instruction tracking procedure is developed to track status of subtask and selectively mask out the completed tasks' instruction. Authors claimed both the loss and instruction tracking can benefit sample efficiency.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Please provide discussion or additional analysis on \"instruction tracking help the agent focus on the remaining tasks by preventing the repetition of completed sub-tasks.\" and on \"improve cross-modal grounding between language instructions and environmental observations\" or consider refine those claims.\n2. For instruction tracking experiment, how is subtask being determined? an example would be helpful.\n3. The time consumption analysis and the auxiliary loss coefficient can be fit into appendix for more room for additional discussion.\n4. Experimental setting in consistent: instruction tracking experiment only using S_{E-W} to compute similarity, Vanilla CAREL is not using action embedding, I am not sure why the authors make such choice, as this would make it hard to know if the entire framework works. In addition, I notice the GoToSeq and OpenDoorOrderN4 experiments in Figure 2 and Figure 4 are done with different settings (the former not using action embedding, the latter uses action embedding), is there a reason for this?", "broader_impact_concerns": "Na", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9Yt4X1F7qj", "reviewer": "Reviewer_xPer", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n(1)  A novel CAREL framework based on cross-modal auxiliary loss functions and instruction tracking mechanism was introduced for instruction-guided reinforcement learning.\n\n(2) Experiments on MiniGrid and BabyAI environments showed the effectiveness of the proposed modules in terms of the overall performance and sample efficiency.\n\nWeaknesses:\n\n(1) Some technical details can be further explained.\n- It is stated that \"as the auxiliary loss improves the similarity scores through time, and the improved similarities enhance the instruction tracking.\" This can be verified in the experiments to demonstrate the benefits of auxiliary loss and instruction tracking.\n- It is stated that \"one can constrain the probability of masking and relax this constraint gradually as the learning progresses\". More discussion can be provided, e.g., how such constraints can be designed for the training process, and how it will prevent false positives, etc.\n\n(2) More experiments and discussion can be conducted to verify the effectiveness of CAREL.\n- The discussion of time consumption in subsection 3.1.2 is unclear. Why is it stated that the results in Table 2 don't show a substantial challenge? Furthermore, what is the \"number of interactions with the environment needed for training\" for CAREL in the experiments?\n- Subsection 3.1.3 shows that the values above 0.01 in Figure 3 result in a decline in performance and 0.001 corresponds to the best performance. However, it is unclear whether smaller values of auxiliary loss coefficient can lead to improved performance.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduced a novel CAREL framework for instruction-guided reinforcement learning. This framework contained two crucial components: cross-modal auxiliary loss functions and instruction tracking mechanism. Experimental results demonstrated the effectiveness of the proposed framework over baselines in terms of the overall performance and sample efficiency.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see above", "broader_impact_concerns": "No concerns on the ethical implications of the work needs to be considered.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "1nVK2PmMp0", "reviewer": "Reviewer_3YgW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths:\n1. The proposed methods, CAREL and instruction tracking, are both well-motivated and can be easily plugged into existing instruction-guided RL frameworks.\n2. In the experiments, the proposed methods improve the performance comparing to the baseline method. Especially, the proposed method significantly improves the sample efficiency of the RL models. With the same numbers of training steps, the methods with the proposed CAREL and instruction tracking always yield better or on-par performance than the baseline.\n3. The paper provides pseudo code of the proposed methods, which help the readers to understand the pipeline.\n\n\n## Weaknesses:\n1. Although the proposed methods improve the sample efficiency, the final performance is not always improved. For example, in PickupLoc and PutNextLocalS6N4 in Figure 2, and MiniGrid-GoToDoor-6x5-N2-v0 in Figure 5, the final performance is just on-par or even below the baseline.\n2. As the authors mentioned in the second paragraph of Section 1, there are a few prior works exploring instruction-guided RL and internal representation improvement in RL. However, the paper does not compare the performance of the proposed method with any of these works.\n3. There are a few technical details remain unclear:\n+ In the instruction tracking method, the paper mentions that instruction can be parsed into a set of sub-tasks via rule-based heuristics. However, it is unclear how the rule-based heuristics are implemented. Some examples would be helpful for the readers to understand.\n+ In the instruction tracking method, the algorithm need to remove sub-tasks from the instruction. As the instruction is in natural language, how is this achieve? Is there any NLP tool used here?\n+ It is unclear how frequently is the auxiliary loss calculated. Is it computed after every one rollout of an episode? Providing a pseudo code about the entire RL framework would be helpful to clarify this point.\n+ In Section 3.1.4, the method uses \"a certain possibility that follows a hyperbolic tangent function in terms of training steps\". What is exactly the expression of this function?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposed CAREL and instruction tracking to improve instruction-guided reinforcement learning models. CAREL serves as an auxiliary training objective to improve the internal observation and instruction representations. Instruction tracking keeps track of the progress of the task completion and remove the completed sub-tasks from the instruction. In the experiments on two benchmarks, the proposed methods are shown to be more performant and sample efficient than the baseline method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. It would be appreciated if the authors could clarify the details mentioned in Weakness 3.\n2. It would be appreciated if the paper could compare the performance of the proposed method with some of the previous methods mentioned in Section 1.\n3. In Equation (14), a hyperparameter $k$ is required. Although the authors set $k=2$ in the experiments, it would be appreciated if the authors could discuss the impact of these hyperparameter. Especially, would this hyperparameter be task-dependent?\n4. When plotting Figure 2 to show the data efficiency of the proposed method, the x-axis is number of frames. However, as Table 2 suggests, the proposed method has a lower FPS than the baseline. It would be appreciated if the paper could show a performance comparison plot with the running time as x-axis to further demonstrate the efficiency.\n\nTypos:\n1. In Equation (12), $1/n$ should be $1/N$.", "broader_impact_concerns": "No ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)