EAGER: Asking and Answering Questions for Automatic Reward Shaping in Language-guided RL
Authors: Thomas Carta
Pierre-Yves Oudeyer
Olivier Sigaud
sylvain lamprier
Venue: NeurIPS 2022 Accept (2022)
Keywords: language-conditioned RL
automatic reward shaping
intrinsic rewards
exploration
auxiliary objectives
question generation
question answering

Abstract: Reinforcement learning (RL) in long horizon and sparse reward tasks is notoriously difficult and requires a lot of training steps. A standard solution to speed up the process is to leverage additional reward signals, shaping it to better guide the learning process.
In the context of language-conditioned RL, the abstraction and generalisation properties of the language input provide opportunities for more efficient ways of shaping the reward.
In this paper, we leverage this idea and propose an automated reward shaping method where the agent extracts auxiliary objectives from the general language goal. These auxiliary objectives use a question generation (QG) and a question answering (QA) system: they consist of questions leading the agent to try to reconstruct partial information about the global goal using its own trajectory.
When it succeeds, it receives an intrinsic reward proportional to its confidence in its answer. 
This incentivizes the agent to generate trajectories which unambiguously explain various aspects of the general language goal.
Our experimental study using various BabyAI environments shows that this approach, which does not require engineer intervention to design the auxiliary objectives, improves sample efficiency by effectively directing the exploration.

EAGER: Asking and Answering Questions for Automatic Reward Shaping in Language-guided RL
Thomas Carta Inria Flowers team Université de Bordeaux thomas. carta@inria.fr
Sylvain Lamprier
ISIR Sorbonne Université Univ Angers, LERIA, SFR MATHSTIC, F-49000 Angers, France
Pierre-Yves Oudeyer Inria Flowers team
ISIR Sorbonne Université, Paris, France olivier. sigaud@isir. uumc..
Université de Bordeaux Microsoft Research Montreal
pierre-yves www.eeee..ueeeeeeeeeee oudeyer@inrian
Abstract
Reinforcement learning (RL) long horizon and sparse reward tasks is notoriously difficult and requires a lot of training steps. A standard solution to speed up the process is leverage additional reward signals, shaping it better guide the learning process. In the context of language-condittione RL, the abstraction and generalisation properties of the language input provide opportunities for more efficient ways of shaping the reward. In this paper, we leverage this idea and propose an automated reward shaping method where the agent extracts auxiliary question generation (QG) and question answering (QA) system: they consist of questions leading the agent to try reconstruct partial information about the global goal using its own trajectory. When it succeeds, it receives an intrinsic reward proportional its confidence in its answer. This incentivizes the agent generate trajectories which unambiguously explain various aspects of the general language goal. Our experimental study shows that this approach, which does not require engineer intervention to design the auxiliary objectives, improves sample efficiency by effectively directing exploration.
objectives from the general language goal. These auxiliary objectives
Introduction
One of the main challenges of Reinforcement Learning (RL) research is to train agents able of abstraction, generalisation and communication. Languages, they natural or formal, afford these desirable properties [I]. Based on this insight, many papers have tried to leverage the abilities of language in RL to enable communication and improve generalisation and sample efficiency be subdivided into language-conditionee RL (LC-RL), in which language conditions the formulation of the 22 and language-assisted RI., where language helps the agent learn [15] TO I 91. In the present paper, we focus on the LC-RL framework where the agent initially receives language instruction and must act to optimise the corresponding reward function. Unfortunately, the corresponding RL algorithms are sample inefficient, especially due to the fact that the reward function is sparse when it is restricted to completing the goal.
3 22 12] 36). The domain
To tackle the reward sparsity issue, idea is to densify the reward by decomposing the general goal into sub-goals and rewarding them individually. This idea is based on a decomposition principle which
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Current stats
Updatte 7
Goal
Reward
box proybull
Question Generation
Reward
Vy pat the hax red the gres
dg huu
4: pst the the giry bell
Question Answering
du
put the biie bee neet the bel
dy
Ue ast the bike bee neet the grey...
the
Frustig srowered quattion
Figure 1: During training, the agent uses the goal to generate relevant questions using its question- generation module QG. Then, it attempts at answering them from current trajectories at each step with its intrinsic reward proportional to its confidence in answer. Then removes the answered questions from the list of questions. This incentivizes the agent to produce trajectories that enable to reconstruct unambiguously partial information about the general language goal, enabling to shape rewards and guide learning.
module QA, by looking at the trajectory. When it succeeds, it obtains an
hypothesises that general goal can be decomposed into set of easier ones. Previous works assume strict decomposition of the general goal into sub-goals B 17. However, strict decomposition requires that that each high-level goal can be reached through an exact series of low-level policies, an assumption which fails when all the primitive actions are not known in advance. More recently, the ELLA method was proposed as a language reward shaping technique that is sample efficient and does not require strict decomposition: it rewards interesting auxiliary objectives, without requiring rigid ordering and correspondence with sub-goals [23].
However, all these methods suffer from the need for expert input. Indeed, they require task-specific engineering for determining at least one of the following elements: the set of sub-goals or auxiliary objectives, the relevant ones and when they are achieved, and the appropriate reward. While all these points can be addressed more or less easily in simple game-like environments, they get demanding when the environment is more complex. Therefore, it seems desirable to find an alternative reward shaping approach that minimises expert involvement. We would like the agent to generate its own auxiliary objectives and create intrinsic rewards, just like a human would do.
The Natural Language Processing (NLP) field often suffers from the same need of expert input, e.g. for the evaluation of automatic summary tasks. In that context, some successful approaches have developed reference-less metrics 331. 29), based on question generation (QG) and question answering (QA). These techniques assess the quality of a generated text by measuring the quantity of information from the source conserved in the generated text (broadly, how well one can answer questions about the original text using the generated text). These metrics are said reference-less because they do not require a comparison with man made example to evaluate the quality of a text.
Contributions In work, build on these reference- metries to circumvent the need of expert input for generating auxiliary objectives. We adapt it and propose a novel QG/QA framework for RL. called FAGER In EAGER, agent reuses the initial language goal sentence generate set of questions (QG): each of these self-generated questions defines an auxiliary objective. Here, generating question consists in masking word of the initial language goal. Then the agent tries to answer these questions (guess the missing word) only by observing its trajectory so far. When manages to answer question correctly (QA) it obtains an intrinsic reward proportional to its confidence in the answer. The QA module is trained using a set of successful example trajectories. If the agent follows path too different from correct ones at some point in its trajectory, the QA module
'xplloit learning, see https: http://///stuuu..eoooe//eeeeeeee EAGER for access the code.
Grounding for effective Exploration in language-conditoneed conditioned Reinforcement

---

will not answer the question correctly, resulting in zero intrinsic reward. The sum of all the intrinsic rewards measures the quality trajectory in relation the given goal. other words, maximizing this intrinsic reward incentivizes the agent to produce behaviour that unambiguously explains various aspects of the given goal.
To the best of our knowledge, EAGER is the only framework that can automatically 1) generate relevant auxiliary objectives. 2) determine their completion and 3) return the appropriate intrinsic reward. This approach only assumes the agent has access to dataset of demonstrated behaviours associated global language commands, which enables it to pre-train its question answering module.
Thus our work brings the following contributions:
We create a QG/QA metric providing to agent an information-rich measure of the quality of its trajectory given a goal.
We propose the EAGER framework that lets the agent guide its own learning process by generating auxiliary objectives and producing intrinsic rewards without requiring any expert intervention.
We show that EAGER retains the good properties of ELLA without requiring task-specific expert knowledge, by leveraging properties of language.
We experiment EAGER with the BabyAl platform 7 we compare our approach against ELLA (SOTA on AI) and RIDE (a non-language based reward shaping approach using intrinsic motivation), showing robustness and sample efficiency. Furthermore, although we use example trajectories to train the QA, their use is much more parsimonious than training an agent using behavioural cloning 7 as we show in Appendix C.
Related work
Language-conditionee RT. We place work in the LC-RL setting, where agent learns policy to execute language commands [24, [1]. We reuse the Baby platform (7]. widely used in this domain as it enables decouple exploration challenges from perception challenges. uses synthetic language exhibiting interesting combinatorial properties with possible conjunction of properties, and procedural generation avoid overfitting g Here, we consider instruction following
agents which receive external instructions and rewards 14 5
Language as an abstraction in hierarchical RL. Several approaches leverage language for abstraction in hierarchical RL. One approach uses language for training a low-level instruction- following policy, then learns a high-level policy that generates the sequence of low-level goals [17] Another one explicitly decomposes high-level tasks into low-level ones as in policy sketches 3
ELLA 23 also uses language decompose high-level tasks but relaxes the strict decomposition constraint by replacing sub-goals with auxiliary objectives. We adopt the same flexible framework in our approach but automatise the decomposition.
Language for exploration and reward shaping. Reward shaping is form of guidance that supplies additional rewards to the agent to direct its learning process. Among approaches studying how language can shape rewards and exploration, LEARN [12 proposes to map intermediate natural language instruction to intermediate rewards. Similarly,
35 enables reward shaping using natural language through a narration- carration-guided guided method. The high- tasks are decomposed into low-level tasks and rewarded using narration. ELLA [23] is positioned in the same paradigm but with fewer assumptions about the environment or the structure of the task.
Other approaches assume that an oracle provides language descriptions of environment states which are used as state abstraction to generate novelty intrinsic rewards and guide exploration 25 34]
This extends classical approaches to intrinsically motivated exploration 44, such as count-based methods [6] or RIDE [28). Here, we also use language to generate intrinsic rewards, but we do not need language descriptions of states. Besides, other forms of intrinsic motivation systems, such as IMAGINE [10), first learn aanguage-parameterize reward functions through interaction with social peer, then autonomously use language to generate diverse and novel goals, using the learned reward function for self-supervision.
3

---

Asking questions in RL. Beyond reward shaping, some methods consider agents that use language ask questions external knowledge sources. In QWA [36,, questions are used to identify sub-tasks and prune irrelevant actions. In AFK 19], questions are used to obtain world knowledge that helps completing tasks.
Natural Language Processing One of the sources of QG/QA methods is the thriving field of question generation from natural language processing and information retrieval [16.. Our approach is inspired from text generation methods where QG and QA are used to measure the quality of generated text without using a human reference [3 29
3 Problem statement
We set ourselves in the standard framework of LC-RL with augmented Partial Observation Markov Decision Process 33 M defined by the tuple (S, A, 2, T, O, g. V. I, R, y), with S the state space, Z the observation space, A the actions space, 9 the goal space, the vocabulary of goal instructions. O stands as the observation function O : S + Z, which maps states to the observations space. I is the instruction function
+ Visize which maps any goal in to the set of language instructions, which correspond to sequences of isize symbols (the empty symbol c belongs to V to allow variable instruction sizes). R is a goal-conditioned state action reward function, with R S x Ax x 9 + [o, 1] the extrinsic reward received for some goal G and the discount factor. For simplicity, we note in the following r = R(+, Ot: g) as the reward obtained at step of any episode with goal g.
At each time step I, the agent receives an observation O E 2 following the observation function : + 2 and selects an action an A reach a goal g E G, expressed by w = I(g).
T x A P(S) is the transition function. Using RL, we search for an optimal goal-conditioned policy x², such that I²' S Visize + A maximises the discounted expected return R²
E, G 1thh 7 r4+141l- We consider in this work sparse reward problems, where r(s, a, g) returns 1 for any state s such that d(s,g) g) S e, for a given distance function d and a specified threshold e, and 0 otherwise. Moreover, we assume a limited number of steps at most H steps. These two conditions result in a hard exploration problem.
To deal with those problems, many methods aim at densifying rewards, by focusing on auxiliary objectives during training, whose accomplishment can help the agent to reach the goal g at hand. In previous work, such as ELLA [23), the selection of relevant objectives required the intervention of an expert (in the form of expert annotations and example trajectories), which can be problematic because new expert trade-offs have to be established for each new environment. Thus, an automated way must be found to recover the relevant auxiliary objectives, measure their completion, and associate the appropriate reward.
Rather than relying on expert knowledge for defining auxiliary objectives, we assume that we have access to a set of trajectories of successful examples coupled to their respective instructions where Tn = (o,, ag [)) with i the number of steps. For any goal instruction wa, we consider a function f that aims at generating set of auxiliary goals of g. such that all y E ((w9) belong to G with G C the set of goals that help training the agent towards g. Then, for trajectory r and any g' E (w:9), function h determines the probability h(T,,g') ) E [o, 1] that the auxiliary objective g' has been achieved. We train using the example demonstrations such that we can leverage its generalisation abilities to use h for unseen trajectories.
(T). (T... w()].
Please note that Behavioural Cloning methods also rely on the exploitation of expert trajectories, by learning the policy BBC that maximises the log- ligliikld L
LBC log P(T.) where log P(r) le((Tpo(aalo)/7(sssssss...
However, the latter uses demonstrations to train an agent to copy an expert based on data set of example trajectories. This technique is only effective when the agent is close enough to the demonstrated examples. large number demonstrations are therefore required to cover the space of states that the agent may encounter. Methods such as GAII. (18] can partially circumvent this problem by forcing the agent to stay on known trajectories, but this impairs generalisation. Our method is much more parsimonious in the use of such examples: because it generalises well, the

---

agent can receive intrinsic rewards even in areas not encountered in the expert demonstrations (see Appendix C).
4
Method
In this section, we first introduce EAGER, our automated reward shaping method based on QG/QA, then we present practical implementation in the context of the BabyAl benchmark. Figure provides a graphical overview of EAGER.
4.1 EAGER
We need an automatic evaluation method that is fine-grained enough to rank various trajectories depending on a language instruction. But various successful policies generate set of valid trajectories for the same goal. It could be deceiving to rank them based only on their final results, as an overly complex trajectory seems intuitively worse than a simpler one even if the result is the same.
To address this issue, the EAGER framework consists of an agent learning module, a question generation module QG (automatic, but not learned) and learned question answering module QAA. These two modules fulfil the role of the functions f and defined in Section 3 EAGER is inspired from works like QuestEval [3]] and Data- QuestEval [29). developed for natural language generation. For instance, for abstractive summarization, by generating questions from the original text (QG) and trying answer them using the summary QA, this method measures the quantity of information shared between both texts.
In our work, we draw the analogy with the NLP task with the goal replacing the original text and the trajectory replacing the summary. We use the QG/QA system to verify that a trajectory contains the same level of information as the language instruction, meaning that the goal is contained in the trajectory. As the goal can be contained in a lot of different trajectories, we also favour simple trajectories. If one can easily answer the question, that means the trajectory is simple.
The QG module We assume that the goal linguistic instruction w of E G is highly expressive language instruction (e.g Put the red ball next to the blue box, open the red door, ...), containing by themselves enough world knowledge to generate questions by masking words. Thus, the QG module returns a list of questions QGx (w:) that can be seen as auxiliary objectives. Fach question is formed by masking one word in the linguistic instruction. Crucially, the choice of words to mask can be done automatically without any expert knowledge of the task, or the environment, for instance masking all nouns and adjectives. These questions can be seen as auxiliary objectives guiding the agent during training. Besides, being formulated natural language, these auxiliary objectives are easily interpreted.
The QA module Let À be the set of possible answers generated automatically from the list of tokens masked by the QG. Thus, Á contains (g, a*) pairs of questions and expected answers. The QA module returns the probability for all E A, that answers question q. given a trajectory Tr, where T, (o,, (i)se(ju.[]] are state-action pairs and ć the time step. We note ä' QA(g, T.) the answer greedily generated from the module and QA(ā" |g. 7:) the associated probability. The auxiliary objectives from the QG are considered achieved when the QA answers them successfully. In our work, the QA module is pre-trained using full example trajectories generated by bot, see Section 5.T without any type of annotations to guide it. Besides, at time step t, there is no guarantee that the trajectories contain enough information to correctly answer a question. Thus we include <RO_amswer= token in Á to prevent the QA from answering correctly by chance. Moreover, since the QA module takes the whole trajectory, once has answered a question, can also answer at the next step. To avoid giving reward that does not have direct link to the current step, every time question q is answered, it is removed from the set of questions: We note Q, the active set of questions, the initial set is Qo QG& (w&) and once question is answered, we apply Q, Q.-1 Mg).
Architecture of the QA The QA is used to compositionally chain low-level tasks. To do this, relies on the episodic transformer (27) This architecture uses multimodal transformers (over language, visual observation, and a list of actions) that have access to the full episode. For any time
5

---

step t and any question, a (I1-L, Pi:e, 01:t) tuple is given as input to the QA module. The language input F1. is the question. is sequence of tokens with 2; N. The visual input U1 the list of observations E &R W xxx3, Finally, the action input d1 is a list of discrete actions. As output, the network returns the probability distribution over the set of possible answers A.
Intrinsic reward The main difficulty in ranking trajectories is that many trajectories can share the same description and thus can answer questions correctly. However, human assessing several trajectories, with the same extrinsic reward, relatively to the same goal would prioritise the simplest one. To account for this, we make the reward proportional to the confidence in the answer at time step QA(āa I q. Te). An overly complex trajectory being harder to understand for the QA, the answer should be given with less confidence, it should obtain less reward than direct and clear trajectory, even with the same number of correct answers.
In this paper, we keep the same desired properties for the shaped reward function as the ones given in ELLA: the reward shaping should not change the optimal policy that prevails before the reward shaping (policy invariance), and the reward should encourage sample efficient exploration based on auxiliary-objectives. Using intrinsic rewards, we modify the global reward from to r' through a policy invariant transformation [26], which ensure that the new policy x² is optimal for both M and M' (S, A, A. T, To.To do so, we ensure that only successful trajectories get the same return with or without the reward shaping by substracting the shaped reward at the final time step N of a successful trajectory: r'y = TN- - ΣnETs r r with r/ = À Egeo, Lā' is correct answer QA(ā* q, Te), where T3 the set of time steps where bonus applied. rz is the reward given at time step t, in the case of sparse reward studied here A 0 only if corresponds to the last step of successful trajectory. In ELLA Mirchandani et al. ( the authors prove that this transformation is policy invariant. Thus, as long as the policy produces unsuccessful trajectories, the agent is guided by the shaping reward. Then once it has learn to successfully complete an instruction, the shaping reward is substracted at the last step and the agent improves using only the extrinsic reward. Further details are provided in the appendix.
The QA system is pre-trained with successful trajectories, which prevents reward hacking. Indeed, if the agent individually completes the auxiliary objectives without meaningful trajectory, the QA does not consider the trajectory meaningful and answers: hho_answer", preventing the agent to get rewarded.
Using the above notations and concepts, we can define a metric to measure the adequacy of a trajectory goal, that corresponds to our cumulative intrinsic reward over a trajectory J of length N, up to A factor:
mOc
QA(a" [g T.) llā"" correct answer to q]-
t= 9E
Algorithm Our algorithmic procedure is given in Algorithm At the beginning of an episode, the QG takes the goal and returns a set of questions related to it. Then the QA module is applied at each step over the active set of questions Q,- When question is answered correctly, the shaped reward function returns bonus A QA(@"
TI), where À is scaling factor, to the agent and the answered question is removed from the set of active questions.
Then we tune À to ensure that no unsuccessful trajectories can get more reward than a successful one from the optimal policy x² The higher bound for the shaped reward of unsuccessful trajectories is k and the lower bound for the reward of a successful trajectory is y rHe where k is the number of questions generated by the QG and H is the maximum length of an episode. Thus we obtain
H1
A
k
Note that by making a less conservative hypothesis, i.e. assuming the worst case the successful trajectory takes N<H steps, we could obtain higher À leading to faster learning [23].

---

Algorithm Automatic auxiliary goal generation and reward shaping using EAGER
Input: 8g initial policy parameters, À bonus scaling factor,
function QA JA_SSAAPEQQ. Tr, for q in 2r do a" + QA Tr)
ENV the environment and OPTIMISE an RL optimiser
for k=O.... Ratep do w ,o, doneg ENV EEV.resee()
if ā is correct answer to q
Qn = {41, .... 9& + QGx(w9)
then
t 5
r. + A QAA(a" gq.
while done; not True do
Q = 2 QM4
at f 0 (o:)
end if end for return ry, Q end function
0 T,, done <ENV((;))
Te 2.+1 +- DA_SHAPPS Qr Tep Tz)
if done,+1 is True then
++
IN
NEUTRALISS(('..)
function NEUTRALISEOI F'N < r'y EieT, 7--N return T'y end function
end if end while
Update 0x-1 < OPTIMISE end for
4.2
A particular instance of the method in the BabyAl framework
We now explain how to adapt the EAGER method to train an RL agent in BabyAI, a language- conditioned environment where the agent has limited number of steps to complete a language goal. In this environment the agent receives a reward if and only if it finishes the task successfully.
The BabyAl benchmark contains tasks with highly expressive language instructions e.g Put the red box next to the green key. Thus they are rich enough to generate questions by masking words. In practice, we mask nouns and adjectives: this form of QG is very simple and can be automated using standard NLP techniques, thus it does not require expert knowledge. For instance, for the goal Put the red bull next to the blue box, using the token <question> as a mask we generate 4 questions among which Put the aquestion> ball next to the blue box.
The environments in our experiments are partially observable. Thus, our agent takes sequences of observations (01, Oz) 0t) as inputs of a recurrent network [3]
5
Experiments
5.1
Experimental settings
We use the BabyAl 7 platform run our experiments. This platform relies on gridworld environment (MiniGrid) to generate set of complex instructioss-ollloiing environments. It has been specifically designed for research on grounded language learning and related sample efficiency problems. The gridworld environment is populated with several entities: the agent, boxes, balls, doors, and keys of 6 different colors. These entities are placed rooms of 8 x 8 tiles that are connected by doors that could be locked or closed. The agent can do 6 primitive navigation actions such as forward, toggle, pick up to solve the language instruction (for instance Pick up the red box). It only has access to partial observations of its environment inside which irrelevant objects are randomly added. Moreover, the observations are a symbolic space using a compact encoding, with 3 input values per grid cell 8 x 3 values in total. When the agent completes the task after N steps, it receives the reward I'v training, all rewards are scaled up by a factor of 20 to ensure good propagation of the rewards. If the agent fails, the reward is O. We focus our tests on tasks of varying complexity: PutNextTo, Unlock and Sequence. The task can also take place in room local or two rooms Medium.
- 0.9 Ty, where H is the maximum number of steps. During
N.
To train the QA module through supervised learning, we build a dataset of example trajectories associated to language goals using a bot provided in BabyAl, then we generate related questions and
'3 integers: one representing the shape of the object, one its color, and one its state. For instance, (4, 1, 1) represents closed green door
7

---

PutNextTo-lan To-Local
PutocxlTo-Local
Scquence-Medion
06
0.-
03
01
PutNext-Medium
Unlock-Medium
Sequence-Medium
0.5
0.4
0.2
0.0
Prames
Frames
EAGER
ELLA
RIDE
PPO
Figure 2: Average reward for EAGER and baselines for four tasks, with error regions to indicate standard deviation over four random seeds. For the PutNext-Local and Sequence Medium tasks, we give an example of possible tasks with environment at time step 0.
answers. To obtain QA that can operate on various tasks, we use mix of the PutNextTo, PickUp, Open, and Sequence tasks for generating training trajectories. This dataset is only used to train the give input of the QA the full trajectory the list of observations and actions - and all the questions generated by the QG and we use the cross-entropy loss its output distribution to update it. To train the QA answer: "no answer" and prevent it from guessing the answer by chance, we randomly associate some trajectories and questions from unrelated goals. For instance, associate the trajectory from the goal: take the red box next to the blue ball and associate with the question: take the red <question> next to the blue key generated from the goal take the red box next the blue key. The QA must learn to pay attention to all the details of the question. It has to see that the trajectory describes an agent taking the red box but placing it next question is no answer and not box.
QA, not to bootstrap policy learning. During training,
object other than the in the question. Thus the good answer the
Moreover, empirically show in Section 5.4 that the QA is more efficient when it learns from broad distribution of trajectories for similar tasks. The intuition behind such behaviour is that a QA trained narrow distribution of successful bot-generated trajectories will not recognise the noisy trajectories of the agent when it starts training. Thus the QA will too often answer "no_ answer". resulting in intrinsic reward and hurting the reward shaping efficiency. To produce wider distribution using the given procedural bot, we force the bot to take with a certain probability a training examples.
random action each time step and only keep successful noisy trajectories More details on the QA pre-training are given in Appendix B. All the tasks we use for training the QA and the agent are summarised with examples Appendix A.
To evaluate our reward shaping framework, we use the Proximal Policy Optimization (PPO) algorithm 30J. but our reward shaping method is algorithm agnostic. We compare framework to PPO without reward shaping, ELLA and RIDE. RIDE 28 an exploration method that does not use language and addresses sparse reward problem by rewarding impactful change in state. We use Nvidia Tesla V100 with 10 cores to train our model and we use seeds in each experiment.
Figure 2 presents learning curves for EAGER, ELLA and RIDE across 4 environments. Table describes the assumptions and the type of expert knowledge required by the three reward shaping methods. It clearly appears that EAGER requires less expert human intervention than ELLA.

---

Table 1: All the assumptions and expert knowledge required for RIDE, EAGER, and ELLA
Number expert
Human Expert Intervention
Automated parts done by the agent
Method
demonstration per task
RIDE
None
Determining if a new state is impacting
7.500 noisy trajectories
Determining what words are Bouns or adjectives
Determining relevant auxilaary objectives
EAGER
(see supplementaty
Determining auxiliary ubjectives completion Determining auxiliary objectives associated reward
Wide distribution of trajectories)
Determining auxiliary objectives Determining auxiliary objectives associated reward
relevant
Determining relevant auxiliary objectives
ELLA
15,000 bot trajectories
among the predetermine class Determining auxiliary objectives completion
5.2 How does EAGER perform when sparsity increases?
In the PutNextTo and Unlock tasks. EAGER obtains results better than ELLA (SOTA in BabyAl) without using expert knowledge. also performs significantly better than RIDE for the tasks PutNext and Sequence and slightly better for Unlock. The better performance of EAGER with respect to RIDE is not surprising as the EAGER agent receives some indications based on example trajectories through the QA module.
EAGER overcomes bottleneck. The general goal being Open the <colour> door, the agent has to first pick up the key of the corresponding colour before reaching the door to open ELLA rewards picking up keys, via the PICK low-level instruction chosen via expert knowledge. EAGER reaches better performance (see the statistical test in Appendix C) without the need for expert knowledge. Moreover, although this is not its main purpose, EAGER gets a similar or better sample efficiency for most tasks.
For
5.3 How does EAGER perform with sequence of tasks under temporal constraint?
The Sequence task adds a temporal constraint by chaining two tasks using 'before' or 'after' together with high number of instructions (over 1.5M instructions in comparison with PutNext- 1440 instructions). Moreover both EAGER and ELLA decompose the goal into auxiliary objectives. This decomposition does not retain the temporal constraint, there no notion of doing one auxiliary objective before another.
with
Our tests show that EAGER retains strong performance, doing better than RIDE and ELLA. The slow progress of EAGER at the beginning can be attributed to the time when the agent is not good enough to efficiently trigger an intrinsic reward signal from the QA module. Indeed, at the beginning. trajectories are noisy and it is more difficult for the QA to exploit a trajectory with more rooms. As result, it over-desponds "no_ answerr leading lesser intrinsic reward.
5.4 Is EAGER robust to QA performance?
At first glance, the reliability of the QA looks crucial to our method. However, the QA could be difficult to train in some environments e.g. if you want the QA to learn to answer in large set of answers from small number of example trajectories. This is why we tested the robustness of our method using the PutNext'To-Local task with two metrics: the success rate SR of the QA after pre-training and the distribution of example trajectories. For the former, we take the same QA at different training epochs and we determine its SR over test dataset, then train the agent using the reward shaping provided by this QA. Figure lefft shows the robustness of our method with agents that display similar training curves soon SR 0.56.
In Figure righht, we plot the SR of the QA when training the agent. Initially, the QA with SR < 0.56 at pre-training time tends to have higher SR. Indeed, the distribution of answers is less peaked and answers are often correct by chance. On the opposite, the QA with a SR 0.56 answers "no_anse" "no_ar" and obtains no reward. However, in this case, the agent learns faster because it only receives reward for meaningful answers. For the QA with a SR S 0.56, the SR along training first grows then decreases. First, the agent is biased by the intrinsic reward to follow path that improves

---

PutNost Median
Sequzece Modiun
Wide Octribution
Narrow Distrbbttin
succusess Eate Sriniig!
x 0,20 x 0,41
0,56
9,45
0,73
Figure 4: Average reward in tasks PutNextTo and Sequence for two distributions of bot tra- jectories used to train QA. A narrow and a wide distribution where noise is added to bot trajec- tories
Figure 3: Average reward (left) and success rate of the QA (right) for the PutNex(To-Local tasks. The agent are trained with QA having different success rate after pre-training.
the SR, but once the agent learns to complete the trajectory leading to the extrinsic reward, the SR converges to pre-training SR.
As explained Section [.1] we added noise to trajectories generated by the bot to compensate for too narrow trajectory distribution. Figure shows training curves for two environments for QA trained on wide trajectory distribution (WD) and narrow distribution (ND). The reward shaping method trained on (WD) learns faster because they efficiently reward the agent early in training.
5.5 How do design choices on the QA module affect EAGER's performance?
In Section 4.1 made two choices for the QA mod- ule and the associated intrinsic reward: first we added "no answer" response, second swer by the confidence the agent had in its own answer. To verify the influence of these choices over EAGER's s performance, we use the PutNextTo task to compare EA- GER against "EAGER \no_ moo_answer", "EAGER Simple", and "EAGER Simple ino__answwa moo_anwwer.. "Simple" means that the agent received binary intrinsic reward for a good answer, otherwise) and "'no "nnswerr" means we sup- pressed the "no answer" solution. Figure ( gives the re- sults of these ablations. We can see that both the use of "no answer" and the non- non-binary reward independently boost sample efficiency.
Auerage Reward
rewarded each
EAGERV AR,ARIE
0.0 0
EAGER Siep (rd, STAAM
1
2
Frames
Figure 5: Average reward in Po with various ablations of the QA module.
6 Conclusion
In this work, we have proposed to leverage the abstraction and generalisation properties of language to build an automatic reward shaping method in the context of long horizon and sparse reward tasks. Our learning agent generates its own questions from the goal and rewards itself for correctly answering them, resulting in an efficient curriculum over auxiliary objectives. This to be contrasted with ELLA [ where expert knowledge is required for choosing auxiliary objectives. Besides, we do not call upon an oracle for getting linguistic description of environment states as in (25).
Limitations and Future Work EAGER assumes the QA system was pre-trained using pre- existing set of example trajectories. Next steps will consist in investigating how to remove this limitation, e.g. by implementing autotelic strategies based on QG/QA learned online. Besides, in this work we tested our method on BabyAl, 2D environment with synthetic language. In the future, we would like to consider a more complex language, generating more complex questions than the one obtained by masking, and testing our method on more realistic environments with true human instructions, as in the ALFRED dataset [22.
Acknowledgments and Disclosure of Funding
This work benefited from the use of the Jean Zay supercomputer associated with the Genci grant A009101 1996, well from the ANR DeepCuriosity Al chair project.
This work has been supported by the Horizon 2020 PI.LLA-rrobos project (grant number 101070381)
10

---

References
[1] Ahmed Akakzia, Cédric Colas, Pierre-Yvee Oudeyer, Mohamed Chetouani, and Olivier Sigaud. Grounding language to autonomously--aqquired skills via goal generation. arXiv preprint arXiv:2006.077 185, 2020.
[2] Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Henge. Vision-and-langun navigation: Interpreting visually-grounde navigation instructions in real environments. In CVPR, 2018.
[3] J. Andreas, D. Klein, and S. Levine. Modular multitask reinforcement learning with policy sketches. International Conference on Machine Learning (ICMI.) 2017.
[4] Arthur Aubret, Laetitia Matignon, and Salima Hassas. A survey on intrinsic motivation in reinforcement learning. arXiv preprint arXiv:1908.069: 2019.
[5] Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Arian Hosseini, Pushmeet Kohli, and Edward Grefenstette. Learning to understand goal specifications by modelling reward. In arXiv preprint 1806.00 049406 2018
[6] Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems, 29, 2016.
[7] Maxime Chevalier- Boisvert, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. Babyai: A platform to study the sample effi- ciency of grounded language learning. In International Conference on Learning Representations (ICLR), 2019.
[8] K. Cobbe, C. Hesse, Hilton, and J. Schulman. Leveraging procedural generation benchmark reinforcement learning. In International Conference on Machine Learning (ICML), 2020.
[9] Cédric Colas, Tristan Karch, Olivier Sigaud, and Pierre-Yvel Oudeyer. Autotelic agents with intrinsically motivated goal-conditioned reinforcement learning: short survey. Journal of Artificial Intelligence Research, 2022.
[10] Cédric Colas, Tristan Karch, Nicolas Lair, Jean-Michel Dussoux, Clément Moulin-Frier, Pe- ter Ford Dominey, and Pierre-Y yves Oudeyer. Language as a cognitive tool to imagine goals in curiosity-driven exploration. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[11) Alison Gopnik and Andrew Meltzoff. The development of categorization in the second year and relation to other cognitive and linguistic developments. In Child development, 1987.
[12] Goyal, Niekum, and Mooney. Using natural language for reward shaping in reinforcement learning. In International Joint Conference Artificial Intelligence (IJCAI), 2019.
[13] M. Hausknecht and P. Stone. Deep recurrent q-learning for partially observable mdps. In arXiv preprint
2015.
[14] Karl Moritz Hermann, Felix Hill, Simon Green, Fumin Wang. Ryan Faulkner, Hubert Soyer, David Szepesvari, Wojeiech Marian Czarnecki, Max Jaderberg, and Denis Teplyashin. Grounded language learning in simulated 3d world. In arXiv preprint
2017.
[15] Hengyuan Hu, Denis Yarats, Qucheng Gong, Yuandong Tian, and Mike Lewis. Hierarchical decision making by generating and following natural language instructions. In arXiv preprins 1rXiv:1906.00744 2019.
[16] U. Jain. S. Lazebnik, and A. G. Schwing. Two can play this game: Visual dialog with discriminative question generation and answering. In Proc. CVPR, 2018.
[17] Y. Jiang, S. S. Gu, K. P. Murphy, and C. Finn. Language an abstraction for hierarchical deep reinforcement learning. Advances Neural Information Processing Systems (NeurIPS), 2019.
11

---

[18] Stefano Ermon Jonathan Ho. Generative adversarial imitation learning, In arXiv preprint 13Xx:16606..33466v,. 2016.
[19] Iou-Jen Liu, Xingdi Yuan, Alex-nlex Côté, Pierre- Yves Oudeyer, and Alexander G. Schwing. Asking for knowledge: Training agents to query external knowledge using language. In International Conference on Machine Learning (ICML), 2022.
[20] Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward Grefenstette, Shimon Whiteson, and Tim Rocktäschel. A survey of reinforcement learning informed by natural language. arXiv preprint arXiv: /906003926 2019.
[21] C. Lynch and P. Sermanet. Grounding language in play. In arXiv preprint arXiv:2005.07648. 2020.
[22] Hongyuan Mei, Mohit Bansal, and Matthew R. Walter. Listen, attend, and walk: Neural mapping of navigational instructions to action sequences. In Association for the Advancement of Artificial Intelligence (AAAI), 2016.
[23] Suvir Mirchandani, Siddharth Karamcheti, and Dorsa Sadigh. Ella: Exploration through learned language abstraction. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[24] Dipendra Misra, John Langford, and Yoav Artzi. Mapping instructions and visual observations to actions with reinforcement learning. In arXiv preprint arXiv:1704.08770
2017.
[25] Jesse Mu, Victor Zhong, Roberta Railcanu, Minqi Jiang, Noah Goodman, Tim Rocktäschel, and Edward Grefenstette. Improving intrinsic exploration with language abstractions. In arXiv preprint arXiv:
2022.
[26] A. Y. Ng, D. Harada, and S. Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In International Conference on Machine Learning (ICML), volume 99, pages 278-287, 1999.
[27] Alexander Pashevich, Cordelia Schmid, and Chen Sun. Episodic transformer for vision-and- language navigation. In arXiv preprint (tt.:2/05.064550 2021.
[28] Roberta Raileanu and Tim Rocktäschel. Ride: Rewarding impact-driven exploration for
environments. In International Conference Learning Representations
(ICLR), 2020.
[29] Clement Rebuffel, Thomas Scialom, Laure Soulier, Benjamin Piwowarski, Sylvain Lamprier, Jacopo Staiano, Geoffrey Scoutheeten, and Patrick Gallinari. Data- Data-QuesttEva:: A referenceless metric for data-to-text semantic evaluation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8029-8036, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/0221 munlp-main.633 URL https:://aclatthhllleeeeooe///ee org.llloooooogg////// 2021 2021.emmnpp.........mmmmm..22m.p emnlp- main. 633
[30] Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. In arXiv preprint arXiv:1707.063477 2017.
[31] Thomas Scialom, Paul-Alexis Dray, Patrick Gallinari, Sylvain Lamprier, Benjamin Piwowarski. Jacopo Staiano, and Alex Wang. Questeval: Summarization asks for fact-based evaluation. In arXiv preprint arXiv:2 103. 22693v2, 2021.
[32] Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mot- taghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In arXiv preprint arXiv: 19Xiv:1912.01770 192.01134v2, 2019.
[33] R. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. MIT press, 2018.
[34] Allison Tam, Neil C. Rabinowitz, Andrew K Lampinen, Nicholas A Roy, Stephanie CY Chan, DJ Strouse, Jane Wang, Andrea Banino, and Felix Hill. Semantic exploration from language abstractions and pretrained representations. arXiv preprint
2022.
12

---

[35] Nicholas Waytowich, Sean L. Barton, Vernon Lawhern, and Garrett Warnell. A narration- based reward shaping approach using grounded natural language commands. In International Conference Machine Learning (ICML), 2019.
[36] Yunqiu Xu, Meng Fang, Ling Chen, Yali Du, Joey Tianyi Zhou, and Chengqi Zhang. Perceiving the world: Question-gue -uideed reinforcement learning for text- text-based games. In arXiv preprint 220. 2022.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See the Method and Experiments Section 4 and 5.
(b) Did you describe the limitations of your work? [Yes] See the Limitations paragraph of in Section 6
(c) Did you discuss any potential negative societal impacts of your work? [N/A] We do not foresee major negative societal impacts of this work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] We do believe our work conforms to them.
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] We give the URL in the introduction.
(b) Did you specify all training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See the Section 5 and the Appendix
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] We ran our main experiments with multiple random seeds and plotted curves with standard deviations.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See the end section 5.1
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] We use open-source environments, cited in the code and in the paper text.
(b) Did you mention the license of the assets? [Yes] We inherit the license in our attached
code repository.
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] We include the code of our approach
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "mb2JRm3yMKQ", "reviewer": "Reviewer_JBKz", "summary": "This paper presents a simple but novel reward shaping method for language-conditioned RL. The paper mainly proposes the auxiliary objectives for question generation and question answering models. Then they calculate the intrinsic reward which is proportional to its confidence in its answer. In the experiments, the authors showed that the proposed algorithm learns more efficiently on various tasks than the baseline algorithms.\n", "questions": "- In lines 280-281, it is expressed as without using expert knowledge, but is the expert dataset not used for pre-training when learning the QA module? (The paper explains as if it used less information than the ELLA, but it requires the same or similar information as the ELLA.)\n- If there is a pre-trained dataset (for the QA module), isn't it more efficient to use knowledge in the form of a prior policy learned from it? It seems that there should be an experiment about it.\n- What if it is used simultaneously with intrinsic reward methods such as RIDE? It is also important to check whether the effect of both methods is still helpful even if they are used at the same time.\n", "limitations": "The authors adequately addressed the limitations and potential negative societal impact of their work.\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The main idea of the proposed method in the paper is simple but novel. \n- The experiments include not only the results of sample efficient learning but also various analyzes.\n\nWeaknesses\n- A near expert dataset is needed for pre-training the QA module, which seems to be a quite strong assumption.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "kmuMuFlrjv9", "reviewer": "Reviewer_aQK3", "summary": "In this paper, the authors proposed a reward shaping method for language-conditioned RL. The authors proposed to generate a set of questions based on natural language instructions, and then evaluate how likely the current trajectory answers the questions. The probabilities of the questions being correctly answered are used as intrinsic rewards. In the experiment, the authors worked on Baby AI benchmark and showed that the proposed reward shaping method helped to improve sample complexity and model performance.", "questions": "Section 4.1 mentioned that if a question is answered, then it is removed from the set of questions. How to determine if a question is answered? Does the QA model have to predict the correct answer with high probability?", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe idea of using question generation and answering to do reward shaping for LC-RL is novel and interesting. It is like doing a self-check when a human follows a task instruction.\n\nThe paper showed that this idea works in the Baby AI benchmark.\n\nWeaknesses:\nWhile the idea is interesting, I think it is not sophisticated enough to generalized to more complex setting.\n\nFirst of all, the QG module generates a questions by masking a word (such as nouns and adjectives) in the linguistic instruction. However, for complex instructions, it is likely that the question can by answered by the masked instruction. For example, an instruction \"pick the [red ball], open the red door, ...., put the [red ball] on the chair\". If we mask the 'red ball' at the beginning of the sentence, we can still infer the answer from the 'red ball' at the end of the sentence. Therefore, even without the trajectory, the questions can be answered. I'd assume that it requires more than just masking words to generate a valid question.\n\nSecondly, the QA model is trained by using the full trajectory, however, when calculating intrinsic rewards, partial trajectories (trajectories up to the current timestep) are used. I would assume that when the partial trajectories are short, the QA model performance is bad. Have the authors investigated this?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "YG5NTaTm8bp", "reviewer": "Reviewer_piZu", "summary": "This paper proposes a new reward shaping method, EAGER, in Language-guided reinforcement learning (RL).  By extracting auxiliary Question Answering (QA) tasks, the agent is given intrinsic rewards based on how the generated trajectories could help answer these questions.  The authors claim that such auxiliary tasks can be designed without engineer interventions.  Specifically, the answer to these questions is the output of the pre-trained QA transformer model after feeding the generated trajectories.  For the BabyAI experiments, this QA model is pre-trained using example trajectories provided by the bot that can already solve the environment. The method is compared against two recent baselines RIDE and ELLA.  EAGER improves sample-efficiency in some tasks from BabyAI.", "questions": "- Is it possible to use behavior cloning to train an agent using those example trajectories from the bot?  This might serve a good baseline.\n", "limitations": "Yes, the authors have addressed the limitations", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n-This seems like a novel method that is inspired by reference-less metrics in NLP. The idea is interesting, and the authors provided detailed explanations on how each component, QA module, QA module, is implemented. \n- Overall, the paper is well-structured and clearly written from the method section to the experiment section.  \n- I believe the future work could be influenced by the idea of reference-less metrics proposed here.\n\nWeaknesses: \n- I think the experiment part is weak due to EAGER still using “engineer intervention” in BabyAI.  If I understand correctly, RIDE [1] does not use any expert information at all while EAGER currently requires a bot that can already solve some of the tasks from BabyAI.  In addition,  QG assumes the answer to the generated questions is one of the words that appear in the goal sentence.  The output of the QA model also requires prior knowledge about the potential words that can appear in the goal sentence though this might be okay if you assume these  are all nouns and you could extract them in advance.  I recommend adding a table to list all the assumptions and expert knowledge required for RIDE, EAGER, and ELLA [2] in a table.  \n- Although EAGER outperforms RIDE, RIDE does not use the bot from BabyAI so I think it’s not too surprising that EAGER can outperform RIDE all the time.  Since EAGER and ELLA both excel in certain tasks, it is difficult to judge whether the method is actually an improvement to ELLA given the reasons in the previous bullet point.  \n\nMinor points:\n- Title 5.2 “How does EAGER performs” → “How does EAGER perform” “Sparsity increase” → increases\n- Line245: I think it’ll be helpful to explain what these 3 input values represent in BabyAI.  How are objects represented in the gridworld.  \n\n\n[1] Roberta Raileanu and Tim Rocktäschel. Ride: Rewarding impact-driven exploration for 418 procedurally-generated environments. In International Conference on Learning Representations 419 (ICLR), 2020\n\n[2] Suvir Mirchandani, Siddharth Karamcheti, and Dorsa Sadigh. Ella: Exploration through learned 406 language abstraction. In Advances in Neural Information Processing Systems (NeurIPS), 2021.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Exkbrib98-8", "reviewer": "Reviewer_ZfBg", "summary": "The paper proposes an auxiliary reward scheme (EAGER) for instruction following RL (BabyAI framework) based on question generation (QG) and answering (QA). Concretely, QG is based on masked word modeling of the instruction, and QA is pre-trained with demo instructions and trajectories and fixed when training RL. The intuition is good and successful RL agent trajectories should make QA correct and easy, thus when QA is successful, an auxiliary reward proportional to correct answer confidence is given. The method is evaluated on several BabyAI tasks and shown on par with other methods (ELLA, RIDE) with more expert inductive bias for subgoal specification and judgement. Further analysis shows EAGER is fairly robust to QA performance, but some QA design choices are important.", "questions": "\nSome minor points:\n\n- Maybe in abstract, mention the domain (babyAI)?\n- Is there any ablation for the policy invariance thing? ", "limitations": "They are discussed in the paper, see the weakness part.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\n- The idea is neat, novel, effective, and provides a means to auxiliary reward for instruction following only using example instructions and trajectories, and minimal linguistic assumptions (what is noun/adj). In contrast, previous methods usually require domain knowledge about subgoal formulation/proposal/completion judgement. \n\n- Some design choices for QA are reasonably explained in theory (e.g. policy invariance) and practice (e.g. no_answer, using confidence instead of binary reward).\n\n- The motivation and idea are well explained, and the paper is well structured and easy to read for most parts.\n\n\nWeakness:\n\nAs mentioned in the last part of the paper, two main limitations (and some of my extensions) are \n\n- QA needs expert trajectories with instructions to pre-train. Arguably it is a weaker assumption compared to prior work, but I feel two important ablations are missing: direct PPO without the auxiliary reward (a trivial low-bound baseline), and just using the QA training trajectories to do imitation learning and warmstarting for RL agent (a trivial way to use these pre-training trajectories). \n\n- BabyAI is perhaps too simple, and experiments seem a bit limited. I can see some sample efficiency advantage on PutNext tasks, but ELLA seems better than EAGER on Unlock and Sequence. What about other tasks in BabyAI, let alone more complex tasks like Alfred (which I feel this method might not easily work, given the visual observation and language instruction are both much harder). Some more experiments or tasks would make me believe the method is more general and robust.\n\nAlso pointed out in the paper (line 295),'\n\n- EAGER might not work well on Sequence or tasks with temporal subgoals. In general, I don't have an intuition what properties of env/task does EAGER work better on and what not. Some analysis or examples of agent behavior or even how auxiliary reward changes with time would give me more concrete ideas.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept