Natural Language Instruction-following with Task-related Language Development and Translation
Authors: Jing-Cheng Pang
Xinyu Yang
Si-Hang Yang
Xiong-Hui Chen
Yang Yu
Venue: NeurIPS 2023 poster (2023)
Keywords: Reinforcement learning
instruction-following
autonomous agent
TL;DR: We propose a reinforcement learning algorithm for enabling efficient policy learning in natural language instruction following.

Abstract: Natural language-conditioned reinforcement learning (RL) enables agents to follow human instructions. Previous approaches generally implemented language-conditioned RL by providing the policy with human instructions in natural language (NL) and training the policy to follow instructions. In this is outside-in approach, the policy must comprehend the NL and manage the task simultaneously. However, the unbounded NL examples often bring much extra complexity for solving concrete RL tasks, which can distract policy learning from completing the task. To ease the learning burden of the policy, we investigate an inside-out scheme for natural language-conditioned RL by developing a task language (TL) that is task-related and easily understood by the policy, thus reducing the policy learning burden. Besides, we employ a translator to translate natural language into the TL, which is used in RL to achieve efficient policy training. We implement this scheme as TALAR (TAsk Language with predicAte Representation) that learns multiple predicates to model object relationships as the TL. Experiments indicate that TALAR not only better comprehends NL instructions but also leads to a better instruction-following policy that significantly improves the success rate over baselines and adapts to unseen expressions of NL instruction. Besides, the TL is also an effective sub-task abstraction compatible with hierarchical RL.

Natural Language Instruction-folloinng with Task-related Language Development and Translation
Jing-Cheng Pang", Xinyu Yang", Si-Hang Yang, Xiong-Hui Chen, Yang Yu! National Key Laboratory of Novel Software Technology, Nanjing University Polixir Technology
{pangjc, yanggx, baanggyyyanghh. chenxh) ajj.edu.. cn, yuyānju. eoydnnu.edu..oo edu.
Abstract
Natural Iinguage-conditione reinforcement learning (RL) enables agents to fol- low human instructions. Previous approaches generally implemented language- conditioned RL by providing the policy with human instructions in natural language (NL) and training the policy follow instructions. In this is outside-in approach, the policy must comprehend the NL and manage the task simultaneously. How- ever, the unbounded NL examples often bring much extra complexity for solving concrete RL tasks, which can distract policy learning from completing the task. To ease the learning burden of the policy, investigate inside-oul scheme for natural sunguage-conditionn task-related and easily understood by the policy, thus reducing the policy learning burden. Besides, we employ translator to translate natural language into the TL, which used in RL to achieve efficient policy training. We implement this scheme as TALAR (TAsk Language with predicAte Representation) that learns multiple predicates model object relationships as the TL. Experiments indicate that TALAR not only better comprehends NL instructions but also leads to better eestruction-folloiinn policy that significantly improves lines and adapts to unseen expressions of NL instruction. Besides, the TL is also an effective sub-task abstraction compatible with hicrarchical RL.
RL by developing task language (TL) that is
success rate over base-
Introduction
Enabling robots collaborate effectively with humans is a crucial aspect of machine intelligence. Natural language is potent tool that reflects human intention and has been widely researched for instructing robot execution, designing rewards, and serving as an observation or action in reinforcement learning [R]) We are particularly interested in developing agents to follow human instructions in this broad context. Natural language-conditioned reinforcement learning (NI.C-IL) is promising approach to training instruction- -owructiiooooooiiiggg. agents. It provides the policy with human instructions natural language (NI.) and trains to follow instructions with RL. algorithms. outside-in learning (OIL) scheme, the natural language instructions are directly exposed to the policy, requiring it to understand and interpret them while accomplishing RL tasks (Fig. I-left).
this
However, natural language is a complex and unbounded representation of human instruction. It places an additional burden on the policy to understand diverse natural languages when solving specific RL task. To address this issue, previous studies have attempted to convert the natural language to simpler latent embeddings [2] using pre-trained language models such as BERT [3] and GPT [4]. However, these embeddings are typically learned independently of the RL task, making capturing the reltedd information in NI instructions difficult. Alternatively, some methods
'Equal contribution. Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

[5, 6, 7] design rule-based, task-related semantic vectors to represent NL instructions. These methods show promising results but require a laborious design of semantic vectors and have limitations in handling complex NL instructions.
This paper proposes novel Inside-Out Learn- ing (IOL) scheme for NLC-RL. As shown in Fig. I-right, IOL automatically generates task language (TL) that is both task-related and con- cise, enabling easy comprehension by the policy. IOL consists of three key components: (1) TL generator that develops task language. (2) translator that translates NL into TL, which facilitates policy learning in RI., and (3) a policy that solves the RI. tasks. We introduce spe- cific implementation of IOL., called TALAR for TAsk Language with predicAte Representation. In particular, TALAR models the TL genera- tion process as referential game, known for its effectiveness in developing effective language. Besides, TALAR learns to model the relation- ships between objects as TL. Given the labelled NL to TL, the translator is trained to minimize the difference between the translated TL with the target TL. Finally, arbitrary RL policy optimization algorithms can optimize the instraction-follooiing policy.
Llck red bull
lock guage develeament
human
natural anguage anooding
Nummnn
tasik banguage
mandstne
apen
policy training
policy training
Outside-in scheme
Inside-out scheme
Figure 1: An illustration of OIL and IOL schemes in NLC-RL. Left: OIL, the policy directly takes the (outside) NL instructions embedding as the input. Right: IOL first develops an (inside) TL, and policy takes TL as the input. The dashed lines represent TL development and translation, while the solid lines represent the instruction following the process.
Our paper presents several contributions to the field of NLC-RL. Firstly, we propose a novel NLC- RL scheme called IOL, which introduces a task language that enhances policy understanding and improves the learning efficiency of cestructiio-following
Secondly, we introduce specific im- plementation of IOL that models TL development as a referential game and uses multiple learned predicates to represent TL. Thirdly, we conduct comprehensive experiments in two environments, FrankaKitchen [8], and CLEVR- Robbot [9], demonstrating that the policy learned by our proposed method outperforms existing approaches in terms of its ability to follow natural language instructions and adapt to previously unseen instructions. Furthermore, the resulting TL effectively represents human instructions, providing solid baseline for sub-task abstraction in hierarchical RL [10].
2 Related Work
This section begins with a summary of prior research on instruction following with RL, followed by two paragraphs discussing works pertinent to our methodology. i.e., language generation in RL and language translation.
Instruction following with RL. Instruction Instruction-follloingg problems require agents to perform tasks specified by natural language instructions. A line of works solves the problems with semantics vectors policies [14, 15, 16]. To train the policies, previous methods typically process NL using a language model and train a following policy with RL. algorithms. For example, [17] encodes a single-sentence instruction with pre- ire-trained language model and feeds the policy with the NL encoding. [18] learns a policy that maps NL instructions to action sequences by marginalizing implied goal locations. [19] combines human instructions with agent observations via multiplication-bas based mechanism and then pre-trains the instruction-following policy using behaviour cloning [20]. Instead of directly converting NL to encoding. [5] encodes NI to manually-designed binary vector in which each element has semantics meaning. Besides, the instruction- -folloiinnn policy has close ties to Hierarchical RL [10] because the instructions can be naturally viewed as a task abstraction for a low-level policy [21]. HAL [9] takes advantage of the compositional structure of NL and makes decisions directly at the NL level to solve long-term, complex RL tasks. These previous methods expose the unbounded NL instructions directly to the policy or encode the NL instructions to scenario scenario-specifie manual vector, both of which have limitations. In contrast, our method automatically develops a task language, which is easily understood by the policy and facilitates policy learning.
[11, 12, 13], or Lunguage-conditionn
Language generation in RI.. Language generation plays a crucial role in developing task language. In reinforcement learning, researchers often approach this issue in a multi-agent setting, where agents
2

---

must learn effective message protocols to communicate with their partners and complete tasks. For example, the study by [22] investigates emergent language in a referential game. At the same time, line of works has explored multi-agent communication [23, 24] for practical cooperation, considering the communication message as language. Regarding language representation, prior research has often used discrete vectors due to the discrete nature of human language. For example, [25] enables agents to communicate using discrete messages and shows that discrete representation performs similarly to continuous representation with much smaller vocabulary size. One-hot representation [26, 22] and binary representation [5, 27] are popular forms of discrete language representation. For instance, [26] uses one-hot language representation to enable two agents to communicate and differentiate between images. In this paper, we develop task language following the discrete form of language representation while using the predication representation.
Language translation. In this paper, TALAR translates natural language into task language, which lies in the domain of language translation [28]. The field of natural language processing (NLP) has extensively investigated various approaches to language translation [29, 300. Among these approaches, the encoder-decoder architecture is promising method because it can extract effective features from input sentences [31]. For example, the study in [32] introduces a continuous latent variable as an efficient feature for language translation, utilizing variational auto-encoder [33]. In this paper, our primary focus is not on language translation. Therefore, we employ a more straightforward method for translation, utilizing the pre-trained BERT model as the NL encoder. In this approach, we consider NL as the source language and the target language (TL) as the target language.
3 Background
3.1 RL and NLC-RL
A typical RL task can be formulated as a Markov Decision Process (MDP) [34,35], which described a tuple M = (S, A. P.r, do). Here S represents the state space. AA is the finite action space defined by AA = {oo. a1, ... -- P represents the probability transition while r represents the reward function. 7 is the discount factor determining the weights of future rewards, whereas do is the initial state distribution. A policy π S + (AA) maps state space to the probability space over action space. In NLC-RL, the agent receives NL instruction (T. that reflects the human's instruction on the agent. An instruction- followiinn policy (-|s;; L) is trained to make decisions based on the current state 9; and NL instruction L. The overall objective of NLC-RL is maximize the expected return under different NL instructions:
y'r(s,, ae, 10) do, a (-18,, l.). t=0
(1)
For the accuracy of sake, use LN and Lt to denote NL and TL, respectively.
3.2 Referential Game
Referential game (RG) has been established as multi-agent communication task [22], which consists of two agents: a sender and a receiver. The two agents communicate using their language to identify a particular object or concept a specific task. In a classic referential game, a pair of images is presented to the sender and the receiver, with the sender being informed which of the two images is the target. The sender must generate an effective communication message that enables the receiver to identify the target image successfully. Through interactive communication, the agents develop meaningful language. As the sender must capture the key feature of the task, the RG presents an ideal setting for generating task language. In TALAR, which will be claborated in Section 4.
extension to the basic RG to generate TL,
3.3 Predicate Representation
Predicate representation utilizes discrete binary vectors to describe the relationships between objects or abstract concepts. The predicate representation formally defined as comprising two key com- ponents: (1) predicate signifies the relationship between objects or concepts, which may involve one or multiple objects. (2) argument serves as the input for the predicate, indicating the specific objects between which the predicate measures the relationship. By employing these components,
3

---

predicate representation offers a clear and concise depiction of the relation between objects. For instance, the predicate representation vector* [1. 1, 0, 0, 1. 0] could represent a predicate expression Pred (a, (a,b). bb) In this case, Pred is predicate that signifies a relationship, and the symbols and b are its arguments. relationship holds). In contrast, the following red and blue one-hot codes represent the indexes of arguments a and b, respectively.
the vector, the initial code [1] indicates that the value of Pred is True (i.e., the
Prior research has shown that predicate representation can be learned by employing networks to output predicate values. In this manner, the learning system can automatically identify relationships in block stacking task [36]. In TALAR, neural networks learn both predicates and their arguments. For additional discussions on predicate representation, refer Appendix A.
3.4 Natural Language as Input for the Neural Network
NL sentences are variable lengths and cannot be fed directly into fully connected network. standard solution is to encode each word as an embedding [37] and loop over each embedding using a recurrent model. Besides the recurrent model, the pre-trained language model, e.g., Bert [3], tokenizes the words in the sentence and extracts sentence embedding based on these tokens. This process involves passing the input sentence through series transformer-basee encoder layers that capture contextual information and create representations of the input tokens. BERT then applies pooling layer to obtain a fixed-length vector representation the input sentence. In this study, we employ Bert process NL sentences fixed-length vectors because it is lightweight and efficient.
4 Inside-out Learning and Implementation
This section introduces our primary contribution to efficient policy learning in NLC-RL, the IOL framework and its specific implementation, TALAR. We begin by formulating the IOL framework.
4.1 Inside-out Learning Formulation
We consider NLC-RL tasks, where the agent is provided with natural language instruction goal to be completed. The key idea of IOL is to develop a task-related language to facilitate policy understanding. To implement IOL, three key components must be considered in IOL. We will elaborate on these components and discuss their impact on solving NLC-RL tasks.
TL generator. The TL generator is the critical component in IOL, designed to develop a task-related language. We expect the resulting TL to effectively and succinctly convey the task's 's objective to the policy. In this way, IOL mitigates the issues encountered in OIL, where the policy struggles to comprehend unseen natural language instructions. We believe that expressiveness and conciseness vital properties of TL. Expressiveness ensures the TL accurately conveys the task goal, while conciseness promotes policy comprehension. To fulfil these requirements, we propose representing TL using predicate representation, considered both expressive and concise [38].
Translator and Policy learning. The translator serves as a tool that connects policy with TL, which translates NL to TL. Utilizing the translator, the policy can be optimized through various RL algorithms to improve its ability to follow human instructions. The policy learning is goal- conditioned RL [39] problem. For complex tasks, plenty of well-established techniques in machine translation and goal-com coal-conditioned RL domains can be employed to improve policy learning in IOL further. Note that the primary focus of this paper is to investigate the effectiveness of IOL. Therefore, we employ straightforward translator and policy learning implementations, as detailed in Section 4.2.
4.2 Our Method
We first introduce the task dataset, which is used for TL. development and translation in TALAR. Definition 1 (Task dataset). A task dataset D = {(s, s', LN)s) consists of multiple triplets. Each triplet contains a natural language instruction Ln and task state pair (8, 8'), where Lx describes stute chunge from to s' in natural language, e.g., move the red bull to the blue ball.
Thhe vector could be extended represent multiple predicate relationships simultaneously.

---

(c)
(a)
(b)
Referential Game
LRG
Multi-laee perceptron
Prodhet,
PredNet,
Prediett
Receiver
ss
Argument list
Concatenation
IN
encoding
Pre-trained language model
TL generator
arg:
ang
Gradient detachment
Translator
ArgNet
ArgNetz
LN
LN
(s,s')
(s.s))
Figure 2: Overall framework of TALAR. (a) Overall training process: The TL generator develops TL via playing referential game with receiver, and the translator translates NL to TL. (b) Network architecture of the TL generator. (c) Network architecture of the translator. The number of predicate arguments and networks can be adjusted according to the task scale.
We use state pair instead of single state for the following reasons: (1) NL instruction typically describes a state change, e.g., turning the wheel to the left; (2) it is not straightforward to describe single state concisely complex task scenarios. Fig. 2 illustrates how TALAR makes use of task dataset for task language development and translation. The subsequent subsections will elaborate on training three critical components of TALAR, i.e., task language development, natural language translation and policy optimization.
4.2.1 Task Language Development in Predicate Representation
The TL generator denoted as go 8'), aims to develop language in predicate representation (Section 3.3). In TALAR, the TL generator is represented by neural networks and parameterized with 0. It takes a state pair (8, s') as the input and outputs task language Lr- The We will discuss how TALAR constructs referential game to facilitate the development of task language.
Referential game. The RG process presented in the right part of Fig. 2. Based on the RG framework in Section 3.2, TALAR adapts it such that the TL. generator serves as the sender 4 and generates the TL LT (s,s,'') s') to describe the state change of the given state pair. Conditioned on the generated TL Lr. a receiver aims to predict the word in the NL instruction corresponding to (8, d'). In this way, the TL generator must capture the critical information the given state pair and generate effective TL, ensuring that the receiver can predict correctly. We use pre-trained BERT model followed by fully-connected layer to implement the receiver to predict the next token. We denote this as (T, LT, ((..),...-))), where f is the fully-connected layer after the BERT output, b is the BERT model, T; is the i-th token of the sentence, and represents the parameters of the fully-connected layer. The token to be predicted T, is randomly sampled from the whole sentence. The objective for training the TL generator and receiver is to minimize the negative log- likelioiddood of the predicted token given the generated TL and the context tokens, which is expressed as:
LRG (0,
| log (Ta s'), ((T......)))))
(2)
(a.5' L.))
where Tzy is the token set of Lx. In practice, we fix the parameters of BERT and only optimize the fully-connected layer.
Network architecture of TL generator. We design a specific network architecture for learning the predicate representation (we refer readers to Section 3.3 for elaborations on predicate representation). As illustrated in Fig. 2(b), the TL generator first extracts N, arguments (arg, arg....... , arg based on the input state pair, and subsequently determines the Boolean values of Np predicates, given the extracted argument list. The predicate values concatenated with the argument list and form the task language Lip = (pred,.". : pred N,n arg ... arg ). The number of predicate networks Npn and arguments networks N₂ can be adjusted according to the RL task scale.
"To avoid confusion, will "TL generator" instead of "sender" for the remainder of this paper.
5

---

Specifically, the arguments for the predicate are extracted by argument networks, denoted by ArgNet,ss ss.).. An argument network implemented as fully-connected network ending with Gumbel-Softmax activation layer [40]. Through the Gumbel-Softman the argument network can output a discrete one-hot vector arg, in form like (0, 1, 0), which represents an ab- stract object concept in the task. TALAR utilizes multiple predicate networks, denoted by PredNet,(s; s', argg, ... argN,). to determine the Boolean values of a set of predicates. These predicates are anonymous we do not need to pre-define them in advance. This learning manner requires predicate networks to automatically discover meaningful relationships in the task. Each predicate network outputs 0-1 value, ending with Combel--oftmas layer. All these 0-1 values concatenated with the argument list, yielding the task language Lr, a discrete binary vector. Note that without the argument list in Lr, the resulting language cannot express different objects and therefore loses its expressiveness. The Gumbel-Softme activation technique permits the differentiation of the entire TL generation procedure.
4.2.2 Natural Language Translation
The objective of the translator is to translate the natural language to the task language. TALAR uses a pre-trained BERT model convert Lx into fixed-length embedding, followed by fully-connected layers to predict the task language corresponding to IN. Fig. 2(d) presents the structure of the translator. We let LT denote the TL generated by the translator, and Po the fully-connected layers parameterized with ạ). The translator trained to minimize the difference between the predicted TI. with the target TL: CT(0) (a.*' -x)~ [- log pg (')((((((T))))]
Here T(NN) denotes the tokenized natural language sentence. The optimized translator trains an instruction-following policy complete the human instructions, as described below.
4.2.3 Policy Optimization
TALAR uses reinforcement learning to train an Instruction Instruction-Following Policy (IFP) TT Lr). When the agent collects samples from the environment, the task generates random human instruction in NL, which is then translated into the task language LT by the translator. Next, the IFP makes decisions for the entire episode based on the current observation and LT until completing the instruction or reaching the maximum timestep. The IFP can be optimized with an arbitrary RL algorithm using the samples collected from the environments. We use PPO [41] for TALAR and all baselines in our implementation. Note that during IFP training, the translator's parameters fixed to prevent the translator from overfitting the current IFP.
Experiments
We conduct series of experiments to evaluate the effectiveness of TALAR and answer the following questions: (1) How does TALAR perform compared to existing NLC-RL approaches when learning an cnerruction-oollwwiii
policy? (Section 5.1)
(2) Can TALAR learn effective task language?
(Section 5.2) (3) Can TL acquire any composi- tional structure and serve as an abstraction for hierarchical RL? (Section 5.3) (4) What is the impact of each component on the overall perfor- mance of TALAR? (Section 5.4)
We conduct experi- ments in FrankaKitchen [8] and CLEVR-Robor [9] environments, as shown in Fig. 3. FrankaK- itchen aims control 9-DoF robot to manip- ulate various objects in a kitchen, involving six sub-tasks: open the microwave door, move the kettle to the top left burner, turn on the light switch, open the slide cabinet, activates the top burner and activate the bottom burner. We treat
Evaluation environments.
FrankaKitchen
(b) CLEVR-Rohot
Figure 3: A visualization of environments in our experiments. (a) FrankaKitchen. The agent con- trols 9-DoF robot manipulate various objects a kitchen. (b) CLEVR-Robo The agent (sil- verpoint) manipulates five movable balls to reach a specific goal configuration An example of NL instruction is: Can you move the red ball to the left of the blue ball?

---

Will Ciltiotonn
Tharap
Teuskg
FrankaKitchen-TT Train
FrankaKitchen-Tean
LLEVR-Robot--Taiie
Figure 4: Training curves of different methods on two evaluation environments. The x-axis represents the timesteps agent interacts with the environment, and the y-axis represents the success rate of completing the instructions. The shaded area stands for the standard deviation over five random seeds.
each sub-task as a different goal configuration. In each trajectory, the environment randomly generates an NL instruction that describes goal configuration. We let ChatGPT [42] generate 50 different NL instructions to describe each goal configuration, yielding 300 different NL instructions in this task. CLEVR-Robo an environment for object interaction based on the MuJoCo physics engine [43]. It contains five movable balls and an agent (silver point). In each trajectory, the agent aims to achieve a goal configuration that requires moving specific ball in target direction (front, behind, left, right) relative to a target ball. An example of NL instruction is Move the green ball to the left of the blue ball. There are 80 different goal configurations in total. We use 18 natural language sentence patterns to describe each goal, resulting in 1440 distinct NL instructions. We split the NI. instructions into two tasks: training and the testing set. The training for each goal configuration in FrankaKitchen and CLEVR-Robar respectively, resulting in a total of 240 NL instructions for FrankaKitchen and 720 for CLEVR-RoboL. During training, the agent can only interact with the NL instructions in the training set. The testing set consists of the remaining NL instructions, which include 120 NL instructions for FrankaKitchen and 720 for CLEVR-RoboL Refer to Appendix D.1 for more details about the environments and the NL instructions in the training and testing sets.
comprises 40 and 9 different NL instructions
Task dataset collection. The task dataset can be obtained from pre-collected data, where state pairs annotated with natural language instructions through human input mulebbase functions. In our experiment, we utilize PPO to train a goal-conditioned policy capable of achieving any goal configuration. We then the trained policy collect trajectory data (150, Gg $1, $1 ... ST) Ln) across various environments, where Ln is the natural language that describes the goal configuration of this trajectory and is randomly sampled from the training instructions set. The collected trajectory data rearranged, with {(N;, 8T, Lx) T= task, and 100,000 for the CLEVR-Robor task. Note that the collected trajectory the FrankaKitche
tuples in the task dataset, resulting in 50,000 tuples for
data can be used pre-train the policy. With tuple data such (s,, 04 IN), the policy x (ss, Lx) can be pre-trained to output the action a. Two baselines (Pre-trained BERT- in Section 5.1) utilize this data for pre-training purposes.
Implementation Details. For experiments, we utilize the pre-trained BERT-base base---as--nceeed model[3] the NL encoder. We conduct all experiments with five random seeds, and the shaded area in the figures represents the standard deviation across all five trials. We refer readers to Appendix for more implementation details about the experiments.
5.1 Performance of Instruction-ffolloingg Policy
Baselines for comparison. We consider multiple representative methods in NLC-RL as baselines:
(1) One-hot encodes the representation of all possible natural language instructions (including training and testing) to a one-hot vector, where each instruction is assigned a unique position. (2) BERT-binary processes the natural language with pre-trained BERT model, and the resulting language encoding is then transformed into binary vector using a fully-connected network. This binary vector's size equals the TL generated by TALAR. To ensure differentiability. we employ reparameterization trick [44] that converts continuous vector into binary vector. (3) BERT- continuous is similar to BERT-binary, except it uses continuous vector of the same size. (4) Pre-trained BELRT-binary/ooniinous initially pre-trains the policy networks and the BERT model using behavior cloning [45] with pre-collected (8;, 0z IN) tuples, and subsequently trains the policy networks with RI. The data collection process described above alongside the task dataset collection. (5) GLA (Grounding Language to latent Action) [46] first employs auto-encoder to learn latent
7

---

open
open The Inkoowan cody
(a) TALAR
(b) GLA
(c) BERT
(d) OIL baseline
Figure 5: The t-SNE projections of different NL encoding types on FrankaKitchen. Points with the same marker encode forty different NL expressions that describe the same goal configuration. We add a slight noise to the overlapping points for better presentation. (a) The translator's t-SNE representations of the TL output. (b) The encoding output by the language encoder in GLA. (c) The encoding output by the NL encoding layer the BERT model. (d) The encoding output by the NL encoding layer of the Pre-trained BERT-continuous.
action embedding and then encodes the NL into the learned latent action space, which is binary vector with the same size as TL.
Experimental results. Fig. presents the training curves of the different natural language instruction datasets. In general, TALAR achieves a superior instruction- following policy that enhances the success by 63.2% FrankaKitchen and 13.4% CLEVR- Robot compared to baseline method and generalizes well to the NI instruction expressions that were previously encountered. On the FrankaKitchen task requiring more difficult robot manipulation. TALAR significantly outperforms all baselines. This result demonstrates the advantage of IOL, which eases the policy's burden on understanding natural language. On the training NL instruction set, TALAR achieves high success rate within a reasonable number of timesteps, significantly faster than the other baselines. Additionally, TALAR demonstrates a remarkable success rate on testing sets in both environments, showcasing greater capacity than the baselines. Although One-hot performs adequately on the training NL set, its ability to generalize to the testing NL sets is limited. BERT-based baselines, built on the OIL. framework, improve slower than TALAR on the training NL instruction set. addition, the NL instructions on the FrankaKitchen task are typically more complex. OIL. methods become increasingly more challenging to achieve high task success rate.
policies
5.2 Analysis on the Generated Task Language
We further investigate the reasons behind TALAR's more efficient learning compared baseline methods. To explore this issue, we employ the optimized translator to convert NL instructions into TL and project the resulting TL onto a dio-dimensional plane using t-SNE [47]. The projection results are displayed in Fig. 5a. The results show that the learned TL effectively distinguishes NL instructions representing different goal configurations, developing unique representation for each configuration This conclusion is supported by the fact that various NI. expressions for the same goal configuration translate into similar, or even identical, TL. Consequently, TALAR can easily recognize these expressions and concentrate on learning how to complete the RL task. For comparison, we also project the NL encoding output generated by other baseline methods, as illustrated Fig. 5(b-d). The NL encoding output from these baseline methods exhibits greater diversity compared to TALAR. These results suggest that the OIL baseline treats diverse NL expressions for the same task objective as different goal configurations, potentially distracting from RL and slowing policy learning. It is worth noting that BERT, an established pre-trained model for natural language understanding, also fails to generate unique NL encoding. Even when fine-tuned on specific tasks, the output encoding of the BERT model is divergent, as shown in Fig. 5d.
5.3 TL Is an Effective Abstraction for Hierarchical RL
Previous experiments have demonstrated that the resulting TL can compactly and uniquely represent the NL instructions, facilitating policy learning in the following instruction. In this section, we further explore whether the TL learns any compositional structure by examining if it can serve as an effective goal abstraction for hierarchical RI. Specifically, we train a high-level policy outputting a TL, instructing the IFP to complete low-level task. We define long-ten task as in which the

---

Tracding
Twrtin
TALAR
140
HALL
EALL
14:
Epliodic Revurd
42
0.2
00
02
3.5
025
1.
Timesepo
Timestpps
Nyn
N.
(a) CLEVR-
(h) FrankaKitchen
(c) Ablation study
Figure Experiments of long-horizon tasks on different environments. The agent obtains reward of +1 when completing one goal configuration.
Figure 7: Ablation study on the number of predicate and argument networks.
agent must achieve multiple goal configurations simultaneously. For CLEVR-Robor we constructed the long-term task following the approach outlined in [9]. In this task, the agent's objective is to arrange objects to satisfy all goal configurations that are implicitly contained within it. For FrankaKitchen, the agent is required to complete all six sub-tasks at the same time. Refer to Appendix D.1.3 for additional details regarding the long-term tasks and the hierarchical RL setting.
For comparison, we consider a baseline method called HAL [9]. which leverages the compositional structure of NL and directly acts at the NL level. Following HAL's original implementation, the high-level policy outputs the index the NL instruction rather than the NL words. To ensure fair comparison, we align all other modules except for the high-level policy of both methods, e-g., using the same IFP trained by TALAR as the low-level policy. Fig. 6 presents the comparison results. The high-level policy that uses TL low-level goal abstraction outperforms that using NL in terms of sample efficiency. This result demonstrates that TL can be an effective goal abstraction naturally compatible with hierarchical RL. In practice, the TL be generated with task dataset and serves as sub-task abstraction for high-level policy.
5.4 Ablation Study
We conduct experiments investigate how the performance of TALAR is affected by varying the number of predicate networks and the number arguments (i.e., Np and N.). Specifically, we choose Nr from the set (2, 4, 6) and N, from 3, 4). We trained the IFPs for 4 million timesteps on the FrankaKitchen environment. The experiment results are presented in Fig. 7. In general, increasing Npn and N, leads to improved performance of the IFP on both the training and testing sets. This result can be attributed to the enhanced representation abilities of the resulting TL, which enables more efficient learning by the policy. However, it worth noting that excessively high values of Np and N, may lead to reduced training efficiency due to the simultaneous learning of multiple networks. Npa and N₂ 4 sufficient for the robot manipulation tasks like FrankaKitchen.
Conclusion and Future Work
In this study, we introduce novel learning scheme for NLC-RL, called IOL, which develops task- related language to facilitate policy learning. Our experiments demonstrate that IOL is a promising approach for efficiently training instraction-folloiing agents. However, there limitations in our proposed implementation of IOI., TALAR. Firstly, TALAR develops the task language using a static task dataset and, therefore. can not be directly applied to the open-world environment, where the NL instructions could vary hugely from the dataset. It is possible to mitigate this issue by dynamically extending the task dataset and fine-tuning the TL generator/translatrr during the policy learning process. Additionally, TALAR requires manual reward function for policy training, which may be inaccessible the reward design complex. Fortunately, there have been vellivaldd methods for solving sparse reward problems [48, 49, 50], an effective substitute for the manual reward function. Furthermore, incorporating basic properties of predicate relationships (such as transitivity, reflexivity, and symmetry) when training the TL generator could make the resulting TL more meaningful and self-contained. Lastly, TALAR implements the translator as a fully-connected layer with a pre-trained BERT model, which may be inadequate for more complex natural language instructions. This

---

limitation could be addressed by employing other Natural Language Processing (NLP) literature techniques, such as semantic parsing.
Recently there has been a growing interest in building instruction-folloviig agent with LLMs as the brain [51, 52, 53]. Typically, LLMS function as planners, while low-level policy is responsible for executing the planned tasks. However, the separation between the LLM planner and the task execution process may impede the overall task completion complex language instructions in open environments remains an unresolved issue. Furthermore, is worth investigating whether LLMs can serve effective task language generator. We hope that future research will explore these intriguing questions and contribute to developing agents that interact with humans more effectively.
The challenge of managing increasingly
Acknowledgements
This work is partly supported by the National Key Research and Development Program of China (202AAAA0107200) and the National Science Foundation of China (61921006). The authors extend their appreciation to Wang- Zhou Dai and Yuxuan Huang for their valuable discussions predicate representation, Xinchun and Yikai Zhang for their insights on language model training. Jingxuan Han for experiment details, Lei Yuan for discussing the potential applications of TALAR, and the anonymous reviewers for their useful feedback on this study.
10

---

References
[1] Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob N. Foerster, Jacob Andreas, Edward Grefenstette, Shimon Whiteson, and Tim Rocktäschel. A survey of reinforcement learning informed by natural language. IJCAI, 2019.
[2] Felix Hill, Sona Mokrá, Nathaniel Wong. and Tim Harley. Human instruction-following with deep reinforcement learning via transfer-leaming from text. arXiv, 2005.09382, 2020.
[3] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT. Association for Computational Linguistics, 2019.
[4] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurlPS, 2020.
[5] Ahmed Akakzia, Cédric Colas, Pierre-Yves Oudeyer, Mohamed Chetouani, and Olivier Sigaud. Grounding language to autonomously-acqqinn skills via goal generation. ICLR, 2021.
[6] Muhannad Al-Omari, Paul Duckworth, David C. Hogg, and Anthony G. Cohn. Natural language acquisition and grounding for embodied robotic systems. In Proceedings of the Thirty- AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA, 2017.
[7] Stefanie Tellex, Thomas Kollar, Steven Dickerson, Matthew R. Walter, Ashis Gopal Banerjee, Seth Teller, and Nicholas Roy. Approaching the symbol grounding problem with probabilistic graphical models. Al Magazine, 32(4):64-76, 2011.
[8] Abhishek Gupta, Vikash Kumar, Corey Lynch. Sergey Levine, and Karol Hausman. Relay policy learning: Solving Long-horizon tasks via imitation and reinforcement learning. In CoRL, 2019.
[9] Yiding Jiang, Shixiang Gu, Kevin Murphy, and Chelsea Finn. Language an abstraction for hierarchical deep reinforcement learning. In NeurIPS, 2019.
[10] Andrew G. Barto and Sridhar Mahadevan. Recent advances in hierarchical reinforcement learning. Discrete Event Dynamic Systems, 13:41-77, 2003.
[11] Matt MacMahon and Brian Stankiewicz. Human and automated indoor route instruction
following. In COGSCI, 2006.
[12] Stefanie Tellex, Thomas Kollar, Steven Dickerson, Matthew R. Walter, Ashis Gopal Banerjee, Seth J. Teller, and Nicholas Roy. Understanding natural language commands for robotic navigation and mobile manipulation. In AAAI, 2011.
[13] David L. Chen and Raymond J. Mooney. Learning interpret natural language navigation instructions from observations. In AAAI, 2011.
[14] S. R. K. Branavan, Harr Chen, Luke Zettlemoyer, and Regina Barzilay. Reinforcement learning for mapping instructions to actions. ACL, 2009.
[15] Jacob Andreas and Dan Klein. Alignment-based compositional semantics for instruction following. In EMNLP, 2015.
[16] Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian D. Reid, Stephen Gould, and Anton van den Hengel. sssion-and-languaal navigation: Interpreting visually-grounded navigation instructions in real environments. In CVPR, 2018.
[17] Felix Hill, Sona Mokrá, Nathaniel Wong, and Tim Harley. Human cestruction-folloiing with deep reinforeement learning via transfer-learing from text. arXiv, 2005.09382, 2021.

---

[18] Dipendra Kumar Misra, Andrew Bennett, Valts Blukis, Eyvind Niklasson, Max Shatkhin, and Yoav Artzi. Mapping instructions actions in 3d environments with visual goal prediction. In EMNLP, 2018.
[19] Devendra Singh Chaplot, Kanthashree Mysore Sathyendra, Rama Kumar Pasumarthi, Dheeraj Rajagopal, and Ruslan Salakhutdinov. Gated-attention architectures for task-oriented language grounding. In AAAI, 2018.
[20] Dean Pomerleau. Efficient training artificial neural networks for autonomous navigation. Neural Computation, 3:88-97, 1991.
[21] Valts Blukis, Chris Paxton, Dieter Fox, Animesh Garg, and Yoav Artzi. A persistent spatial semantic representation for high-level natural language instruction execution. In CoRL, 2022.
[22] Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni. Multi-agent cooperation and the emergence of (natural) language. In ICLR, 2017.
[23] Changxi Zhu, Mehdi Dastani, and Shihan Wang. A survey of multi-agent reinforcement learning with communication. arXiv, 2203.08975, 2022.
[24] Afshin Oroojlooyjadid and Davood Hajinezhad. A review of cooperative multi-agent deep reinforcement learning. arXiv, 1908.03999 03963, 2019.
[25] Sheng Li, Yutai Zhou, Ross E. Allen, and Mykel J. Kochenderfer. Learning emergent discrete message communication for cooperative reinforcement learning. In ICRA, 2022.
[26] Shivansh Patel, Saim Wani, Unnat Jain, Alexander G. Schwing, Svetlana Lazebnik, Manolis Savva, and Angel X. Chang. Interpretation of emergent communication in heterogeneous collaborative embodied agents. In ICCV, 2021.
[27] Frans A. Oliehoek and Christopher Amato. Concise Introduction to Decentralized POMDPs. Springer Briefs in Intelligent Systems. Springer, 2016.
[28] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In ICLR, 2015.
[29] Irene Rivera- Trigurr-si...os.... Machine translation systems and quality assessment: systematic review, Language Resources and Evaluation, 56:593-619, 2022.
[30] Felix Stahlberg. Neural machine translation: A review. Journal of Artificial Intelligence Research, 69:343-418, 2020.
[31] Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks. In NewrIPS, 2014.
[32] Artidoro Pagnoni, Kevin Liu, and Shangyan Li. Conditional variational autoencoder for neural machine translation. arXiv, 1812.04405, 2018.
[33] Diederik P. Kingma and Max Welling, Auto-encoding variational bayes. In ICLR, 2014.
[34] Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley Series in Probability and Statistics. Wiley, 1994.
[35] Richard S. Sutton and Andrew G. Barto. Reinforeement learning: An introduction. IEEE Transactions on Neural Networks, 905) 1054, 1998.
[36] Masataro Asai. Unsupervised grounding of plannable first-order logic representation from images. In ICAPS, 2019.
[37] Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. In ICLR, 2013.
[38] Alexander Borgida. On the relative expressiveness of description logics and predicate logics. Artificial Intelligence, 82:353-367, 1996.
12

---

[39] Minghuan Liu, Menghui Zhu, and Weinan Zhang. Goal-conditioned reinforcement learning: Problems and solutions. In IJCAI, 2022.
[40] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In ICLR, 2017.
[41] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv, 1707.06347, 2017.
[42] OpenAl. GPT-4 technical report. arXiv, 2303.08774, 2023.
[43] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model- based control. In IROS, 2012.
[44] Seiya Tokui and Issei Sato. Reparameterization trick for discrete variables. arXiv, 1611.01239, 2016.
[45] Dean Pomerleau. ALVINN: an autonomous land vehicle in neural network. In David S. Touretzky, editor, Advances in Neural Information Processing Systems I, [NIPS Conference, Denver, Colorado, USA, 1988], pages 305-313. Morgan Kaufmann, 1988.
[46] David Gaddy and Dan Klein. Pre-leaming environment representations for data-efficient neural instruction following. arXiv, 1907.09671, 2019.
[47] Van der Maaten, Laurens, and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, --2579-2605, 2008.
[48] Marcin Andrychowicz, Dwight Crow, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. In NeurIPS, 2017.
[49] Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojeiech Zaremba, and Pieter Abbeel. Overcoming exploration in reinforcement learning with demonstrations. In ICRA, 2018.
[50] Martin A. Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Van de Wiele, Vlad Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by playing solving sparse reward tasks from scratch. In ICML, 2018.
[51] Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, et al. Do as i can, not as say: Grounding language in robotic affordances. In CoRL, 2022.
[52] Wenlong Huang. Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. In CoRL, 2022.
[53] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint 10.00662, 2022.
13

---

Appendix
Table of Contents
Discussion about Predicate Representation
15
B Additional Related Work
15
C Algorithm Descriptions
16
D Implementation Details
16
Tasks for Evaluation
D.2 Network architccture Hyper-Parameters
22
D.3
22
E Additional Experiment Results
23
E.1
Training IFP with Different Number of NL Sentence Patterns
23
E.2
Complete Results of T-SNE Projection of Different Representations Aligning the network architecture of baselines with TALAR Ablation study on the size dataset
23
23
23
E.S
Fine-tuning BERT-baed baselines
25
E.6
End-to-end TALAR
25
E.7
Replacing generator with hand-crafted feature
26
E.8
Deployment Examples of IFP Trained by TALAR
28
F
Retrievals Notations and Abbreviations
30
G Societal Impact
30
Appendix References
31
14

---

A Discussion about Predicate Representation
crucial aspect of IOL is how the task language is represented. We believe that expressiveness and conciseness are essential properties of language representation. Expressiveness ensures that the task language accurately reflects the goal configuration. while conciseness promotes ease of policy comprehension. To fulfil these requirements, we implement IOL. by representing task language with predicate representation, which considered both expressive [38] and concise as discrete representation. The expressiveness of predicate representation stems from two key advantages: compositional structure and interpretability.
We first discuss the compositionality, which refers to the ability of a language to denote novel composite meanings. For example, if a language can represent blue circle and red square, it can represent blue square as well. This language has compositional structure. The compositionality is seen as fundamental feature of natural language and a pre-condition for language to generalize at scale [55]. Predicate representation naturally has compositional structure since it can denote the composite meanings by changing the truth value of predicates.
Next, we talk about the interpretability. Predicate representa- tion multiple predicate expressions, such Pred (a1, a2), to describe the relationships in a specific environment. Previ- ous works have found that, even if the predicate symbols are machine-generated by a learning system and anonymous to humans, they still offer some extent of interpretability. For example, if learning system generates predicate expression Pred (cat, grassland andd) =True for Fig. we can guess that Pred means [stand] or [above]. When more figures and pred- icate expressions are provided, the actual meaning of these anonymous predicate symbols would be more apparent to hu- mans. The primary focus of this paper to explore efficient learning methods, and as such, we do not delve into the topic of interpretability.
Figure 8: An illustration of the interpretability of predicate rep- resentation. For an anonymous predicate expression: Pred(cat, grassl and)) we can guess that Pred represents [above].
In conclusion, we propose to use predicate representation to represent task language since nearly all tasks contain relationships. For instance, an Atari game [56] includes relationship between the agent and the antagonist, a first-person shooter game contains relationship between the gun and the bullets, a stock trading task contains a relationship between stocks and price, etc. Even if some tasks are highly abstract and have no apparent relationships, we can implement the IOL by replacing predicate representation binary or continuous vectors.
B Additional Related Work
Hierarchical Reinforcement Learning (HRL) approaches are promising tools for solving complex decision-making tasks [57]. HRL solves the problem by decomposing it into simpler sub-tasks using a hierarchy of policies learned by RL [58, 59, 60]. For example, to put a drink into a fridge, you will (1) take up the drink, (2) open the fridge door, (3) put down the drink, and (4) close the fridge door. Putting drink into fridge is a long-term task, while these four are sub-tasks. Typically, a high-level policy is trained decompose the main task into sub-tasks (three steps in the example), and low-level policies (or single policy) are trained to complete these sub-tasks. A core problem in HRL is how to represent the goal of the sub-task (i.e., goal representation) that the low-level policy follows. Previous works have investigated both concrete and abstract goal representations of a sub-task. Concrete goal representation can be the target position to reach [61], or a target state [62]. As for abstract goal representation, it can directly be natural language [9] or an encoding of action primitives [64]. Back to the topic of this paper, language is natural goal representation for the sub-tasks, such open the fridge door in the above example. Through our experiments (Section 5.3), we observe that the resulting TL an effective abstraction of the goal of the sub-task, which demonstrates the applicability of TL to work together with HRL.
15

---

C
Algorithm Descriptions
Algorithm 1, 2, and present the training procedures of the TL generator, the translator and the instnuction-folloiing policy, respectively.
Algorithm 1 Training procedure of the TL generator.
Input: task dataset D {s,, s', LN): pre-trained BERT model Output: the optimized TL generator.
1: Initialize the TL generator ge.
2: while training not complete do
3: Sample batch of data from D.
4: Update 9 to minimize the RG loss (2).
5: end while
6: return the optimized TI. generator go-
Algorithm 2 Training procedure of the translator.
Input: task dataset D {(s, s', Lx); ). the optimized TL generator go. the translator network pe parameterized with Φ, and the pre-trained BERT model.
Output: the optimized TL translator.
I: Initialize the translator , with parameters Φ1 and Φ2.
2: while training not complete do
3: Sample batch of data {s, _N)j} from D.
4: // Compute the target task language.
5: Calculate the task language LT go(s, 8').
6: // Optimize translator.
7: Update ø to minimize the loss in (3).
8: end while
9: return the optimized translator pg.
Algorithm 3 Training procedure of the instruction-followingg policy.
Input: the optimized translator Pa-
Output: the optimized instruction followiinn policy.
I: Initialize the policy function and the value function.
2: while training not complete do
3: Sample NL instruction LN from the environment.
4: Generate corresponding task language LT (L(N)).
5: // Collecting samples
6: while episode not terminal do
7: Observe current state 8t-
8:
Execute action ~ (-188e, Ir), and receive reward y2 from the environment.
9:
end while
10:
// Training
Update the policy and value functions with the RL algorithm.
12: end while
return the optimized policy n.
D Implementation Details
In our experiments, we utilize the open-sourced RL repository, stable-baselines3 [65], to implement the RL optimization. The parameters of the BERT model are fixed if not otherwise specified. All experiments are run five times with different random seeds. We will now introduce two experimental environments, followed by a detailed explanation of model architecture of our method and the hyper-parameters utilized in experiments.
16

---

D.1 Tasks for Evaluation
D.1.1 FrankaKitchen
Task descriptions As depicted in Fig. 3, the FrankaKitchen environment emulates realistic kitchen setting with objects that can be interacted with, including 9-DoF Franka rohot, a microwave, a kettle, overhead light, cabinets, and oven. The observation space is defined R30 representing the state of each object. The action space is R with 7 DoF for the arm and for the gripper. The actions represented as the joint velocity. between the distance to the target robot's ' posture at the previous step and the distance to the target robot's posture the current step. reward of +1 given for success, and 0 provided for failure.
each timestep, the reward is calculated the difference
Natural language instructions. In FrankaKitchen, six distinct goal configurations are related to operating various objects. To generate natural language instructions that describe these goal configurations, we employ ChatGPT to produce everyday language sentences. For example, we prompt ChatGPT with: "I want ask my robot to open the microwave door. Can you provide with 50 different instructions can use?" We obtain 6 x 50 = 300 unique natural language instructions in this manner. To conserve space, only present 20 (i.e., 10 in the training dataset 10 in the testing dataset) 120 natural language instructions used in our experiments.
Training set)
Please open the door of the microwave.
Can you open the microwave door for me?
would appreciate
you could open the microwave door.
Can you open the microwave door at your earliest convenience?
Would you be able to open the door of the microwave?
need the door of the microwave be opened, please.
Please, could you open the door of the microwave?
require you to open the door of the microwave.
Could you open the microwave door when you get a chance?
Would it be possible for you to open the door of the microwave?
Could you fill the kettle with water and place it on the stove?
It's tea time! Could you turn the stove and place the kettle top?
need some hot water. Could you fill the kettle, put it on the stove, and turn the burner?
Please prepare some hot water by placing the kettle on the burner and turning on the stove.
Can you fill the kettle, put on the stove, and turn on the burner to heat up the water?
Hey robot, can you prepare some hot water by putting the kettle on the burner and turning the heat?
Can you please put the kettle on the stove, robot?
Robot, it's tea time! Please put the kettle on the burner and heat up the water.
Hey robot, could you heat up the water in the kettle on the burner?
I' m feeling chilly, robot. Can you put the kettle on the stove and boil some water?
Could you illuminate the room for me?
need some light, can you help me?
Can you activate the lighting system?
Light up the room, please.
want to see better, please turn on the lights.
Can you switch the room's 's lighting, please?
Can you brighten up the space?
need some light in here, can you turn on the lights?
need some illumination, can you help me?
Would you mind me switching the light?
Please open the cabinet door on the sliding mechanism right away.
Please open the cabinet door by sliding it.
17

---

would appreciate it if you could open the sliding cabinet door.
Please slide the cabinet door open quickly as possible.
Please open the sliding cabinet door promptly.
Would you mind opening the slide cabinet door?
Please open the cabinet door by sliding it as soon as possible.
Can you slide open the cabinet door quickly, please?
Can you please open the cabinet door by sliding it now?
need your help, open the slide cabinet door please.
Could you kindly activate the top burner by turning the knob?
Twist the oven knob to activate the top burner.
Activate the top burner by turning the oven knob clockwise.
Kindly turn the oven knob to activate the top burner.
Please turn the oven knob to switch on the top burner.
The top burner is activated.
The top burner can be activated by turning the oven knob the right.
Activate the top burner by turning the oven knob the right on the stove.
Please turn the oven knob to switch on the top burner on the stove.
Turn the knob on the oven that corresponds to the top burner.
Please turn the knob on the oven that activates the bottom right burner.
Could you turn the knob on the oven that the bottom right burner activates?
Would you kindly
the knob the oven so that the bottom right burner activates?
Activate the bottom right burner by turning the knob on the oven, Robot.
Please turn the oven knob to activate the bottom right burner, Robot.
Would you mind activating the bottom right burner on the oven by turning the knob?
Can you turn the oven knob to activate the bottom right burner, Robot?
Robot, please activate the bottom right burner the oven by turning the knob.
Manipulate the oven knob turn the burner the bottom right.
Turn the oven knob to activate the bottom burner.
Testting set)
Can you please open the door of the microwave?
need the microwave door to be opened.
Could you open the door of the microwave quickly?
Can you open the microwave door quickly?
I'd like you to open the microwave door for me, please.
Would you mind opening the door of the microwave for me?
Can you assist me by opening the microwave door?
Can you kindly open the door of the microwave?
Can you please open the microwave door without delay?
Could you please open the door of the microwave?
Can you start heating up the water by placing the kettle on the burner and turning on the stove?
Please start boiling some water in the kettle the burner, robot.
Robot, could you put the kettle on the burner and heat up the water?
Let's make some tea. Can you put the kettle on the burner and turn on the heat?
Can you please heat up some water by putting the kettle on the burner and turning on the stove?
Please heat up the kettle by putting it on the burner.
Please put the kettle on the burner, robot. I'm craving cup of tea.
need some hot water. Can you put the kettle on the stove and turn on the burner?
Can you please boil some water by putting the kettle on the burner and turning on the stove?
18

---

I'm I'm craving some tea. Can you put the kettle on the burner and turn on the heat?
Please, illuminate the space.
Turn the lights, please.
Could you turn on the lighting system?
need some illumination, can you turn on the lights?
Can you switch on the lights in here, please?
Please, switch on the lights.
Please, turn the lights in this room.
Can you brighten up the room?
Please switch on the light.
Can you brighten up this space for me?
Please slide the cabinet door open.
Please slide the cabinet door open now.
Can you open the cabinet door by sliding it without delay?
Could you please open the cabinet door by sliding it?
Can you open the sliding cabinet door right away?
Could you slide the cabinet door open?
Please, would you mind me opening the slide cabinet door.
need you to open the sliding cabinet door.
Can you open the sliding door the cabinet, please?
Could you please slide the cabinet door open?
Twist the oven knob to turn on the top burner.
Manipulate the oven knob to activate the top burner.
Turn the oven knob to the right to activate the top burner on the stove.
Could you please turn the oven knob to activate the top burner?
Use the oven knob to activate the top burner the stove.
Please turn the oven knob activate the top burner.
Turn the oven knob clockwise to activate the top burner on the stove.
Use the oven knob to switch on the top burner on the stove.
Can you turn on the top burner by twisting the oven knob?
Turning the oven knob will switch on the top burner.
Would you mind turning the oven knob activate the bottom burner?
The bottom burner is activated.
Rotate the knob located on the oven to start the burner at the bottom right.
Please turn the oven knob so that the bottom right burner activates, Robot.
Please activate the bottom right burner on the oven by turning the knob, Robot.
Would you mind turning the oven knob to activate the bottom right burner, Robot?
Activate the bottom right burner the oven by turning the knob, please.
Turn the knob the oven in order activate the bottom right burner.
Adjust the oven knob the position that turns on the burner the bottom right.
Turn the oven knob to activate the bottom right burner, Robot.
In each trajectory, a natural language instruction is randomly selected, after which the agent makes decisions to complete the given instruction. This experiment aims to not only confirm the effectiveness of TALARbut also demonstrate its robustness in handling diverse and realistic scenarios.
D.1.2 CLEVR-Robot
Task descriptions. As depicted in Fig. 3, The CLEVR-Robot environment requires the agent to manipulate five balls in accordance with human instructions. The observation space is R200 which represents the location of each object, and [A] = 40 corresponds to selecting and pushing an object in one of the eight cardinal directions. At each timestep, the reward is calculated as the difference
19

---

between the distance to the target position at the previous step and the distance to the target position the current step, with reward of +100 for success and -10 for failure. Some previous OlL-based methods attempt to train the IFP in a sparse reward environment [66, 67]. We relax the sparse reward constraint to dense reward for two reasons: (1) Sparse reward not the point of this study. We are more concerned with exploring new learning scheme for NLC-RL than with resolving the sparse reward problem; (2) The applicability of current methods. Most RL approaches address the sparse reward problem using HER [48], which relabels the origin goal in the trajectory with the actual achieved state, Nonetheless, the relabeling procedure is contingent on the transformation from the state to the goal, which necessitates additional human amnotation or program design.
NL instructions. We use eighteen different NL sentence patterns to express each goal configura- tion. For example, if we take a goal configuration such as "red ball in front of the blue ball", its corresponding NI. instructions (i.e., eighteen NI. sentence patterns) can be one of the following:
TTriiiing set)
Push the red ball in front of the blue ball.
Can you push the red ball in front of the blue ball?
Can you help me push the red ball in front of the blue ball?
Is the red ball in front of the blue ball?
Is there any red ball front of the blue ball?
The red ball moves in front of the blue ball.
The red ball is being pushed in front of the blue ball.
The red ball is pushed in front of the blue ball.
The red ball was moved in front of the blue ball.
(Testing set)-
Move the red ball in front of the blue ball.
Keep the red ball in front of the blue ball.
Can you move the red ball in front of the blue ball?
Can you keep the red ball in front of the blue ball?
Can you help me move the red ball in front of the blue ball?
Can you help me keep the red ball in front of the blue ball?
The red ball is being moved in front the blue ball.
The red ball is moved in front of the blue ball.
The red ball was pushed in front of the blue ball.
There are kinds of datasets: training and testing. In the training set, goal configurations are expressed using nine NL sentence patterns, while the remaining nine NL sentence patterns are used in the testing At the start of each trajectory, the environment randomly samples goal configuration and an NL sentence pattern to express the human instruction. By constructing these two datasets, we simulate the scenario in an open environment where different individuals instruct the robots using their linguistic preferences.
D.1.3 Hierarchical RI..
Hierarchical RL setting The observation space of the high-level policy is the same as that of the low-level policy, i.e. IFP, resulting in R200 for CLEVR-Robo and R:30 for FitnkaKitchen. Our method utilizes TL space as the action space, while HAL uses NI. instruction space. The action output by the high-level policy is then treated as the goal configuration of the IFP. We roll out the IFP for ten steps for every high-level action. We use sparse reward to verify the effectiveness of the high-level goal abstraction, i.c., the goal configuration assigned to IFP. The reward is calculated at each timestep as the count of the goal configurations achieved at the current timestep.
CLEVR-Robot As demonstrated in Fig.9a. the object arrangement task was proposed by [9], which
aims to rearrange the objects in the environment to satisfy all ten implicit constraints. At the beginning of a trajectory, the environment resets the position of all balls to a random location. Following [9],
the precise arrangement constraints are: (1) red ball to the right of purple ball. (2) green ball to the
right of red ball. (3) green ball to the right of cyan ball. (4) purple ball to the left of cyan ball. (5)
cyan ball to the right of purple ball. (6) red ball in front of blue ball. (7) red ball to the left of green
20

---

(a) CLEVR-Robot
(b) FrankaKitchen
Figure 9: Visualization of completed long-term tasks in the high-level environment. ball. (8) green ball in front of blue ball. (9) purple ball to the left of cyan ball. (10) blue ball behind the red ball.
FrankaKitchen As demonstrated in Fbg.9b, the multitask environment was introduced by [8], which aims to complete all six goal configurations. At the beginning of trajectory, the environment resets all sub-tasks their initial state, as shown in the leftmost column of Fig. 18.
21

---

D.2 Network architecture
To implement TALAR, we employ the IFP network architecture illustrated in Fig. 10. Both the policy network and value network share the same architecture, but they have different output dimensions. In our experiments, all baselines utilize the same IFP network architecture as TALAR. The primary distinction lies in how these baselines encode natural language instructions. The natural language encoding for baselines the same as TI., and the concatenated vectors (i.e., [state, NI. encoding)) serve as input for the policy and value networks.
Natural
Translator
language
language
Action to be executed
State
Figure 10: Network architecture of instruction-foolooiing policy network in TALAR.
D.3 Hyper-Parameters
The hyper-parametesn for implementing TALAR are presented in Table 2. When implementing baseline methods, we use the same hyper-parameters of PPO for policy learning.
Table 2: Syyper-parametrs:
Hyper--paaaeede
Value
Na
2
Np
4
Size of ArgNet's output
5
Learning rate (LR) for LRG
3e-4
LR for Lr
3e-4
MLP networks in translator
[256, 256, 256, 256, [T relu
Predicate network
[128, 128, 21, relu
Argument network
[128, 128, 5], relu
PPO epoch
10
PPO policy LR
3e-4
PPO value LR
3e-4
PPO policy network
[32, 64, 64], tanh
PPO value network
[32, 64, 64), tanh
PPO mini-batch size
128
PPO nums of mini-batch
160
GLA action encoder net
12|88,128,128.641 relu
GLA action decoder net
124,, relu
22

---

E
Additional Experiment Results
E.1
Training IFP with Different Number of NL Sentence Patterns
We evaluate the robustness TALAR by training an instruction-followee bers of natural language sentence patterns. Specifically, utilize the CLEVR-Robot environment and train the IFP with NL patterns number from the set (1, 5, 9). Table 3 displays the experimental results. As the number of sentence patterns increases, TALAR consistently maintains success rate. This is due to TALAR's s ability to uniquely represent NL expressions across different sentence patterns, allowing the IFP comprehend instructions more easily and learn to execute tasks efficiently. These results suggest that developing a task-related language is beneficial for policy learning. In contrast, the performance of baseline methods declines significantly as the number of NL sentence patterns increases, as they must simultancously understand greater variety of NL expressions and acquire the skills execute tasks.
policy under varying num-
100%
Table A summary of the final success rate (%) on the training with different number of NL sentence patterns. Each IFP is trained for 2M timesteps on CLEVR-Robot and evaluated for 40 episodes. The results averaged over seeds.
Pattern nums.
1
5
9
Method
TALAR
100
100% %
99.9%
One-hot
98.2%
91.8%
86.5%
BERT-binary
82.0%
60.9%
64.0%
BERT-continuous
95.3%
63.1%
60.7%
E.2 Complete Results of T-SNE Projection of Different Representations
Figure 11 presents the t-SNE projection results of various representations both training and testing natural language instruction datasets. Upon analyzing the training dataset of natural language instructions, it is obvious that the t-SNE projections of the task language exhibit higher degree of clustering compared to the baselines. This observation implies that the IOL method effectively learns a concise task goal representation, which is suitable for solving NL instruction following task. The BERT-binary representation exhibits a certain capacity to distinguish natural language instructions. Nevertheless, all the data points tend to get close, suggesting less pronounced separation of different NL instructions in comparison to the task language. In the testing dataset, the task language representation surpasses other baselines, as data points with the same marker display higher concentration. This result suggests that the task language is more robust against unseen natural language expressions. On the other hand, the t-SNE projections of the baselines appear scattered across the plane, which imposes natural language understanding challenge on policy learning.
E.3 Aligning the network architecture of baselines with TALAR
In this section, align the network structures of several baselines with TALAR in order to eliminate the impact of the additional networks on performance TALAR. More specifically, we have added new BERT-basee baselines FrankaKitchen task, which employ the same architecture the TL generator in TALAR, adhering to the BERT model. Other experimental settings remain same with the experiments in the paper. These baselines denoted with the prefix "Aligned", and their respective experimental results are presented in the Fig. 12. The performance of the baseline improves with the implementation of a new network architecture. However, it remains inferior to the TALAR approach in terms of learning efficiency and convergence scores. This highlights the effectiveness of the TALAR method.
E.4 Ablation study on the size of dataset
We recognize that the size of task dataset significant factor to consider. Thus we have conducted additional experiments to investigate the impact of task sample number on the performance of the
23

---

Task language
GLA
One-hot
P-BERT-binary
BERT- BERT-Binar BBinrr
Training
Testing
Figure 11: The t-SNE projection of different representations NL expression datasets. Each row represents one kind of representation, and each column stands for different NI instruction datasets. Points with the same marker encode nine different NL expressions that describe the same goal configuration.
24

---

Training
Testing
1.0
1.0
TALAR
TALAR
Ben- cern-binrry
Bert-binary
Aligmed-Bett Bert-binary
0.8
Aligned-Bert bortt-binury
Pretrained- Ben bert-inary
Pretrained- Bert bart--inary
Succoss Rate
Presained-A lighel-Bert-binny
Pretained-A Seetiined-Aligned---ee -ig.-iiince---eetbnaay
0.6
0.6
Success
0.4
0.4
0.2
0.2
0.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
0.0
0.5
1,0
1.5
2,0 Timesteps
2.5
3.0
3.5
4.0
Timesteps
1e6
1e6
Figure 12: Training
FrankaKitchen environments. The baselines with the same network architecture as TALAR are denoted with the prefix "Aligned".
algorithm. The results are presented the Fig. 13. The results demonstrate that 10,000 samples is sufficient train policy that achieves success rate of 76%+, which clearly outperforms the performance of other baseline methods. These experimental results suggest that TALAR can effectively train robust policy even with limited number of samples in the task dataset.
Training
Testing
0.8
0.8
Success Rate
0.6
Success
0.2
TALAR-
0.2
TALAR- 10800
TALAR- 20000
TALAR-20000
TALAR-50000
PALAR-50000
0.0
2.5
Timesteps
1e6
Timesteps
Figure 13: Training curves of TALARtrained on different size of datasets on FrankaKitchen environ- ments. The numbers after TARLAR indicate the size of the dataset used.
E,5 Fine-tuning BERT-baed baselines
We carried further experiments where the parameters of the BERT model in baseline methods are updated during the training process, as depicted in Fig. 14. The results indicate that when achieve successful task
the parameters of BERT optimized, the baseline methods struggle completion. We hypothesize that this outcome may be attributed to the extensive parameter count of the BERT model, which potentially increases the complexity of the learning process.
E.6 End-to-end TALAR
We conduct additional experiment that trains TL generator and translator jointly FrankaKitchen task, following the original experiment setting, as shown in the Fig. 15. The experiment results indicate that training the two modules independently is more effective in training policy.
25

---

Training
Testing
GLA
Bert-binary
Bett-binary
0.25
Bert-contimmous
Ber-continuous
Pretrained-Rert-hinnny
Premainedd-ert-bee
Success Rate
0.20
Pretrained-Rert-oontmmee
Success Rate
0.20
Treriinad-err-contiine
0.15
0.15
0.10
0.05
0.00
0.00
1.0
1.5
2,0
4.0
Timestops
Timestops
Figure 14: Training curves of fine-tuned baselines FrankaKitchen environments. The fine-tuned baselines are denoted with the prefix "Pretrained".
Training
Testing
1.0
1.0
0.8
0.8
Success Rate
0.6
0.6
Success
0.4
0.4
0.2
AALAR-Ind
TALAR- LLLAR--ddppeddnnt
Toit
TALAR-Joia
0.0
0.0
0.0
0.5
1.0
1.5
2.0 Timesteps
2.5
3.5
4.0
1.0
1.5
2.0
2.5
3.0
355
4.0
1e6
Timesteps
1e6
Figure 15: Training curves of end-to-end TALAR FrankaKitchen environments.
E.7 Replacing generator with hand-crafted feature
We conduct supplementary experiment to compare TALAR with a manually crafted structural representation on the FrankaKitchen task. To implement the new baseline, the NL instructions are parsed into a one-hot vector which indicates the current coul-configuration, serving as the structured representation. Subsequently, we implement a translation process same to that of TALAR. All other experimental setting remains consistent with those outlined in our original study. The results of this experiment are presented in the table below, where 'Handcrafted' denotes the new baseline trained based on the handcrafted representation.
The results show that the performance of TALAR is comparable to the handcrafted representation in term of learning speed and the final score (Figure 6). This experiment further justify the effectiveness and conciseness of the task language learned by TALAR.
26

---

Training
1.0
0.8
Success Rate
0.6
0.4
TALAR
0.2
Handerufted
TALAN AR-Handrrfted
TAL fR-Aliigannndddrrefeed
23
30
33
4.0
Timesteps
le6
Figure 16: Training curves hand-crafted features on FrankaKitchen environments.
27

---

E,8 Deployment Examples of IFP Trained by TALAR
We visualize the performance of the IFP trained by TALAR the FrankaKitchen and CLEVR-Robot environment. Fig. 17 and Fig. 18 presents the instruction-followin examples on CLEVR-Robo and FrankaKitchen environments, respectively. The trained IFP demonstrates strong ability to accurately understand natural language instructions and efficiently accomplish the assigned tasks.
Can you push the red ball behind the blue ball?
The ball pushed behind the blue ball.
The red ball moves to the right the oyan ball.
Is there any red ball in front of the purple ball?
ball was moved in the purple ball.
Figure 17: A visualization of the TALAR's IFP deployment process on CLEVR-Robot.
28

---

Please open the door of the microwave.
Can you please put the kettle on the stove, robot?
need some light, can you help me?
Would you mind opening the slide cabinet door?
Can you tum on the top burner by twisting the oven knob?
Turn the oven knob to activate the bottom burner.
Figure 18: A visualization the TALAR's IFP deployment process on FrankaKitchen.
29

---

Retrievals for Notations and Abbreviations
Table 4: Notations and abbreviations in this paper.
Name
Meaning
Notations
LT
task language
Ln
natural language
qo
encoder with parameters Φ1 decoder with parameters Φ2 TL generator with parameters 0
Poz
90
Abbreviations
NL
Natural Language
TL
Task Language
LM
Language Model
RL
Reinforcement Learning
IOL
Inside-Out Learning
OIL
Outside-In Learning
RG
Referential Game
GLA
Grounding Language to latent Action
NLC-RL
Natural Language-Conditioned Reinforcement Learning
G
Societal Impact
This research enhances the success rate and adaptability of instruction-folloiing policies in response to various natural language expressions. The development of NLC-RL agents capable of effectively following human instructions holds the potential to impact multiple aspects society. One of the most immediate societal impacts lies in improving human-robot collaboration. As these RL agents become increasingly proficient in comprehending and executing human instructions, they can be integrated into various applications, including manufacturing, logistics, and service industries. This integration may lead to heightened efficiency, reduced labour costs, and enhanced safety across these sectors. Moreover, these RL agents' advanced natural language understanding can be applied to assistive technologies for individuals with disabilities. For instance, they could be utilized in intelligent home systems to facilitate navigation for those with mobility impairments or support individuals with cognitive impairments in accomplishing daily tasks. Nonetheless, it is important to consider the ethical implications of deploying these RL agents in real sol-world settings. Ensuring that these systems maintain transparency and accountability is of paramount importance.
30

---

Appendix References
[54] Alexander Borgida. On the relative expressiveness of description logics and predicate logics. Artificial Intelligence, 82:353-367, 1996.
[55] Angeliki Lazaridou and Marco Baroni. Emergent multi-agent communication in the deep learning era. arXiv, 2006.02419, 2020.
[56] M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
[57] Shubham Pateria, Budhitama Subagdja, Ah-Hwee Tan, and Chai Quek. Hierarchical rein- forcement learning: A comprehensive survey, ACM Computing Surveys, 54(5):109:1-109:35, 2021.
[58] Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg. David Silver, and Koray Kavukeuoglu. Feudal networks for hierarchical reinforcement learning. In ICML, 2017.
[59] Nicholas K Jong, Todd Hester, and Peter Stone. The utility of temporal abstraction in reinforce- ment learning. AAMAS, 2008.
[60] Ofir Nachum, Haoran Tang, Xingyu Lu, Shixiang Gu, Honglak Lee, and Sergey Levine. Why does hierarchy (sometimes) work well in reinforcement learning? arXiv, 1909.10618, 2019.
[61] Xintong Yang, Ze Jing Wu, Yu-Kun Lai, Changyun Wei, Guoliang Liu, and Rossitza Setchi. Hierarchical reinforcement learning with universal policies for multistep robotic manipulation. IEEE Transactions on Neural Networks and Learning Systems, 33:4727-4741, 2022.
[62] Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. In NeurlPS, 2018.
[63] Yiding Jiang, Shixiang Gu, Kevin Murphy, and Chelsea Finn. Language abstraction for hierarchical deep reinforcement learning. In NeurlPS, 2019.
[64] Karl Pertsch, Youngwoon Lee, and Joseph J. Lim. Accelerating reinforcement learning with learned skill priors. In CoRL. 2020.
[65] Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. stable-baseliness Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22:268:1-268: 2021.
[66] Rishabh Agarwal, Chen Liang, Dale Schuurmans, and Mohammad Norouzi. Learning to generalize from sparse and underspecified rewards. In ICMI. 2019.
[67] Geoffrey Cideron, Mathieu Seurin, Florian Strub, and Olivier Pietquin. Higher: Improving instruction following with hindsight generation for experience replay. In SSCI, 2020.
[68] Marcin Andrychowicz. Dwight Crow, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. In NewrIPS, 2017.
[69] Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving ong-horrizon tasks via imitation and reinforcement learning. In CoRL, 2019.
31


Reviews:
[{"review_id": "qM6JPradkw", "reviewer": "Reviewer_rdPR", "summary": "This work tackles the problem of learning an effective natural language (NL) instruction-following agent using RL algorithm as a goal-conditioned RL setup. The authors propose to learn a task-language (TL), which is a synthetic and vectorized representation containing the abstractive meaning of the instruction, via a referential-game-like manner. A TL generator alongsidse an NL-to-TL translator is learned for deriving an expressive and concise representation for an RL algorithm to effectively utilize. The experiments conducted on simulated environment demonstrate superior performance compared to several instruction-following baselines, as well as better distributed latent representations for the goal-conditioned RL policy.", "questions": "- How does it compare to program synthesis lines of research? PL can be an expressive and logical (procedural), which contain deterministic execution routes. And PL can be more expressive than the utilized predicate representation for more complex and long horizon tasks. I think a discussion could be nice. [2] [3]\n- Referring to the corresponding weakness, how would authors envision/suggest future works to apply similar (and/or extended) paradigm to more complex instruction following tasks, such as navigation, robot manipulation, etc., that are guided with much more sophisticated language?\n- Is L_T^tilt in Section 4.2.2 discrete or continous?\n- Better to include the prompting scheme for ChatGPT instruction generation (basically, paraphrases).\n\n[2] Sun, Shao-Hua, et al. \"Neural program synthesis from diverse demonstration videos.\" ICML-18.\n[3] Sun, Shao-Hua, Te-Lin Wu, and Joseph J. Lim. \"Program guided agent.\" ICLR-19.", "limitations": "- Please refer to the weaknesses for the limitations of this work.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- It is an interesting idea to position referential game in developing the TL, and the efficacy of such a language is the key to effective communication during the guided RL.\n- The proposed framework is well-modularized and component-wise it can be interpretable to certain extent for model analysis.\n- The paper presentation is easy to follow, with nicely illustrated visualizations.\n- Experiments are solid, and the supplementary materials are helpful.", "weaknesses": "- Could you elaborate the rationale behind learning the TL independent to a translator? (I’m guessing better modularization or something similar.) The TL developed this way is not expected to generalize very well across many tasks, and sacrifices the expressiveness of NL to a specifically trained translator. I.e., why not learn the translator (and the TL) in an end-to-end fashion (such as VQ-VAE paradigm or methods like discrete code-book look up), and maybe compare its pros and cons to the proposed framework? (The training can utilize bascially the same settings as the proposed framework in this work.)\n- Following above, if my understanding is correct, the objective of the “receiver” is something like next-word prediction or MLM in BERT. In this sense, the L_T operates similar to a prefix. How would you ensure it is indeed conditioned on the prefix for the word prediction? It is intuitive to think that the model will anyway predict the word correctly if trained well under standard language modeling objective and/or MLM.\n- A discussion in comparisons with recent robotic works that utilize LLM as a strong resource/planner is needed, such as [1].\n- A set of non-frozen LM baselines could be conducted to even further emphasize on the needs of the conciseness of the generated TL.\n\n[1] Ahn, Michael, et al. \"Do as i can, not as i say: Grounding language in robotic affordances.\" arXiv 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for carefully reviewing our paper and providing constructive comments. We hope that our response has addressed your concerns, but if we missed anything please let us know.\n\n> Q1: Could you elaborate the rationale behind learning the TL independent to a translator?\n> \n\nA1: The main reason we separate TL development and translation is we can handle and expand these two modules independently. For example, developing TL during the RL training phase.\n\nBesides, we consider the practical aspects of the training procedure. Training these two modules concurrently could potentially lead to a trivial solution (e.g., both the generator and translator outputting zero to achieve a low translation loss). Furthermore, the back-propagation of loss from the translator could potentially interfere with the effective development of the TL.\n\nTo better illustrate, we conduct additional experiment that trains TL generator and translator jointly on FrankaKitchen task, following the original experiment setting, as shown in the Figure 1 in the rebuttal PDF file. The experiment results indicate that training the two modules independently is more effective in training a policy.\n\n> Q2: The objective of the “receiver” is something like next-word prediction or MLM in BERT. How would you ensure it is indeed conditioned on the prefix for the word prediction?\n\nA2: Yes, the training objective of receiver has similar form like MLM loss in BERT, while it aims at predicting the next token based on the task language and previous tokens. In some cases, the tokens cannot be accurately predicted without the prefix. For example, there could be similar natural language instructions like “Open the refrigerator door” and “Open the microwave door”. The generated $L_{\\rm T}$ must capture the key information in the state pair to assist model in correctly predicting ‘refrigerator’ token.\n\n> Q3: A discussion in comparisons with recent robotic works that utilize LLM as a strong resource/planner is needed.\n\nA3: We would like to discuss related works about language-based robotics control [1,2,3]. SayCan [1] combines low-level tasks with LLMs so that the language model provides high-level knowledge about the procedures for performing complex and temporally extended instructions. Inner Monologue [2] makes further improvements by adding the eponymous “inner monologue\", which is implemented as injected feedback from the environment. ReAct [3] introduces LLMs reasoning to help the model induce, track, and update action plans as well as handle exceptions. Overall, these works design methods that generate high-level language instructions, while our work provides efficient ways to learning to complete the instructions.\n\n> Q4: A set of non-frozen LM baselines could be conducted.\n\nA4: We conducted additional experiments where the parameters of the BERT model in baseline methods are updated during the training process, as depicted in Figure 2 in the rebuttal PDF file. The results indicate that when the parameters of BERT are optimized, the baseline methods struggle to achieve successful task completion. We hypothesize that this outcome may be attributed to the extensive parameter count of the BERT model, which potentially increases the complexity of the learning process.\n\n> Q5: How does it compare to program synthesis lines of research?\n\nA5: That’s a very interesting question and we think that there are two key differences:\n\n1. the expressiveness and complexity of the target domain-specific language (DSL). Program synthesis often deals with general-purpose programming languages, such as Python, which have rich syntax and semantics. TALAR uses a simpler and more restricted DSL. The advantage of using a simpler DSL is that it can be easier to learn and understand by the agent and achieves high learning efficiency. A potential limitation is that it may not fully encapsulate all the subtleties and variations inherent in natural language instructions.\n2. the way the natural language instructions are translated into the DSL. Program synthesis typically depends on semantic parsing techniques, which necessitates extensive linguistic knowledge and domain expertise, and it can be challenging to manage ambiguity, noise, or incompleteness in natural language. In contrast, TALAR employs a neural network-based translator to map natural language to the task language, thereby eliminating the need for prior knowledge about the task.\n\n> Q6: How would authors envision/suggest future works to apply similar (and/or extended) paradigm to more complex instruction following tasks?\n\nA6: One possible direction is to develop a more expressive and flexible task language. Currently TALAR only employs the format of predicate representation as the task language. How to improve the inherent property of the learned task language is a key problem to be solved. For example, we could add some constraints to the learning process to ensure the TL generator could learn symmetric predicate, which makes it more meaningful.\n\n> Q7: Is $\\tilde{L_{T}}$ in Section 4.2.2 discrete or continuous?\n\nA7: $\\tilde{L_{T}}$ outputted by translator is **discrete**.\n\n> Q8: the prompting scheme for ChatGPT instruction generation.\n\nA8: The specific prompt we use is as follows: “Suppose you want to order a home service robot to [task description]. Give me 100 kinds of diverse expressions. Note that you don't need emphasize you are talking with a robot and don't use word ‘robot'.” Here, [task description] is a placeholder for the specific task, such as 'open the microwave door'.\n\n### References:\n\n[1] Ahn, Michael, et al. Do as i can, not as i say: Grounding language in robotic affordances.\n\n[2] Wenlong Huang, et al. Inner monologue: Embodied reasoning through planning with language models.\n\n[3] Shunyu Yao, et al. ReAct: Synergizing Reasoning and Acting in Language Models.\n\n[4] Shao-Hua Sun, et al. Neural program synthesis from diverse demonstration videos.\n\n[5] Shao-Hua Sun, et al. Program guided agent."}, {"review_id": "IonXIWuPlF", "reviewer": "Reviewer_nXhV", "summary": "This paper presents an algorithm that reduces the policy learning burden in a natural language-conditioned reinforcement learning framework. The authors investigate an inside-out scheme for natural language-conditioned RL and then present a new approach, TALAR, that learns multiple predicates to model object relationships as the task language. In the experiments, the authors demonstrated that TALAR outperforms previous baseline algorithms with instruction following policy.", "questions": "- How to construct $s$ and $s’$ of $(s,s’,L_N)$ in task dataset $D$? As mentioned in lines 283-285, when composing $(s, s’, L_N)$, all $s$ from timestep 0 to $T-1$ are used, but doesn’t most of them correspond only to the partial part of $L_N$, not exactly $L_N$?\n- It seems that annotation for predicate representation is also necessary for referential game learning, is that correct? If so, isn't the amount of information required different from other baseline algorithms? It would be better if the annotations required for each algorithm were specified.\n- In lines 256-258, it is mentioned that the NL instruction was created using ChatGPT. How did you create it using it specifically? It would be better to provide a more detailed explanation with an example.", "limitations": "The authors mentioned limitations in their paper.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- This proposed method helps policy learning by learning representations so that natural language instructions are close to semantically similar ones.\n- It shows much better performance than baseline algorithms in various domains.", "weaknesses": "- A separate training dataset is required for TALAR learning, and the cost for this seems expensive.\n- NL instructions must be defined separately for each task and data must be collected.\n- Performance is sensitive depending on the type of NL instructions collected for training.\n- It is difficult to create a training dataset including NL instructions without prior knowledge of the domain.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your insightful comments and constructive feedback on our paper. We are grateful for the time and effort you put into reviewing our work. Below we address each of your concerns and questions.\n\n> Q1: A separate training dataset is required for TALAR learning, and the cost for this seems expensive.\n> \n\nA1: We appreciate your concern regarding the potential expense associated with the need for a separate training dataset for TALAR learning. \n\nWe would like to clarify that the dataset is necessary for connecting the RL agent with the natural language. While some baseline methods do not explicitly use a dataset containing natural language, the language model they depend on also require amounts of corpus data to be trained.\n\nBesides, the acquisition of this data does not inherently imply a high cost, as the dataset for TALAR learning does not necessarily have to be generated from scratch. Existing datasets can be effectively repurposed or augmented to meet our requirements. For instance, pre-collected datasets have been extensively utilized in the realms of offline RL, imitation learning, and language model training. \n\nWe recognize that the size of task dataset is a significant factor to consider. Thus we have conducted additional experiments to investigate the impact of task sample number on the performance of the algorithm. The results are presented in the Figure 3 in the rebuttal PDF file. The results demonstrate that 10,000 samples is sufficient to train a policy that achieves a success rate of 76%+, which clearly outperforms the performance of other baseline methods. These experimental results suggest that TALAR can effectively train a robust policy even with a limited number of samples in the task dataset.\n\n> Q2: NL instructions must be defined separately for each task; It is difficult to create a training dataset including NL instructions without prior knowledge of the domain; Performance is sensitive depending on the type of NL instructions collected for training.\n> \n\nA2: We would like to clarify that these instructions are not 'defined' in any rigid sense. In practical, the NL instructions in the task dataset could be flexibly assigned based on the linguistic habits of the person providing the state transition descriptions. This means that we do not require specific design of NL instructions to align with real-world application scenarios.\n\nTo illustrate, while constructing the task dataset, one could generate NL instructions by observing a video of a robotic manipulation (i.e., the trajectory) and then describing the robot's execution goal in their own words. This process does not necessitate prior knowledge of the task domain. For example, we use ChatGPT to generate the NL instructions to construct the task dataset. Our experiments in Figure 4 in the original paper have shown that TALAR is robust to unseen NL instructions (please refer to Appendix D.1.1 for examples of training/testing NL instructions). We believe the experimental results underscore the robustness of the proposed approach.\n\n> Q3: How to construct $s$ and $s’$ of $(s,s’,L_N)$ in task dataset $D$?\n> \n\nA3: In practical, we could pre-collect the trajectory data $\\{s_0,s_1,\\cdots,s_T\\}_i$, and have a human observer describe the instruction for each trajectory in natural language. Ideally, we would use $(s_0, s_T)$ to construct the task dataset. However, this could potentially make the data collection process both costly and labor-intensive. To mitigate this issue, we utilize a data augmentation strategy. We take the intermediate state in the trajectory as $s$, and the terminal state as $s’$. This approach treats NL instructions similarly to the goal in a goal-conditioned RL setting, where all states in the trajectory share the same goal. Consequently, it is sufficient for the agent to determine its action based on the natural language instruction and current state. We believe this data augmentation strategy not only enhances the efficiency of data collection but also enriches the diversity of the task dataset, thereby improving the robustness and generalizability of our model.\n\n> Q4: It seems that annotation for predicate representation is also necessary for referential game learning.\n> \n\nA4: Good point. While we acknowledge that the annotation for predicate representation might potentially accelerate the learning of task language, these annotations are not necessary.   There are two primary considerations.\n\nFirstly, the central objective of our research is not to match the manually designed NL representations for each NLC-RL task. Instead, our focus is on the automatic discovery of task-specific relationships, which we refer to as task language. This approach aligns with our broader aim of fostering a more autonomous and adaptable system.\n\nSecondly, providing predicate representation could be laborious and costly. In such scenarios, the benefits of our approach become particularly evident. Our system learns predicate networks in an unsupervised manner through engagement in a referential game with the receiver.\n\n> Q5: How did you use ChatGPT specifically?\n> \n\nA5: The specific prompt we use is as follows: \"Suppose you want to order a home service robot to [task description]. Give me 100 kinds of diverse expressions. Note that you don't need emphasize you are talking with a robot and don't use word ‘robot’.\" Here, [task description] is a placeholder for the specific task, such as 'open the microwave door'.\n\nWe hope that these responses can address your concerns and questions. If you had any further concerns, please let us know."}, {"review_id": "BalbK8sEpN", "reviewer": "Reviewer_KoSw", "summary": "This paper proposes the framework of TALAR, an Inside-Out learning framework for training policies that follows language instructions with reinforcement learning. The method leverages predicate representation for building a compact space of task language, and learns the generator of the task language from the game state by reconstructing natural language. A translator from natural language to task language is learned by language translation loss, and a policy based on task language is learned by reinforcement learning. In experiment, TALAR shows better performance than baseline methods that use natural language embeddings on two continuous control environments.  ", "questions": "In figure 2(b), it seems that there is no mechanism for selecting different arguments for different predicates. Do all the predicates have the same set of arguments? If so, since the predicate is a Boolean variable, it is reasonable to suspect that that the multiple predicates cannot convey rich information in the arguments.  \n\nIn line 222, it is not precisely described how multiple predicates are concatenated to form the task language.  ", "limitations": "The limitations are well addressed in the conclusion section.  ", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "This paper has overall great presentations of the motivation, methods, and experiment.  \n\nTALAR has the clear motivation of learning the task-correlated language and optimizing a policy based on task language, instead of using generally pre-trained language embeddings by language models.  \n\nThe experiments in section 5.2 show that TALAR learns better embedding space compared to BERT pre-trained models.  ", "weaknesses": "Some baseline methods are not compared: method using the same network architecture or number of parameters as TALAR but with no Task Language translation loss, trained with behavior cloning and reinforcement learning. This will alleviate the concerns that the performance improvement is caused by additional network capacity introduced by the autoregressive translation, while the baseline methods only have a very small number of parameters being tuned (the FC layer after BERT, if it is the case).   ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your time and effort in reviewing our paper and providing valuable feedback. We are glad that you found our presentation, soundness, and the TALAR method to be strong. We would like to address your concerns and questions as follows.\n\n> Q1: Some baseline methods are not compared: method using the same network architecture or number of parameters as TALAR but with no Task Language translation loss.\n> \n\nA1: We have incorporated a comparison with these baselines, which share the same network architecture as TALAR. More specifically, we have added new BERT-based baselines on FrankaKitchen task, which employ the same architecture as the TL generator in TALAR, adhering to the BERT model. Other experimental settings remain same with the experiments in the paper. These new baselines are denoted with the prefix 'Aligned', and their respective experimental results are presented in the following table:\n\n|  | TALAR | Bert-binary | Aligned-Bert-binary | Pretrained-Bert-binary | Aligned-Pretrained-Bert-binary |\n| --- | --- | --- | --- | --- | --- |\n| Training | **93.5±8.8** | 22.3±6.6 | 45.8±5.6 | 10.8±4.1 | 23.3±7.0 |\n| Testing | **88.3±7.1** | 22.1±4.8 | 45.2±6.3 | 11.1±3.0 | 24.1±6.8 |\n\nThe performance of the baseline improves with the implementation of a new network architecture. However, it remains inferior to the TALAR approach in terms of learning efficiency and convergence scores. This highlights the effectiveness of the TALAR method. Please refer to Figure 4 in the rebuttal attached PDF file for the training curves. We appreciate your suggestions about the network architecture of baselines, and would add the new experiment results to the revised version.\n\n> Q2: In figure 2(b), it seems that there is no mechanism for selecting different arguments for different predicates. Do all the predicates have the same set of arguments?\n> \n\nA2: At present, we have implemented the TL generator such that all predicate networks utilize a shared argument list. This approach proves to be both sufficient and effective when the task language can be represented using a limited number of arguments. However, the TALAR system is inherently flexible and can be readily extended to accommodate more complex tasks. For instance, we can enable each predicate network to input a unique argument list by constructing additional, separate argument networks.\n\n> Q3: In line 222, it is not precisely described how multiple predicates are concatenated to form the task language.\n\n\nA3: We apologize for unclear presentation in the original paper regarding these points. The resulting task language is represented as $[Pred_1,\\cdots,Pred_{N_{pn}}, arg_1, \\cdots, arg_{N_a}]$, where $\\text{Pred}_{i}$ denotes the output of the i-th predicate network. To enhance clarity, we will incorporate the explanation to Section 4.2.1.\n\nWe hope that our response has addressed your concern and questions satisfactorily. If you had any further concerns, we are glad for discussion."}, {"review_id": "EgLBlGw36p", "reviewer": "Reviewer_rFCy", "summary": "The main contribution of the work is TALAR, a method for learning a vector representation of the instruction via a referential game. In TALAR, a generator creates a task vector from (state, next-state) pairs, which is then back-translated into natural language by the receiver. A translation from natural language to task vectors is then learned. At policy learning time, the instruction gets translated into task language and fed to the policy. The results show that TALAR outperforms other methods of learning task embeddings.\n", "questions": "- Line 197 and Line 203. Are these two sentences contradictory? You say you’re predicting the word corresponding to the state, but the word is chosen randomly?\n- Figure 2: is the number of predicate networks linear in the number of objects in the state? How did you choose the number of predicate and argument networks?", "limitations": "The discussion is adequate.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Allowing a task representation to emerge in a referential game, and then using the representation in policy learning is original. The experiments are standard in the RL community and show TALAR's superior performance. The finding that a more linearly separable task representation leads to better instruction following performance is significant. \n", "weaknesses": "My main concern with the work is that I don’t understand why the proposed technique would be better than parsing the instruction from humans into a structured representation that’s formulated ahead of time. What is the utility of having the discrete representation of a task emerge, rather than defining a desired discrete representation to begin with? GLA and the BERT methods make more sense to me, because the embedding methods are fairly generic . \n\nIt seems to me that you could choose the predicate representation vector ahead of time, instead of playing the referential game, and then do the same translation step as before. Handcrafting another structured representation baseline seems like it would elucidate whether the referential game is necessary. One could do the parsing programmatically, or using an LLM like GPT-3.\n\nFurthermore, what happens when the natural language is OOD in task language? The method doesn’t seem set up to handle distributional shifts, which seems likely because, as the authors say, “natural language is a complex and unbounded representation” (line 29). Because the task representation is both discrete and emergent, is the representation manifold going to be well-behaved for OOD natural language?\n\nI also had some difficulty understanding certain paragraphs in the methods section. See questions.\n\nComments:\n- I recommend including “predicate representation in the title.\n- Line 4: I wouldn’t include the terms “outside-in” and “inside-out” in the abstract, because you don’t define them until the intro and the meanings are not immediately intuitive.\n- Line 192: Typo “The We”\n- Line 194: This paragraph is hard to understand. A receiver diagram would be nice.\n\nMy score reflects the fact that after reading the paper, I’m unconvinced on the utility of the core technique in the paper. Happy to discuss this with the authors.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and valuable comments. We have taken every question into consideration and revised our paper to fix the typos. Please find the response below.\n\n> Q1: Why the proposed technique would be better than parsing the instruction from humans into a structured representation that’s formulated ahead of time.\n\nA1: Good point. we acknowledge that a manually designed structured representation could potentially be an effective method for interpreting natural language instructions. However, it is important to highlight the advantages of our approach over this traditional method.\n\n- Universality: The primary advantage of our method, TALAR, is its universality. Converting human instructions into a structured representation typically necessitates a unique design for each task, a process that can be both labor-intensive and limited in its ability to manage complex instructions. In contrast, TALAR automatically learns to develop TL and translates NL to TL, thereby eliminating the need for manual parsing of natural language instruction representation. This learning process is both automatic and universal.\n- Scalability: TALAR's task language is designed to adapt and scale in accordance with the complexity of the environment and the tasks that agents are required to perform. This feature proves particularly beneficial in scenarios involving complex tasks with numerous objects and operations, where creating a structured representation for natural language instructions can be challenging. In contrast, our method learns task language in an unsupervised manner through engagement in a referential game with the receiver. This learning framework can be easily applied to various NLC-RL tasks.\n\n> Q2: Handcrafting another structured representation baseline seems like it would elucidate whether the referential game is necessary.\n\nA2: We conduct a supplementary experiment to compare TALAR with a manually crafted structural representation on the FrankaKitchen task. To implement the new baseline, the NL instructions are parsed into a one-hot vector which indicates the current goal-configuration, serving as the structured representation. Subsequently, we implement a translation process same to that of TALAR. All other experimental setting remains consistent with those outlined in our original study. The results of this experiment are presented in the table below, where ‘Handcrafted’ denotes the new baseline trained based on the handcrafted representation.\n\n|  | TALAR | Handcrafted |\n| --- | --- | --- |\n| Success rate | 93.5±8.8 | 94.1±8.1 |\n\nThe results show that the performance of TALAR is comparable to the handcrafted representation in term of learning speed and the final score (Figure 6). This experiment further justify the effectiveness and conciseness of the task language learned by TALAR.\n\n> Q2: Furthermore, what happens when the natural language is OOD in task language?\n\nA2: At present TALAR handles with OOD natural language based on the generalization ability of translator. The translator's language model is trained on extensive corpus data, thereby demonstrating a certain degree of generalization ability. Preliminary experiments on unseen natural language instructions, although potentially not sufficiently OOD, provide initial evidence of this generalization ability.\n\nSimultaneously, we recognize that managing OOD datasets is a valuable area of exploration. A direct yet potentially effective strategy could involve designing an OOD detection module and processing the OOD natural language instructions differently. For instance, we could consider updating the translation generator or translator module online.\n\n> Q3: I also had some difficulty understanding certain paragraphs in the methods section.\n\nA3: Apologize for the method description not being clear enough. Here we would like to take this opportunity to clarify the particular details of our method.\n\n**Comment1**: Figure 2: is the number of predicate networks linear in the number of objects in the state? How did you choose the number of predicate and argument networks?\n\nResponse1: The number of predicate networks is not necessarily linear to the number of objects in the state. It serves as a representation of the potential relationships that our learning algorithm might identify within a specific task. These relationships can often be abstract and anonymized, which makes their exact quantification a complex endeavor. Thus, the selection of the number of predicate networks is mainly influenced by the scale of the task, which includes factors such as the complexity of the instructions, the number of objects. In our experiment, we opted for a moderate number of 4 for both the FrankaKitchen and CLEVR-Robot tasks. Our ablation study on the number of predicate networks (Figure 7 in the original paper) show that TALAR performs consistently well across different selections of predicate network numbers.\n\n**Comment2**: Are two sentences at line 197 and line 203 contradictory?\n\nResponse2: Sorry for the unclear presentation. The sentence on line 197 provides a high-level overview of the receiver's training objective, while the sentence on line 203 delves into the specifics of our implementation, and they are not contradictory.\n\nIn the context of the $(s,s’,L_N)$ tuple, we initially select a word $T_i$ at random from the $L_N$ sentence. Subsequently, the receiver's goal is to predict this selected word, taking into account the task language (produced by the TL generator) and the previous words (i.e., $T_{1,\\cdots,i-1}$). This process ensures that the TL generator is important to generate effective task language, as its effectiveness directly influence the receiver's prediction accuracy.\n\n**Comment3**: This paragraph is hard to understand. A receiver diagram would be nice.\n\nResponse3: We have incorporated a receiver diagram to enhance the clarity of the discussed concept, which is depicted in Figure 7 in the rebuttal attached PDF file."}]

Author rebuttal:
We would like to express our gratitude to the reviewers and chairs for their valuable time and constructive feedback on our paper. We have carefully considered each comment and provided detailed responses. Thanks to the insightful assessments from the reviewers, we have conducted a more thorough exploration about our method (Reviewer rdPR, nXhV), performed a comprehensive comparison with baselines (Reviewer KoSw, rFCy), further highlighted the effectiveness of our method (Reviewer rdPR, KoSw, rFCy), and improved the presentation regarding predicate representation (Reviewer rdPR, nXhV, KoSw, rFCy). The results of these experiments can be found in the attached PDF file.

We also would like to clarify the advantage of our proposed IOL framework and its implementation, TALAR. Our framework has the capability to automatically develop task-specific language, which can subsequently be utilized in the RL phase to enhance the learning efficiency of the RL agent. In comparison to the baseline method, which utilizes a pretrained language model to encode natural language instructions, our approach produces more concise task language, enabling more efficient policy learning. Additionally, unlike methods that require a pre-defined structured representation, our approach eliminates the need for task-specific design, reducing the need for laborious efforts.

We hope our response could address your concerns about our paper. Please let us know if you have any further questions or concerns.

Meta-review: This work proposes conditioning natural language (NL) instruction-following policies on a “task language” (TL) rather than on the raw NL input. The raw NL is translated to TL, which allows for efficient conditioning that leads to more robust performance. The reviewers generally think that this is a good approach and the author rebuttal alleviated several concerns.

I agree with reviewer rFCy that I would like to see a more explicit comparison to and discussion of semantic parsing (which as the author rebuttal remarks can be a useful comparison since their TL approach might be more efficient as it can specialize to the task much more). The idea of having an intermediate structured representation is not new and I think the paper needs to do a much better job at acknowledging this.
Decision: Accept (poster)