SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Evaluation environments. We conduct experiments in FrankaKitchen [8] and CLEVR-Robot [9] environments, as shown in Fig. 3. FrankaKitchen aims to control a 9-DoF robot to manipulate various objects in a kitchen, involving six sub-tasks: open the microwave door, move the kettle to the top left burner, turn on the light switch, open the slide cabinet, activates the top burner and activate the bottom burner. We treat
(a) FrankaKitchen
(b) CLEVR-Robot
Figure 3: A visualization of environments in our experiments. (a) FrankaKitchen. The agent controls a 9-DoF robot to manipulate various objects in a kitchen. (b) CLEVR-Robot. The agent (silverpoint) manipulates five movable balls to reach a specific goal configuration. An example of NL instruction is: Can you move the red ball to the left of the blue ball?
FrankaKitchen-Train
FrankaKitchen-Test
Figure 4: Training curves of different methods on two evaluation environments. The x-axis represents the timesteps agent interacts with the environment, and the y-axis represents the success rate of completing the instructions. The shaded area stands for the standard deviation over five random seeds.
CLEVR-Robot-Train
CLEVR-Robot-Test
each sub-task as a different goal configuration. In each trajectory, the environment randomly generates an NL instruction that describes a goal configuration. We let ChatGPT [42] generate 50 different NL instructions to describe each goal configuration, yielding 300 different NL instructions in this task. CLEVR-Robot is an environment for object interaction based on the MuJoCo physics engine [43]. It contains five movable balls and an agent (silver point). In each trajectory, the agent aims to achieve a goal configuration that requires moving a specific ball in a target direction (front, behind, left, or right) relative to a target ball. An example of NL instruction is Move the green ball to the left of the blue ball. There are 80 different goal configurations in total. We use 18 natural language sentence patterns to describe each goal, resulting in 1440 distinct NL instructions. We split the NL instructions into two tasks: training and the testing set. The training set comprises 40 and 9 different NL instructions for each goal configuration in FrankaKitchen and CLEVR-Robot, respectively, resulting in a total of 240 NL instructions for FrankaKitchen and 720 for CLEVR-Robot. During training, the agent can only interact with the NL instructions in the training set. The testing set consists of the remaining NL instructions, which include 120 NL instructions for FrankaKitchen and 720 for CLEVR-Robot. Refer to Appendix D.1 for more details about the environments and the NL instructions in the training and testing sets.
Task dataset collection. The task dataset can be obtained from pre-collected data, where state pairs are annotated with natural language instructions through human input or rule-based functions. In our experiment, we utilize PPO to train a goal-conditioned policy capable of achieving any goal configuration. We then use the trained policy to collect trajectory data  $\{(s_0, a_0, s_1, a_1 \cdots s_T), L_{\mathrm{N}})$  across various environments, where  $L_{\mathrm{N}}$  is the natural language that describes the goal configuration of this trajectory and is randomly sampled from the training instructions set. The collected trajectory data is rearranged, with  $\{(s_i, s_T, L_{\mathrm{N}})\}_{i=0}^{T-1}$  as tuples in the task dataset, resulting in 50,000 tuples for the FrankaKitchen task, and 100,000 for the CLEVR-Robot task. Note that the collected trajectory data can be used to pre-train the policy. With tuple data such as  $(s_i, a_i, L_{\mathrm{N}})$ , the policy  $\pi(s, L_{\mathrm{N}})$  can be pre-trained to output the action  $a$ . Two baselines (Pre-trained BERT- $*$  in Section 5.1) utilize this data for pre-training purposes.
Implementation Details. For experiments, we utilize the pre-trained BERT-base-uncased model[3] as the NL encoder. We conduct all experiments with five random seeds, and the shaded area in the figures represents the standard deviation across all five trials. We refer readers to Appendix D for more implementation details about the experiments.
5.1 Performance of Instruction-following Policy
Baselines for comparison. We consider multiple representative methods in NLC-RL as baselines: (1) One-hot encodes the representation of all possible natural language instructions (including training and testing) to a one-hot vector, where each instruction is assigned a unique position. (2) BERT-binary processes the natural language with a pre-trained BERT model, and the resulting language encoding is then transformed into a binary vector using a fully-connected network. This binary vector's size equals the TL generated by TALAR. To ensure differentiability, we employ a reparameterization trick [44] that converts a continuous vector into a binary vector. (3) BERT-continuous is similar to BERT-binary, except it uses a continuous vector of the same size. (4) Pre-trained BERT-binary/continuous initially pre-trains the policy networks and the BERT model using behavior cloning [45] with pre-collected  $(s_i, a_i, L_{\mathrm{N}})$  tuples, and subsequently trains the policy networks with RL. The data collection process is described above alongside the task dataset collection. (5) GLA (Grounding Language to latent Action) [46] first employs an auto-encoder to learn a latent
(a) TALAR
(b) GLA
Figure 5: The t-SNE projections of different NL encoding types on FrankaKitchen. Points with the same marker encode forty different NL expressions that describe the same goal configuration. We add a slight noise to the overlapping points for better presentation. (a) The translator's t-SNE representations of the TL output. (b) The encoding output by the language encoder in GLA. (c) The encoding output by the NL encoding layer of the BERT model. (d) The encoding output by the NL encoding layer of the Pre-trained BERT-continuous.
(c) BERT
(d) OIL baseline
action embedding and then encodes the NL into the learned latent action space, which is a binary vector with the same size as TL.
Experimental results. Fig. 4 presents the training curves of the instruction-following policies on different natural language instruction datasets. In general, TALAR achieves a superior instruction-following policy that enhances the success rate by  $63.2\%$  on FrankaKitchen and  $13.4\%$  on CLEVR-Robot compared to baseline method and generalizes well to the NL instruction expressions that were not previously encountered. On the FrankaKitchen task requiring more difficult robot manipulation, TALAR significantly outperforms all baselines. This result demonstrates the advantage of IOL, which eases the policy's burden on understanding natural language. On the training NL instruction set, TALAR achieves a high success rate within a reasonable number of timesteps, significantly faster than the other baselines. Additionally, TALAR demonstrates a remarkable success rate on testing sets in both environments, showcasing greater capacity than the baselines. Although One-hot performs adequately on the training NL set, its ability to generalize to the testing NL sets is limited. BERT-based baselines, built on the OIL framework, improve slower than TALAR on the training NL instruction set. In addition, the NL instructions on the FrankaKitchen task are typically more complex. OIL methods become increasingly more challenging to achieve a high task success rate.
5.2 Analysis on the Generated Task Language
We further investigate the reasons behind TALAR's more efficient learning compared to baseline methods. To explore this issue, we employ the optimized translator to convert NL instructions into TL and project the resulting TL onto a two-dimensional plane using t-SNE [47]. The projection results are displayed in Fig. 5a. The results show that the learned TL effectively distinguishes NL instructions representing different goal configurations, developing a unique representation for each configuration. This conclusion is supported by the fact that various NL expressions for the same goal configuration translate into similar, or even identical, TL. Consequently, TALAR can easily recognize these expressions and concentrate on learning how to complete the RL task. For comparison, we also project the NL encoding output generated by other baseline methods, as illustrated in Fig. 5(b-d). The NL encoding output from these baseline methods exhibits greater diversity compared to TALAR. These results suggest that the OIL baseline treats diverse NL expressions for the same task objective as different goal configurations, potentially distracting from RL and slowing policy learning. It is worth noting that BERT, an established pre-trained model for natural language understanding, also fails to generate unique NL encoding. Even when fine-tuned on specific tasks, the output encoding of the BERT model is divergent, as shown in Fig. 5d.
5.3 TL Is an Effective Abstraction for Hierarchical RL
Previous experiments have demonstrated that the resulting TL can compactly and uniquely represent the NL instructions, facilitating policy learning in the following instruction. In this section, we further explore whether the TL learns any compositional structure by examining if it can serve as an effective goal abstraction for hierarchical RL. Specifically, we train a high-level policy outputting a TL, instructing the IFP to complete a low-level task. We define a long-term task as one in which the
(a) CLEVR-Robot
(b) FrankaKitchen
(c) Ablation study
Figure 6: Experiments of long-horizon tasks on different environments. The agent obtains a reward of  $+1$  when completing one goal configuration.
Figure 7: Ablation study on the number of predicate and argument networks.
agent must achieve multiple goal configurations simultaneously. For CLEVR-Robot, we constructed the long-term task following the approach outlined in [9]. In this task, the agent's objective is to arrange objects to satisfy all ten goal configurations that are implicitly contained within it. For FrankaKitchen, the agent is required to complete all six sub-tasks at the same time. Refer to Appendix D.1.3 for additional details regarding the long-term tasks and the hierarchical RL setting.
For comparison, we consider a baseline method called HAL [9], which leverages the compositional structure of NL and directly acts at the NL level. Following HAL's original implementation, the high-level policy outputs the index of the NL instruction rather than the NL words. To ensure a fair comparison, we align all other modules except for the high-level policy of both methods, e.g., using the same IFP trained by TALAR as the low-level policy. Fig. 6 presents the comparison results. The high-level policy that uses TL as a low-level goal abstraction outperforms that using NL in terms of sample efficiency. This result demonstrates that TL can be an effective goal abstraction naturally compatible with hierarchical RL. In practice, the TL can be generated with a task dataset and serves as a sub-task abstraction for high-level policy.
We conduct experiments to investigate how the performance of TALAR is affected by varying the number of predicate networks and the number of arguments (i.e.,  $N_{\mathrm{pn}}$  and  $N_{\mathrm{a}}$ ). Specifically, we choose  $N_{\mathrm{pn}}$  from the set  $\{2, 4, 6\}$  and  $N_{\mathrm{a}}$  from  $\{2, 3, 4\}$ . We trained the IFPs for 4 million timesteps on the FrankaKitchen environment. The experiment results are presented in Fig. 7. In general, increasing  $N_{\mathrm{pn}}$  and  $N_{\mathrm{a}}$  leads to improved performance of the IFP on both the training and testing sets. This result can be attributed to the enhanced representation abilities of the resulting TL, which enables more efficient learning by the policy. However, it is worth noting that excessively high values of  $N_{\mathrm{pn}}$  and  $N_{\mathrm{a}}$  may lead to reduced training efficiency due to the simultaneous learning of multiple networks.  $N_{\mathrm{pn}} = 6$  and  $N_{\mathrm{a}} = 4$  are sufficient for the robot manipulation tasks like FrankaKitchen.
In this study, we introduce a novel learning scheme for NLC-RL, called IOL, which develops task-related language to facilitate policy learning. Our experiments demonstrate that IOL is a promising approach for efficiently training instruction-following agents. However, there are limitations in our proposed implementation of IOL, TALAR. Firstly, TALAR develops the task language using a static task dataset and, therefore, can not be directly applied to the open-world environment, where the NL instructions could vary hugely from the dataset. It is possible to mitigate this issue by dynamically extending the task dataset and fine-tuning the TL generator/Translator during the policy learning process. Additionally, TALAR requires a manual reward function for policy training, which may be inaccessible if the reward design is complex. Fortunately, there have been well-validated methods for solving sparse reward problems [48, 49, 50], an effective substitute for the manual reward function. Furthermore, incorporating basic properties of predicate relationships (such as transitivity, reflexivity, and symmetry) when training the TL generator could make the resulting TL more meaningful and self-contained. Lastly, TALAR implements the translator as a fully-connected layer with a pre-trained BERT model, which may be inadequate for more complex natural language instructions. This
limitation could be addressed by employing other Natural Language Processing (NLP) literature techniques, such as semantic parsing.
Recently, there has been a growing interest in building instruction-following agent with LLMs as the brain [51, 52, 53]. Typically, LLMs function as planners, while a low-level policy is responsible for executing the planned tasks. However, the separation between the LLM planner and the task execution process may impede the overall task completion rate. The challenge of managing increasingly complex language instructions in open environments remains an unresolved issue. Furthermore, it is worth investigating whether LLMs can serve as effective task language generator. We hope that future research will explore these intriguing questions and contribute to developing agents that interact with humans more effectively.
This work is partly supported by the National Key Research and Development Program of China (2020AAA0107200) and the National Science Foundation of China (61921006). The authors extend their appreciation to Wang-Zhou Dai and Yuxuan Huang for their valuable discussions on predicate representation, Xinchun Li and Yikai Zhang for their insights on language model training, Jingxuan Han for experiment details, Lei Yuan for discussing the potential applications of TALAR, and the anonymous reviewers for their useful feedback on this study.
[18] Dipendra Kumar Misra, Andrew Bennett, Valts Blukis, Eyvind Niklasson, Max Shatkhin, and Yoav Artzi. Mapping instructions to actions in 3d environments with visual goal prediction. In EMNLP, 2018.
[19] Devendra Singh Chaplot, Kanthashree Mysore Sathyendra, Rama Kumar Pasumarthi, Dheeraj Rajagopal, and Ruslan Salakhutdinov. Gated-attention architectures for task-oriented language grounding. In AAAI, 2018.
[20] Dean Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural Computation, 3:88-97, 1991.
[21] Valts Blukis, Chris Paxton, Dieter Fox, Animesh Garg, and Yoav Artzi. A persistent spatial semantic representation for high-level natural language instruction execution. In CoRL, 2022.
[22] Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni. Multi-agent cooperation and the emergence of (natural) language. In ICLR, 2017.
[23] Changxi Zhu, Mehdi Dastani, and Shihan Wang. A survey of multi-agent reinforcement learning with communication. arXiv, 2203.08975, 2022.
[24] Afshin Oroojlooyjadid and Davood Hajinezhad. A review of cooperative multi-agent deep reinforcement learning. arXiv, 1908.03963, 2019.
[25] Sheng Li, Yutai Zhou, Ross E. Allen, and Mykel J. Kochenderfer. Learning emergent discrete message communication for cooperative reinforcement learning. In ICRA, 2022.
[26] Shivansh Patel, Saim Wani, Unnat Jain, Alexander G. Schwing, Svetlana Lazebnik, Manolis Savva, and Angel X. Chang. Interpretation of emergent communication in heterogeneous collaborative embodied agents. In ICCV, 2021.
[27] Frans A. Oliehoek and Christopher Amato. A Concise Introduction to Decentralized POMDPs. Springer Briefs in Intelligent Systems. Springer, 2016.
[28] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In ICLR, 2015.
[29] Irene Rivera-Trigueros. Machine translation systems and quality assessment: a systematic review. Language Resources and Evaluation, 56:593-619, 2022.
[30] Felix Stahlberg. Neural machine translation: A review. Journal of Artificial Intelligence Research, 69:343-418, 2020.
[31] Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks. In NeurIPS, 2014.
[32] Artidoro Pagnoni, Kevin Liu, and Shangyan Li. Conditional variational autoencoder for neural machine translation. arXiv, 1812.04405, 2018.
[33] Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.
[34] Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley Series in Probability and Statistics. Wiley, 1994.
[35] Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction. IEEE Transactions on Neural Networks, 9(5):1054-1054, 1998.
[36] Masataro Asai. Unsupervised grounding of plannable first-order logic representation from images. In ICAPS, 2019.
[37] Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. In ICLR, 2013.
[38] Alexander Borgida. On the relative expressiveness of description logics and predicate logics. Artificial Intelligence, 82:353-367, 1996.
[39] Minghuan Liu, Menghui Zhu, and Weinan Zhang. Goal-conditioned reinforcement learning: Problems and solutions. In *IJCAI*, 2022.
[40] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In ICLR, 2017.
[41] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv, 1707.06347, 2017.
[42] OpenAI. GPT-4 technical report. arXiv, 2303.08774, 2023.
[43] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In IROS, 2012.
[44] Seiya Tokui and Issei Sato. Reparameterization trick for discrete variables. arXiv, 1611.01239, 2016.
[45] Dean Pomerleau. ALVINN: an autonomous land vehicle in a neural network. In David S. Touretzky, editor, Advances in Neural Information Processing Systems 1, [NIPS Conference, Denver, Colorado, USA, 1988], pages 305-313. Morgan Kaufmann, 1988.
[46] David Gaddy and Dan Klein. Pre-learning environment representations for data-efficient neural instruction following. arXiv, 1907.09671, 2019.
[47] Van der Maaten, Laurens, and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9:2579-2605, 2008.
[48] Marcin Andrychowicz, Dwight Crow, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. In NeurIPS, 2017.
[49] Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Overcoming exploration in reinforcement learning with demonstrations. In ICRA, 2018.
[50] Martin A. Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Van de Wiele, Vlad Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by playing solving sparse reward tasks from scratch. In ICML, 2018.
[51] Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, et al. Do as i can, not as i say: Grounding language in robotic affordances. In CoRL, 2022.
[52] Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. In CoRL, 2022.
[53] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022.
A Discussion about Predicate Representation 15
B Additional Related Work 15
C Algorithm Descriptions 16
D Implementation Details 16
D.1 Tasks for Evaluation 17
D.2 Network architecture 22
D.3 Hyper-Parameters 22
E Additional Experiment Results 23
E.1 Training IFP with Different Number of NL Sentence Patterns 23
E.2 Complete Results of T-SNE Projection of Different Representations 23
E.3 Aligning the network architecture of baselines with TALAR 23
E.4 Ablation study on the size of dataset 23
E.5 Fine-tuning BERT-baed baselines 25
E.6 End-to-end TALAR 25
E.7 Replacing generator with hand-crafted feature 26
E.8 Deployment Examples of IFP Trained by TALAR 28
F Retrievals for Notations and Abbreviations 30
G Societal Impact 30
Appendix References 31
A Discussion about Predicate Representation
A crucial aspect of IOL is how the task language is represented. We believe that expressiveness and conciseness are essential properties of language representation. Expressiveness ensures that the task language accurately reflects the goal configuration, while conciseness promotes ease of policy comprehension. To fulfil these requirements, we implement IOL by representing task language with predicate representation, which is considered both expressive [38] and concise as a discrete representation. The expressiveness of predicate representation stems from two key advantages: compositional structure and interpretability.
We first discuss the compositionality, which refers to the ability of a language to denote novel composite meanings. For example, if a language can represent blue circle and red square, it can represent blue square as well. This language has a compositional structure. The compositionality is seen as a fundamental feature of natural language and a pre-condition for a language to generalize at scale [55]. Predicate representation naturally has compositional structure since it can denote the
composite meanings by changing the truth value of predicates.
Next, we talk about the interpretability. Predicate representation uses multiple predicate expressions, such as Pred(a1, a2), to describe the relationships in a specific environment. Previous works have found that, even if the predicate symbols are machine-generated by a learning system and anonymous to humans, they still offer some extent of interpretability. For example, if a learning system generates a predicate expression Pred(cat, grassland)  $=$  True for Fig. 8, we can guess that Pred means [stand] or [above]. When more figures and predicate expressions are provided, the actual meaning of these anonymous predicate symbols would be more apparent to humans. The primary focus of this paper is to explore efficient learning methods, and as such, we do not delve into the topic of interpretability.
Figure 8: An illustration of the interpretability of predicate representation. For an anonymous predicate expression: Pred(cat, grassland)  $=$  True, we can guess that Pred represents [above].
In conclusion, we propose to use predicate representation to represent task language since nearly all tasks contain relationships. For instance, an Atari game [56] includes a relationship between the agent and the antagonist, a first-person shooter game contains a relationship between the gun and the bullets, a stock trading task contains a relationship between stocks and price, etc. Even if some tasks are highly abstract and have no apparent relationships, we can implement the IOL by replacing predicate representation binary or continuous vectors.