SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We carried out further experiments where the parameters of the BERT model in baseline methods are updated during the training process, as depicted in Fig. 14. The results indicate that when the parameters of BERT are optimized, the baseline methods struggle to achieve successful task completion. We hypothesize that this outcome may be attributed to the extensive parameter count of the BERT model, which potentially increases the complexity of the learning process.
E.6 End-to-end TALAR
We conduct additional experiment that trains TL generator and translator jointly on FrankaKitchen task, following the original experiment setting, as shown in the Fig. 15. The experiment results indicate that training the two modules independently is more effective in training a policy.
Figure 14: Training curves of fine-tuned baselines on FrankaKitchen environments. The fine-tuned baselines are denoted with the prefix "Pretrained".
Figure 15: Training curves of end-to-end TALAR on FrankaKitchen environments.
E.7 Replacing generator with hand-crafted feature
We conduct a supplementary experiment to compare TALAR with a manually crafted structural representation on the FrankaKitchen task. To implement the new baseline, the NL instructions are parsed into a one-hot vector which indicates the current goal-configuration, serving as the structured representation. Subsequently, we implement a translation process same to that of TALAR. All other experimental setting remains consistent with those outlined in our original study. The results of this experiment are presented in the table below, where 'Handcrafted' denotes the new baseline trained based on the handcrafted representation.
The results show that the performance of TALAR is comparable to the handcrafted representation in term of learning speed and the final score (Figure 6). This experiment further justify the effectiveness and conciseness of the task language learned by TALAR.
Figure 16: Training curves of hand-crafted features on FrankaKitchen environments.
E.8 Deployment Examples of IFP Trained by TALAR
We visualize the performance of the IFP trained by TALAR in the FrankaKitchen and CLEVR-Robot environment. Fig. 17 and Fig. 18 presents the instruction-following examples on CLEVR-Robot and FrankaKitchen environments, respectively. The trained IFP demonstrates a strong ability to accurately understand natural language instructions and efficiently accomplish the assigned tasks.
Can you push the red ball behind the blue ball?
The red ball is pushed behind the blue ball.
The red ball moves to the right of the cyan ball.
Is there any red ball in front of the purple ball?
The red ball was moved in front of the purple ball.
Figure 17: A visualization of the TALAR's IFP deployment process on CLEVR-Robot.
Please open the door of the microwave.
Can you please put the kettle on the stove, robot?
I need some light, can you help me?
Would you mind opening the slide cabinet door?
Can you turn on the top burner by twisting the oven knob?
Turn the oven knob to activate the bottom burner.
Figure 18: A visualization of the TALAR's IFP deployment process on FrankaKitchen.
F Retrievals for Notations and Abbreviations
Table 4: Notations and abbreviations in this paper.
Name	Meaning
Notations
LT	task language
LN	natural language
qφ1	encoder with parameters φ1
pφ2	decoder with parameters φ2
gθ	TL generator with parameters θ
Abbreviations
NL	Natural Language
TL	Task Language
LM	Language Model
RL	Reinforcement Learning
IOL	Inside-Out Learning
OIL	Outside-In Learning
RG	Referential Game
GLA	Grounding Language to latent Action
NLC-RL	Natural Language-Conditioned Reinforcement Learning
G Societal Impact
This research enhances the success rate and adaptability of instruction-following policies in response to various natural language expressions. The development of NLC-RL agents capable of effectively following human instructions holds the potential to impact multiple aspects of society. One of the most immediate societal impacts lies in improving human-robot collaboration. As these RL agents become increasingly proficient in comprehending and executing human instructions, they can be integrated into various applications, including manufacturing, logistics, and service industries. This integration may lead to heightened efficiency, reduced labour costs, and enhanced safety across these sectors. Moreover, these RL agents' advanced natural language understanding can be applied to assistive technologies for individuals with disabilities. For instance, they could be utilized in intelligent home systems to facilitate navigation for those with mobility impairments or to support individuals with cognitive impairments in accomplishing daily tasks. Nonetheless, it is important to consider the ethical implications of deploying these RL agents in real-world settings. Ensuring that these systems maintain transparency and accountability is of paramount importance.
[54] Alexander Borgida. On the relative expressiveness of description logics and predicate logics. Artificial Intelligence, 82:353-367, 1996.
[55] Angeliki Lazaridou and Marco Baroni. Emergent multi-agent communication in the deep learning era. arXiv, 2006.02419, 2020.
[56] M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
[57] Shubham Pateria, Budhitama Subagdja, Ah-Hwee Tan, and Chai Quek. Hierarchical reinforcement learning: A comprehensive survey. ACM Computing Surveys, 54(5):109:1-109:35, 2021.
[58] Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. In ICML, 2017.
[59] Nicholas K Jong, Todd Hester, and Peter Stone. The utility of temporal abstraction in reinforcement learning. In AAMAS, 2008.
[60] Ofir Nachum, Haoran Tang, Xingyu Lu, Shixiang Gu, Honglak Lee, and Sergey Levine. Why does hierarchy (sometimes) work so well in reinforcement learning? arXiv, 1909.10618, 2019.
[61] Xintong Yang, Ze Ji, Jing Wu, Yu-Kun Lai, Changyun Wei, Guoliang Liu, and Rossitza Setchi. Hierarchical reinforcement learning with universal policies for multistep robotic manipulation. IEEE Transactions on Neural Networks and Learning Systems, 33:4727-4741, 2022.
[62] Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. In NeurIPS, 2018.
[63] Yiding Jiang, Shixiang Gu, Kevin Murphy, and Chelsea Finn. Language as an abstraction for hierarchical deep reinforcement learning. In NeurIPS, 2019.
[64] Karl Pertsch, Youngwoon Lee, and Joseph J. Lim. Accelerating reinforcement learning with learned skill priors. In CoRL, 2020.
[65] Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22:268:1-268:8, 2021.
[66] Rishabh Agarwal, Chen Liang, Dale Schuurmans, and Mohammad Norouzi. Learning to generalize from sparse and underspecified rewards. In ICML, 2019.
[67] Geoffrey Cideron, Mathieu Seurin, Florian Strub, and Olivier Pietquin. Higher: Improving instruction following with hindsight generation for experience replay. In SSCI, 2020.
[68] Marcin Andrychowicz, Dwight Crow, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. In NeurIPS, 2017.
[69] Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. In CoRL, 2019.