SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hierarchical Reinforcement Learning (HRL) approaches are promising tools for solving complex decision-making tasks [57]. HRL solves the problem by decomposing it into simpler sub-tasks using a hierarchy of policies learned by RL [58, 59, 60]. For example, to put a drink into a fridge, you will (1) take up the drink, (2) open the fridge door, (3) put down the drink, and (4) close the fridge door. Putting a drink into a fridge is a long-term task, while these four are sub-tasks. Typically, a high-level policy is trained to decompose the main task into sub-tasks (three steps in the example), and low-level policies (or single policy) are trained to complete these sub-tasks. A core problem in HRL is how to represent the goal of the sub-task (i.e., goal representation) that the low-level policy follows. Previous works have investigated both concrete and abstract goal representations of a sub-task. Concrete goal representation can be the target position to reach [61], or a target state [62]. As for abstract goal representation, it can directly be natural language [9] or an encoding of action primitives [64]. Back to the topic of this paper, language is a natural goal representation for the sub-tasks, such as open the fridge door in the above example. Through our experiments (Section 5.3), we observe that the resulting TL is an effective abstraction of the goal of the sub-task, which demonstrates the applicability of TL to work together with HRL.
C Algorithm Descriptions
Algorithm 1, 2, and 3 present the training procedures of the TL generator, the translator and the instruction-following policy, respectively.
Algorithm 1 Training procedure of the TL generator.
Input: task dataset  $\mathcal{D} = \{(s,s^{\prime},L_{\mathrm{N}})_i\}$  , pre-trained BERT model  $b$    
Output: the optimized TL generator. 1: Initialize the TL generator  $g_{\theta}$  2: while training not complete do 3: Sample a batch of data from  $\mathcal{D}$  4: Update  $\theta$  to minimize the RG loss (Eq.(2)). 5: end while 6: return the optimized TL generator  $g_{\theta}$
Algorithm 2 Training procedure of the translator.
Input: task dataset  $\mathcal{D} = \{(s,s^{\prime},L_{\mathrm{N}})_i\}$  , the optimized TL generator  $g_{\theta}$  , the translator network  $p_{\phi}$  parameterized with  $\phi$  , and the pre-trained BERT model.   
Output: the optimized TL translator..   
1: Initialize the translator  $t_{\phi_1,\phi_2}$  with parameters  $\phi_{1}$  and  $\phi_{2}$    
2: while training not complete do   
3: Sample a batch of data  $\{(s,s^{\prime},L_{\mathrm{N}})_j\}$  from  $\mathcal{D}$    
4: // Compute the target task language.   
5: Calculate the task language  $L_{\mathrm{T}} = g_{\theta}(s,s^{\prime})$    
6: // Optimize the translator.   
7: Update  $\phi$  to minimize the loss in Eq.(3).   
8: end while   
9: return the optimized translator  $p_{\phi}$
Algorithm 3 Training procedure of the instruction-following policy.
Input: the optimized translator  $p_{\phi}$    
Output: the optimized instruction-following policy.   
1: Initialize the policy function  $\pi$  , and the value function.   
2: while training not complete do   
3: Sample a NL instruction  $L_{\mathrm{N}}$  from the environment.   
4: Generate corresponding task language  $\widetilde{L}_{\mathrm{T}} = p_{\phi}(L_{\mathrm{N}})$    
5: // Collecting samples   
6: while episode not terminal do   
7: Observe current state  $s_t$    
8: Execute action  $a_{t}\sim \pi (\cdot |s_{t},\widetilde{L_{\mathrm{T}}})$  , and receive a reward  $r_t$  from the environment.   
9: end while   
10: // Training   
11: Update the policy and value functions with the RL algorithm.   
12: end while   
13: return the optimized policy  $\pi$
In our experiments, we utilize the open-sourced RL repository, stable-baselines3 [65], to implement the RL optimization. The parameters of the BERT model are fixed if not otherwise specified. All experiments are run five times with different random seeds. We will now introduce two experimental environments, followed by a detailed explanation of the model architecture of our method and the hyper-parameters utilized in our experiments.
D.1 Tasks for Evaluation
D.1.1 FrankaKitchen
Task descriptions As depicted in Fig. 3, the FrankaKitchen environment emulates a realistic kitchen setting with objects that can be interacted with, including a 9-DoF Franka robot, a microwave, a kettle, an overhead light, cabinets, and an oven. The observation space is defined as  $\mathbb{R}^{30}$ , representing the state of each object. The action space is  $\mathbb{R}^9$ , with 7 DoF for the arm and 2 for the gripper. The actions are represented as the joint velocity. At each timestep, the reward is calculated as the difference between the distance to the target robot's posture at the previous step and the distance to the target robot's posture at the current step. A reward of  $+1$  is given for success, and a 0 is provided for failure.
Natural language instructions. In FrankaKitchen, six distinct goal configurations are related to operating various objects. To generate natural language instructions that describe these goal configurations, we employ ChatGPT to produce everyday language sentences. For example, we prompt ChatGPT with: "I want to ask my robot to open the microwave door. Can you provide me with 50 different instructions I can use?" We obtain  $6 \times 50 = 300$  unique natural language instructions in this manner. To conserve space, we only present  $6 \times 20$  (i.e., 10 in the training dataset + 10 in the testing dataset) = 120 natural language instructions used in our experiments.
- (Training set)
- Please open the door of the microwave.
- Can you open the microwave door for me?
- I would appreciate it if you could open the microwave door.
- Can you open the microwave door at your earliest convenience?
- Would you be able to open the door of the microwave?
- I need the door of the microwave to be opened, please.
- Please, could you open the door of the microwave?
- I require you to open the door of the microwave.
- Could you open the microwave door when you get a chance?
- Would it be possible for you to open the door of the microwave?
- Could you fill the kettle with water and place it on the stove?
- It's tea time! Could you turn on the stove and place the kettle on top?
- I need some hot water. Could you fill the kettle, put it on the stove, and turn on the burner?
- Please prepare some hot water by placing the kettle on the burner and turning on the stove.
- Can you fill the kettle, put it on the stove, and turn on the burner to heat up the water?
- Hey robot, can you prepare some hot water by putting the kettle on the burner and turning on the heat?
- Can you please put the kettle on the stove, robot?
- Robot, it's tea time! Please put the kettle on the burner and heat up the water.
- Hey robot, could you heat up the water in the kettle on the burner?
- I'm feeling chilly, robot. Can you put the kettle on the stove and boil some water?
- Could you illuminate the room for me?
- I need some light, can you help me?
- Can you activate the lighting system?
Light up the room, please.
- I want to see better, please turn on the lights.
- Can you switch on the room's lighting, please?
- Can you brighten up the space?
- I need some light in here, can you turn on the lights?
- I need some illumination, can you help me?
- Would you mind me switching the light?
- Please open the cabinet door on the sliding mechanism right away.
- Please open the cabinet door by sliding it.
- I would appreciate it if you could open the sliding cabinet door.
- Please slide the cabinet door open as quickly as possible.
- Please open the sliding cabinet door promptly.
- Would you mind opening the slide cabinet door?
- Please open the cabinet door by sliding it as soon as possible.
- Can you slide open the cabinet door quickly, please?
- Can you please open the cabinet door by sliding it now?
- I need your help, open the slide cabinet door please.
- Could you kindly activate the top burner by turning the oven knob?
- Twist the oven knob to activate the top burner.
- Activate the top burner by turning the oven knob clockwise.
- Kindly turn the oven knob to activate the top burner.
- Please turn the oven knob to switch on the top burner.
- The top burner is activated.
- The top burner can be activated by turning the oven knob to the right.
- Activate the top burner by turning the oven knob to the right on the stove.
- Please turn the oven knob to switch on the top burner on the stove.
- Turn the knob on the oven that corresponds to the top burner.
- Please turn the knob on the oven that activates the bottom right burner.
- Could you turn the knob on the oven so that the bottom right burner activates?
- Would you kindly turn the knob on the oven so that the bottom right burner activates?
- Activate the bottom right burner by turning the knob on the oven, Robot.
- Please turn the oven knob to activate the bottom right burner, Robot.
- Would you mind activating the bottom right burner on the oven by turning the knob?
- Can you turn the oven knob to activate the bottom right burner, Robot?
- Robot, please activate the bottom right burner on the oven by turning the knob.
- Manipulate the oven knob to turn on the burner at the bottom right.
- Turn the oven knob to activate the bottom burner.
- (Testing set)
- Can you please open the door of the microwave?
- I need the microwave door to be opened.
- Could you open the door of the microwave quickly?
- Can you open the microwave door quickly?
- I'd like you to open the microwave door for me, please.
- Would you mind opening the door of the microwave for me?
- Can you assist me by opening the microwave door?
- Can you kindly open the door of the microwave?
- Can you please open the microwave door without delay?
- Could you please open the door of the microwave?
- Can you start heating up the water by placing the kettle on the burner and turning on the stove?
- Please start boiling some water in the kettle on the burner, robot.
- Robot, could you put the kettle on the burner and heat up the water?
- Let's make some tea. Can you put the kettle on the burner and turn on the heat?
- Can you please heat up some water by putting the kettle on the burner and turning on the stove?
- Please heat up the kettle by putting it on the burner.
- Please put the kettle on the burner, robot. I'm craving a cup of tea.
- I need some hot water. Can you put the kettle on the stove and turn on the burner?
- Can you please boil some water by putting the kettle on the burner and turning on the stove?
- I'm craving some tea. Can you put the kettle on the burner and turn on the heat?
- Please, illuminate the space.
- Turn on the lights, please.
- Could you turn on the lighting system?
- I need some illumination, can you turn on the lights?
- Can you switch on the lights in here, please?
- Please, switch on the lights.
- Please, turn on the lights in this room.
- Can you brighten up the room?
- Please switch on the light.
- Can you brighten up this space for me?
- Please slide the cabinet door open.
- Please slide the cabinet door open now.
- Can you open the cabinet door by sliding it without delay?
- Could you please open the cabinet door by sliding it?
- Can you open the sliding cabinet door right away?
- Could you slide the cabinet door open?
- Please, would you mind me opening the slide cabinet door.
- I need you to open the sliding cabinet door.
- Can you open the sliding door on the cabinet, please?
- Could you please slide the cabinet door open?
- Twist the oven knob to turn on the top burner.
- Manipulate the oven knob to activate the top burner.
- Turn the oven knob to the right to activate the top burner on the stove.
- Could you please turn the oven knob to activate the top burner?
- Use the oven knob to activate the top burner on the stove.
- Please turn the oven knob to activate the top burner.
- Turn the oven knob clockwise to activate the top burner on the stove.
- Use the oven knob to switch on the top burner on the stove.
- Can you turn on the top burner by twisting the oven knob?
- Turning the oven knob will switch on the top burner.
- Would you mind turning the oven knob to activate the bottom burner?
- The bottom burner is activated.
- Rotate the knob located on the oven to start the burner at the bottom right.
- Please turn the oven knob so that the bottom right burner activates, Robot.
- Please activate the bottom right burner on the oven by turning the knob, Robot.
- Would you mind turning the oven knob to activate the bottom right burner, Robot?
- Activate the bottom right burner on the oven by turning the knob, please.
- Turn the knob on the oven in order to activate the bottom right burner.
- Adjust the oven knob to the position that turns on the burner at the bottom right.
- Turn the oven knob to activate the bottom right burner, Robot.
In each trajectory, a natural language instruction is randomly selected, after which the agent makes decisions to complete the given instruction. This experiment aims to not only confirm the effectiveness of TALAR but also demonstrate its robustness in handling diverse and realistic scenarios.
D.1.2 CLEVR-Robot
Task descriptions. As depicted in Fig. 3, The CLEVR-Robot environment requires the agent to manipulate five balls in accordance with human instructions. The observation space is  $\mathbb{R}^{10}$ , which represents the location of each object, and  $|\mathcal{A}| = 40$  corresponds to selecting and pushing an object in one of the eight cardinal directions. At each timestep, the reward is calculated as the difference
between the distance to the target position at the previous step and the distance to the target position at the current step, with a reward of  $+100$  for success and -10 for failure. Some previous OIL-based methods attempt to train the IFP in a sparse reward environment [66, 67]. We relax the sparse reward constraint to dense reward for two reasons: (1) Sparse reward is not the point of this study. We are more concerned with exploring a new learning scheme for NLC-RL than with resolving the sparse reward problem; (2) The applicability of current methods. Most RL approaches address the sparse reward problem using HER [48], which relabels the origin goal in the trajectory with the actual achieved state. Nonetheless, the relabeling procedure is contingent on the transformation from the state to the goal, which necessitates additional human annotation or program design.
NL instructions. We use eighteen different NL sentence patterns to express each goal configuration. For example, if we take a goal configuration such as "red ball in front of the blue ball", its corresponding NL instructions (i.e., eighteen NL sentence patterns) can be one of the following:
- (Training set)
- Push the red ball in front of the blue ball.
- Can you push the red ball in front of the blue ball?
- Can you help me push the red ball in front of the blue ball?
- Is the red ball in front of the blue ball?
- Is there any red ball in front of the blue ball?
- The red ball moves in front of the blue ball.
- The red ball is being pushed in front of the blue ball.
- The red ball is pushed in front of the blue ball.
- The red ball was moved in front of the blue ball.
- (Testing set)
- Move the red ball in front of the blue ball.
- Keep the red ball in front of the blue ball.
- Can you move the red ball in front of the blue ball?
- Can you keep the red ball in front of the blue ball?
- Can you help me move the red ball in front of the blue ball?
- Can you help me keep the red ball in front of the blue ball?
- The red ball is being moved in front of the blue ball.
- The red ball is moved in front of the blue ball.
- The red ball was pushed in front of the blue ball.
There are two kinds of datasets: training and testing. In the training set, goal configurations are expressed using nine NL sentence patterns, while the remaining nine NL sentence patterns are used in the testing set. At the start of each trajectory, the environment randomly samples a goal configuration and an NL sentence pattern to express the human instruction. By constructing these two datasets, we simulate the scenario in an open environment where different individuals instruct the robots using their linguistic preferences.
D.1.3 Hierarchical RL.
Hierarchical RL setting The observation space of the high-level policy is the same as that of the low-level policy, i.e. IFP, resulting in  $\mathbb{R}^{10}$  for CLEVR-Robot and  $\mathbb{R}^{30}$  for FrankaKitchen. Our method utilizes TL space as the action space, while HAL uses NL instruction space. The action output by the high-level policy is then treated as the goal configuration of the IFP. We roll out the IFP for ten steps for every high-level action. We use sparse reward to verify the effectiveness of the high-level goal abstraction, i.e., the goal configuration assigned to IFP. The reward is calculated at each timestep as the count of the goal configurations achieved at the current timestep.
CLEVR-Robot As demonstrated in Fig.9a, the object arrangement task was proposed by [9], which aims to rearrange the objects in the environment to satisfy all ten implicit constraints. At the beginning of a trajectory, the environment resets the position of all balls to a random location. Following [9], the precise arrangement constraints are: (1) red ball to the right of purple ball. (2) green ball to the right of red ball. (3) green ball to the right of cyan ball. (4) purple ball to the left of cyan ball. (5) cyan ball to the right of purple ball. (6) red ball in front of blue ball. (7) red ball to the left of green
(a) CLEVR-Robot
Figure 9: Visualization of completed long-term tasks in the high-level environment.
ball. (8) green ball in front of blue ball. (9) purple ball to the left of cyan ball. (10) blue ball behind the red ball.
FrankaKitchen As demonstrated in Fig.9b, the multitask environment was introduced by [8], which aims to complete all six goal configurations. At the beginning of a trajectory, the environment resets all sub-tasks to their initial state, as shown in the leftmost column of Fig. 18.
D.2 Network architecture
To implement TALAR, we employ the IFP network architecture illustrated in Fig. 10. Both the policy network and value network share the same architecture, but they have different output dimensions. In our experiments, all baselines utilize the same IFP network architecture as TALAR. The primary distinction lies in how these baselines encode natural language instructions. The natural language encoding for baselines is the same size as TL, and the concatenated vectors (i.e., [state, NL encoding]) serve as input for the policy and value networks.
Figure 10: Network architecture of instruction-following policy network in TALAR.
D.3 Hyper-Parameters
The hyper-parameters for implementing TALAR are presented in Table 2. When implementing baseline methods, we use the same hyper-parameters of PPO for policy learning.
Table 2: Hyper-parameters.
Hyper-parameters	Value
Na	2
Npn	4
Size of ArgNet's output	5
Learning rate (LR) for LRG	3e-4
LR for LT	3e-4
MLP networks in translator	[256, 256, 256, 256, |LT|], relu
Predicate network	[128, 128, 2], relu
Argument network	[128, 128, 5], relu
PPO epoch	10
PPO policy LR	3e-4
PPO value LR	3e-4
PPO policy network	[32, 64, 64], tanh
PPO value network	[32, 64, 64], tanh
PPO mini-batch size	128
PPO nums of mini-batch	160
GLA action encoder net	[2|S|,128,128,64], relu
GLA action decoder net	[64,128,128,|S|], relu
E.1 Training IFP with Different Number of NL Sentence Patterns
We evaluate the robustness of TALAR by training an instruction-following policy under varying numbers of natural language sentence patterns. Specifically, we utilize the CLEVR-Robot environment and train the IFP with NL patterns number from the set  $\{1, 5, 9\}$ . Table 3 displays the experimental results. As the number of sentence patterns increases, TALAR consistently maintains a near  $100\%$  success rate. This is due to TALAR's ability to uniquely represent NL expressions across different sentence patterns, allowing the IFP to comprehend instructions more easily and learn to execute tasks efficiently. These results suggest that developing a task-related language is beneficial for policy learning. In contrast, the performance of baseline methods declines significantly as the number of NL sentence patterns increases, as they must simultaneously understand a greater variety of NL expressions and acquire the skills to execute tasks.
Table 3: A summary of the final success rate (\%) on the training set with a different number of NL sentence patterns. Each IFP is trained for 2M timesteps on CLEVR-Robot and evaluated for 40 episodes. The results are averaged over 5 seeds.
Pattern nums. Method	1	5	9
TALAR	100%	100%	99.9%
One-hot	98.2%	91.8%	86.5%
BERT-binary	82.0%	60.9%	64.0%
BERT-continuous	95.3%	63.1%	60.7%
E.2 Complete Results of T-SNE Projection of Different Representations
Figure 11 presents the t-SNE projection results of various representations on both training and testing natural language instruction datasets. Upon analyzing the training dataset of natural language instructions, it is obvious that the t-SNE projections of the task language exhibit a higher degree of clustering compared to the baselines. This observation implies that the IOL method effectively learns a concise task goal representation, which is suitable for solving NL instruction following task. The BERT-binary representation exhibits a certain capacity to distinguish natural language instructions. Nevertheless, all the data points tend to get close, suggesting a less pronounced separation of different NL instructions in comparison to the task language. In the testing dataset, the task language representation surpasses other baselines, as data points with the same marker display a higher concentration. This result suggests that the task language is more robust against unseen natural language expressions. On the other hand, the t-SNE projections of the baselines appear scattered across the plane, which imposes a natural language understanding challenge on policy learning.
E.3 Aligning the network architecture of baselines with TALAR
In this section, we align the network structures of several baselines with TALAR in order to eliminate the impact of the additional networks on performance in TALAR. More specifically, we have added new BERT-based baselines on FrankaKitchen task, which employ the same architecture as the TL generator in TALAR, adhering to the BERT model. Other experimental settings remain same with the experiments in the paper. These new baselines are denoted with the prefix "Aligned", and their respective experimental results are presented in the Fig. 12. The performance of the baseline improves with the implementation of a new network architecture. However, it remains inferior to the TALAR approach in terms of learning efficiency and convergence scores. This highlights the effectiveness of the TALAR method.
E.4 Ablation study on the size of dataset
We recognize that the size of task dataset is a significant factor to consider. Thus we have conducted additional experiments to investigate the impact of task sample number on the performance of the
Figure 11: The t-SNE projection of different representations on two NL expressions datasets. Each row represents one kind of representation, and each column stands for different NL instruction datasets. Points with the same marker encode nine different NL expressions that describe the same goal configuration.
Figure 12: Training curves on FrankaKitchen environments. The new baselines with the same network architecture as TALAR are denoted with the prefix "Aligned".
algorithm. The results are presented in the Fig. 13. The results demonstrate that 10,000 samples is sufficient to train a policy that achieves a success rate of  $76\%+$ , which clearly outperforms the performance of other baseline methods. These experimental results suggest that TALAR can effectively train a robust policy even with a limited number of samples in the task dataset.
Figure 13: Training curves of TALARtrained on different size of datasets on FrankaKitchen environments. The numbers after TARLAR indicate the size of the dataset used.
E.5 Fine-tuning BERT-baed baselines