SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jing-Cheng Pang*, Xinyu Yang*, Si-Hang Yang, Xiong-Hui Chen, Yang Yu†
National Key Laboratory of Novel Software Technology, Nanjing University
Polixir Technology
{pangjc, yangxy, yangsh, chenxh}@lamda.nju.edu.cn, yuy@nju.edu.cn
Natural language-conditioned reinforcement learning (RL) enables agents to follow human instructions. Previous approaches generally implemented language-conditioned RL by providing the policy with human instructions in natural language (NL) and training the policy to follow instructions. In this is outside-in approach, the policy must comprehend the NL and manage the task simultaneously. However, the unbounded NL examples often bring much extra complexity for solving concrete RL tasks, which can distract policy learning from completing the task. To ease the learning burden of the policy, we investigate an inside-out scheme for natural language-conditioned RL by developing a task language (TL) that is task-related and easily understood by the policy, thus reducing the policy learning burden. Besides, we employ a translator to translate natural language into the TL, which is used in RL to achieve efficient policy training. We implement this scheme as TALAR (TAsk Language with predicAte Representation) that learns multiple predicates to model object relationships as the TL. Experiments indicate that TALAR not only better comprehends NL instructions but also leads to a better instruction-following policy that significantly improves the success rate over baselines and adapts to unseen expressions of NL instruction. Besides, the TL is also an effective sub-task abstraction compatible with hierarchical RL.
Enabling robots to collaborate effectively with humans is a crucial aspect of machine intelligence. Natural language is a potent tool that reflects human intention and has been widely researched for instructing robot execution, designing rewards, and serving as an observation or action in reinforcement learning (RL)[1]. We are particularly interested in developing agents to follow human instructions in this broad context. Natural language-conditioned reinforcement learning (NLC-RL) is a promising approach to training instruction-following agents. It provides the policy with human instructions in natural language (NL) and trains it to follow instructions with RL algorithms. In this outside-in learning (OIL) scheme, the natural language instructions are directly exposed to the policy, requiring it to understand and interpret them while accomplishing RL tasks (Fig. 1-left).
However, natural language is a complex and unbounded representation of human instruction. It places an additional burden on the policy to understand diverse natural languages when solving a specific RL task. To address this issue, previous studies have attempted to convert the natural language to simpler latent embeddings [2] using pre-trained language models such as BERT [3] and GPT [4]. However, these embeddings are typically learned independently of the RL task, making capturing the task-related information in NL instructions difficult. Alternatively, some methods
[5, 6, 7] design rule-based, task-related semantic vectors to represent NL instructions. These methods show promising results but require a laborious design of semantic vectors and have limitations in handling complex NL instructions.
This paper proposes a novel Inside-Out Learning (IOL) scheme for NLC-RL. As shown in Fig. 1-right, IOL automatically generates a task language (TL) that is both task-related and concise, enabling easy comprehension by the policy. IOL consists of three key components: (1) a TL generator that develops task language, (2) a translator that translates NL into TL, which facilitates policy learning in RL, and (3) a policy that solves the RL tasks. We introduce a specific implementation of IOL, called TALAR for TAsk Language with predicAte Representation. In particular, TALAR models the TL generation process as a referential game, known for its effectiveness in developing effective language. Besides, TALAR learns to model the relation
Figure 1: An illustration of OIL and IOL schemes in NLC-RL. Left: In OIL, the policy directly takes the (outside) NL instructions embedding as the input. Right: IOL first develops an (inside) TL, and policy takes TL as the input. The dashed lines represent TL development and translation, while the solid lines represent the instruction following the process.
ships between objects as TL. Given the labelled NL to TL, the translator is trained to minimize the difference between the translated TL with the target TL. Finally, arbitrary RL policy optimization algorithms can optimize the instruction-following policy.
Our paper presents several contributions to the field of NLC-RL. Firstly, we propose a novel NLC-RL scheme called IOL, which introduces a task language that enhances policy understanding and improves the learning efficiency of instruction-following. Secondly, we introduce a specific implementation of IOL that models TL development as a referential game and uses multiple learned predicates to represent TL. Thirdly, we conduct comprehensive experiments in two environments, FrankaKitchen [8], and CLEVR-Robot [9], demonstrating that the policy learned by our proposed method outperforms existing approaches in terms of its ability to follow natural language instructions and adapt to previously unseen instructions. Furthermore, the resulting TL effectively represents human instructions, providing a solid baseline for sub-task abstraction in hierarchical RL [10].
This section begins with a summary of prior research on instruction following with RL, followed by two paragraphs discussing works pertinent to our methodology, i.e., language generation in RL and language translation.
Instruction following with RL. Instruction-following problems require agents to perform tasks specified by natural language instructions. A line of works solves the problems with semantics vectors [11, 12, 13], or language-conditioned policies [14, 15, 16]. To train the policies, previous methods typically process NL using a language model and train a following policy with RL algorithms. For example, [17] encodes a single-sentence instruction with a pre-trained language model and feeds the policy with the NL encoding. [18] learns a policy that maps NL instructions to action sequences by marginalizing implied goal locations. [19] combines human instructions with agent observations via a multiplication-based mechanism and then pre-trains the instruction-following policy using behaviour cloning [20]. Instead of directly converting NL to encoding, [5] encodes NL to a manually-designed binary vector in which each element has semantics meaning. Besides, the instruction-following policy has close ties to Hierarchical RL [10] because the instructions can be naturally viewed as a task abstraction for a low-level policy [21]. HAL [9] takes advantage of the compositional structure of NL and makes decisions directly at the NL level to solve long-term, complex RL tasks. These previous methods expose the unbounded NL instructions directly to the policy or encode the NL instructions to a scenario-specific manual vector, both of which have limitations. In contrast, our method automatically develops a task language, which is easily understood by the policy and facilitates policy learning.
Language generation in RL. Language generation plays a crucial role in developing task language. In reinforcement learning, researchers often approach this issue in a multi-agent setting, where agents
must learn effective message protocols to communicate with their partners and complete tasks. For example, the study by [22] investigates emergent language in a referential game. At the same time, a line of works has explored multi-agent communication [23, 24] for practical cooperation, considering the communication message as language. Regarding language representation, prior research has often used discrete vectors due to the discrete nature of human language. For example, [25] enables agents to communicate using discrete messages and shows that discrete representation performs similarly to continuous representation with a much smaller vocabulary size. One-hot representation [26, 22] and binary representation [5, 27] are popular forms of discrete language representation. For instance, [26] uses a one-hot language representation to enable two agents to communicate and differentiate between images. In this paper, we develop task language following the discrete form of language representation while using the predication representation.
Language translation. In this paper, TALAR translates natural language into task language, which lies in the domain of language translation [28]. The field of natural language processing (NLP) has extensively investigated various approaches to language translation [29, 30]. Among these approaches, the encoder-decoder architecture is a promising method because it can extract effective features from input sentences [31]. For example, the study in [32] introduces a continuous latent variable as an efficient feature for language translation, utilizing a variational auto-encoder [33]. In this paper, our primary focus is not on language translation. Therefore, we employ a more straightforward method for translation, utilizing the pre-trained BERT model as the NL encoder. In this approach, we consider NL as the source language and the target language (TL) as the target language.
3.1 RL and NLC-RL
A typical RL task can be formulated as a Markov Decision Process (MDP) [34, 35], which is described as a tuple  $\mathcal{M} = (\mathcal{S},\mathcal{A},P,r,\gamma ,d_0)$ . Here  $\mathcal{S}$  represents the state space.  $\mathcal{A}$  is the finite action space defined by  $\mathcal{A} = \{a_0,a_1,\dots ,a_{|\mathcal{A}| - 1}\}$ .  $P$  represents the probability of transition while  $r$  represents the reward function.  $\gamma$  is the discount factor determining the weights of future rewards, whereas  $d_0$  is the initial state distribution. A policy  $\pi :\mathcal{S}\to \Delta (\mathcal{A})$  maps state space to the probability space over action space. In NLC-RL, the agent receives an NL instruction  $(L)$  that reflects the human's instruction on the agent. An instruction-following policy  $\pi (\cdot |s_t,L)$  is trained to make decisions based on the current state  $s_t$  and NL instruction  $L$ . The overall objective of NLC-RL is to maximize the expected return under different NL instructions:
$$
\mathbb {E} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} r \left(s _ {t}, a _ {t}, L\right) \mid s _ {0} \sim d _ {0}, a _ {t} \sim \pi (\cdot \mid s _ {t}, L) \right]. \tag {1}
$$
For the accuracy of sake, we use  $L_{\mathrm{N}}$  and  $L_{\mathrm{T}}$  to denote NL and TL, respectively.
3.2 Referential Game
Referential game (RG) has been established as a multi-agent communication task [22], which consists of two agents: a sender and a receiver. The two agents communicate using their language to identify a particular object or concept in a specific task. In a classic referential game, a pair of images is presented to the sender and the receiver, with the sender being informed which of the two images is the target. The sender must generate an effective communication message that enables the receiver to identify the target image successfully. Through interactive communication, the agents develop meaningful language. As the sender must capture the key feature of the task, the RG presents an ideal setting for generating task language. In TALAR, we use an extension to the basic RG to generate TL, which will be elaborated in Section 4.
3.3 Predicate Representation
Predicate representation utilizes discrete binary vectors to describe the relationships between objects or abstract concepts. The predicate representation is formally defined as comprising two key components: (1) predicate signifies the relationship between objects or concepts, which may involve one or multiple objects. (2) argument serves as the input for the predicate, indicating the specific objects between which the predicate measures the relationship. By employing these components,
predicate representation offers a clear and concise depiction of the relation between objects. For instance, the predicate representation vector $^3$ $[1, 1, 0, 0, 0, 1, 0]$  could represent a predicate expression Pred(a, b). In this case, Pred is a predicate that signifies a relationship, and the symbols a and b are its arguments. In the vector, the initial code [1] indicates that the value of Pred is True (i.e., the relationship holds). In contrast, the following red and blue one-hot codes represent the indexes of arguments a and b, respectively.
Prior research has shown that predicate representation can be learned by employing networks to output predicate values. In this manner, the learning system can automatically identify relationships in a block stacking task [36]. In TALAR, neural networks learn both predicates and their arguments. For additional discussions on predicate representation, refer to Appendix A.
3.4 Natural Language as Input for the Neural Network
NL sentences are variable lengths and cannot be fed directly into a fully connected network. A standard solution is to encode each word as an embedding [37] and loop over each embedding using a recurrent model. Besides the recurrent model, the pre-trained language model, e.g., Bert [3], tokenizes the words in the sentence and extracts sentence embedding based on these tokens. This process involves passing the input sentence through a series of transformer-based encoder layers that capture contextual information and create representations of the input tokens. BERT then applies a pooling layer to obtain a fixed-length vector representation of the input sentence. In this study, we employ Bert to process NL sentences to fixed-length vectors because it is lightweight and efficient.
4 Inside-out Learning and Implementation
This section introduces our primary contribution to efficient policy learning in NLC-RL, the IOL framework and its specific implementation, TALAR. We begin by formulating the IOL framework.
4.1 Inside-out Learning Formulation
We consider NLC-RL tasks, where the agent is provided with a natural language instruction as a goal to be completed. The key idea of IOL is to develop a task-related language to facilitate policy understanding. To implement IOL, three key components must be considered in IOL. We will elaborate on these components and discuss their impact on solving NLC-RL tasks.
TL generator. The TL generator is the critical component in IOL, designed to develop a task-related language. We expect the resulting TL to effectively and succinctly convey the task's objective to the policy. In this way, IOL mitigates the issues encountered in OIL, where the policy struggles to comprehend unseen natural language instructions. We believe that expressiveness and conciseness are vital properties of TL. Expressiveness ensures the TL accurately conveys the task goal, while conciseness promotes policy comprehension. To fulfil these requirements, we propose representing TL using predicate representation, considered both expressive and concise [38].
Translator and Policy learning. The translator serves as a tool that connects policy with TL, which translates NL to TL. Utilizing the translator, the policy can be optimized through various RL algorithms to improve its ability to follow human instructions. The policy learning is a goal-conditioned RL [39] problem. For complex tasks, plenty of well-established techniques in machine translation and goal-conditioned RL domains can be employed to improve policy learning in IOL further. Note that the primary focus of this paper is to investigate the effectiveness of IOL. Therefore, we employ straightforward translator and policy learning implementations, as detailed in Section 4.2.
4.2 Our Method
We first introduce the task dataset, which is used for TL development and translation in TALAR. Definition 1 (Task dataset). A task dataset  $\mathcal{D} = \{(s,s',L_{\mathrm{N}})_i\}$  consists of multiple triplets. Each triplet contains a natural language instruction  $L_{\mathrm{N}}$  and a task state pair  $(s,s')$ , where  $L_{\mathrm{N}}$  describes state change from  $s$  to  $s'$  in natural language, e.g., move the red ball to the blue ball.
<sup>3</sup>The vector could be extended to represent multiple predicate relationships simultaneously.
Figure 2: Overall framework of TALAR. (a) Overall training process: The TL generator develops TL via playing a referential game with a receiver, and the translator translates NL to TL. (b) Network architecture of the TL generator. (c) Network architecture of the translator. The number of predicate arguments and networks can be adjusted according to the task scale.
We use a state pair instead of a single state for the following reasons: (1) NL instruction typically describes a state change, e.g., turning the wheel to the left; (2) it is not straightforward to describe a single state concisely in complex task scenarios. Fig. 2 illustrates how TALAR makes use of task dataset for task language development and translation. The subsequent subsections will elaborate on training three critical components of TALAR, i.e., task language development, natural language translation and policy optimization.
4.2.1 Task Language Development in Predicate Representation
The TL generator denoted as  $g_{\theta}(s,s')$ , aims to develop a language in predicate representation (Section 3.3). In TALAR, the TL generator is represented by neural networks and parameterized with  $\theta$ . It takes a state pair  $(s,s')$  as the input and outputs task language  $L_{T}$ . The We will discuss how TALAR constructs a referential game to facilitate the development of task language.
Referential game. The RG process is presented in the right part of Fig. 2. Based on the RG framework in Section 3.2, TALAR adapts it such that the TL generator serves as the sender  $^4$  and generates the TL  $L_{\mathrm{T}} = g_{\theta}(s,s^{\prime})$  to describe the state change of the given state pair. Conditioned on the generated TL  $L_{\mathrm{T}}$ , a receiver aims to predict the word in the NL instruction corresponding to  $(s,s^{\prime})$ . In this way, the TL generator must capture the critical information in the given state pair and generate effective TL, ensuring that the receiver can predict correctly. We use a pre-trained BERT model followed by a fully-connected layer to implement the receiver to predict the next token. We denote this as  $f_{\delta}(T_i|L_{\mathrm{T}},b(T_{1,2\dots i - 1}))$ , where  $f$  is the fully-connected layer after the BERT output,  $b$  is the BERT model,  $T_{i}$  is the  $i$ -th token of the sentence, and  $\delta$  represents the parameters of the fully-connected layer. The token to be predicted  $T_{i}$  is randomly sampled from the whole sentence. The objective for training the TL generator and receiver is to minimize the negative log-likelihood of the predicted token given the generated TL and the context tokens, which is expressed as:
$$
\mathcal {L} _ {\mathrm {R G}} (\theta , \delta) = \underset {(s, s ^ {\prime}, L _ {\mathrm {N}}) \sim \mathcal {D}, T _ {i} \sim T _ {L _ {\mathrm {N}}}} {\mathbb {E}} \left[ - \log f _ {\delta} \left(T _ {i} \mid g _ {\theta} (s, s ^ {\prime}), b \left(T _ {1, \dots , i - 1}\right)\right) \right], \tag {2}
$$
where  $T_{L_N}$  is the token set of  $L_N$ . In practice, we fix the parameters of BERT and only optimize the fully-connected layer.
Network architecture of TL generator. We design a specific network architecture for learning the predicate representation (we refer readers to Section 3.3 for elaborations on predicate representation). As illustrated in Fig. 2(b), the TL generator first extracts  $N_{\mathrm{a}}$  arguments  $(\arg_1, \arg_2, \dots, \arg_{N_{\mathrm{a}}})$  based on the input state pair, and subsequently determines the Boolean values of  $N_{\mathrm{pn}}$  predicates, given the extracted argument list. The predicate values are concatenated with the argument list and form the task language  $L_{\mathrm{T}}^{i} = (\operatorname{pred}_{1}, \dots, \operatorname{pred}_{N_{p}n}, \arg_1, \dots, \arg_{N_{\mathrm{a}}})$ . The number of predicate networks  $N_{\mathrm{pn}}$  and arguments networks  $N_{\mathrm{a}}$  can be adjusted according to the RL task scale.
4To avoid confusion, we will use "TL generator" instead of "sender" for the remainder of this paper.
Specifically, the arguments for the predicate are extracted by argument networks, denoted by  $\mathrm{ArgNet}_i(s,s')$ . An argument network is implemented as a fully-connected network ending with a Gumbel-Softmax activation layer [40]. Through the Gumbel-Softmax, the argument network can output a discrete one-hot vector  $\arg_{i}$  in form like  $(0,1,\dots ,0)$ , which represents an abstract object or concept in the task. TALAR utilizes multiple predicate networks, denoted by  $\mathrm{PredNet}_i(s,s',\arg_1,\dots ,\arg_{N_{\mathrm{a}}})$ , to determine the Boolean values of a set of predicates. These predicates are anonymous as we do not need to pre-define them in advance. This learning manner requires predicate networks to automatically discover meaningful relationships in the task. Each predicate network outputs a 0-1 value, ending with a Gumbel-Softmax layer. All these 0-1 values are concatenated with the argument list, yielding the task language  $L_{\mathrm{T}}$ , a discrete binary vector. Note that without the argument list in  $L_{\mathrm{T}}$ , the resulting language cannot express different objects and therefore loses its expressiveness. The Gumbel-Softmax activation technique permits the differentiation of the entire TL generation procedure.
4.2.2 Natural Language Translation
The objective of the translator is to translate the natural language to the task language. TALAR uses a pre-trained BERT model to convert  $L_{\mathrm{N}}$  into a fixed-length embedding, followed by fully-connected layers to predict the task language corresponding to  $L_{\mathrm{N}}$ . Fig. 2(d) presents the structure of the translator. We let  $\widetilde{L_{\mathrm{T}}}$  denote the TL generated by the translator, and  $p_{\phi}$  the fully-connected layers parameterized with  $\phi$ . The translator is trained to minimize the difference between the predicted TL with the target TL:
$$
\mathcal {L} _ {\mathrm {T}} (\phi) = \underset {(s, s ^ {\prime}, L _ {\mathrm {N}}) \sim \mathcal {D}} {\mathbb {E}} \left[ - \log p _ {\phi} \left(g _ {\theta} \left(s, s ^ {\prime}\right) | b \left(T \left(L _ {\mathrm {N}}\right)\right)\right) \right]. \tag {3}
$$
Here  $T(L_{\mathrm{N}})$  denotes the tokenized natural language sentence. The optimized translator trains an instruction-following policy to complete the human instructions, as described below.
4.2.3 Policy Optimization
TALAR uses reinforcement learning to train an Instruction-Following Policy (IFP)  $\pi(\cdot | s, \widetilde{L}_{\mathrm{T}})$ . When the agent collects samples from the environment, the task generates a random human instruction in NL, which is then translated into the task language  $\widetilde{L}_{\mathrm{T}}$  by the translator. Next, the IFP makes decisions for the entire episode based on the current observation and  $\widetilde{L}_{\mathrm{T}}$  until completing the instruction or reaching the maximum timestep. The IFP can be optimized with an arbitrary RL algorithm using the samples collected from the environments. We use PPO [41] for TALAR and all baselines in our implementation. Note that during IFP training, the translator's parameters are fixed to prevent the translator from overfitting the current IFP.
We conduct a series of experiments to evaluate the effectiveness of TALAR and answer the following questions: (1) How does TALAR perform compared to existing NLC-RL approaches when learning
an instruction-following policy? (Section 5.1) (2) Can TALAR learn effective task language? (Section 5.2) (3) Can TL acquire any compositional structure and serve as an abstraction for hierarchical RL? (Section 5.3) (4) What is the impact of each component on the overall performance of TALAR? (Section 5.4)