Venue: ACL 2017

# Interactive Learning of Grounded Verb Semantics towards

Human-Robot Communication

Lanbo She &Joyce Y. Chai

Department of Computer Science and Engineering

Michigan State University

East Lansing, Michigan 48824, USA

{shelanbo, jchai}@cse.msu.edu

###### Abstract

To enable human-robot communication and collaboration, previous works represent grounded verb semantics as the potential change of state to the physical world caused by these verbs. Grounded verb semantics are acquired mainly based on the parallel data of the use of a verb phrase and its corresponding sequences of primitive actions demonstrated by humans. The rich interaction between teachers and students that is considered important in learning new skills has not yet been explored. To address this limitation, this paper presents a new interactive learning approach that allows robots to proactively engage in interaction with human partners by asking good questions to learn models for grounded verb semantics. The proposed approach uses reinforcement learning to allow the robot to acquire an optimal policy for its question-asking behaviors by maximizing the long-term reward. Our empirical results have shown that the interactive learning approach leads to more reliable models for grounded verb semantics, especially in the noisy environment which is full of uncertainties. Compared to previous work, the models acquired from interactive learning result in a 48% to 145% performance gain when applied in new situations.

## 1 Introduction

In communication with cognitive robots, one of the challenges is that robots do not have sufficient linguistic or world knowledge as humans do. For example, if a human asks a robot to _boil the water_ but the robot has no knowledge what this verb phrase means and how this verb phrase relates to its own actuator, the robot will not be able to execute this command. Thus it is important for robots to continuously learn the meanings of new verbs and how the verbs are grounded to its underlying action representations from its human partners.

To support learning of grounded verb semantics, previous works [30, 6] rely on multiple instances of human demonstrations of corresponding actions. From these demonstrations, robots capture the state change of the environment caused by the actions and represent verb semantics as the desired goal state. One advantage of such state-based representation is that, when robots encounter the same verbs/commands in a new situation, the desired goal state will trigger the action planner to automatically plan a sequence of primitive actions to execute the command.

While the state-based verb semantics provides an important link to connect verbs to the robot's actuator, previous works also present several limitations. First of all, previous approaches were developed under the assumption of perfect perception of the environment [37, 39, 38]. However, this assumption does not hold in real-world situated interaction. The robot's representation of the environment is often incomplete and error-prone due to its limited sensing capabilities. Thus it is not clear whether previous approaches can scale up to handle noisy and incomplete environment.

Second, most previous works rely on multiple demonstration examples to acquire grounded verb models. Each demonstration is simply a sequence of primitive actions associated with a verb. No other type of interaction between humans and robots is explored. Previous cognitive studies [2] on how people learn have shown that social interaction (e.g., conver

sation with teachers) can enhance student learning experience and improve learning outcomes. For robotic learning, previous work [16] has also demonstrated the necessity of question answering in the learning process. Thus, in our view, interactive learning beyond demonstration of primitive actions should play a vital role in the robot's acquisition of more reliable models of grounded verb semantics. This is especially important because the robot's perception of the world is noisy and incomplete, human language can be ambiguous, and the robot may lack the relevant linguistic or world knowledge during the learning process.

To address these limitations, we have developed a new interactive learning approach where robots actively engage with humans to acquire models of grounded verb semantics. Our approach explores the space of interactive question answering between humans and robots during the learning process. In particular, motivated by previous work on robot learning [20], we designed a set of questions that are pertinent to verb semantic representations. We further applied reinforcement learning to learn an optimal policy that guides the robot in deciding when to ask what questions. Our empirical results have shown that this interactive learning process leads to more reliable representations of grounded verb semantics, which contribute to significantly better action performance in new situations. When the environment is noisy and uncertain (as in a realistic situation), the models acquired from interactive learning result in a performance gain between 48% and 145% when applied in new situations. Our results further demonstrate that the interaction policy acquired from reinforcement learning leads to the most efficient interaction and the most reliable verb models.

## 2 Related Work

To enable human-robot communication and collaboration, recent years have seen an increasing amount of works which aim to learn semantics of language that are grounded to agents' perception [37; 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20] and action [21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32]. Specifically for verb semantics, recent works explored the connection between verbs and action planning [23, 24, 25, 26, 27], for example, by representing grounded verbs semantics as the desired goal state of the physical world that is a result of the corresponding actions. Such representations are learned based on example actions demonstrated by humans. Once acquired, these representations will allow agents to interpret verbs/commands issued by humans in new situations and apply action planning to execute actions. Given its clear advantage in connecting verbs with actions, our work also applies the state-based representation for verb semantics. However, we have developed a new approach which goes beyond learning from demonstrated examples by exploring how rich interaction between humans and agents can be used to acquire models for grounded verb semantics.

This approach was motivated by previous cognitive studies [1] on how people learn as well as recent findings on robot skill learning [16]. One of the principles for human learning is that "learning is enhanced through socially supported interactions". Studies have shown that social interaction with teachers and peers (e.g., substantive conversation) can enhance student learning experience and improve learning outcomes. In recent work on interactive robot learning of new skills [16], researchers identified three types of questions that can be used by a human/robot student to enhance learning outcomes: 1) _demonstration query_ (i.e., asking for a full or partial demonstration of the task), 2) _label query_ (i.e., asking whether an execution is correct), and 3) _feature query_ (i.e., asking for a specific feature or aspect of the task). Inspired by these previous findings, our work explores interactive learning to acquire grounded verb semantics. In particular, we aim to address when to ask what questions during interaction to improve learning.

## 3 Acquisition of Grounded Verb Semantics

This section gives a brief review on acquisition of grounded verb semantics and illustrates the differences between previous approaches and our approach using interactive learning 



### State-based Representation

As shown in Figure 1, the verb semantics (e.g., \(boil(x)\)) is represented by the goal state (e.g., \(Status(x,TempHigh)\)) which is the result of the demonstrated primitive actions. Given the verb phrase _boil the water_ (i.e., \({\cal L}_{i}\)), the human teaches the robot how to accomplish the corresponding action based on a sequence of primitive actions \(\overrightarrow{{\cal A}_{i}}\). By comparing the final environment \({\cal E}^{\prime}_{i}\) with the initial environment \({\cal E}_{i}\), the robot is able to identify the state change of the environment, which becomes a hypothesis of goal state to represent verb semantics. Compared to procedure-based representations, the state-based representation supports automated planning at the execution time. It is environment-independent and more generalizable. In She and Chai [2016], instead of one hypothesis, it maintains a specific-to-general hypothesis space as shown in Figure 2 to capture all goal hypotheses of a particular verb frame. Specifically, it assumes that one verb frame may lead to different outcomes under different environments, where each possible outcome is represented by one node in the hierarchical graph and each node is a conjunction of multiple atomic fluents. 1

Footnote 1: In this work, we assume the set of atomic fluents representing environment state are given and do not address the question of whether these predicates are adequate to represent a domain.

Given a language command (i.e., a verb phrase), a robot will engage in the following processes:

* **Execution.** In this process, the robot will select a hypothesis from the space of hypotheses that is most relevant to the current situation and use the corresponding goal state to plan for actions to execute.
* **Learning.** When the robot fails to select a hypothesis or fails to execute the action, it will ask the human for a demonstration. Based on the demonstrated actions, the robot will learn a new representation (i.e., new nodes) and update the hypothesis space.

### Noisy Environment

Previous works represent the environment \({\cal E}_{i}\) as a conjunction of grounded state fluents. Each fluent consists of a predicate and one or more arguments (i.e., objects in the physical world, or object status), representing one aspect of the perceived environment. An example of a fluent is "\(Has(Kettle_{1},WATER)\)" meaning object \(Kettle_{1}\) has some water inside, where \(Has\) is the predicate, and \(Kettle_{1}\) and \(WATER\) are arguments. The set of fluents include the status of the robot (e.g., \(Grasping(Kettle_{1})\)), the status of different objects (e.g., \(Status(WATER,TempHigh)\)), and relations between objects (e.g., \(On(Kettle_{1},Stove)\)). One limitation of the previous wor

Figure 3: An example probabilistic sensing result.

Figure 2: An example hypothesis space for the verb frame \(fill(x,y)\).



ronment has a perfect, deterministic representation, as shown in Figure 1. This is clearly not the case in the realistic physical world.

In reality, given limitations of sensor capabilities, the environment representation is often partial, error prone, and full of uncertainties. Figure 3 shows an example of a more realistic representation where each fluent comes with a confidence between 0 and 1 to indicate how likely that particular fluent can be detected in the current environment. Thus, it is unclear whether the previous work is able to handle representations with uncertainties. Our interactive learning approach aims to address these uncertainties through interactive question answering with human partners.

## 4 Interactive Learning

### Framework of Interactive Learning

Figure 4 shows a general framework for interactive learning of action verbs. It aims to support a life-long learning cycle for robots, where the robot can continuously (1) engage in collaboration and communication with humans based on its existing knowledge; (2) acquire new verbs by learning from humans and experiencing the change of the world (i.e., grounded verb semantics as in this work); and (3) learn how to interact (i.e., update interaction policies). The lifelong learning cycle is composed by a sequence of interactive learning episodes (Episode 1, 2...) where each episode consists of either an execution phase or a learning phase or both.

The execution phase starts with a human request for action (e.g., _boil the water_). According to its interaction policy, the robotic agent may choose to ask one or more questions (i.e., \(Q^{+}_{i}\)) and wait for human answers (i.e., \(A^{+}_{i}\)), or select a hypothesis from its existing knowledge base to execute the command (i.e., _Execute_). With the human feedback of the execution, the robot can update its interaction policy and existing knowledge.

In the learning phase, the robot can initiate the learning by requesting a demonstration from the human. After the human performs the task,

\begin{table}
\begin{tabular}{|p{85.4pt}||p{176.9pt}|} \hline \multicolumn{1}{|c||}{**Only demonstration[She and Chai, 2016]**} & \multicolumn{2}{c|}{**Interactive learning (this work)**} \\ \hline \multicolumn{2}{|c||}{**Execution Example**} & \multicolumn{2}{c|}{**Execution Example**} \\ H1: Boil the water. & H1: Boil the water. & H1: Boil the water. \\ R1: (_select a hypothesis and execute the action accordingly_) H2: (_provide true/false feedback on whether the execution is correct_) & R1: I see a silver object on the table, is that a pot? \\ R2: (_provid true/false feedback on whether the execution is correct_) & H2: Yes. & H2: Should I leave the pot on the stove after boil? \\ R3: (_select a hypothesis to execute_) & H3: Yes. & H3: Yes. \\ \cline{2-3}  & **Learning Example** & **K2erning Example** \\ H3: Boil the water. & H5: Boil the water. & \\ R3: I don’t know how to boil, could you show me an example? H4: (_human shows the robot how to achieve the corresponding action through a sequence of primitive actions_) & R5: Ioa't know how to boil, could you show me an example? H6: Ok. (_demonstrate the task, at the end put the pot on a tenat pad_) & \\ R4: (_indicate the acceptance and induce a hypothesis to represent verb semantics_) & R6: So after boiling something, should I put it on a heat pad? H7: No, you don’t have to. \\ \cline{2-3}  & **R6**: Team 1 see. (_induce goal state hypothesis_) & R7: I see. (_induce goal state hypothesis_) \\ \hline \end{tabular}
\end{table}
Table 1: Examples to show differences between learning thousands as in the previous works [She and Chai, 2016] and the proposed learning from interaction.

Figure 4: A general framework of robot interactive learning. KB stands for knowledge base, \(\theta_{E}\) stands for Interaction Strategy for Execution, and \(\theta_{D}\) stands for Interaction Strategy for Learning.

 

the robotic agent can either choose to update its knowledge if it feels confident, or it can choose to ask the human one or more questions before updating its knowledge.

### Examples of Interactive Learning

Table 1 illustrates the differences between the previous approach that acquires verb models based solely on demonstrations and our current work that acquires models based on interactive learning. As shown in Table 1, under the _demonstration_ setting, humans only provide a demonstration of primitive actions and there's no interactive question answering. In the _interactive learning_ setting, the robot can proactively choose to ask questions regarding the uncertainties either about the environment (e.g., R1), the goal (e.g., R2), or the demonstrations (e.g., R6). Our hypothesis is that rich interactions based on question answering will allow the robot to learn more reliable models for grounded verb semantics, especially in a noisy environment.

Then the question is how to manage such interaction: when to ask and what questions to ask to most efficiently acquire reliable models and apply them in execution. Next we describe the application of reinforcement learning to manage interactive question answering for both the execution phase and the learning phase.

### Formulation of Interactive Learning

Markov Decision Process (MDP) and its closely related Reinforcement Learning (RL) have been applied to sequential decision-making problems in dynamic domains with uncertainties, e.g., dialogue/interaction management Singh et al. (2002); Paek and Pieraccini (2008); Williams and Zweig (2016), mapping language commands to actions Branavan et al. (2009), interactive robot learning Knox and Stone (2011), and interactive information retrieval Li et al. (2017). In this work, we formulate the choice of when to ask what questions during interaction as a sequential decision-making problem and apply reinforcement learning to acquire an optimal policy to manage interaction.

Specifically, each of the execution and learning phases is governed by one policy (i.e., \(\theta_{E}\) and \(\theta_{D}\)), which is updated by the reinforcement learning algorithm. The use of RL intends to obtain optimal policies that can lead to the highest long-term reward by balancing the cost of interaction (e.g., the length of interaction and difficulties of questions) and the quality of the acquired models. The reinforcement formulation for both the execution phase and the learning phase are described below.

**State** For the execution phase, each state \(s_{e}\in S_{E}\) is a five tuple: \(s_{e}=<l,e,KB,Grd,Goal>\). \(l\) is a language command, including a verb and multiple noun phrases extracted by the Stanford parser. For example, the command "_Microwave the ramen_" is represented as \(l=microwave(ramen)\). The environment \(e\) is a probabilistic representation of the currently perceived physical world, consisting of a set of grounded fluents and the confidence of perceiving each fluent (an example is shown in Figure 3). \(KB\) stands for the existing knowledge of verb models. \(Grd\) accounts for the agent's current belief of object grounding: the probability of each noun in the \(l\) being grounded to different objects. \(Goal\) represents the agent's belief of different goal state hypotheses of the current command. Within one interaction episode, command \(l\) and knowledge \(KB\) will stay the same, while \(e\), \(Grid\), and \(Goal\) may change accordingly due to interactive question answering and robot actions. In the execution phase, \(Grd\) and \(Goal\) are initialized with existing knowledge of learned verb models. For the learning phase, a state \(s_{d}\in S_{D}\) is a four tuple: \(s_{d}=<l,e_{start},e_{end},Grd>\). \(e_{start}\) and \(e_{end}\) stands for the environment before the demonstration and after the demonstration.

**Action** Motivated by previous studies on how humans ask questions while learning new skills Cakmak and Thomaz (2012), the agent's question set includes two categories: yes/no questions and wh- questions. These questions are designed to address ambiguities in noun phrase grounding, uncertain environment sensing, and goal states. They are domain independent in nature. For example, one of the questions is np_grd_ynq(\(n,o\)). It is a yes/no question asking whether the noun phrase \(n\) refers to an object \(o\) (e.g., "_I see a silver object, is that the pot?_"). Other questions are env_pred_ynq(\(p\)) (i.e., whether a fluent \(p\) is present in the environment; e.g., "_Is the microwave door open?_") and goal_pred_ynq(\(p\)) (i.e., whether a predicate \(p\) should be part of the goal; "_Should the pot be on a pot stand?_"). Table 2 lists all the actions available in the execution and learning phases. The select_hypo action (i.e., select a goal hypothesis to execute) is only for the execution. Ideally, after asking questions, the agent should be more likely to select a goal hy 

pothesis that best describes the current situation. For the learning phase, the include_fluent(\(\wedge p\)) action forms a goal hypothesis by conjoining a set of fluents \(p\)s where each \(p\) should have high probability of being part of the goal.

**Transition** The transition function takes action \(a\) in state \(s\), and gives the next state \(s^{\prime}\) according to human feedback. Note that the command \(l\) does not change during interaction. But the agent's belief of environment \(e\), object grounding \(Grd\), and goal hypotheses \(Goal\) is changed according to the questions and human answers. For example, suppose the agent asks whether noun phrase \(n\) refers to the object \(o\), if the human confirms it, the probability of \(n\) being grounded to \(o\) becomes 1.0, otherwise it will become 0.0.

**Reward** Finding a good reward function is a hard problem in reinforcement learning. Our current approach has followed the general practice in the spoken dialogue community Schatzmann et al. (2006); Fang et al. (2014); Su et al. (2016). The immediate robot questions are assigned small costs to favor shorter and more efficient interaction. Furthermore, motivated by how humans ask questions Cakmak and Thomaz (2012), yes/no questions are easier for a human to answer than the open questions (e.g., wh-questions) and thus are given smaller costs. A large positive reward is given at the end of interaction when the task is completed successfully. Detailed reward assignment for different actions are shown in Table 2.

```
Input: e, l (, \(e_{end}\)); Feature function \(\phi\); Old policy \(\theta\) (i.e., a weight vector) Verd Goal States Hypotheses \(\mathcal{H}_{i}\) Initialize: state is initialized with \(e\), l (, \(e_{end}\)); first action \(a\sim P(a|\mathbf{\delta};\theta)\) with \(\epsilon\) greedy while\(s\) is not terminaldo Take action \(a\), receive reward \(r\); \(s^{\prime}=T(s,a)\); Choose \(a^{\prime}\sim P(a^{\prime}|s^{\prime};\theta)\) with \(\epsilon\) greedy; \(\delta\gets r+\gamma\cdot\theta^{T}\cdot\phi(s^{\prime},a^{\prime})- \theta^{T}\cdot\phi(s,a)\); \(\theta\leftarrow\theta+\delta\cdot\eta\cdot\phi(s,a)\);  end whileifif\(s\) terminates with positive feedbackthen Update \(\mathcal{H}\);  end if Output: Updated \(\mathcal{H}\) and \(\theta\).
```

**Algorithm 1**Policy learning. The execution and learning phases share the same learning process, but with different state \(s\), action \(a\) spaces, and feature vectors \(\phi\). The \(e_{end}\) is only available to the learning phase.

**Learning** The _SARSA_ algorithm with linear function approximation is utilized to update policies \(\theta_{E}\) and \(\theta_{D}\) Sutton and Barto (1998). Specifically, the objective of training is to learn an optimal value function \(Q(s,a)\) (i.e., the expected cu

\begin{table}
\begin{tabular}{|c|l|l|l|} \hline Action Name & Explanation & Question Example & Reward \\ \hline \hline
1. np\_grd\_whq(\(n\)) & Ask for the grounding of a np. & “_Which is the cup, can you show me?_” & -6.51  \\ \hline
2. np\_grd\_ynq(\(n\), o) & Confirm the grounding of a np. & “_I see a silver object, is that the pot?_” & -1.0 / -2.0 \\ \hline
3. env\_pred\_ynq(\(p\)) & Confirm a predicate in current environment. & “_Is the microwave door open?_” & -1.0 / -2.0 \\ \hline
4. goal\_pred\_ynq(\(p\)) & Confirm whether a predicate \(p\) should be in the final environment. & “_Is it true the pot should be on the counter?_” & -1.0 / -2.0 \\ \hline
5. select\_hypo(\(h\)) & Choose a hypothesis to use as goal and execute. & & 100 / -2.0 \\ \hline \hline
6. bulk\_np\_grd\_ynq(\(n,o\)) & Confirm the grounding of multiple nps. & “_I think the pot is the red object and milk is in the white box, am 1 right!_” & -3.0 / -6.02  \\ \hline
7. pred\_change\_ynq(\(p\)) & Ask whether a predicate \(p\) has been changed by the action demonstration. & “_The pot is on a stand after the action, is that correct?_” & -1.0 / -2.0 \\ \hline
8. include\_fluent(\(\wedge p\)) & Include \(\times p\) into the goal state representation. Update the verb semantic knowledge. & & 100 / -2.0 \\ \hline \end{tabular}
\end{table}
Table 2: The action space for reinforcement learning, where \(n\) stands for a noun phrase, \(o\) a physical object, \(p\) a fluent representation of the current state of the world, \(h\) a goal hypothesis. Action 1 and 2 are shared by both the execution and learning phases. Action 3, 4, 5 are for the execution phase, and 6, 7, 8 are only used for the learning phase. -1.0/-2.0 are typically used for yes/no questions. When the human answers the question with a “yes”, the reward is -1.0, otherwise it’s -2.0.

 

mulative reward of taking action \(a\) in a state \(s\)). This value function is approximated by a linear function \(Q(s,a)=\theta^{intercal}\cdot\phi(s,a)\), where \(\phi(s,a)\) is a feature vector and \(\theta\) is a weight updated during training. Details of the algorithm is shown in Algorithm 1. During testing, the agent can take an action \(a\) that maximizes the \(Q\) value at a state \(s\).

**Feature** Example features used by the two phases are listed in Table 3. These features intend to capture different dimensions of information such as specific types of questions, how well noun phrases are grounded to the environment, uncertainties of the environment, and consistencies between a hypothesis and the current environment.

## 5 Evaluation

### Experiment Setup

**Dataset.** To evaluate our approach, we utilized the benchmark made available by Misra et al. (2015). Individual language commands and corresponding action sequences are extracted similarly as She and Chai (2016). This dataset includes common tasks in the kitchen and living room domains, where each data instance comes with a language command (e.g., "_boil the water_", "_throw the beer into the trashcan_") and the corresponding sequence of primitive actions. In total, there are 979 instances, including 75 different verbs and 215 different noun phrases. The length of primitive action sequences range from 1 to 51 with an average of 4.82 (+/-4.8). We divided the dataset into three groups: (1) 200 data instances were used by reinforcement learning to acquire optimal interaction policies; (2) 600 data instances were used by different approaches (i.e., previous approaches and our interactive learning approach) to acquire grounded verb semantics models; and (3) 179 data instances were used as testing data to evaluate the learned verb models. The performance on applying the learned models to execute actions for the testing data is reported.

To learn interaction policies, a simulated human model is created from the dataset Schatzmann et al. (2006) to continuously interact with the robot learner3. This simulated user can answer the robot's different types of questions and make decisions on whether the robot's execution is correct. During policy learning, one data instance can be used multiple times. At each time, the interaction sequence is different due to _exploitation and exploration_ in RL in selecting the next action. The RL discount factor \(\gamma\) is set to 0.99, the \(\epsilon\) in \(\epsilon\)-greedy is 0.1, and the learning rate is 0.01.

Footnote 3: In our future work, interacting with real humans will be conducted through Amazon Mechanical Turk. And the policies acquired with a simulated user in this work will be used as initial policies.

**Noisy Environment Representation.** The original data provided by Misra et al. (2015) is based on the assumption that environment sensing is perfect and deterministic. To enable incomplete and noisy environment representation, for each fluent (e.g., \(grasping(Cup_{3})\), \(near(robot_{1},Cup_{3})\)) in the original data, we independently sampled a confidence value to simulate the likelihood that a particular fluent can be detected correctly from the environment. We applied the following four different variations in sampling the confidence values, which correspond to different levels of sensor reliability.

_(1) PerfectEnv_ represents the most reliable sensor. If a fluent is true in the original data, its sampled confidence is 1, and 0 otherwise.

_(2) NormStd3_ represents a relatively reliable sensor. For each fluent in the original environment, a confidence is sampled according to a normal distribution \(\mathcal{N}(1,0.3^{2})\) with an interval [0,1]. This distribution has a large probability of sampling a number larger than 0.5, meaning the corresponding fluent is still more likely to be true.

_(3) NormStd5_ represents a less reliable sensor. The sampling distribution is \(\mathcal{N}(1,0.5^{2})\), which has a larger probability of generating a number smaller than 0.5 compared to _NormStd3_.

\begin{table}
\begin{tabular}{|p{227.6pt}|} \hline
**Features shared by both phases** \\ If \(a\) is a np.pdf\_whq(\(n\)). \\ The entropy of candidate groundings of \(n\). \\ If \(h\) has more than 4 grounding candidates. \\ If \(a\) is a np.pdf\_ynq(\(n\), \(o\)). \\ The probability of \(n\) grounded to \(o\). \\
**Additional Features specific for the Execution phase** \\ If \(a\) is a select.hypo(\(h\)) action. \\ The probability of hypo \(h\) not satisfied in current environment. \\ Similarity between the \(ns\) used by command \(l\) and the commands from previous experiences. \\
**Additional Features specific for the Learning phase** \\ If \(a\) is a pred.change\_ynq(\(p\)). \\ The probability of \(p\) been changed by demo. \\ \end{tabular}
\end{table}
Table 3: Example features used by the two phases. \(a\) stands for action. Other notations are the same as used in Table 2. The “If” features are binary, and the other features are real-valued.

 

_(4) UniEnv_ represents an unreliable sensor. Each number is sampled with a uniform distribution between 0 and 1. This means the sensor works randomly. A fluent has a equal change to be true or false no matter what the true environment is.

**Evaluation Metrics.** We used the same evaluation metrics as in the previous works Misra et al. (2015); She and Chai (2016) to evaluate the performance of applying the learned models to testing instances on action planning.

* _IED_: Instruction Editing Distance. This is a number between 0 and 1 measuring the similarity between the predicted action sequence and the ground-truth action sequence. _IED_ equals 1 if the two sequences are exactly the same.
* _SJI_: State Jaccard Index. This is a number between 0 and 1 measuring the similarity between the predicted and the ground-truth state changes. _SJI_ equals 1 if action planning leads to exactly the same state change as in the ground-truth.

**Configurations.** To understand the role of interactive learning in model acquisition and action planning, we first compared the interactive learning approach with the previous leading approach (presented as _She16_). To further evaluate the interaction policies acquired by reinforcement learning, we also compared the learned policy (i.e., _RLPolicy_) with the following two baseline policies:

* _RandomPolicy_ which randomly selects questions to ask during interaction.
* _ManualPolicy_ which continuously asks for yes/no confirmations (i.e., object grounding questions (\(GroundQ\)), environment questions (\(EnvQ\)), goal prediction questions (\(GoalQ\))) until there's no more questions before making a decision on model acquisition or action execution.

### Results

#### 5.2.1 The Effect of Interactive Learning

Table 4 shows the performance comparison on the testing data between the previous approach _She16_ and our interactive learning approach based on environment representations with different levels of noise. The verb models acquired by interactive learning perform better consistently across all four environment conditions. When the environment becomes noisy (i.e., _NormStd3_, _NormStd5_, and _UnEmv_), the performance of _She16_ that only relies on demonstrations decreases significantly. While the interactive learning improves the performance under the perfect environment condition, its effect in noisy environment is more remarkable. It leads to a significant performance gain between 48% and 145%. These results validate our hypothesis that interactive question answering can help to alleviate the problem of uncertainties in environment representation and goal prediction.

Figure 5 shows the performance of the various learned models on the testing data, based on a varying number of training instances and different interaction policies. The interactive learning guided by the policy acquired from RL outperforms the previous approach _She16_. The RL policy slightly outperforms interactive learning using manually defined policy (i.e., _ManualPolicy_). However, as shown in the next section, the _Man

\begin{table}
\begin{tabular}{|l|c||c||c|c||c|c|} \hline  & \multicolumn{2}{c||}{_She16_} & \multicolumn{2}{c||}{_RL policy_} & \multicolumn{2}{c|}{\% improvement} \\ \cline{2-7}  & _IED_ & _SJII_ & _IED_ & _SJI_ & _IED_ & _SJI_ \\ \hline \hline _PerfectEnv_ & 0.430 & 0.426 & 0.453 & 0.468 & 5.3\%\({}^{*}\) & 9.9\%\({}^{*}\) \\ \hline _NormStd3_ & 0.284 & 0.273 & 0.420 & 0.431 & 47.9\%\({}^{*}\) & 57.9\%\({}^{*}\) \\ \hline _NormStd5_ & 0.172 & 0.168 & 0.392 & 0.411 & 127.9\%\({}^{*}\) & 144.6\%\({}^{*}\) \\ \hline _UnEnv_ & 0.168 & 0.163 & 0.332 & 0.347 & 97.6\%\({}^{*}\) & 112.9\%\({}^{*}\) \\ \hline \end{tabular}
\end{table}
Table 4: Performance comparison between _She16_ and our interactive learning based on environment representations with different levels of noise. All the improvements (marked *) are statistically significant (\(p<0.01\)).

Figure 5: Performance (\(SJI\)) comparison by applying models acquired based on different interaction policies to the testing data.

 

ualPolicy_ results in much longer interaction (i.e., more questions) than the RL acquired policy.

#### 5.2.2 Comparison of Interaction Policies

Table 5 compares the performance of different interaction policies. It shows the average number of questions asked under different policies. It is not surprising the _RandomPolicy_ has the worst performance. For the _ManualPolicy_, its performance is similar to the _RLPolicy_. However, the average interaction length of _ManualPolicy_ is 6.792, which is much longer than the _RLPolicy_ (which is 3.127). These results further demonstrate that the policy learned from RL enables efficient interactions and the acquisition of more reliable verb models.

## 6 Conclusion

Robots live in a noisy environment. Due to the limitations in their external sensors, their representations of the shared environment can be error prone and full of uncertainties. As shown in previous work Mourao et al. (2012), learning action models from the noisy and incomplete observation of the world is extremely challenging. The same problem applies to the acquisition of verb semantics that are grounded to the perceived world. To address this problem, this paper presents an interactive learning approach which aims to handle uncertainties of the environment as well as incompleteness and conflicts in state representation by asking human partners intelligent questions. The interaction strategies are learned through reinforcement learning. Our empirical results have shown a significant improvement in model acquisition and action prediction. When applying the learned models in new situations, the models acquired through interactive learning leads to over 140% performance gain in noisy environment.

The current investigation also has several limitations. As in previous works, we assume the world can be described by a closed set of predicates. This causes significant simplification for the physical world. One of the important questions to address in the future is how to learn new predicates through interaction with humans. Another limitation is that the current utility function is learned based on a set of pre-identified features. Future work can explore deep neural network to alleviate feature engineering.

As cognitive robots start to enter our daily lives, data-driven approaches to learning may not be possible in new situations. Human partners who work side-by-side with these cognitive robots are great resources that the robots can directly learn from. Recent years have seen an increasing amount of work on task learning from human partners Saunders et al. (2006); Chernova and Veloso (2008); Cantrell et al. (2012); Mohan et al. (2013); Asada et al. (2009); Mohseni-Kabir et al. (2015); Nejati et al. (2006); Liu et al. (2016). Our future work will incorporate interactive learning of verb semantics with task learning to enable autonomy that can learn by communicating with humans.

This work was supported by the National Science Foundation (IIS-1208390 and IIS-1617682) and the DARPA SIMPLEX program under a subcontract from UCLA (N66001-15-C-4035). The authors would like to thank Dipendra K. Misra and colleagues for providing the evaluation data, and the anonymous reviewers for valuable comments.

\begin{table}
\begin{tabular}{|c|c|c|c||c|c|c|c||c||c||} \hline  & \multicolumn{6}{c|}{Average number of questions} & Performance \\ \cline{2-10}  & \multicolumn{6}{c|}{Learning Phase} & \multicolumn{6}{c|}{Execution Phase} \\ \cline{2-10}  & \(GroundQ\) & \(EnvQ\) & _TotalQ_ & \(GroundQ\) & \(EnvQ\) & \(GoalQ\) & _TotalQ_ & \(IED\) & \(SJI\) \\ \hline \hline \multirow{2}{*}{_RLPolicy_} & 2.130\({}^{*}\) & 2.615\({}^{*}\) & 4.746\({}^{*}\) & 0.383\({}^{*}\) & 0.650\({}^{*}\) & 2.626 & 3.665\({}^{*}\) & 0.420 & 0.430\({}^{*}\) \\  & +/-0.231 & +/-0.317 & +/-0.307 & +/-0.137 & +/-0.366 & +/-0.331 & +/-0.469 & +/-0.015 & +/-0.018 \\ \hline \multirow{2}{*}{_ManualPolicy_} & 2.495 & 5.338 & 7.833 & 1.236 & 3.202 & 2.353 & 6.792 & 0.406 & 0.404 \\  & +/-0.025 & +/-0.008 & +/-0.025 & +/-0.002 & +/-0.012 & +/-0.023 & +/-0.025 & +/-0.002 & +/-0.004 \\ \hline \multirow{2}{*}{_RandomPolicy_} & 0.545 & 0.368 & 0.913 & 0.678 & 0.081 & 0.151 & 0.909 & 0.114 & 0.113 \\  & +/-0.016 & +/-0.033 & +/-0.040 & +/-0.055 & +/-0.030 & +/-0.024 & +/-0.018 & +/-0.025 & +/-0.029 \\ \hline \end{tabular}
\end{table}
Table 5: Comparison between different policies including the average number (and standard deviation) of different types of questions asked during the execution phase and the learning phase respectively, and the performance on action planning for the testing data. The results are based on the noisy environment sampled by _NormStd3_. * indicates statistically significant difference (\(p<0.05\)) comparing _RLPolicy_ with _ManualPolicy_.

 



## References

* Artzi and Zettlemoyer (2013) Yoav Artzi and Luke Zettlemoyer. 2013. Weakly supervised learning of semantic parsers for mapping instructions to actions. _Transactions of the Association for Computational Linguistics_ Volume1(1):49-62.
* Asada et al. (2009) Minoru Asada, Koh Hosoda, Yasuo Kuniyoshi, Hiroshi Ishiguro, Toshio Inui, Yuichiro Yoshikawa, Masaki Ogino, and Chisato Yoshida. 2009. Cognitive developmental robotics: A survey. _IEEE Transactions on Autonomous Mental Development_ 1(1):12-34.
* Volume 1_. Association for Computational Linguistics, Stroudsburg, PA, USA, ACL '09, pages 82-90.
* Bransford et al. (2000) John D. Bransford, Ann L. Brown, and Rodney R. Cocking. 2000. _How People Learn: Brain, Mind, Experience, and School: Expanded Edition_. National Academy Press., Washington, DC.
* Cakmak and Thomaz (2012) Maya Cakmak and Andrea L. Thomaz. 2012. Designing robot learners that ask good questions. In _Proceedings of the 7th Annual ACM/IEEE International Conference on Human-Robot Interaction_. ACM, New York, NY, USA, HRI '12, pages 17-24.
* Cantrell et al. (2012) R. Cantrell, K. Talamadupula, P. Schermerhorn, J. Benton, S. Kambhampati, and M. Scheutz. 2012. Tell me when and why to do it! run-time planner model updates via natural language instruction. In _Proceedings of the 7th Annual ACM/IEEE International Conference on Human-Robot Interaction_. ACM, New York, NY, USA, HRI '12, pages 471-478.
* Chernova and Veloso (2008) Sonia Chernova and Manuela Veloso. 2008. Teaching multi-robot coordination using demonstration of communication and state sharing. In _Proceedings of the 7th international joint conference on Autonomous agents and multilingual systems-Volume 3_. International Foundation for Autonomous Agents and Multiagent Systems, pages 1183-1186.
* Fang et al. (2014) Rui Fang, Malcolim Doering, and Joyce Y. Chai. 2014. Collaborative models for referring expression generation in situated dialogue. In _Proceedings of the 28th AAAI Conference on Artificial Intelligence_. AAAI Press, AAAI'14, pages 1544-1550.
* Gao et al. (2016) Qiaozi Gao, Malcolim Doering, Shaohua Yang, and Joyce Y. Chai. 2016. Physical causality of action verbs in grounded language understanding. In _ACL (1)_. The Association for Computer Linguistics.
* Gorniak and Roy (2007) P. Gorniak and D. Roy. 2007. Situated language understanding as filtering perceived affordances. In _Cognitive Science_, volume 31(2), pages 197-231.
* Kim and Mooney (2012) Joohyun Kim and Raymond J. Mooney. 2012. Unsupervised pcfg induction for grounded language learning with highly ambiguous supervision. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing and Natural Language Learning (EMNLP-CoNLL '12)_. Jeju Island, Korea, pages 433-444.
* Knox and Stone (2011) W. Bradley Knox and Peter Stone. 2011. Understanding human teaching modalities in reinforcement learning environments: A preliminary report. In _IJCAI 2011 Workshop on Agents Learning Interactively from Human Teachers (ALHIT)_.
* Li et al. (2017) Jiwei Li, Alexander H. Miller, Sumit Chopra, Marc'Aurelio Ranzato, and Jason Weston. 2017. Learning through dialogue interactions. In _ICLR_.
* Liu and Chai (2015) Changsong Liu and Joyce Y. Chai. 2015. Learning to mediate perceptual differences in situated human-robot dialogue. In _Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence_. AAAI Press, AAAI'15, pages 2288-2294.
* Liu et al. (2014) Changsong Liu, Lanbo She, Rui Fang, and Joyce Y. Chai. 2014. Probabilistic labeling for efficient referential grounding based on collaborative discourse. In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, pages 13-18.
* Liu et al. (2016) Changsong Liu, Shaohua Yang, Sari Saba-Sadiya, Nishant Shukla, Yunzhong He, Song-chun Zhu, and Joyce Y. Chai. 2016. Jointly learning grounded task structures from language instruction and visual demonstration. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Austin, Texas, pages 1482-1492.
* Matuszek et al. (2012a) Cynthia Matuszek, Nicholas Fitzgerald, Luke Zettlemoyer, Liefeng Bo, and Dieter Fox. 2012a. A joint model of language and perception for grounded attribute learning. In John Langford and Joelle Pineau, editors, _Proceedings of the 29th International Conference on Machine Learning (ICML-12)_. ACM, New York, NY, USA, pages 1671-1678.
* Matuszek et al. (2012b) Cynthia Matuszek, Evan Herbst, Luke S. Zettlemoyer, and Dieter Fox. 2012b. Learning to parse natural language commands to a robot control system. In Jaydev P. Desai, Gregory Dudek, Oussama Khatib, and Vijay Kumar, editors, _ISER_. Springer, volume 88 of _Springer Tracts in Advanced Robotics_, pages 403-415.
* Misra et al. (2014) Dipendra K Misra, Jaeyong Sung, Kevin Lee, and Ashutosh Saxena. 2014. Tell me dave: Context-sensitive grounding of natural language to manipulation instructions. _Proceedings of Robotics: Science and Systems (RSS), Berkeley, USA_ .
* Misra et al. (2015) Dipendra Kumar Misra, Kejia Tao, Percy Liang, and Ashutosh Saxena. 2015. Environment-driven lexicon induction for high-level instructions. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the

7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. Association for Computational Linguistics, Beijing, China, pages 992-1002.
* 12th International Conference on Cognitive Modeling_.
* Mohseni-Kabir et al. (2015) Anahita Mohseni-Kabir, Charles Rich, Sonia Chernova, Candace L. Sidner, and Daniel Miller. 2015. Interactive hierarchical task learning from a single demonstration. In _Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction_. ACM, HRI '15, pages 205-212.
* Mourao et al. (2012) Kira Mourao, Luke S. Zettlemoyer, Ronald P. A. Petrick, and Mark Steedman. 2012. Learning STRIPS operators from noisy and incomplete observations. In _Proceedings of the Twenty-Eighth Conference on Uncertainty in Artificial Intelligence_. Catalina Island, CA, USA, pages 614-623.
* Nejati et al. (2006) Negin Nejati, Pat Langley, and Tolga Konik. 2006. Learning hierarchical task networks by observation. In _Proceedings of the 23rd international conference on Machine learning_. ACM, pages 665-672.
* Paek and Pieraccini (2008) Tim Piek and Roberto Pieraccini. 2008. Automating spoken dialogue management design using machine learning: An industry perspective. _Speech Communication_ 50(8-9):716-729.
* Saunders et al. (2006) Joe Saunders, Chrystopher L Nehaniv, and Kerstin Dautenhahn. 2006. Teaching robots by moulding behavior and scaffolding the environment. In _Proceedings of the 1st ACM SIGCHISIGART conference on Human-robot interaction_. ACM, pages 118-125.
* Schatzmann et al. (2006) Jost Schatzmann, Karl Weilhammer, Matt Stuttle, and Steve Young. 2006. A survey of statistical user simulation techniques for reinforcement-learning of dialogue management strategies. _Knowl. Eng. Rev._ 21(2):97-126.
* She and Chai (2016) Lanbo She and Joyce Y. Chai. 2016. Incremental acquisition of verb hypothesis space towards physical world interaction. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers_.
* She et al. (2014) Lanbo She, Shaohua Yang, Yu Cheng, Yunyi Jia, Joyce Y. Chai, and Ning Xi. 2014. Back to the blocks world: Learning new actions through situated human-robot dialogue. In _Proceedings of the SIGDIAL 2014 Conference, The 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue, 18-20 June 2014, Philadelphia, PA, USA_. pages 89-97.
* Singh et al. (2002) Satinder Singh, Diane Litman, Michael Kearns, and Marilyn Walker. 2002. Optimizing dialogue management with reinforcement learning: Experiments with the nifjun system. _Journal of Artificial Intelligence Research_ 16:105-133.
* Su et al. (2016) Pei-Hao Su, Milica Gasic, Nikola Mrksic, Lina M. Rojas Barahona, Stefan Ultes, David Vandyke, Tsung-Hsien Wen, and Steve Young. 2016. On-line active reward learning for policy optimisation in spoken dialogue systems. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, Berlin, Germany, pages 2431-2441.
* Sutton and Barto (1998) Richard S. Sutton and Andrew G. Barto. 1998. _Introduction to Reinforcement Learning_. MIT Press, Cambridge, MA, USA, 1st edition.
* Tellexx et al. (2014) Stefanie Tellexx, Prattissha Thaker, Joshua Joseph, and Nicholas Roy. 2014. Learning perceptually grounded word meanings from unaligned parallel data. _Machine Learning_ 94(2):151-167.
* Thomason et al. (2016) Jesse Thomason, Jivko Sinapov, Maxwell Svetlik, Peter Stone, and Raymond J. Mooney. 2016. Learning multi-modal grounded linguistic semantics by playing "i spy". In _Proceedings of the 25th International Joint Conference on Artificial Intelligence (IJCAI-16)_. New York City, pages 3477-3483.
* Thomason et al. (2015) Jesse Thomason, Shiqi Zhang, Raymond Mooney, and Peter Stone. 2015. Learning to interpret natural language commands through human-robot dialog. In _Proceedings of the 24th International Joint Conference on Artificial Intelligence (IJCAI)_. pages 1923-1929.
* Williams and Zweig (2016) Jason D Williams and Geoffrey Zweig. 2016. End-to-end lstm-based dialog control optimized with supervised and reinforcement learning. _arXiv preprint arXiv:1606.01269_ .
* Yang et al. (2016)Shaohua Yang, Qiaozi Gao, Changsong Liu, Caiming Xiong, Song-Chun Zhu, and Joyce Y. Chai. 2016. Grounded semantic role labeling. In _NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, 2016_. pages 149-159.