SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xiaohan Wang
Institute of Artificial Intelligence and Robotics
Xi'an Jiaotong University
wuhanxh2016@stu.xjtu.edu.cn
Yuehu Liu
liuyh@mail.xjtu.edu.cn
Xinhang Song
Institute of Computing Technology
University of Chinese Academy of Science
xinhang.song@vipl.ict.ac.cn
Beibei Wang
wangbb@stu.xjtu.edu.cn
Shuqiang Jiang
sqjiang@ict.ac.cn
Visual navigation has been widely studied under the assumption that there may be several clear routes to reach the goal. However, in more practical scenarios such as a house with several messy rooms, there may not. Interactive Navigation (InterNav) considers agents navigating to their goals more effectively with object interactions, posing new challenges of learning interaction dynamics and extra action space. Previous works learn single vision-to-action policy with the guidance of designed representations. However, the causality between actions and outcomes is prone to be confounded when the attributes of obstacles are diverse and hard to measure. Learning policy for long-term action planning in complex scenes also leads to extensive inefficient exploration. In this paper, we introduce a causal diagram of InterNav clarifying the confounding bias caused by obstacles. To address the problem, we propose a multi-policy model that enables the exploration of counterfactual interactions as well as reduces unnecessary exploration. We develop a large-scale dataset containing 600k task episodes in 12k multi-room scenes based on the ProcTHOR simulator and showcase the effectiveness of our method with the evaluations on our dataset.
Embodied AI, incorporating internet intelligence such as vision, language, and reasoning into an embodiment, aims to deploy autonomous agents in real-world environments [16]. In Visual Navigation, a cornerstone task, agents navigate to the goal in static environments where the movement of scene objects is avoided. In a practical scenario an indoor robot may need to navigate in a house with multiple rooms and get blocked by obstacles (see Figure 1). Interactive Navigation (InterNav)[39, 36] considers navigating more efficiently with object interaction rather than merely moving forward and turning around.
Blocked Positions (harder to interact from P1 to P3)
small-size lightweight
Figure 1: Illustration of InterNav in a 4-room scene. Trying to reach the goal as a blue circle, the agent is blocked by obstacles in yellow boxes spawned in three rooms. The agent needs to make decisions at three blocked positions marked in blue boxes: to bypass or to interact with obstacles? The outcomes depend on the interactive attributes of obstacles (whether they can be pushed away or picked up, e.g. heavy, small, etc.). The green arrow is a path the agent may take to reach the target and the nodes denote interaction with obstacles. Markings in colors are invisible to the agent.
too large to pick up lightweight
too large and heavy to pick or push
Years of visual navigation exploration witnessed the growing interest in long-horizon navigation in complex environments [13]. Zeng et al.[39] build a dataset benchmarking interactive navigation task on the AI2-THOR simulator, where agents receive visual observation and output the sequence of discrete actions. New challenges are raised compared to visual navigation: First, the agent needs to understand the outcome of object interaction. Second, the need for interaction with obstacles leads to a larger action space for reinforcement learning (RL). However, the dataset is limited to a simple scenario where a few objects are lined up between the starting and target point in a single room. Navigation in multi-room scenes requires agents to plan longer action sequences and understand more diverse states of the environment.
Since the dynamics of object interaction follow the physical rules that are structurally invariant in the environment, there naturally exists a cause-effect relation between the action and its outcome. From the perspective of causality, an RL agent performs interventions to the environment (by taking actions) and estimates the effect (rewards) in order to learn a policy that maximizes the long-term reward. However, learning causality through RL training is challenging due to the existence of unobserved confounders (UCs), which affect both the decision-making of actions and the generation of rewards [41, 18]. In InterNav, the agent may decide to navigate or interact based on whether it is blocked by an object (referred to as an obstacle) whose distribution is unobserved, namely unmeasurable. The existence or type of obstacles is also decisive to the outcome (see Figure 1). For instance, when blocked by a desk the agent may choose to push it aside since empirically it leads to a high reward, while the interaction will turn out otherwise if the desk gets stuck. Without knowing the true causality in interaction, the agent is prone to learn a sub-optimal policy. Prior work [39] on InterNav attempts to predict the change of object pose as a representation to understand the interaction outcomes. The model performance relies on effective obstacle detection and being familiar with obstacle distribution, which may not work in complex scenes with various obstacles. Long-term planning in complex environments also leads to inefficient exploration. Moving forward at blocked positions or trying to interact with no object around makes no difference to the environment and the agent can learn little about the causality from those experiences. Common solutions to visual navigation [35] and InterNav [39] train single policy with end-to-end RL, which requires extensive training to learn a universal policy tackling various scenarios. However, an agent may come up with multiple intents but can only conduct one behavior at once due to its embodiment. In a short-term view, the agent can either interact with objects or navigate somewhere. Hence it is more efficient to learn separate policies for interaction and navigation.
In this paper, we introduce a structural causal model, analyzing InterNav through causal lenses and clarifying the key challenge as the confounding bias caused by obstacles. We show that learning a policy conditioned on its primary decision (referred to as "intent") can enable the agent to learn from counterfactual experiences and make better decisions. As a solution to InterNav, we propose a Causal Multi-Policy model (CaMP), which reduces unnecessary exploration by breaking down the task into simple sub-tasks and enables counterfactual exploration by trialing conditioned on the agent's intent. In particular, three action policies (i.e. navigation, pushing interaction, and picking interaction) are developed by training three actor-critic networks in designed auxiliary tasks. We then train a global policy that receives both observation and action intent and calls action policies in sequence. An intent policy is developed to duplicate the decision of the global policy so that action policies can be called to produce their action intents. To study the InterNav in more practical scenarios, we collect a dataset containing 600k task episodes with various obstacle arrangements, based on the ProcTHOR simulator [14] of large-scale multi-room scenes. We also introduce a new metric measuring the time efficiency of task completion due to the need for interaction in InterNav. Evaluations on our dataset showcase the effectiveness of the proposed model CaMP.
Interactive Navigation comes up as the downstream task of visual navigation, which has been extensively studied in Embodied AI literature in the past decade. Most works address PointNav [5, 27, 35] and ObjNav [4, 6, 44, 42, 43], while others develop Visual-language Navigation [1, 29, 33, 21], AudioNav [8, 7], etc that bring in new intelligence components. Environments of sim-to-real [13] or large-scale [14] indoor scenes are also proposed for more complexity. Zeng et al. [39] focus on vision-based agents interacting with directional pushes on iTHOR environment [20]. However, they consider elementary scenarios where obstacles are spawned in a single room for agents to push out of the way. In this work we construct a new dataset based on ProcTHOR environment [14] that provides 12k multi-room scenes with diverse floorplans and various assets, supporting large-scale training.
Causal Reinforcement Learning. Causal inference [24, 25, 26] has been a longstanding methodology adopted to pursue the causal effect. Recent years have witnessed the progress of applying causal theory in computer vision tasks including visual question answering [23, 37, 9], scene graph generation [32, 10], visual recognition [22, 31, 37, 38], etc. Causal reinforcement learning (CRL) [18, 17, 40, 3, 41] is motivated by the challenges of RL including data inefficiency and lack of interpretability, and has been applied in traditional RL tasks such as multi-armed bandit problem [3], Markov decision process [41]. Naturally, the causal bias also exists in Embodied AI tasks, but there exists little work that helps train RL agents to complete tasks applying the theory of causal reasoning. We make the first attempt to model the causalities among the interactive navigation and address the confounding bias of obstacles by implementing a counterfactual policy.
Hierarchical Models for Embodied AI. To address complex and long-horizon tasks, one of the most popular solutions is the hierarchical approach inspired by Hierarchical Reinforcement learning (HRL) [30, 15]. The main idea is to divide the whole task into solvable sub-tasks through temporal abstraction which allows taking actions at different time scales [2]. Previous attempts applied on embodied question answering [11, 12], and interactive question answering [19] mainly follow the flavor of finding sub-goals for the low-level module to achieve. However, learning policy associated with sub-goals can be data expensive and requires much handcraft design for vision-based tasks. Our model is different from those works following the idea of the option framework [2] that develops multiple policies for different task situations without the need for any sub-goals.
3 A Causal View of InterNav
We start by grounding the problem of InterNav through the causal lens and clarify the challenges brought by obstacles. Then in the face of the confounding bias of obstacles, we let the agent learn counterfactual policy conditioned on its original intent so that it can gain experiences of counterfactual situations and make better decisions.
(a) Causal diagram of InterNav
(b) Intent-aware causal diagram
(c) Implementation of intent-aware policy
Figure 2: Causal diagram of proposed SCM. The part in gold of (b) denotes the causal entity of the counterfactual policy. The action policy  $\pi$  shares the same architecture with the intent policy so that they can receive the state observation in the same form.
3.1 Structural Modelling
We first formulate the underlying problem as a time-discrete Markov Decision Process (MDP) defined by a tuple  $\langle S, A, T, R, \gamma \rangle$ , where  $S$  is the state space,  $A$  is the action space,  $T: S \times A \times S \to [0,1]$  is a transition function,  $R$  is a reward function, and  $\gamma \in [0,1]$  is the discount factor. At each step  $t$ , the agent observes state  $s_t = (m_t, p_t)$  representing an egocentric visual observation and a goal coordinate, executes an action  $a_t \in A$ , and receives a reward  $r_t$ . Here, the action space  $A = \{MoveAhead, RotateLeft, RotateRight, LookUp, LookDown, PushAhead, PushBack, PushLeft, PushRight, PickUp, Drop, Done\}$ .
We then analyze the above MDP with the form of Structural Causal Model (SCM, see Figure 2). Formally, the SCM  $M$  is defined by a tuple  $\langle O,V,F,P(u)\rangle$ , where  $V = S\cup A\cup R$  is a set of observable variables,  $O$  denotes the obstacles that hinder the navigation of agents,  $F = \{f_s,f_a,f_r\}$  is the set of structural functions relative to  $V$ , and  $P(o)$  is the probability distribution over obstacles which is unobserved in InterNav. In particular, objects that currently serve as obstacles are decided by the environmental state such as the layout of surroundings  $(\mathrm{S}\rightarrow \mathrm{O})$ . At step  $t$ , action  $a_{t} = f_{a}(s_{t},o_{t})$  is decided by the agent based on state observation and its recognition of obstacles  $(\mathrm{S}\rightarrow \mathrm{A}\leftarrow \mathrm{O})$ . Practically, a stochastic policy  $\pi (a_t|s_t):S\times A\to [0,1]$  is learned to cover the process. Note that although  $S$  denotes the state encompassing all information of the environment, it causally influences agent's actions in a partially observable way through visual observations. Reward  $r_t = f_r(s_t,a_t,o_t)$  is returned by the environment based on state, action and obstacles to which the action is applied  $(\mathrm{S,A,O}\rightarrow \mathrm{R})$ . The current state  $s_t = f_s(s_{t - 1},a_{t - 1})$  is decided by the state and action at last time step. The goal of the agent is to learn a policy  $\pi^{*} = argmaxV_{\pi}(s_{t})$  that maximizes the expected cumulative reward:
$$
V _ {\pi} \left(s _ {t}\right) = \mathbb {E} \left[ \sum_ {k = 0} ^ {\infty} \gamma^ {k} r _ {t + k + 1} \mid s _ {t} \right] \tag {1}
$$
which is formulated in the form of state-value function  $V_{\pi}$  and  $\gamma$  is the discount factor.
Through interaction with the environment, the agent performs intervention  $do(A = a)$ , that actively sets  $A$  as  $a$  rather than passively observes when  $A$  appears to be  $a$ , and learns from the effect which can be written probabilistically as  $P(R = r|do(A = a), S = s)$ . Since obstacles influence both the decision of actions and the generation of rewards (A←O→R, see dashed arrows in Figure 2(a)), the causal relation between A and R is confounded such that  $P(r|do(a)) \neq P(r|a)$ , to which the confounding bias refers [25]. During RL training, the model estimates spurious value of actions (in the form of reward) due to the confounding bias which may result in sub-optimal policy. For example, frequent encounters with heavy obstacles (e.g. chairs, tables) make the agent predict a higher value of pushing them than picking them up. The agent can get familiar with limited distribution  $P(o)$  with extensive training. However, without considering the causalities concerning obstacles (A←O→R), it's hard to generalize to unseen environments or large-scale datasets.
3.2 Counterfactual Decision-making
The underlying confounding bias can be tackled with the ability of counterfactual decision-making. A counterfactual effect  $P(R_{A = a}|A = a')$  of taking action can be read as "The reward the agent
Figure 3: Model overview. Our model CaMP is composed of a global policy, three action policies, and an intent policy. Given an observation, the intent policy predicts a policy  $\omega_{t}^{\prime}$  and low-level policies predict an action  $a_{t}^{\prime}$ , which is then weighted by the policy distribution as the integrated intent  $i_{t}$  (equation (7)). Finally the global policy decides the policy  $\omega_{t}$  based on the state representation and the intent, and the chosen policy executes the action  $a_{t}$ .
would obtain had it taken action  $A = a$  (contrary to the fact), given  $A = a''$ , which is imaginary thus usually not estimable from data. However, Baireinboim et al. [3] show that the above counterfactual quantity can be estimated if the action is interrupted before its decision is executed (ETT, Effect of Treatment on the Treated). We call the decision before execution as agent's intent  $I = i_{t} = f_{i}(s_{t},o_{t})$  which shares the same causal parents of action  $A$ . Our goal is to learn a counterfactual policy  $\pi_{ctf}^{*} = \arg\max V_{\pi_{ctf}}(s_{t}')$  (see Figure 2(b)) that maximizes the long-term value conditioned on the intent-specific state  $s_{t}' = (s_{t},i_{t})$ :
$$
V _ {\pi_ {c t f}} \left(s _ {t} ^ {\prime}\right) = \mathbb {E} \left[ \sum_ {k = 0} ^ {\infty} \gamma^ {k} r _ {t + k + 1} \mid s _ {t}, i _ {t} \right] \tag {2}
$$
The action distribution of the counterfactual policy  $P(a_{t}|i_{t}) = \pi_{cft}(s_{t},i_{t})$  is posterior to the intent distribution  $P(i_{t}) = \pi (s_{t})$ . Thus the agent can obtain both experimental experiences (ones the original policy "would have collected" when  $a_{t} = i_{t}$ ) and counterfactual experiences (when  $a_{t}\neq i_{t}$ ) that explore external strategies. The intent also provides context about the obstacles due to their causal relation. Thus the counterfactual policy obtains more value than standard policy [41]:
$$
V _ {\pi_ {c t f} ^ {*}} \left(s _ {t}, i _ {t}\right) \geq V _ {\pi^ {*}} \left(s _ {t}\right) \tag {3}
$$
where equality does not hold when unobserved confounders (obstacles) exist. Intuitively, an agent that explores different counterfactual situations (e.g. "what if I push/pick up the box instead of bypassing it?") has a better understanding of the expected value of the current state, compared to the agent taking actions out of intuition. We illustrate the idea of how we implement a counterfactual policy in Figure 2(c).
4 The Proposed Solution
In this section, we first introduce a hierarchical framework for multiple policies to cooperate. Then we implement a global policy with the ability of counterfactual decision-making which plans three action policies based on the integrated intent of the model. The model architecture is shown in Figure 3.
4.1 Hierarchical Policy Framework
Essentially, we reduce the task complexity following a prior understanding that the InterNav task is temporally separable. 3 action policies are defined in Table 1 to split the action space. Policy
Table 1: Descriptions of action policies
Policy	Action space	Objective
Navigate	MoveAhead, RotateLeft, Rotate-Right, LookUp, LookDown, Done	Navigating to the goal point within a distance of 0.2m.
Push	PushAhead, PushBack, PushLeft, PushRight, Done	Moving obstacles until there is a reachable path towards the goal or the existing path gets shorter.
Pick	PickUp, Drop, RotateLeft, Rotate-Right, Done	Picking and dropping small obstacles until there is a reachable path towards the goal or the existing path gets shorter.
Navigate focuses on the ability of point navigation without considering the environment dynamics. While Push and Pick policies aim for the ability of object manipulation according to the goal and agent position. Formally, assume action policy  $\pi_{\omega}$  is called at step  $t$  mapping the transition from state observation  $s_t$  to the distribution of action  $a_t$ :
$$
P \left(a _ {t}\right) = \pi_ {\omega} \left(s _ {t}, h _ {t - 1}\right), a _ {t} \in \mathcal {A} _ {\omega} \tag {4}
$$
where  $h_{t-1}, \mathcal{A}_{\omega}$  denotes the recurrent hidden state and the action space of policy  $\pi_{\omega}$ . In our model,  $s_t$  is the concatenation of goal embedding and visual features extracted with a convolutional neural network.  $\pi_{\omega}$  is implemented with a GRU and two linear layers for both policy (actor) and value (critic). We pretrain each action policy with an auxiliary task designed according to their objectives.
With the short-term abilities of action policies, a global policy  $\pi_{\Omega}$  aims to plan a sequence of policies  $\omega_{t} \in \Omega = \{\text{Navigate}, \text{Push}, \text{Pick}, \text{Done}\}$  for the whole task completion. We let the agent follow the Navigate policy by default and the global policy decides whether to interact during the long horizon navigation. Once an interaction policy is called, it does not return the control until the sub-task termination (output Done).  $\pi_{\Omega}$  shares the same model architecture with  $\pi_{\omega}$ .
4.2 Intent-aware policy planning
Since the global policy plans when to interact with obstacles, it's crucial to learn action-outcome causality (e.g. whether calling push policy results in a clear path). Moreover, unlike low-level policies, the global policy does not receive direct rewards for its decisions during training, which makes their causal relation harder to learn. Thus we introduce a counterfactual global policy which makes plans based on the theory in section 3.2, in order to enable more effective policy learning. The intent of the agent is combined with its observation as the intent-specific state that is fed to the global policy. Instead of planning policies based on its own intent of policy,  $\pi_{\Omega}$  is provided with the integrated intent of basic actions  $i_t \in \mathcal{A}$ :
$$
P \left(\omega_ {t}\right) = \pi_ {\Omega} \left(s _ {t}, P \left(i _ {t}\right), h _ {t - 1}\right) \tag {5}
$$
$$
P \left(\omega_ {t} ^ {\prime}\right) = \pi_ {\Omega} ^ {\prime} \left(s _ {t}, P \left(\bar {i _ {t}}\right), h _ {t - 1}\right) \tag {6}
$$
$$
P \left(i _ {t}\right) = \sum_ {\omega^ {j} \in \Omega} \pi_ {\omega^ {j}} \left(s _ {t}, h _ {t - 1}\right) \cdot P \left(\omega_ {t} ^ {\prime} = \omega^ {j}\right) \tag {7}
$$
where  $\pi_{\Omega}^{\prime}$  is the intent policy that duplicates the decision-making of global policy and  $\overline{i_t}$  is a plane vector denoting an intent without any inclination. Each action intent (zero-padded to the same size) produced by policy  $\pi_{\omega^j}$  is weighted by the probability that the decision of  $\pi_{\Omega}^{\prime}$  is  $\omega^{j}$ . Note that  $P(i_{t})$  is encoded with a linear function and concatenated to the observation  $v_{t}$  as the input of GRU. By providing information of the intents of low-level policies, the new context bridges the relation between actions and rewards for the global policy.
We train the model with Proximal Policy Optimization (PPO) [28] algorithm which updates the model parameters for  $k$  iterations with a rollout of data. To ensure that the intent represents the decision of the agent, we synchronize the parameters of two networks  $(\pi_{\Omega}^{\prime} \gets \pi_{\Omega})$  during training. Since the global policy does trials conditioned on the distribution of model intent, similar distributions of actions and intents are more likely to be sampled identically (experimental case,  $a_{t} = i_{t}$ ). On the contrary, the distinctive distributions of actions and intents lead to more counterfactual cases  $(a_{t} \neq i_{t})$ , which brings entropy increase for the system. In other words, the agent tends to exploit its internal knowledge about influencing UCs (obstacles) receiving its own intent, while exploring the external space of policy when setting off from the old intent. Since the global policy is becoming distinct from the intent policy after iterations, we set the synchronization frequency as  $K$  rollouts to balance the exploration and exploitation of policy learning.
1 rooms scene
2 rooms scene
Figure 4: Top view examples of our dataset. Blue circles denote the starting and target point. Yellow boxes denote the obstacles we spawn on the path.
4 rooms scene (across 2 rooms)
9 rooms scene (across 6 rooms)
Table 2: Statistics of proposed dataset split by the number of rooms. The ratio of each split, number of spawned obstacles, initial distance between start and target, and number of times the agent needs to across the room are counted.
Dataset splits (room num)	ratio	obstacle num	distance(m)	cross-room num
1~2	39.0%	2.05	5.45	1.44
3~5	42.3%	3.37	9.92	2.50
6~10	18.7%	4.58	14.67	3.29
5.1 Experiment Setup
Data collection. We perform experiments with the ProcTHOR simulator [14] on our dataset, which contains 600k task episodes in 12k scenes (50 per scene). For each episode, we first randomly set the starting and target points in two different rooms. Second, we calculate all room paths (e.g. room1→room4→room7) and all node paths (e.g. coordinates of start→door3→target). At last we randomly pick obstacle assets (655 in total) at random positions between adjacent nodes until there is no clear path or lengthen the shortest path (50% probability each). The dataset is split by scenes so the models are evaluated in unseen scenes: 11.8k training scenes, 100 validation scenes (5k episodes), and 100 testing scenes (500 episodes). In Table 2 the statistics of our dataset show that the agent is challenged by multiple obstacles, long-distance navigation, and multi-door passing. Figure 4 shows several top-view examples of our dataset.