Title: CaMP: Causal Multi-policy Planning for Interactive Navigation in  Multi-room Scenes
Venue: NeurIPS 2023
Accepted: yes
Keywords: Embodied AI

Interactive Navigation

Causal Reinforcement Learning

Hierarchical Reinforcement Learning
Abstract: Visual navigation has been widely studied under the assumption that there may be several clear routes to reach the goal. However, in more practical scenarios such as a house with several messy rooms, there may not. Interactive Navigation (InterNav) considers agents navigating to their goals more effectively with object interactions, posing new challenges of learning interaction dynamics and extra action space. Previous works learn single vision-to-action policy with the guidance of designed representations. However, the causality between actions and outcomes is prone to be confounded when the attributes of obstacles are diverse and hard to measure. Learning policy for long-term action planning in complex scenes also leads to extensive inefficient exploration. In this paper, we introduce a causal diagram of InterNav clarifying the confounding bias caused by obstacles. To address the problem, we propose a multi-policy model that enables the exploration of counterfactual interactions as well as reduces unnecessary exploration. We develop a large-scale dataset containing 600k task episodes in 12k multi-room scenes based on the ProcTHOR simulator and showcase the effectiveness of our method with the evaluations on our dataset.

CaMP: Causal Multi-policy Planning for Interactive Navigation in Multi-room Scenes

Abstract

Visual navigation has been widely studied under the assumption that there may be several clear routes to reach the goal. However, in more practical scenarios such as a house with several messy rooms, there may not. Interactive Navigation (InterNav) considers agents navigating to their goals more effectively with object interactions, posing new challenges of learning interaction dynamics and extra action space. Previous works learn single vision-to-action policy with the guidance of designed representations. However, the causality between actions and outcomes is prone to be confounded when the attributes of obstacles are diverse and hard to measure. Learning policy for long-term action planning in complex scenes also leads to extensive inefficient exploration. In this paper, we introduce a causal diagram of InterNav clarifying the confounding bias caused by obstacles. To address the problem, we propose a multi-policy model that enables the exploration of counterfactual interactions as well as reduces unnecessary exploration. We develop a large-scale dataset containing 600k task episodes in 12k multi-room scenes based on the ProcTHOR simulator and showcase the effectiveness of our method with the evaluations on our dataset.

1 Introduction

Embodied AI, incorporating internet intelligence such as vision, language, and reasoning into an embodiment, aims to deploy autonomous agents in real-world environments [16]. In Visual Navigation, a cornerstone task, agents navigate to the goal in static environments where the movement of scene objects is avoided. In a practical scenario an indoor robot may need to navigate in a house with multiple rooms and get blocked by obstacles(see Figure 1). Interactive Navigation (InterNav)[39, 36] considers navigating more efficiently with object interaction rather than merely moving forward and turning around.

Blocked Positions (harder to interact from P1 to P3)

bypass

Goal

Agent

Top view

Desk

push

P2

Sofa

P3

pick up

P1

Lettuce

Microwave

P2

P3

P1

small-size

lightweight

too large to pick up

lightweight

too large and heavy to

pick or push

Goal

narrow

wide

wide

Years of visual navigation exploration witnessed the growing interest in long-horizon navigation in complex environments [13]. Zeng et al.[39] build a dataset benchmarking interactive navigation task on the AI2-THOR simulator, where agents receive visual observation and output the sequence of discrete actions. New challenges are raised compared to visual navigation: First, the agent needs to understand the outcome of object interaction. Second, the need for interaction with obstacles leads to a larger action space for reinforcement learning (RL). However, the dataset is limited to a simple scenario where a few objects are lined up between the starting and target point in a single room. Navigation in multi-room scenes requires agents to plan longer action sequences and understand more diverse states of the environment.

Since the dynamics of object interaction follow the physical rules that are structurally invariant in the environment, there naturally exists a cause-effect relation between the action and its outcome. From the perspective of causality, an RL agent performs interventions to the environment (by taking actions) and estimates the effect (rewards) in order to learn a policy that maximizes the long-term reward. However, learning causality through RL training is challenging due to the existence of unobserved confounders (UCs), which affect both the decision-making of actions and the generation of rewards [41, 18]. In InterNav, the agent may decide to navigate or interact based on whether it is blocked by an object (referred to as an obstacle) whose distribution is unobserved, namely unmeasurable. The existence or type of obstacles is also decisive to the outcome (see Figure 1). For instance, when blocked by a desk the agent may choose to push it aside since empirically it leads to a high reward, while the interaction will turn out otherwise if the desk gets stuck. Without knowing the true causality in interaction, the agent is prone to learn a sub-optimal policy. Prior work [39] on InterNav attempts to predict the change of object pose as a representation to understand the interaction outcomes. The model performance relies on effective obstacle detection and being familiar with obstacle distribution, which may not work in complex scenes with various obstacles. Long-term planning in complex environments also leads to inefficient exploration. Moving forward at blocked positions or trying to interact with no object around makes no difference to the environment and the agent can learn little about the causality from those experiences. Common solutions to visual navigation [35] and InterNav [39] train single policy with end-to-end RL, which requires extensive training to learn a universal policy tackling various scenarios. However, an agent may come up with multiple intents but can only conduct one behavior at once due to its embodiment. In a short-term view, the agent can either interact with objects or navigate somewhere. Hence it is more efficient to learn separate policies for interaction and navigation.

In this paper, we introduce a structural causal model, analyzing InterNav through causal lenses and clarifying the key challenge as the confounding bias caused by obstacles. We show that learning a policy conditioned on its primary decision (referred to as "intent") can enable the agent to learn from counterfactual experiences and make better decisions. As a solution to InterNav, we propose a Causal Multi-Policy model (CaMP), which reduces unnecessary exploration by breaking down the task into simple sub-tasks and enables counterfactual exploration by trialing conditioned on the agent's intent. In particular, three action policies (i.e. navigation, pushing interaction, and picking interaction) are developed by training three actor-critic networks in designed auxiliary tasks. We then train a global policy that receives both observation and action intent and calls action policies in sequence. An intent policy is developed to duplicate the decision of the global policy so that action policies can be called to produce their action intents. To study the InterNav in more practical scenarios, we collect a dataset containing 600k task episodes with various obstacle arrangements, based on the ProcTHOR simulator [14] of large-scale multi-room scenes. We also introduce a new metric measuring the time efficiency of task completion due to the need for interaction in InterNav. Evaluations on our dataset showcase the effectiveness of the proposed model CaMP.

2 Related Works

Interactive Navigation comes up as the downstream task of visual navigation, which has been extensively studied in Embodied AI literature in the past decade. Most works address PointNav [5, 27, 35] and ObjNav [4, 6, 44, 42, 43], while others develop Visual-language Navigation [1, 29, 33, 21], AudioNav [8, 7], etc that bring in new intelligence components. Environments of sim-to-real [13] or large-scale [14] indoor scenes are also proposed for more complexity. Zeng et al. [39] focus on vision-based agents interacting with directional pushes on iTHOR environment [20]. However, they consider elementary scenarios where obstacles are spawned in a single room for agents to push out of the way. In this work we construct a new dataset based on ProcTHOR environment [14] that provides 12k multi-room scenes with diverse floorplans and various assets, supporting large-scale training.

Causal Reinforcement Learning. Causal inference [24, 25, 26] has been a longstanding methodology adopted to pursue the causal effect. Recent years have witnessed the progress of applying causal theory in computer vision tasks including visual question answering [23, 37, 9], scene graph generation [32, 10], visual recognition [22, 31, 37, 38], etc. Causal reinforcement learning (CRL) [18, 17, 40, 3, 41] is motivated by the challenges of RL including data inefficiency and lack of interpretability, and has been applied in traditional RL tasks such as multi-armed bandit problem [3], Markov decision process [41]. Naturally, the causal bias also exists in Embodied AI tasks, but there exists little work that helps train RL agents to complete tasks applying the theory of causal reasoning. We make the first attempt to model the causalities among the interactive navigation and address the confounding bias of obstacles by implementing a counterfactual policy.

Hierarchical Models for Embodied AI. To address complex and long-horizon tasks, one of the most popular solutions is the hierarchical approach inspired by Hierarchical Reinforcement learning (HRL) [30, 15]. The main idea is to divide the whole task into solvable sub-tasks through temporal abstraction which allows taking actions at different time scales [2]. Previous attempts applied on embodied question answering [11, 12], and interactive question answering [19] mainly follow the flavor of finding sub-goals for the low-level module to achieve. However, learning policy associated with sub-goals can be data expensive and requires much handcraft design for vision-based tasks. Our model is different from those works following the idea of the option framework [2] that develops multiple policies for different task situations without the need for any sub-goals.

3 A Causal View of InterNav

We start by grounding the problem of InterNav through the causal lens and clarify the challenges brought by obstacles. Then in the face of the confounding bias of obstacles, we let the agent learn counterfactual policy conditioned on its original intent so that it can gain experiences of counterfactual situations and make better decisions.

A

O

R

S

A: Action

R: Reward

S: State

O: Obstacles

I: Intent

(a) Causal diagram of InterNav

πctf

(c) Implementation of intent-aware policy

state

observation

S

action policy

action

A

intent policy

intent

I

parameter

sharing

A

O

R

S

I

(b) Intent-aware causal diagram

3.1 Structural Modelling

We first formulate the underlying problem as a time-discrete Markov Decision Process (MDP) defined by a tuple ⟨ S, A, T, R, γ ⟩ , where S is the state space, A is the action space, T : S × A × S → [0 , 1] is a transition function, R is a reward function, and γ ∈ [0 , 1] is the discount factor. At each step t , the agent observes state s t = ( m t , p t ) representing an egocentric visual observation and a goal coordinate, executes an action a t ∈ A , and receives a reward r t . Here, the action space A = { MoveAhead, RotateLeft, RotateRight, LookUp, LookDown, PushAhead, PushBack, PushLeft, PushRight, PickUp, Drop, Done } .

We then analyze the above MDP with the form of Structural Causal Model (SCM, see Figure 2). Formally, the SCM M is defined by a tuple ⟨ O,V,F,P ( u ) ⟩ , where V = S ∪ A ∪ R is a set of observable variables, O denotes the obstacles that hinder the navigation of agents, F = { f s , f a , f r } is the set of structural functions relative to V , and P ( o ) is the probability distribution over obstacles which is unobserved in InterNav. In particular, objects that currently serve as obstacles are decided by the environmental state such as the layout of surroundings (S → O). At step t , action a t = f a ( s t , o t ) is decided by the agent based on state observation and its recognition of obstacles (S → A ← O). Practically, a stochastic policy π ( a t | s t ) : S × A → [0 , 1] is learned to cover the process. Note that although S denotes the state encompassing all information of the environment, it causally influences agent's actions in a partially observable way through visual observations. Reward r t = f r ( s t , a t , o t ) is returned by the environment based on state, action and obstacles to which the action is applied (S,A,O → R). The current state s t = f s ( s t -1 , a t -1 ) is decided by the state and action at last time step. The goal of the agent is to learn a policy π ∗ = argmaxV π ( s t ) that maximizes the expected cumulative reward:

which is formulated in the form of state-value function V π and γ is the discount factor.

̸

Through interaction with the environment, the agent performs intervention do ( A = a ) , that actively sets A as a rather than passively observes when A appears to be a , and learns from the effect which can be written probabilistically as P ( R = r | do ( A = a ) , S = s ) . Since obstacles influence both the decision of actions and the generation of rewards (A ← O → R, see dashed arrows in Figure 2(a)), the causal relation between A and R is confounded such that P ( r | do ( a )) = P ( r | a ) , to which the confounding bias refers [25]. During RL training, the model estimates spurious value of actions (in the form of reward) due to the confounding bias which may result in sub-optimal policy. For example, frequent encounters with heavy obstacles (e.g. chairs, tables) make the agent predict a higher value of pushing them than picking them up. The agent can get familiar with limited distribution P ( o ) with extensive training. However, without considering the causalities concerning obstacles (A ← O → R), it's hard to generalize to unseen environments or large-scale datasets.

3.2 Counterfactual Decision-making

The underlying confounding bias can be tackled with the ability of counterfactual decision-making. A counterfactual effect P ( R A = a | A = a ′ ) of taking action can be read as "The reward the agent

(x, y)

goal position

egocentric view

Global Policy

Action Policies

Push

Pick

Navigate

Visual

Encoding

Intent Policy

vt

state

representation

Embedding

State

Encoding

call

action

'

PushRight

'

i

t

intent

ωt

at

ωt

`

i

t

void intent

'

at

'

policy intent

policy decision

sum.

would obtain had it taken action A = a (contrary to the fact), given A = a ′ ", which is imaginary thus usually not estimable from data. However, Bareinboim et al. [3] show that the above counterfactual quantity can be estimated if the action is interrupted before its decision is executed (ETT, Effect of Treatment on the Treated). We call the decision before execution as agent's intent I = i t = f i ( s t , o t ) which shares the same causal parents of action A . Our goal is to learn a counterfactual policy π ∗ ctf = argmaxV π ctf ( s ′ t ) (see Figure 2(b)) that maximizes the long-term value conditioned on the intent-specific state s ′ t = ( s t , i t ) :

̸

The action distribution of the counterfactual policy P ( a t | i t ) = π cft ( s t , i t ) is posterior to the intent distribution P ( i t ) = π ( s t ) . Thus the agent can obtain both experimental experiences (ones the original policy "would have collected" when a t = i t ) and counterfactual experiences (when a t = i t ) that explore external strategies. The intent also provides context about the obstacles due to their causal relation. Thus the counterfactual policy obtains more value than standard policy [41]:

where equality does not hold when unobserved confounders (obstacles) exist. Intuitively, an agent that explores different counterfactual situations (e.g. "what if I push/pick up the box instead of bypassing it?") has a better understanding of the expected value of the current state, compared to the agent taking actions out of intuition. We illustrate the idea of how we implement a counterfactual policy in Figure 2(c).

4 The Proposed Solution

In this section, we first introduce a hierarchical framework for multiple policies to cooperate. Then we implement a global policy with the ability of counterfactual decision-making which plans three action policies based on the integrated intent of the model. The model architecture is shown in Figure 3.

4.1 Hierarchical Policy Framework

Essentially, we reduce the task complexity following a prior understanding that the InterNav task is temporally separable. 3 action policies are defined in Table 1 to split the action space. Policy

Navigate focuses on the ability of point navigation without considering the environment dynamics. While Push and Pick policies aim for the ability of object manipulation according to the goal and agent position. Formally, assume action policy π ω is called at step t mapping the transition from state observation s t to the distribution of action a t :

where h t -1 , A ω denotes the recurrent hidden state and the action space of policy π ω . In our model, s t is the concatenation of goal embedding and visual features extracted with a convolutional neural network. π ω is implemented with a GRU and two linear layers for both policy (actor) and value (critic). We pretrain each action policy with an auxiliary task designed according to their objectives.

With the short-term abilities of action policies, a global policy π Ω aims to plan a sequence of policies ω t ∈ Ω = { Navigate, Push, Pick, Done } for the whole task completion. We let the agent follow the Navigate policy by default and the global policy decides whether to interact during the long horizon navigation. Once an interaction policy is called, it does not return the control until the sub-task termination (output Done ). π Ω shares the same model architecture with π ω .

4.2 Intent-aware policy planning

Since the global policy plans when to interact with obstacles, it's crucial to learn action-outcome causality (e.g. whether calling push policy results in a clear path). Moreover, unlike low-level policies, the global policy does not receive direct rewards for its decisions during training, which makes their causal relation harder to learn. Thus we introduce a counterfactual global policy which makes plans based on the theory in section 3.2, in order to enable more effective policy learning. The intent of the agent is combined with its observation as the intent-specific state that is fed to the global policy. Instead of planning policies based on its own intent of policy, π Ω is provided with the integrated intent of basic actions i t ∈ A :

where π ′ Ω is the intent policy that duplicates the decision-making of global policy and i t is a plane vector denoting an intent without any inclination. Each action intent (zero-padded to the same size) produced by policy π ω j is weighted by the probability that the decision of π ′ Ω is ω j . Note that P ( i t ) is encoded with a linear function and concatenated to the observation v t as the input of GRU. By providing information of the intents of low-level policies, the new context bridges the relation between actions and rewards for the global policy.

̸

We train the model with Proximal Policy Optimization (PPO) [28] algorithm which updates the model parameters for k iterations with a rollout of data. To ensure that the intent represents the decision of the agent, we synchronize the parameters of two networks ( π ′ Ω ← π Ω ) during training. Since the global policy does trials conditioned on the distribution of model intent, similar distributions of actions and intents are more likely to be sampled identically (experimental case, a t = i t ). On the contrary, the distinctive distributions of actions and intents lead to more counterfactual cases ( a t = i t ), which brings entropy increase for the system. In other words, the agent tends to exploit its internal knowledge about influencing UCs (obstacles) receiving its own intent, while exploring the external space of policy when setting off from the old intent. Since the global policy is becoming distinct from the intent policy after iterations, we set the synchronization frequency as K rollouts to balance the exploration and exploitation of policy learning.

1 rooms scene

2 rooms scene

4 rooms scene (across 2 rooms)

9 rooms scene (across 6 rooms)

5 Experiments

5.1 Experiment Setup

Data collection. We perform experiments with the ProcTHOR simulator [14] on our dataset, which contains 600k task episodes in 12k scenes (50 per scene). For each episode, we first randomly set the starting and target points in two different rooms. Second, we calculate all room paths (e.g. room1 → room4 → room7) and all node paths (e.g. coordinates of start → door3 → target). At last we randomly pick obstacle assets (655 in total) at random positions between adjacent nodes until there is no clear path or lengthen the shortest path (50% probability each). The dataset is split by scenes so the models are evaluated in unseen scenes: 11.8k training scenes, 100 validation scenes (5k episodes), and 100 testing scenes (500 episodes). In Table 2 the statistics of our dataset show that the agent is challenged by multiple obstacles, long-distance navigation, and multi-door passing. Figure 4 shows several top-view examples of our dataset.

Environment settings. Following the standard settings, we let MoveAhead move the agent ahead by 0.25 meters, RotateRight and RotateLeft change the agent's azimuth angle by ±90 degrees, LookUp and LookDown rotate the agent's camera elevation angle by ±30 degrees. The DirectionalPushs let the agent push (along ±z and ±x axis) the closest visible object with a constant force. The agent takes the END to indicate that it has completed an episode.

Evaluation metrics. We adopt Success Rate (SR), Final Distance to Target (FDT), and Success weighted by Path Length (SPL). SR is the ratio of successful episodes in total episodes, FDT is the average geodesic distance between the agent and the goal when the episode is finished, and SPL is calculated as 1 N ∑ N n =1 Suc n L n max ( P n ,L n ) , where N is the total episodes number, Suc n is the successful indicator of n -th episode, L n is the shortest path length, and P n is the length of the real path.

In non-interactive navigation, the metric SPL that measures path efficiency is also a usable indicator of time efficiency, since the length of a path is usually proportional to the time it takes to execute the corresponding navigation actions (e.g. move ahead, rotate right). However, a shorter path does not guarantee less time consumption for task completion in InterNav due to the object interaction. For instance, the agent may take the shortest path to the target through extensive interactions which cost a large amount of time and end up less effective than taking a detour path. Thus we evaluate an additional metric STS (Success rate weighted by Time Steps) to measure the time cost of task completion: STS = 1 N ∑ N n =1 Suc n L n /grid TS n , where L n is the shortest path length, TS n is the timesteps the agent takes to complete the task, and grid = 0 . 25 m is the unit distance of a step. Thus L n /grid represents the number of timesteps it takes to navigate to the target by merely moving forward. STS can be regarded as a time-measurement variant of SPL and the score is higher when the agent accomplishes the task with less time. The ideal situation is that the target is directly ahead and there is no need for interaction where STS is 1.

5.2 Implementation Details

Our method is implemented and evaluated using the AllenAct [34] framework. The egocentric observation is set as 300*300 RGB and depth images. We first train three action policies then the whole model, utilizing Adam with an initial learning rate of 3 · 10 -4 for policy training and 1 · 10 -4 for joint model training. The hidden size of CNN, GRU is set as 512. The intent embedding size is set as 12 (the same as the intent size and the length of action space). The frequency of synchronization K and epoch number k are set as 3 and 1. All models are trained for 10 million steps with 2 million warm-up steps in single-room scenes (1m for action policies and 1m for joint training) and 8 million in all scenes 1 .

Reward shaping. We set the reward for our task as three parts: r = r success +∆ dis -r sp , where r success = 10 is provided if the agent takes action Done meanwhile achieving the goal. ∆ dis represents the difference of geodesic distance between the agent and the goal position, and r sp = 0 . 01 denotes the step penalty.

Three auxiliary tasks are designed as: (1) The Navigate policy is pretrained with the PointNav task where the agent moves to the goal point in an obstacle-free environment. The task shares the same reward r nav = r as above. (2) To pretrain the policy of Push and Pick , the agent is placed aside an obstacle opposite to the goal. It achieves success by taking Done when the obstacle is cleared. The reward r inter = r nav + r as -r af , where r as , r af denote the successful reward and failed penalty of chosen action (e.g. fail to conduct Drop before picking up something). Since effective pushes or picks change the length of the shortest path, ∆ dis also applies to interaction reward.

5.3 Baselines

Random : This baseline randomly takes actions with the same probability until the agent reaches the target or reaches the maximum number of steps.

PPO : We set a common baseline trained with PPO algorithm according to [35]. Similar to our method, it extracts visual features with an encoder and reads the goal position with an embedding layer, then a single policy network predicts actions based on fused features.

NIE : We implement the method NIE [39] which predicts the state change of observed objects conditioned on actions with a module in addition to the PPO baseline. The representation of object change is fused with visual features and goal embedding.

HRL : To study the effect of our multi-policy framework, we implement a common hierarchical policy supported by AllenAct [34] that shares the same three sub-spaces of action. The difference is that the low-level policies are called at each step, not able to plan actions in a period.

PPO+intent : We develop a counterfactual policy based on the PPO baseline according to Figure 2(c) to study the effect of intent-aware decision-making.

5.4 Results Analysis

Comparison with baselines. We pick model parameters on the validation set and report their performance on the whole testing set, hard scenes with more rooms ( N ≥ 4 ), and the splits of episodes that exist clear paths (non-interactive). As quantitatively shown in Table 3, CaMP outperforms all baselines and obtains +13.6/+7.5, +18.1/+9.2, and +4.3/+3.5 improvement in SR, SPL, and STS (all/ N ≥ 4 , %) respectively compared to the model PPO. Compared with PPO+intent, CaMP obtains more improvement in hard scenes (+3.4 and +3.3 in SR, SPL) which shows the robustness of multi-policy facing complex environment. Besides, the relatively poorer performances of HRL and Random model reflect the challenge of our dataset. The limited performance in hard scenes shows that effective methods for navigation in plenty of rooms are still in need. The contrary trends across STS and SPL when comparing PPO and HRL on the hard split and the non-interactive split indicate that both path and time efficiency should be measured considering InterNav scenarios.

Ablation study. Our ablation study aims to explore the effect of two model designs. First, we change the synchronization frequency of model PPO+intent as 1 epoch, 1 rollout, 3 rollouts, and 5 rollouts. As shown in Table 4, the variant with K = 3 gains the relatively best performance, which reflects how the balanced exploration-exploitation benefits policy learning.

Second, we study how the model intent influences performance: We implement (1) CaMP without the intent network as wo/intent , (2) CaMP that inferences without the intent network as test wo/intent , (3) CaMP provided with the intent recursively generated based on the prior intent as recursive intent , and (4) CaMP provided with the intent of the global policy as policy intent . Here our model CaMP is referred to as integrated intent . As shown in Table 4, the integrated intent effectively improves the decision-making ability of the global policy compared with wo/intent (+5.5/+3.3 and +10.4/+8.0 in SR, SPL). Without the information of the intent of action policies, policy intent shows less gain effect compared with integrated intent (-3.8/-2.3 and -5.6/-4.0 in SR, SPL). The limited performance of test wo/intent suggests that the inference process of our model also relies on the intent. The results of recursive intent suggest that although the secondary intent still provides incremental information about the system, the counterfactual situation based on the "counterfact" may become less meaningful for learning causality.

PPO

CaMP

Top view (scene_1999)

Egocentric view at key steps

Tomato

Tomato

Sofa

Pillow

Sofa

Goal

Table

t=0

t=0

t=25

t=97

t=25

t=35

t=70

t=50

t=95

Case study. We qualitatively compare our model with a baseline PPO on a typical instance in a 4-room scene in Figure 5. The blue trajectory of CaMP shows that the agent successfully removes obstacles of tomato and pillow , and attempts to bypass an immovable sofa to reach the target. While the red trajectory shows that PPO intends to interact with a table and a bed which are too heavy to be pushed aside and ends up getting stuck in a corner. The comparison suggests that our model can make better decisions on whether and how to interact with obstacles.

Study of obstacle interaction. To study how intent-aware policy learns the obstacle interaction and its outcome, we make a direct evaluation on obstacle interaction of several models in Table 5: (1) Interaction Reward (IR): the reduction of geodesic distance ∆ dis caused by interactions. (2) Interaction Success Rate (ISR): the ratio of taking interactions successfully (a more detailed measurement compared to the overall metric SR measuring task completion). A failed case of interaction is when the agent tries to pick up a heavy table or attempts to push a chair out of sight. We also report the success rate of Push and Pick interactions separately as PuSR (Push Success Rate) and PiSR (Pick Success Rate). The results (comparison between PPO and PPO+intent, CaMP and CaMP wo/intent) show the outcome of our causal method in pursuing the interaction efficacy of reducing the distance to the target (IR) and avoiding invalid actions (ISR). The results on PuSR and PiSR also suggest that successfully pushing an obstacle is more challenging since the obstacle may get stuck and is not movable in all directions.

6 Conclusions

In this paper, we tackle the task of Interactive Navigation (InterNav) which requires mastery of long-horizon navigation and object interaction in complex environments. We take a causal view of InterNav and clarify the challenge caused by the confounding bias of obstacles. A Causal Multi-policy Planning (CaMP) model is proposed following the main idea of reducing inefficient exploration with multi-policy and exploring counterfactual interaction by considering the primary intent of model's decision-making. We construct a large-scale dataset of interactive navigation with various obstacles spawned across multiple rooms. Experiments on our dataset illustrate the effectiveness of our method and support our causal modeling of InterNav.

Acknowledgment

This work is supported by the National Key Research and Development Project of New Generation Artificial Intelligence of China, Grant Number: 2018AAA0102504. This work is in part by Beijing Natural Science Foundation under Grant JQ22012, and in part by the National Natural Science Foundation of China under Grant 62125207.

References

Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3674-3683, 2018.

Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 31, 2017.

Elias Bareinboim, Andrew Forney, and Judea Pearl. Bandits with unobserved confounders: A causal approach. Advances in Neural Information Processing Systems , 28, 2015.

Dhruv Batra, Aaron Gokaslan, Aniruddha Kembhavi, Oleksandr Maksymets, Roozbeh Mottaghi, Manolis Savva, Alexander Toshev, and Erik Wijmans. Objectnav revisited: On evaluation of embodied agents navigating to objects. arXiv preprint arXiv:2006.13171 , 2020.

Devendra Singh Chaplot, Dhiraj Gandhi, Saurabh Gupta, Abhinav Gupta, and Ruslan Salakhutdinov. Learning to explore using active neural slam. arXiv preprint arXiv:2004.05155 , 2020.

Devendra Singh Chaplot, Dhiraj Prakashchand Gandhi, Abhinav Gupta, and Russ R Salakhutdinov. Object goal navigation using goal-oriented semantic exploration. Advances in Neural Information Processing Systems , 33:4247-4258, 2020.

Changan Chen, Unnat Jain, Carl Schissler, Sebastia Vicenc Amengual Gari, Ziad Al-Halah, Vamsi Krishna Ithapu, Philip Robinson, and Kristen Grauman. Audio-visual embodied navigation. environment , 97:103, 2019.

Changan Chen, Unnat Jain, Carl Schissler, Sebastia Vicenc Amengual Gari, Ziad Al-Halah, Vamsi Krishna Ithapu, Philip Robinson, and Kristen Grauman. Soundspaces: Audio-visual navigation in 3d environments. In European Conference on Computer Vision , pages 17-36. Springer, 2020.

Long Chen, Xin Yan, Jun Xiao, Hanwang Zhang, Shiliang Pu, and Yueting Zhuang. Counterfactual samples synthesizing for robust visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10800-10809, 2020.

Long Chen, Hanwang Zhang, Jun Xiao, Xiangnan He, Shiliang Pu, and Shih-Fu Chang. Counterfactual critic multi-agent training for scene graph generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4613-4623, 2019.

Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Embodied question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 1-10, 2018.

Abhishek Das, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Neural modular control for embodied question answering. In Conference on Robot Learning , pages 53-62. PMLR, 2018.

Matt Deitke, Winson Han, Alvaro Herrasti, Aniruddha Kembhavi, Eric Kolve, Roozbeh Mottaghi, Jordi Salvador, Dustin Schwenk, Eli VanderBilt, Matthew Wallingford, et al. Robothor: An open simulation-to-real embodied ai platform. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 3164-3174, 2020.

Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Jordi Salvador, Kiana Ehsani, Winson Han, Eric Kolve, Ali Farhadi, Aniruddha Kembhavi, et al. Procthor: Large-scale embodied ai using procedural generation. arXiv preprint arXiv:2206.06994 , 2022.

Thomas G Dietterich. Hierarchical reinforcement learning with the maxq value function decomposition. Journal of artificial intelligence research , 13:227-303, 2000.

Jiafei Duan, Samson Yu, Hui Li Tan, Hongyuan Zhu, and Cheston Tan. A survey of embodied ai: From simulators to research tasks. IEEE Transactions on Emerging Topics in Computational Intelligence , 2022.

Andrew Forney and Elias Bareinboim. Counterfactual randomization: rescuing experimental studies from obscured confounding. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pages 2454-2461, 2019.

Andrew Forney, Judea Pearl, and Elias Bareinboim. Counterfactual data-fusion for online reinforcement learners. In International Conference on Machine Learning , pages 1156-1164. PMLR, 2017.

Daniel Gordon, Aniruddha Kembhavi, Mohammad Rastegari, Joseph Redmon, Dieter Fox, and Ali Farhadi. Iqa: Visual question answering in interactive environments. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4089-4098, 2018.

Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Matt Deitke, Kiana Ehsani, Daniel Gordon, Yuke Zhu, et al. Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint arXiv:1712.05474 , 2017.

Xiangyang Li, Zihan Wang, Jiahao Yang, Yaowei Wang, and Shuqiang Jiang. Kerm: Knowledge enhanced reasoning for vision-and-language navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2583-2592, 2023.

Ruyang Liu, Hao Liu, Ge Li, Haodi Hou, TingHao Yu, and Tao Yang. Contextual debiasing for visual recognition with causal mechanisms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12755-12765, 2022.

Yulei Niu, Kaihua Tang, Hanwang Zhang, Zhiwu Lu, Xian-Sheng Hua, and Ji-Rong Wen. Counterfactual vqa: A cause-effect look at language bias. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12700-12710, 2021.

Judea Pearl. Causal inference in statistics: An overview. Statistics surveys , 3:96-146, 2009.

Judea Pearl et al. Models, reasoning and inference. Cambridge, UK: CambridgeUniversityPress , 19(2), 2000.

Donald B Rubin. Essential concepts of causal inference: a remarkable history and an intriguing future. Biostatistics & Epidemiology , 3(1):140-155, 2019.

Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. Habitat: A platform for embodied ai research. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9339-9347, 2019.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10740-10749, 2020.

Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence , 112(12):181-211, 1999.

Kaihua Tang, Jianqiang Huang, and Hanwang Zhang. Long-tailed classification by keeping the good and removing the bad momentum causal effect. Advances in Neural Information Processing Systems , 33:1513-1524, 2020.

Kaihua Tang, Yulei Niu, Jianqiang Huang, Jiaxin Shi, and Hanwang Zhang. Unbiased scene graph generation from biased training. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 3716-3725, 2020.

Zihan Wang, Xiangyang Li, Jiahao Yang, Yeqi Liu, and Shuqiang Jiang. Gridmm: Grid memory map for vision-and-language navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 15625-15636, 2023.

Luca Weihs, Jordi Salvador, Klemen Kotar, Unnat Jain, Kuo-Hao Zeng, Roozbeh Mottaghi, and Aniruddha Kembhavi. Allenact: A framework for embodied ai research. arXiv preprint arXiv:2008.12760 , 2020.

Erik Wijmans, Abhishek Kadian, Ari Morcos, Stefan Lee, Irfan Essa, Devi Parikh, Manolis Savva, and Dhruv Batra. Dd-ppo: Learning near-perfect pointgoal navigators from 2.5 billion frames. arXiv preprint arXiv:1911.00357 , 2019.

Fei Xia, William B Shen, Chengshu Li, Priya Kasimbeg, Micael Edmond Tchapmi, Alexander Toshev, Roberto Martín-Martín, and Silvio Savarese. Interactive gibson benchmark: A benchmark for interactive navigation in cluttered environments. IEEE Robotics and Automation Letters , 5(2):713-720, 2020.

Xu Yang, Hanwang Zhang, Guojun Qi, and Jianfei Cai. Causal attention for vision-language tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9847-9857, 2021.

Zhongqi Yue, Tan Wang, Qianru Sun, Xian-Sheng Hua, and Hanwang Zhang. Counterfactual zero-shot and open-set visual recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15404-15414, 2021.

Kuo-Hao Zeng, Luca Weihs, Ali Farhadi, and Roozbeh Mottaghi. Pushing it out of the way: Interactive visual navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9868-9877, 2021.

Dong Zhang, Hanwang Zhang, Jinhui Tang, Xian-Sheng Hua, and Qianru Sun. Causal intervention for weakly-supervised semantic segmentation. Advances in Neural Information Processing Systems , 33:655-666, 2020.

Junzhe Zhang and Elias Bareinboim. Markov decision processes with unobserved confounders: A causal approach. Technical report, Technical report, Technical Report R-23, Purdue AI Lab, 2016.

Sixian Zhang, Weijie Li, Xinhang Song, Yubing Bai, and Shuqiang Jiang. Generative metaadversarial network for unseen object navigation. In Computer Vision - ECCV 2022 - 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIX , volume 13699 of Lecture Notes in Computer Science , pages 301-320.

Sixian Zhang, Xinhang Song, Yubing Bai, Weijie Li, Yakui Chu, and Shuqiang Jiang. Hierarchical object-to-zone graph for object navigation. In Proceedings of the IEEE/CVF international conference on computer vision , pages 15130-15140, 2021.

Sixian Zhang, Xinhang Song, Weijie Li, Yubing Bai, Xinyao Yu, and Shuqiang Jiang. Layoutbased causal inference for object navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 10792-10802, June 2023.

A Appendix

A.1 Proof of equation (3)

For a state s t and intent i t at step t , the optimal counterfactual policy π ∗ ctf = argmaxV π ctf ( s t , i t ) , and the optimal standard policy π ∗ = argmaxV π ( s t ) , we have V π ∗ ctf ( s t , i t ) ≥ V π ∗ ( s t , i t ) . For a standard optimal policy, since π ∗ executes its intent i t as an optimal action a ′ t without re-consideration, we have:

Thus considering the information of its own intent, the agent can learn a counterfactual policy gaining more value.

A.2 Impact of different intents on policy exploration

We train our model with PPO [28] which encourages the exploration with an entropy loss L S t = c 2 S [ π ] = -c 2 ∑ a t ∈ A P ( a t ) logP ( a t ) where c 2 < 0 is the coefficient. Since our agent does trials conditioned on its intent, it maximizes the conditional entropy H ( A | I ) = -∑ a t ∈ A P ( a t | i t ) logP ( a t | i t ) . We are curious about how this conditional exploration can make the new policy distribution different from the conditional distribution (intent). To study the question and also how the similarity between intent and action affects policy exploration, we evaluate the mean KL divergence between the current and old policy distributions under several settings throughout the training. In particular, we vary the frequency of synchronizing the action policy and the intent policy as 1 epoch, 3 epochs, 6 epochs, 9 epochs, 12 epochs, 15 epochs, and without intent. In Figure 6, we report the mean KL divergence between the distributions of current policy and different old policy (from 1 epoch behind to 15 epochs behind). The trend of each curve shows that the counterfactual information of intent encourages the agent to explore new strategies. The comparison between curves shows that the more different actions and intents, the more the model tends to explore new policies rather than exploit the old policy. Thus the frequency of synchronization can be utilized to adjust how much the agent explores new policies under counterfactual situations.

/epoch

/3epochs

/6epochs

/9epochs

→/12epochs

→/15epochs

wo/intent

0.0084

Divergence

0.0079

0.0074

0.0069

0.0064

0.0059

0.0054

1

2

3

4

5

6

7

9

10

11

12

13

14

15

Gap number of epochs between the current policy and old policy

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the multi-room interactive navigation problem and proposes a novel model that is motivated by counterfactual reasoning. In particular, the paper posits that obstacle objects serve as a confounding factor when understanding the relationship between actions taken and the outcomes observed / reward received. To address this, a counterfactual reasoning based model is proposed, which explicitly encourages exploring actions outside the distribution predicted by the policy (i.e., what would happen if I did action X instead of Y?). The proposed model is hierarchical in nature, with low-level policies for navigation, picking, and pushing skills, and higher-level policies for selecting a skill to execute. The higher-level policy is conditioned on an intent, specifying where the policy is likely navigating to next. This is expected to allow the policy to account for the intent and explore actions that go against the intent.  Results on the ProcThor simulated dataset demonstrate the superiority of the proposed policy over alternative baselines.\n\nSTRENGTHS:\n* The idea of performing counterfactual reasoning is interesting and novel in the context of embodied AI.\n* The problem setting proposed is a good extension to prior work on single-room interactive navigation and is valuable for the community to work on.\n* The paper clarity is good, but it makes assumption about how knowledgable the reader is with causal inference (see weaknesses). \n* The experiments are well designed and ablation studies convey useful information to understand the overall model. The proposed model also performs much better than reasonable baselines.\n\nWEAKNESSES:\n# Post-rebuttal comments\n* The authors have sufficiently addressed my concerns and provided new experiments to quantify the improvement in interaction-ability. I'm happy to raise my rating to accept.\n\n--------------------------------------------------------------------------------------------------\n# Paper writing clarity\n* The paper writing clarity can be improved quite a bit with regards to causal inference. Since this is a relatively new topic to the embodied AI space, most readers may be unfamiliar with the topic and jargon like \"counterfactual reasoning\", \"confounding factors\", \"structural causal model\", \"structural functions\", etc. also, L129 was not obvious from my first reading (i.e., the difference between do(A) and A). \n* More clarity can be provided about what the counterfactual situations here are (e.g., more examples like L131-133). \n* In Figure 3, is the entire model differentiable? For example, are gradients from the loss propagated through to intent predictions w_t^{'} and a_t^{'}?\n* In the experiments, an explicit connection should be made to how the proposed model is using counterfactual reasoning and how it addresses the issue of confounding factors. \n\n# State vs. obstacles as confounding factors \nThe idea of treating obstacles as confounding variables makes sense. But at a high level, isn't the state variable itself a confounding factor? What is the value is isolating only obstacles here?\n\n# Experiment section can be improved\n* Error bars are missing in Tables 3 and 4. It will be useful to have results from training and evaluating on multiple seeds, especially for the top-3 methods (NIE, PPO+intent, CaMP). Similar issue for Table 4.\n* In Table 4, why is \"sync. /3*rollout\" worse than \"integrated intent\"? Aren't they both the same models?\n* In Table 4, why is \"sync. /epoch\" worse than \"wo/intent\"? If CaMP is synchronized frequently, I would expect the intent to be selected as the action itself, leaving little to no room for counterfactual exploration. So \"sync. /epoch\" should match \"wo/intent\", right?\n* There is no analysis on how well each models interact with the \"relevant\" obstacles, i.e., obstacles that lie along the shortest path and can be moved. This can be measured via precision and recall metrics. \n    * Precision = fraction of interacted objects that are \"relevant\" obstacles\n    * Recall = fraction of \"relevant\" obstacles interacted with\n\nQUESTIONS:\nKindly address the weaknesses stated above.\n\nLIMITATIONS:\nNo limitations have been discussed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nBroadly, the paper tackles the Interactive Navigation task: navigating to a goal and interacting with obstacles as necessary, e.g. pushing a chair out of the way. \n\nThey use the ProcTHOR simulator with 12k multi-room scenes and generate navigation episodes that are suitably cluttered with obstacles. Their embodiment is abstract, with a discrete high-level action space (e.g. PushLeft, PickUp, Drop, RotateLeft).\n\nTheir approach is a hierarchical model, with three pre-trained low-level \"action policies\" (skills): navigate, push, and pick. A master policy sequences the three. Specifically, the navigation policy is run by default; the master policy can interrupt it by invoking an interaction policy (push or pick); the interaction policy then runs exclusively until it self-terminates, returning control to the master policy.\n\nTheir main contribution is the addition of an intent policy; they claim this helps the master policy make better decisions in the presence of obstacles. I'll describe this more below.\n\nThey evaluate their approach against several baselines:\n* random actions\n* a monolithic (non-hierarchical) sensors-to-actions policy trained with end-to-end RL, labeled as PPO\n* a prior approach labeled as NIE that aims to predict state change of observed objects\n* a hierarchical policy that uses the same nav/push/pick action subspaces as CaMP but differs in how they are combined\n* a monolithic sensors-to-actions policy with the addition of CaMP's intent policy\n\nThey also study ablations related to the intent policy.\n\nI'll now describe the intent policy in detail, with the caveat that I'm looking for further clarification from the authors here (see Questions section).\n\nThe architecture of the intent and master policy are mostly identical; their inputs include goal embedding and extracted visual features and they feature recurrent units.\n\nAt timestep t, the intent policy outputs w'.t, a distribution over the three action policies, essentially choosing between push/pick/navigate. Meanwhile, each action policy j ouputs a'.t^j, a distribution over discrete actions. Finally, the intent is the sum of all a'.t weighted by w'.t. For example, if the intent policy on a given step is biased towards push and the push action policy is biased towards the PushLeft action, the overall intent will be biased towards PushLeft.\n\nThis intent is fed to the master policy alongside observations (together, \"intent-specific state\"), which chooses an action policy (either allowing the nav policy to continue, or interrupting by invoking push or pick).\n\nThe intent policy and master policy generally share parameters and generally behave similarly, except the intent policy is synced from the master policy only periodically during training, such that in practice the intent will differ from the taken action some of the time.\n\nThey describe this intent-informed policy as \"exploring counterfactuals\", and a component here is learning a value function for the above-mentioned intent-specific state.\n\nSTRENGTHS:\nThe InterNav task is a challenging, relevant task for Embodied AI.\n\nThe task, scene/episode dataset, and model architecture are clearly communicated in figures and text, with minor exceptions noted below.\n\nThe scenes and episodes appear diverse and high-quality.\n\nBaselines and ablation study are rigorous.\n\nThe authors show strong performance against their baselines.\n\nWEAKNESSES:\nThere's ambiguity on some paper details. I'll list bullets here; see my questions for more details:\n\n= action space and positioning of objects\n\n= object dynamics in ProcTHOR\n\n= details of invoking and terminating interaction subtasks\n\n= learning for the intent policy parameters\n\nThe paper doesn't discuss sim-to-real transfer or otherwise discuss how CaMP might be applied in a real-world setting. \n\nThe biggest weakness is that the intuition and conceptual value of the intent component aren't clearly articulated. For details, see my questions below.\n\nQUESTIONS:\nIn Figure 5 and in your video at 0:33, we see the agent making only 90-degree turns. However, the PPO baseline visualized in Figure 5 shows other turn angles. Can you explain this? Is your CaMP agent trained using a more restricted action space (versus action space in baselines)? If so, does this make your comparison results to baselines less clear? I would speculate that restriction to 90-degree orientations would simplify learning particularly in your scenes, which appear to be mostly axis-aligned (Figure 4; walls and most furniture). And a related question: 5.1 mentions \"adjacent nodes\". Is there a discrete structure to the position of objects, e.g. a grid?\n\nCan you explain a bit about the object dynamics of ProcTHOR, specifically, how does the simulator decide which objects are nonmovable, pushable, or pickable? And can you speculate on how the trained CaMP policy is able to predict this? I.e. is it overfitting to a specific object set, generalizing based on apparent object size in the image sensors, or something else?\n\n>Once an interaction policy is called, it does not return the control until the sub-task termination (output Done).\n\nCan you elaborate on how this works? For example, does the master policy include a recurrent unit, and does this unit receive continuous observations even while an interaction sub-task is running? Related: it would help to modify Figure 3. As-is, Figure 3 suggests that the master policy is free to select any action policy on any step.\n\n>To ensure that the intent represents the decision of the agent, we synchronize the parameters of two networks (π'Ω ← πΩ) during training.\n\nIn between these syncs, is the intent policy being updated during the learning phase of PPO, or is it frozen?\n\nCan you discuss sim-to-real? How might CaMP be used in a real-world setting?\n\nRelated to sim2real, can you also address one specific concern? First, a few assumptions:\n\n= you wish to keep the same hierarchical architecture\n\n= you could build low-level push/pick/nav policies that work on a real robot\n\n= you require most training of the master policy to happen in simulation due to the required scale of experience-collection\n\nMy specific sim2real concern: from 0:29 in the video, it appears that the push subtask doesn't move the agent or change its view (it's a \"magic push\"). The agent isn't required to approach the object in a certain way or move itself to effectively push the object. So, I speculate that this makes it remarkably easy for the master policy to chain navigation and pushing: the navigation policy essentially resumes after push completion from exactly its last pose, almost as if the obstacle were teleported away. Chaining real-world nav and push subtasks in this way would likely fail due to the push task moving the agent.\n\nI'd like greater clarity in section 3.2, on counterfactual decision-making and the basis for your intent policy and overall causal policy. A few specific questions: \n\n>Trained based on its own intent, the agent can obtain both experimental experiences (when a.t = i.t) and counterfactual experiences \n(when a/t != i.t), boosting the exploration of new strategies\n\nCan you elaborate? How does the causal policy architecture \"boost\" this exploration? How does this produce better exploration than, say, simply tuning temperature on a stochastic policy?\n\n>The intent also provides context about the obstacles due to their causal relation\n\nCan you elaborate? Can you experimentally verify/quantify what context the intent is providing about obstacles?\n\n>Intuitively, an agent that explores different counterfactual situations (e.g. \"what if I push/pick up the box instead of bypassing it?\") has a better understanding of the expected value of the current state, compared to the agent taking actions out of intuition.\n\nIt seems to me that a stochastic policy will try pushing, picking, and bypassing the box many times over the course of large-scale training and thus get this understanding of the expected value. What is the essential difference with your causal policy?\n\nLIMITATIONS:\nNo concerns here", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a causally-inspired hierarchical policy framework for the interactive navigation task in the AI2THOR environment. The framework consists of a master policy, intent policy, and three sub-control policies. The intent policy embeds intuitive intents from the sub-control policies into the master policy, enabling it to make counterfactual decisions. The proposed approach, named CaMP, is evaluated on a newly collected dataset in the ProcTHOR multi-room scenes. The experimental results show that CaMP outperforms the baselines, achieving the best performance on the interactive task.\n\nSTRENGTHS:\n+) The interactive navigation (IN) task is an interesting and challenging task in the field of embodied AI. This paper contributes to the progress in this area by introducing a causally-inspired hierarchical reinforcement learning (HRL) policy. The use of a newer and larger dataset with complex scene layouts and diverse objects enhances the realism of the task. The results demonstrate the effectiveness of the proposed HRL policy, highlighting the advancements in tackling the challenges of interactive navigation.\n\n+) The decomposition of the embodied policy into a master policy and sub-control policies is a reasonable and effective approach for the IN task. This work demonstrates the potential of HRL in tasks where interactions with the environment are crucial for achieving goals.\n\n+) The explanation of confounding bias and the causal diagram provided in Fig. 2(a) help clarify the concept and its relevance to the IN task. However, the discussion of heavy obstacles as an example may need further refinement.\n\n+) The experimental results presented in the paper provide strong evidence for the effectiveness of CaMP. It outperforms several baselines, including PPO, NIE, and HRL, on the IN task using the newly collected large-scale dataset in ProcTHOR multi-room scenes.\n\nWEAKNESSES:\n-) The connection between confounding bias resulting from unmeasurable obstacles and the counterfactual policy design is not adequately explained. It remains unclear how the counterfactual policy, through the intent policy, effectively addresses the bias caused by obstacles. The paper lacks a convincing explanation for the direct line from $O$ to $A$ in Fig. 2(b). It is not clear how the weighted-sum of action logits from the sub-control policies can fully capture the policies' intents and uncover the causality depicted in Fig. 2(a) ($O$ -> $A$ and $O$ -> $R$).\n\n-) The model design lacks intuition. Instead of introducing an additional Intent Policy to generate intents, a more straightforward approach would be to recursively use the Master Policy to obtain intents $P^j(i_t)$ and provide feedback to the Master Policy. This recursive feedback mechanism could be extended to multiple levels, denoted by $j$ ∈ $J$, with $P^0(i_t)$ representing a void intent. Additionally, rather than using the action logits from the sub-control policies to represent intents, exploring the utilization of hidden features returned from the GRU in each sub-policy would offer a more intuitive approach.\n\n-) Several implementation details are missing, such as determining which object to interact with when using Push/Pick actions, the force applied during the Push action, and whether the force is correlated to the object's mass. Further clarification on these aspects would greatly enhance the understanding of the proposed approach. More questions regarding missing details can be found in the Question.\n\nQUESTIONS:\nPlease address following points for a more comprehensive understanding of the proposed approach.\n\nClarifications:\n\no) Please provide further clarification on the connection between the confounding bias resulting from unmeasurable obstacles and the CaMP model design, specifically the relationship between Sec. 3 and Sec. 4.\n\no) Clarify why using the weighted-sum of action logits is an effective way to represent intent and its relationship to the confounding bias.\n\nMissing details:\n\no) How to determine which object to interact with when using the Push/Pick actions?\n\no) What is the amount of force applied during the Push action, and is it correlated with the object's mass?\n\no) Provide the exact dimension of the intent embedding $P(i_t)$. Is it equivalent to the size of the total action space?\n\no) Provide architectural details of the policy, such as whether the CNN is a simple CNN or a CLIP pretrained ResNet, and the number of layers in the GRU.\n\no) Clarify the meaning of \"epochs with a rollout of data\" at L191. Does it refer to the number of update iterations using a rollout of data in the PPO implementation in AllenAct?\n\no) Missing implementation details of the HRL baseline. Does the HRL apply all outputs from the sub-control policies in a single execution, or does it follow a traditional HRL policy where the master policy calls one of the sub-control policies at each step?\n\nConfusions:\n\no) Clarify whether $r_{tp}$ at L227 is the same as $r_{sp}$ at L229.\n\no) At L235, it is mentioned that $r_{nav}$ is used to learn the push and pick sub-control policies. This appears to contradict the context provided at L235. Please clarify the requirements for task success, specifically if the agent needs to reach the target position.\n\nLIMITATIONS:\n-) While the proposed CaMP is interesting and evaluated in a complex environment with non-trivial tasks, it would be beneficial to see more results in a simpler environment. For example, evaluating the model on a 2D environment like PettingZoo, where a dot agent performs the interactive navigation task by interacting with obstacles, could provide insights into how the model disentangles intertwined factors by the causally-inspired design in a more controlled setting.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the problem of interactive visual navigation; i.e., an agent navigating in an enviornment where it is allowed to affect the configuration of the environment (e.g., by moving objects around or picking them up), to improve navigation performance. The key idea is to learn a hierarchical policy that factors in agent intent to propose an action that either results in navigation towards the goal, or interaction with an obstacle. A dataset based on the PROC-THOR simulation environment is also introduced, to facilitate evaluation.\n\nSTRENGTHS:\n[S1] The proposed method is sound; and for the most part, well-defined. The problem formulation is easy to follow, and the description of the method is clear.\n\n[S2] Interactive navigation is a challenging and relevant robotics/embodied-AI problem to the Neurips community. While a large fraction of existing approaches focues on non-interactive visual navigation, this paper explicitly considers affecting state changes by manipulating obstacles, resulting in a novel problem setting where there isn't much prior work.\n\n[S3] I find the positioning of this paper w.r.t. existing literature fair. The baseline methods considered for evaluation are representative of the various flavors of non-interactive visual navigation approaches that have, over the years, been proposed for PointGoal navigation.\n\n[S4] The paper discusses enough implementation details that a reasonable practitioner may be able to replicate the key aspects of the model architecture and the state, action, reward structures.\n\nWEAKNESSES:\nMeta-comment: I have one major concern with the experiment design and evaluation setup; which unfortunately results in the key claims of the paper not being substantiated. I have tried to elaborate the issue, and also the rationale behind it. I also suggest a few mitigation strategies (note: these aren't the only possible ones; other strategies welcome too). While I am unsure whether these may be addressed in the short author response window, if these are adequately addressed, I would have no reservations in bumping my score up.\n\n\n[W1] **[Major] Evaluation metrics and experiment design**:\n\nInteractive navigation is very tricky to evaluate. In (non-interactive) navigation scenarios, the path length metrics used in literature are often highly correlated to execution time. Assuming that each atomic action executed by the agent takes nearly the same time to execute, a longer path length would mean that the agent takes longer to reach the goal. However, in the interactive navigation scenario, typical path length metrics like SPL are no longer good indicators of the amount of time it would take to complete the task. This is because, the time spent in picking up or moving objects will also count towards the overall \"time-to-reach-goal-state\"; and picking up or moving objects is heavily dependent on object states, which voids the assumption that each action takes roughly the same amout of time.\n\nI understand the rationale behind using typical metrics like SR (success rate), SPL (success-weighted inverse path length), and FDT (final distance to target) -- these metrics allow for easy benchmarking with existing approaches (esp. non-interactive navigation techniques); these are also easily accessible via modern simulators that support PointGoal navigation.\n\nThat said, reporting only these above metrics only portrays the benefits of an interactive navigation strategy, while masking away the disadvantages. Execution time is often lost when interacting with objects, which goes unaccounted for. This becomes apparent when you consider the following counterexample: assume that an agent spends more than half its time moving (or picking up) every object encountered in its way; but ends up taking the shortest possible (i.e., optimal) path to goal. It will then end up with a success rate and SPL of 1 and an FDT value of 0. However, in reality, an agent that takes a twice as long path to the goal (but does not interact with any object) will end up reaching the goal at about the same time, and will have the same SPL, SR, and FDT values (recall that, in this hypothetical scenario, the interactive agent spends half its time interacting with obstacles in its path). This calls for a different approach to evaluate interactive navigation approaches; ideally the SPL metric should also account for time lost due to interaction. Measuring this time could be challenging (or impossible) to do in the first place, so here are a few potential mitigation strategies to consider.\n\n* While episode lengths are fixed, to ensure the agent does not take an indefinite amount of time, to compare fairly against non-interactive agents, each episode length must be capped to the number of timesteps in which a non-interactive agent completes the task. This will ensure that a fair comparison is possible, when a reasonably accurate time estimate is available for each interaction.\n* An alternative would be to consider evaluating solely in scenarios where it is impossible to reach the goal without interacting with objects. (more nuance on this follows in [W2] below).\n\n[W2] **Eval on data subsets**: I would have liked to seen more granularity in terms of the quantitative results presented in Table 3. It would, for instance, be useful to split the dataset into a variety of categories, depending on whether or not interaction is required to solve an episode, and the level of difficulty of the episode. This becomes more important, because the dataset is claimed as a contribution. Table 2 goes a bit along this direction, indicating the number of obstacles per room; however, it is not clear how many of these episodes absolutely require interactive navigation (i.e., cannot be solved by a non-interactive optimal agent). (The dataset construction seems to ensure a 50% likelihood that a path is unavailable -- I would argue in favor of forming two splits of the dataset, one where no non-interactive agent trajectory exists; and one where an optimal non-interactive agent will need to take a longer path).\n\n\n[W3] **[Minor]** The paper, in its current form, falls short in terms of technical rigor when discussing aspects of causal inference. E.g., lines 134-135 \"However, without considering the causalities from obstacles, it’s hard to generalize to unseen environments or large-scale datasets.\"; line 38 \"... , learning causality through RL training is challenging due to the existence of unobserved...\"; line 137 \"The above confounding bias can be tackled...\". Neither the \"causalities stemming from obstacles\" nor the \"confounding bias\" have been clearly been defined in the paper. It is also important to clearly define the type of causal relationships that are being learned by design (perhaps section 4.2 is a good place to do so).\nRevising these aspects of the paper will make the descriptions of the approach more accurate.\n\nQUESTIONS:\nOverall, I like where this paper is going. The hierarchical policy presented herein is sound. My concerns are not with the method; rather with the choice of evaluation metrics. In my review (see weaknesses above), I have attempted to also provide a few mitigation strategies (these are by no means extensive -- merely initial thoughts on how newer evaluation metrics may be designed for the task at hand).\n\nI would like to see this discussed further in the author response phase. If addressed adequately, I have no hesitation in bumping my score up.\n\nLIMITATIONS:\nAdequately addressed", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}