CaMP: Causal Multi-policy Planning for Interactive Navigation in  Multi-room Scenes
Authors: Xiaohan Wang
Yuehu Liu
Xinhang Song
Beibei Wang
Shuqiang Jiang
Venue: NeurIPS 2023 poster (2023)
Keywords: Embodied AI
Interactive Navigation
Causal Reinforcement Learning
Hierarchical Reinforcement Learning

Abstract: Visual navigation has been widely studied under the assumption that there may be several clear routes to reach the goal. However, in more practical scenarios such as a house with several messy rooms, there may not. Interactive Navigation (InterNav) considers agents navigating to their goals more effectively with object interactions, posing new challenges of learning interaction dynamics and extra action space. Previous works learn single vision-to-action policy with the guidance of designed representations. However, the causality between actions and outcomes is prone to be confounded when the attributes of obstacles are diverse and hard to measure. Learning policy for long-term action planning in complex scenes also leads to extensive inefficient exploration. In this paper, we introduce a causal diagram of InterNav clarifying the confounding bias caused by obstacles. To address the problem, we propose a multi-policy model that enables the exploration of counterfactual interactions as well as reduces unnecessary exploration. We develop a large-scale dataset containing 600k task episodes in 12k multi-room scenes based on the ProcTHOR simulator and showcase the effectiveness of our method with the evaluations on our dataset.

CaMP: Causal Multi-policy Planning for Interactive Navigation in Multi-room Scenes
Xiaohan Wang
Yuehu Liu
Institute of Artificial Intelligence and Robotics
Institute of Artificial Intelligence and Robotics an Jiaotong University
Xi'an Jiaotong University
wuhanwxh201 10suuu... xj tu. edu.
liuyh@mail xjtu. edu.
Xinhang Song
Beibei Wang
Institute of Computing Technology
Institute Artificial Intelligence and Robotics
Chinese Academy of Sciences
Xi' an Jiaotong University wangbbustu. tigesttuxjjuu xjju. edu, cn
University of Chinese Academy of Science
xinhang. songévápl.ict.acc
Shuqiang Jiang
Institute of Computing Technology
Chinese Academy of Sciences
University of Chinese Academy of Science aqjiangDict.ec cn
Abstract
Visual navigation has been widely studied under the assumption that there may several clear routes to reach the goal. However, in more practical scenarios such as a house with several messy rooms, there may not. Interactive Navigation (InterNav) considers agents navigating to their goals more effectively with object interactions, posing new challenges of learning interaction dynamics and extra action space. Previous works learn single vision-to-action policy with the guidance of designed representations. However, the causality between actions and outcomes is prone to be confounded when the attributes of obstacles are diverse and hard to measure. Learning policy for long-term action planning in complex scenes also leads to extensive inefficient exploration. In this paper, we introduce a causal diagram of InterNav clarifying the confounding bias caused by obstacles. To address the problem, we propose multi-police model that enables the exploration counterfactual interactions well as reduces unnecessary exploration. We develop a large-scale dataset containing 600k task episodes in 12k multi-room scenes based on the ProcTHOR simulator and showcase the effectiveness of method with the evaluations on our dataset.
Introduction
Embodied AI, incorporating internet intelligence such as vision, language, and reasoning into embodiment, aims deploy autonomous agents in real-world environments [16]. In Visual Navigation, a cornerstone task, agents navigate to the goal in static environments where the movement of scene objects is avoided. In practical scenario indoor robot may need navigate in house with multiple rooms and get blocked by obstacles(see Figure I). Interactive Navigation (InterNav) 36] considers navigating more efficiently with object interaction rather than merely moving forward and turning around.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Blocked Positions (harder interact from P1 P3)
Top view
Goal
Microware
Goo!
Desk
Sola
Lettuce
small-sae sizee
large to pick up lightwlight
too large and heavy pick or push
lightweight
Jok
PUSH
by BYSS
A gent
Figure I: Illustration of InterNav in a 4-room scene. Trying to reach the goal as a blue circle, the agent is blocked by obstacles in yellow boxes spawned in three rooms. The agent needs to make decisions at three blocked positions marked in blue boxes: to bypass or to interact with obstacles? The outcomes depend on the interactive attributes of obstacles (whether they be pushed away picked up, e.g. heavy, small, etc.). The green arrow is a path the agent may take to reach the target and the nodes denote interaction with obstacles. Markings in colors are invisible to the agent.
Years of visual navigation exploration witnessed the growing interest in long-horizon navigation in complex environments [13]. Zeng et al.[39] build a dataset benchmarking interactive navigation task on the A12-THOR simulator, where agents receive visual observation and output the sequence of discrete actions. New challenges are raised compared to visual navigation: First, the agent needs to understand the outcome of object interaction. Second, the need for interaction with obstacles leads to larger action space for reinforcement learning (RL). However, the dataset is limited to a simple scenario where few objects are lined up between the starting and target point in single room. Navigation in multi-room scenes requires agents to plan longer action sequences and understand more diverse states of the environment.
Since the dynamics of object interaction follow the physical rules that are structurally invariant the environment, there naturally exists cause-effect relation between the action and its outcome. From the perspective of causality, an RL agent performs interventions to the environment (by taking actions) and estimates the effect (rewards) in order learn policy that maximizes the long-term reward. However, learning causality through RL training is challenging due to the existence of unobserved confounders (UCs), which affect both the decision-making of actions and the generation of rewards [41, 18). In InterNav, the agent may decide to navigate or interact based on whether it is blocked by object (referred The existence or type of obstacles is also decisive to the outcome (see Figure 1). For instance, when blocked by desk the agent may choose push it aside since empirically it leads to a high reward, while the interaction will turn out otherwise if the desk gets stuck. Without knowing the true causality in interaction, the agent is prone to learn sub-optimal policy. Prior work [39] on InterNav attempts to predict the change of object pose as a representation to understand the interaction outcomes. The model performance relies on effective obstacle detection and being familiar with obstacle distribution, which may not work in complex scenes with various obstacles. Long-term planning in complex environments also leads to inefficient exploration. Moving forward at blocked positions or trying to interact with no object around makes no difference to the environment and the agent can learn little about the causality from those experiences. Common solutions to visual navigation [35] and InterNav [39] train single policy with end-to-end RL, which requires extensive training to learn universal policy tackling various scenarios. However, an agent may come up with multiple intents but can only conduct one behavior at once due to its embodiment. In short-term view, the agent can either interact with objects or navigate somewhere. Hence is more efficient to learn separate policies for interaction and navigation.
obstacle) whose distribution is unobserved, namely unmeasurable.

---

In this paper, we introduce structural causal model, analyzing InterNav through causal lenses and clarifying the key challenge as the confounding bias caused by obstacles. We show that learning a policy conditioned on its primary decision (referred to as "intent") can enable the agent to learn from counterfactual experiences and make better decisions. As solution to InterNav, we propose Causal Multi-Policy model (CaMP), which reduces unnecessary exploration by breaking down the task into simple sub-tasks and enables counterfactual exploration by trialing conditioned on the agent's 's intent. In particular, three action policies (i.e. navigation, pushing interaction, and picking interaction) are developed by training three actor-critic networks in designed auxiliary tasks. We then train a global policy that receives both observation and action intent and calls action policies in sequence. An intent policy is developed to duplicate the decision of the global policy so that action policies can be called collect dataset
to produce their action intents. To study the InterNav in more practical scenarios, containing 600k task episodes with various obstacle arrangements, based on the ProcTHOR simulator [14] of large-scale multi-room scenes. We also introduce new metric measuring the time efficiency of task completion due to the need for interaction in InterNav. Evaluations on our dataset showcase the effectiveness of the proposed model CaMP.
2 Related Works
Interactive Navigation comes up as the downstream task of visual navigation, which has been
extensively studied in Embodied AT literature the past decade. Most works address PointNav [5, 27, 35] and ObjNav [4, 6, 44, 42, 43], while others develop Visual-language Navigation [1, 29, 33, 21], AudioNav [8. 7]. etc that bring in new intelligence components. Environments of sim-to-reat [13] or large-scale [14] indoor scenes are also proposed for more complexity. Zeng et al. [39] focus on vision-based agents interacting with directional pushes on iTHOR environment [20]. However, they consider elementary scenarios where obstacles are spawned in single room for agents to push out of the way. In this work we construct a new dataset based on ProcTHOR environment [14] that provides 12k multi-room scenes with diverse floorplans and various assets, supporting large-scale training.
Causal Reinforcement Learning. Causal inference [24, 25, 26] has been a longstanding method- ology adopted to pursue the causal effect. Recent years have witnessed the progress of applying causal theory in computer vision tasks including visual question answering [23, 37, 9], scene graph generation [32, 10]. visual recognition [22, 31, 37, 38], etc. Causal reinforcement learning (CRL) [18, 17, 40, 3, 41] is motivated by the challenges of RL including data inefficiency and lack of interpretability, and has been applied in traditional RL tasks such multi-armed bandit problem [3], Markov decision process [41]. Naturally, the causal bias also exists in Embodied AI tasks, but there exists little work that helps train RL agents to complete tasks applying the theory of causal reasoning. We make the first attempt to model the causalities among the interactive navigation and address the confounding bias of obstacles by implementing a counterfactual policy.
Hierarchical Models for Embodied AI. To address complex and long- mogghooizoon tasks, one of the most popular solutions is the hierarchical approach inspired by Hierarchical Reinforcement learning (HRL) [30, 15]. The main idea is to divide the whole task into solvable sub-tasks through temporal abstraction which allows taking actions at different time scales [2]. Previous attempts applied on embodied question answering [11. 12], and interactive question answering [19] mainly follow the flavor of finding sub-goal, for the level module to achieve. However, learning policy associated with sub-goals can be data expensive and requires much handcraft design for vision-based tasks. Our
model is different from those works following the idea of the option framework [2] that develops multiple policies for different task situations without the need for any sub- sub-goals.
3 A Causal View of InterNav
We start by grounding the problem of InterNav through the causal lens and clarify the challenges brought by obstacles. Then in the face of the confounding bias of obstacles, we let the agent learn counterfactual policy conditioned on its original intent so that it can gain experiences of counterfactual situations and make better decisions.
3

---

Restals 31
intent policy
Action
meals)
Reward
observation
parameter
State
5
sharing
action policy
action.
R
(e) Causal diagram of InterNavv
causal diagram
Implementation inteniiaaaer policy
Figure 2: Causal diagram of proposed SCM. The part in gold of (b) denotes the causal entity of the counterfactual policy. The action policy shares the same architecture with the intent policy so that they can receive the state observation the same form.
3.1 Structural Modelling
We first formulate the underlying problem as a time-discrete Markov Decision Process (MDP) defined by tuple (S, A, A, R, 7), where S is the state space, A is the action space, T is a reward function, and E [0 1] is the discount factor. At each step t, the agent observes state St (m, pt) represent- ing an egocentric visual observation and a goal coordinate, executes an action Or E A, and receives reward Tt. Here, the action space = (MoveAhead, RotateLeft, RighteRight, LookUp, LookDown, PushAhead, PushBack, PushLeft, PushRight, PickUp, Drop, Done).
S x A x S - 1] is transition function,
We then analyze the above MDP with the form of Structural Causal Model (SCM, see Figure 2). Formally, the SCM M is defined by a tuple (O, V, F(u)), where V = U AU R is a set of observable variables, O denotes the obstacles that hinder the navigation of agents, F = the set of structural functions relative to V, and P(o) is the probability distribution over obstacles which is unobserved in InterNav. In particular, objects that currently the environmental state such as the layout of surroundings (S (S~O)). At step t. action a; = fa (St, 0z) is decided by the agent based state observation and its recognition of obstacles (S A)). Practically, stochastic policy x (a (t) : S x A + [0, 1] is learned to cover the process. Note that although denotes the state encompassing all information of the environment, causally influences agent's is actions in a partially observable way through visual observations. Reward = [= ar, 0,) returned by the environment based on state, action and obstacles to which the action is applied (S.A,O~R). The current state fa (st at-1) is decided by the state and action at last time step. The goal of the agent is to learn policy x' argmarV, (st) that maximizes the expected cumulative reward:
fr)
obstacles are decided by
(st =
x r14s+1|s0
k=0
which is formulated in the form of state-value function Vj and is the discount factor.
Through interaction with the environment, the agent performs intervention do(A = a). that actively sets A as rather than passively observes when A appears be and learns from the effect which can written probabilistically as P(R = r rido(A= = a), S = s). Since obstacles influence both the decision of actions and the generation of rewards (A+0-+R. see dashed arrows in Figure 2(a)), the causal relation between A and R is confounded such that P(r(do(a)) + P(rla). to which the confounding bias refers [25]. During RL training, the model estimates spurious value of actions (in the form of reward) due to the confounding bias which may result in sub-optimal policy. For example, frequent encounters with heavy obstacles (e.g. chairs, tables) make the agent predict higher value of pushing them than picking them up. The agent can get familiar with limited distribution P(o) with extensive training. However, without considering the causalities concerning obstacles (A-O~*R) it's hard to generalize to unseen environments or large-scale datasets.
3.2 Counterfactual Decision-making
The underlying confounding bias can be tackled with the ability of counterfactual decision-making A counterfactual effect P(R AA a') of taking action can be read "The reward the agent

---

policy decision
Causal Multi- colti-policy Planning
Wt
Global Policy
1t
call
intent
Vi
Push
30B
Visual Encoding
State Encoding
Pick
Navigatte
a
state representation
Action Polliies
egocentric view
(x, y) goal position
Embedding
Wt
Policy
policy intent
void intent
Figure 3: Model overview. Our model CaMP is composed of global policy, three action policies, and am intent policy. Given an observation, the intent policy predicts a policy w/ and low-level policies predict action o/. which is then weighted by the policy distribution as the integrated intent & (equation (7)). Finally the global policy decides the policy We based on the state representation and the intent, and the chosen policy executes the action dς.
would obtain had it taken action A = a (contrary to the fact), given A a"", which is imaginary thus usually not estimable from data. However, Bareinboim et al. [3] show that the above counterfactual quantity can be estimated the action is interrupted before its decision executed (ETT, Effect of Treatment the Treated), We call the decision before execution agent's intent which shares the same causal parents of action A. Our goal is to learn counterfactual policy Taf ar argmaxx Vear (5() (see Figure 2(b)) that maximizes the long-term value conditioned on the sppent- state s/ (st, it):
(i oz)
V Rett (3))
E
Te+ St, de]
kmm 0
The action distribution of the counterfactual policy P(a.li:) (efe(Sr it) is posterior the intent distribution P(t:) (8t). Thus the agent can obtain both experimental experiences (ones the original policy "would have collected" when d, it) and counterfactual experiences (when a A i,) that explore external strategies. The intent also provides context about the obstacles due to their causal relation. Thus the counterfactual policy obtains more value than standard policy [41]:
Vasa (sc. it) V (s:)
(3)
where equality does not hold when unobserved confounders (obstacles) exist. Intuitively, an agent that explores different counterfactual situations (e.g. "what if push/pick the box instead of bypassing it?") has a better understanding of the expected value of the current state, compared the agent taking actions out of intuition. We illustrate the idea of how we implement counterfactual policy Figure 2(c).
4 The Proposed Solution
In this section, we first introduce hierarchical framework for multiple policies to cooperate. Then we implement global policy with the ability of counterfactual decision-making which plans three action policies based on the integrated intent of the model. The model architecture is shown in Figure
4.1 Hierarchical Policy Framework
Essentially, we reduce the task complexity following prior understanding that the InterNav task is temporally separable. 3 action policies are defined in Table to split the action space. Policy
5

---

Table 1: Descriptions of action policies
Policy
Action space
Objective
MoveAhead, RotateLeft, Rotate- Right, LookUp, LookDown, Done
Navigate
Navigating to the goal point within distance of 0.2m.
PushAhead, PushBack, PushLeft, PushRight, Done
Moving obstacles until there is reachable path towards the goal or the existing path gets shorter.
Push
PickUp, Drop, RotateLeft, Rotate- Right, Done
Picking and dropping small obstacles until there is reach- able path towards the goal or the existing path gets shorter.
Pick
Navigate focuses on the ability of point navigation without considering the environment dynamics. While Push and Pick policies aim for the ability of object manipulation according to the goal and agent position. Formally, assume action policy Te called at step mapping the transition from state observation to the distribution of action o::
P(a+)= = (8;, h1-1), at E Au
where An denotes the recurrent hidden state and the action space of policy Tu. In our model, 8t is the concatenation of goal embedding and visual features extracted with a convolutional neural network. Tu is implemented with a GRU and two linear layers for both policy (actor) and value (critic). We pretrain each action policy with an auxiliary task designed according to their objectives.
With the short-term abilities of action policies. a global policy To aims to plan a sequence of policies Wi Ω = (Navigate, Push, Pick, Done) for the whole task completion. We let the agent follow the Navigate policy by default and the global policy decides whether to interact during the long horizon navigation. Once an interaction policy is called, it does not return the control until the sub-task termination (output Done). xq shares the same model architecture with Tw..
4.2 Intent-aware policy planning
Since the global policy plans when to interact with obstacles, it's crucial to learn action-outcome causality (c.g. whether calling push policy results in a clear path). Moreover, unlike low-level policies, the global policy does not receive direct rewards for its decisions during training, which makes their causal relation harder to learn. Thus we introduce a counterfactual global policy which makes plans based the theory in section 3.2, in order enable more effective policy learning. The intent of the agent is combined with its observation as the intent-specific state that is fed to the global policy. Instead of planning policies based on its own intent of policy, my is provided with the integrated intent of basic actions in E A:
P(wx) Ta((S; P(ir), 14-1) P(w[)= = TT( P(t),, h4-1) P(ie) Σ Two (St,/1 11- P(w/ = =²)
(6)
(7)
WAEN
where T/2 is the intent policy that duplicates the decision- of global policy and & is a plane vector denoting an intent without any inclination. Each action intent (zero-padded to the same size) produced by policy Twi is weighted by the probability that the decision of To is w² Note that P(ix) encoded with linear function and concatenated to the observation U/ as the input of GRU. By providing information of the intents of low-level policies, the new context bridges the relation between actions and rewards for the global policy.
We train the model with Proximal Policy Optimization (PPO) [28] algorithm which updates the model parameters for l iterations with rollout of data. To ensure that the intent represents the decision of the agent, we synchronize the parameters of two networks (T'2 + TT) during training. Since the global policy does trials conditioned on the distribution of model intent, similar distributions of actions and intents are more likely to be sampled identically (experimental case, at it). On the contrary, the distinctive distributions of actions and intents lead to more counterfactual cases (a: F it), which brings entropy increase for the system. In other words, the agent tends to exploit its internal knowledge about influencing UCs (obstacles) receiving its own intent, while exploring the external space of policy when setting off from the old intent. Since the global policy is becoming distinct from the intent policy after iterations, we set the synchronization frequency as K rollouts to balance the exploration and exploitation of policy learning.

---

rooms scene
2 rooms scene
4 rooms scene (across 2 rooms)
9 rooms scene (across rooms)
Figure 4: Top view examples of our dataset. Blue circles denote the starting and target point. Yellow boxes denote the obstacles we spawn on the path.
Table Statistics proposed dataset split by the number of rooms. The ratio of each split, number of spawned obstacles, initial distance between start and target, and number of times the agent needs to across the room are counted.
Dataset splits (room num)
ratio
obstacle num
distance(m)
cross-room num
1~2
39.0%
2.05
5.45
1.44
3~5
42.3%
3.37
9.92
2.50
6~~10
18.7%
4.58
14.67
3.29
Experiments
5.1 Experiment Setup
Data collection. We perform experiments with the ProcTHOR simulator [14] on our dataset, which contains 600k task episodes in 12k scenes (50 per scene). For each episode, first randomly set the starting and target points two different rooms. Second, we calculate all room paths (e.g. rooml-rroome -rroomm---oomm) and all node paths (e.g. coordinates of start Mtart=xdoor3->aargee randomly pick obstacle assets (655 in total) at random positions between adjacent nodes until there is no clear path lengthen the shortest path (50% probability each). The dataset is split by the models are evaluated in unseen scenes: 1 1.8k training scenes, 100 validation scenes (5k episodes), and 100 testing scenes (500 episodes). Table the statistics of our dataset show that the agent challenged by multiple obstacles, long-distance navigation, and multi-door passing. Figure 4 shows several tip-view examples of our dataset.
At last
so
Environment settings. Following the standard settings, we let MoveAhead move the agent ahead by 0.25 meters, RotateRight and RotateLefi change the agent's azimuth angle by +90 degrees, LookUp Up and LookDown rotate the agent's ' camera elevation angle by #30 degrees. The DirectionalPuste let the agent push (along tz and axis) the closest visible object with constant force. The agent takes the END to indicate that it has completed an episode.
Evaluation metrics. We adopt Success Rate (SR), Final Distance to Target (FDT), and Success weighted by Path Length (SPL). SR the ratio of successful episodes in total episodes, FDT is the average geodesic distance between the agent and the goal when the episode is finished, and SPL is calculated as 1 >A Sucn I.a) where N is the total episodes number, Suc, the successful indicator n-th episode, Tip the shortest path length, and P, the length of the real path.
7

---

In non intiitteractive navigation, the metric SPL that measures path efficiency is also usable indicator of time efficiency, since the length of a path is usually proportional to the time it takes to execute the corresponding navigation actions (e.g. move ahead, rotate right). However, shorter path does not guarantee less time consumption for task completion in InterNav due to the object interaction. For instance, the agent may take the shortest path to the target through extensive interactions which cost a large amount of time and end up less effective than taking a detour path. Thus we evaluate an additional metric STS (Success rate weighted by Time Steps) to measure the time cost of task La/grid where Ln is the shortest path length, TS, is the completion: STS N Em=1 Suc,, TS.
timesteps the agent takes to complete the task, and grid 0.25m is the unit distance of step. Thus Ln/ Ln/grid represents the number of timesteps it takes to navigate to the target by merely moving forward. STS can be regarded as a time-measee measureeenem variant of SPL and the score is higher when the agent accomplishes the task with less time. The ideal situation is that the target is directly ahead and there is no need for interaction where STS is
5.2 Implementation Details
Our method is implemented and evaluated using the AllenAct [34] framework. The egocentric observation is set as 300*300 RGB and depth images. We first train three action policies then the whole model, utilizing Adam with initial learning rate of 3 10-4 4 for policy training and 1 10-4 for joint model training. The hidden size of CNN, GRU is set as 512. The intent embedding size is set as 12 (the same the intent size and the length of action space). The frequency of synchronization K' and epoch number & are set as 3 and 1, All models are trained for 10 million steps with 2 million warm-up steps in single-room scenes (Im for action policies and 1m for joint training) and million in all scenes
"spp where T Tauccess = 10 is provided if the agent takes action Done meanwhile achieving the goal. Adis represents the difference of geodesic distance between the agent and the goal position, and rssp = 0.01 denotes the step penalty.
Reward shaping. We set the reward for our task three parts: r F'suceess Adis
Three auxiliary tasks are designed as: (1) The Navigate policy is pretrained with the PointNav task where the agent moves to the goal point in an obstacle-free environment. The task shares the same reward r nan as above. (2) To pretrain the policy of Push and Pick, the agent is placed aside an obstacle opposite to the goal. It achieves success by taking Done when the obstacle is cleared. The reward Tinter Tuav + Tas rafs where ras, Taf denote the successful reward and failed penalty of chosen action (e.g. fail to conduct Drop before picking up something). Since effective pushes or picks change the length of the shortest path, Adis also applies to interaction reward.
5.3 Baselines
Random:
This baseline randomly takes actions with the same probability until the agent reaches the target or reaches the maximum number of steps.
PPO: We set a common baseline trained with PPO algorithm according to [35]. Similar to our
method, it extracts visual features with an encoder and reads the goal position with an embedding layer, then single policy network predicts actions based on fused features.
NIE: We implement the method NIE [39] which predicts the state change of observed objects conditioned on actions with a module in addition to the PPO baseline. The representation of object change fused with visual features and goal embedding.
HRL: To study the effect of our multi-policy framework, we implement a common hicrarchical policy supported by AllenAct [34] that shares the same three sub-spaces of action. The difference is that the low-level policies are called at each step, not able to plan actions in period.
PPO+intent: We develop a counterfactual policy based on the PPO baseline according to Figure 2(c) to study the effect of antent-aware recision-making
'The project is available at: attps://gitthe wtt./ooooooo///// com//p//llalllaaanntttraaaa
8

---

Table 3: Quantitative results.
N2
non-interactive STS
Methods
SR(%)T
FDT(
SPL
STS
SR
FDT
SPI.
STS
SR
FDT
SPI.
Random
1.32
7.68
0.002
0.001
12.29
0
2.42
7.35
0.003
0.001
PPO [35]
42.7
4.84
0.252
0.134
23.3
7.79
0.139
0,086
51.5
3.44
0.306
0.181
NIE [39]
52.0
3,94
0.287
0.155
37.3
6.54
0.195
0.102
58.8
3.01
0.345
0.188
HRL
43.4
4.80
0.253
0.135
24.3
7.01
0.169
0.084
51.9
3.40
0.316
0.176
PPO+intent
54.7
3.84
0.305
0.163
38.0
6.29
0.198
0.109
70.4
2.38
0.390
0.222
CaMP
56.3
3.67
0.327
0.177
41.4
5.76
0.231
0.121
72.3
2.05
0.407
0.236
Table 4: Ablation results.
all FDT(m)
N2 4
non-interactive STS
Methods
SR(%)
SPL1
STS
SR
FDT
SPL
STS
SR
FDT
SPL
Sync. variants:
sync. /epoch
53.0
4.16
0.316
0.160
36.7
6.95
0.196
0.107
63.4
2.53
0.375
0.219
syne. /rollout
50.7
4.22
0.277
0.145
36.0
6.82
0.175
0.105
57.9
3.10
0.345
0.191
syne. /3*rollout
54.7
3.84
0.305
0.163
37.3
6.54
0.195
0.109
70.4
2.38
0.390
0.222
sync. ro*lol
50.3
4.25
0.279
0.150
30.7
6.85
0.147
0.098
55.9
3.21
0.337
0.185
intent variants:
wo/intent
50.8
4.03
0.294
0.151
31.0
6.79
0.151
0.095
59.2
2.98
0.347
0.202
test wo/intent
41.7
4.95
0.202
0.111
24.7
7.72
0.110
0.085
50.5
3.56
0.296
0.168
recursive intent
51.4
4.06
0.292
0.156
32.7
7.04
0.159
0.096
62.8
2.61
0.353
0.215
policy intent
52.5
4.32
0.304
0.155
35.8
0.191
0.100
62.6
2.91
0.357
0.216
integrated intent
56.3
3.67
0.327
0.177
414
5.76
0.231
0.121
72.3
2.05
0.407
0.236
5.4 Results Analysis
Comparison with baselines. We pick model parameters on the validation set and report their performance on the whole testing set, hard scenes with more rooms (N 2 4), and the splits of episodes that exist clear paths (non-interactive). As quantitatively shown in Table 3, CaMP outperforms all baselines and obtains +13.6/47.5, 1/49.2, and +4.3/+3.5 improvement in SR. SPL, and STS (all/N 2 4, %) respectively compared to the model PPO. Compared with PPO+intent, CaMP obtains more improvement in hard scenes (+3.4 and +3.3 SR, SPL) which shows robustness of multi-policy facing complex environment. Besides, the relatively poorer performances of HRL and Random model reflect the challenge of our dataset. The limited performance hard scenes shows that effective methods for navigation in plenty of rooms trends across STS and SPL when comparing PPO and HRL on the hard split and the non-interactive split indicate that both path and time efficiency should be measured considering InterNav scenarios.
still in need. The contrary
Ablaiio study. Our ablation study aims to explore the effect of two model designs. First, we change the synchronization frequency of model PPO+intent as epoch. rollout, 3 rollouts, and 5 rollouts. As shown in Table 4. the variant with K gains the relatively best performance, which reflects how the balanced exploration-exploitatio benefits policy learning.
We implement (1) CaMP without the intent network as wofintent, (2) CaMP that inferences without the intent network as test wo/intent, (3) CaMP provided with the intent recursively generated based on the prior intent as recursive intent, and (4) CaMP provided with the intent of the global policy as policy intent. Here our model CaMP is referred to integrated intent. As shown in Table 4, the integrated intent effectively improves the decision-makin ability of the global policy compared with wo/intent (+5.5/+3.3 and +1 +10.4/+8. in SR, SPL). Without the information of the intent of action policies. policy intent shows less gain effect compared with integrated intent (-3.8/-2.3 and -5.6/-4.0 4.00 in SR. SPL). The limited performance of test wo/intent suggests that the inference process of our model also relies on the intent. The results of recursive intent suggest that although the secondary intent still provides incremental information about the system, the counterfactual situation based the "counterfact" may become less meaningful for learning causality.
Second, study how the model intent influences performance:

---

Top view (scene 1999)
Egocentric viewat key steps
b=25
1-35
PPQ
Take
1=11
t-25
1=50
t=1 15
CaMP
Safa
Figure 5: Qualitative cases. The starting and target points marked by blue circles and obstacles are marked by yellow boxes. The blue arrows and red arrows denote the trajectories from CaMP and PPO, respectively. The nodes and crosses denote successful object interactions and failed ones. Markings in colors are invisible to agents.
Case study. We qualitatively compare our model with baseline PPO on a typical instance 4-room scene in Figure 5. The blue trajectory of CaMP shows that the agent successfully removes obstacles of tomato and pillow, and attempts bypass an immovable safa to reach the target. While the red trajectory shows that PPO intends to interact with table and bed which are too heavy to be pushed aside and ends up getting stuck corner. The comparison suggests that our model can make better decisions on whether and how to interact with obstacles.
Table 5: Obstacle interaction results.
Methods
IR(m)
ISR %
PuSR %)
PiSR %
PPO [35]
0.271
23.2
23.8
21.9
PPO+intent
0.371
27.0
25.1
28.6
CaMP wo/intent
0.287
20.4
19.9
23.5
CaMP
0.402
27.6
27.1
29.8
Study of obstacle interaction. To study how intent-aware policy learns the obstacle interaction and its outcome, make direct evaluation on obstacle interaction of several models in Table 5: (I) Interaction Reward (IR): the reduction of geodesic distance Adia caused by interactions. (2) Interaction Success Rate (ISR): the ratio of taking interactions successfully (a more detailed measurement compared to the overall metric SR measuring task completion). A failed case of interaction when the agent tries to pick up heavy table attempts to push chair out of sight. We also report the success rate of Push and Pick interactions separately as PuSR (Push Success Rate) and PiSR (Pick Success Rate). The results (comparison between PPO and PPO+intent, CaMP and CaMP wo/intent) show the outcome of our causal method in pursuing the interaction efficacy of reducing the distance to target (IR) and avoiding invalid actions (ISR). The results on PuSR and PiSR also suggest that successfully pushing an obstacle is more challenging since the obstacle may get stuck and is not movable in all directions.
6 Conclusions
In this paper, we tackle the task of Interactive Navigation (InterNav) which requires mastery of long-horizon navigation and object interaction in complex environments. We take a causal view of InterNav and clarify the challenge caused by the confounding bias of obstacles. Causal Multi policy Planning (CaMP) model is proposed following the main idea of reducing inefficient exploration with multi-policy and exploring counterfactual interaction by considering the primary intent of model's necision-making. We construct a large-scale dataset of interactive navigation with various obstacles spawned across multiple rooms. Experiments on our dataset illustrate the effectiveness of our method and support causal modeling of InterNav.
10

---

Acknowledgment
This work is supported by the National Key Research and Development Project of New Generation Artificial Intelligence of China, Grant Number: 2018AAA0102504. This work is in part by Beijing Natural Science Foundation under Grant JQ22012, and in part by the National Natural Science Foundation of China under Grant 62125207.
References
[1] Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3674-3683, 2018.
[2] Puerre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Proceedings
of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
[3] Elias Bareinboim, Andrew Forney, and Judea Pearl. Bandits with unobserved confounders: A
causal approach. Advances in Neural Information Processing Systems, 28, 2015.
[4] Dhruv Batra, Aaron Gokaslan, Aniruddha Kembhavi, Oleksandr Maksymets, Roozbeh Mottaghi, Manolis Savva, Alexander Toshev, and Erik Wijmans. Objectnav revisited: On evaluation of embodied agents navigating to objects, arXiv preprint vrXiv:2006.13177 2020.
[5] Devendra Singh Chaplot, Dhiraj Gandhi, Saurabh Gupta, Abhinav Gupta, and Ruslan Salakhut- dinov. Learning to explore using active neural slam. arXiv preprint arXiv:2004. 15500 2020.
[6] Devendra Singh Chaplot, Dhiraj Prakashchand Gandhi, Abhinav Gupta, and Russ R Salakhut- dinov. Object goal navigation using goal-oriented semantic exploration. Advances in Neural Information Processing Systems, 33:4247-4258, 2020.
[7] Changan Chen, Unnat Jain, Carl Schissler, Sebastia Vicene Amengual Gari, Ziad Al-Halah, Vamsi Krishna Ithapu, Philip Robinson, and Kristen Grauman. Audio-visual embodied naviga-
tion. environment, 97:103, 2019.
[8] Changan Chen, Unnat Jain, Carl Schissler, Sebastia Vicene Amengual Gari, Ziad Al-Halah, Vamsi Krishna Ithapu, Philip Robinson, and Kristen Grauman. Soundspaces: Audio-visual navigation in 3d environments. In European Conference on Computer Vision, pages 17-36.
Springer, 2020.
[9] Long Chen, Xin Yan, Jun Xiao, Hanwang Zhang, Shiliang Pu, and Yueting Zhuang. Counterfac- tual samples synthesizing for robust visual question answering. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 10800-10809, 2020.
[10] Long Chen, Hanwang Zhang, Jun Xiao, Xiangnan He, Shiliang Pu, and Shih-Fu Chang. Counterfactual critic multi-agent training for scene graph generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4613-4623, 2019.
[11] Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Embodied question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1-10, 2018.
[12] Abhishek Das, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Neural modular control for embodied question answering. In Conference on Robot Learning, pages 53-62.
PMLR, 2018
[13] Matt Deitke, Winson Han, Alvaro Herrasti, Aniruddha Kembhavi, Eric Kolve, Roozbeh Mot- taghi, Jordi Salvador, Dustin Schwenk, Eli VanderBilt, Matthew Wallingford, et al. Robothor: An open simulation-to-ret
embodied ai platform. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3164-3174, 2020.
[14] Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Jordi Salvador, Kiana Ehsani, Winson Han, Eric Kolve, Ali Farhadi, Aniruddha Kembhavi, al. Procthor: Large-scale embodied ai using procedural generation. arXiv preprini arXiv:2206.060 06994, 2022.
[15] Thomas G Dietterich. Hierarchical reinforcement learning with the maxq value function decomposition. Journal of artificial intelligence research, 13:227-303, 2000.
[16] Jiafei Duan, Samson Yu, Hui Li Tan, Hongyuan Zhu, and Cheston Tan. A survey of embodied ai: From simulators research tasks. IEEE Transactions on Emerging Topics in Computational Intelligence, 2022.
[17] Andrew Forney and Elias Bareinboim. Counterfactual randomization: rescuing experimental studies from obscured confounding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 2454-2461, 2019.

---

[18] Andrew Forney, Judea Pearl, and Elias Bareinboim. Counterfactual data-fusion for online reinforcement learners. In International Conference on Machine Learning, pages 1156-1164.
PMLR, 2017.
[19]
Daniel Gordon, Aniruddha Kembhavi. Mohammad Rastegari, Joseph Redmon, Dieter Fox, and Ali Farhadi. Iqa: Visual question answering in interactive environments. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4089-4099 2018
[20] Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Matt Deitke, Kiana Ehsani, Daniel Gordon, Yuke Zhu, et al. Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint urXivv:7122 05444,, 2017.
[21] Xiangyang Li, Zihan Wang, Jiahao Yang, Yaowei Wang, and Shuqiang Jiang. Kerm: Knowledge navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2583-2592, 2023.
enhanced reasoning for
[22] Ruyang Liu, Hao Liu, Ge Li, Haodi Hou, TingHao Yu, and Tao Yang. Contextual debiasing for visual recognition with causal mechanisms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12755-12765, 2022.
[23] Yulei Niu, Kaihua Tang, Hanwang Zhang, Zhiwu Lu, Xian-Sheng Hua, and Ji-Rong Wen. Counterfactual vqa: A cause-effect look at language bias. In Proceedings of the IEEE/CVF
Conference Computer Vision and Pattern Recognition, pages 12700-12710, 2021.
[24]
Judea Pearl. Causal inference in statistics: An overview. Statisties surveys, 3:96 146, 2009. Judea Pearl et al. Models, reasoning and inference. Cambridge, UK: CanbridgeUniversity 19(2). 2000.
[25]
[26]
Donald B Rubin. Essential concepts of causal inference: a remarkable history and an intriguing
future. Biostatistics & Epidemiology. 3(1): 1000::401110 140-155, 2019.
[27] Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub. Jia Liu, Vladlen Koltun, Jitendra Malik, et al. Habitat: A platform for embodied ai research. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pages 1339-93477 2019.
[28] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 060700 2017.
[29] Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mot- taghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10740-10749, 2020.
[30] Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 11( 2):181-211, 1999.
[31] Kaihua Tang. Jianqiang Huang, and Hanwang Zhang. Long-tailed classification by keeping the good and removing the bad momentum causal effect. Advances in Neural Information Processing Systems, 33:1513-1524 2020.
[32] Kaihua Tang, Yulei Niu, Jianqiang Huang, Jiaxin Shi, and Hanwang Zhang. Unbiased scene graph generation from biased training. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 37 12166337 3725, 2020.
[33] Zihan Wang, Xiangyang Li, Jiahao Yang. Yeqi Liu, and Shuqiang Jiang. Gridmm: Grid memory map for vision-and lann-uuee navigation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 15625-15636, 2023.
[34] Luca Weihs, Jordi Salvador, Klemen Kotar, Unnat Jain, Kuo-Hao Zeng, Roozbeh Mottaghi, and Aniruddha Kembhavi. Allenact: A framework for embodied ai research. arXiv preprint arXiv:2008. 12760, 2020.
[35] Erik Wijmans, Abhishek Kadian, Ari Morcos, Stefan Lee, Irfan Essa, Devi Parikh, Manolis Savva, and Dhruv Batra. Dd-ppo: Learning near- pear-eerfectppee pointgoal navigators from 2.5 billion 0e:::1111000555 2019.
frames. arXiv preprint
[36] Fei Xia. William B Shen, Chengshu Li, Priya Kasimbeg, Micael Edmond Tchapmi, Alexan- der Toshev, Roberto Martín Martin,,,,,,,, and Silvio Savarese. Interactive gibson benchmark: A benchmark for interactive navigation in cluttered environments. IEEE Robotics and Automation Letters, 5(2):733-700, 2020.
[37] Xu Yang. Hanwang Zhang, Guojun Qi, and Jianfei Cai. Causal attention for vision-languse tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9847-9857, 2021.
[38] Zhongqi Yue, Tan Wang, Qianru Sun, Xian-Sheng Hua, and Hanwang Zhang Counterfactual zero-shot and open-set visual recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15404-15411 2021.
12

---

[39]
Kuo-Hao Zeng, Luca Weihs, Ali Farhadi, and Roozbeh Mottaghi. Pushing it out of the way: Interactive visual navigation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9168-9877.2 2021.
[40] Dong Zhang, Hanwang Zhang, Jinhui Tang, Xian-Sheng Hua, and Qianru Sun. Causal interven- tion for weakly-supervised semantic segmentation. Advances in Neural Information Processing Systems, 33:655--:6666 2020.
[41] Junzhe Zhang and Elias Bareinboim. Markov decision processes with unobserved confounders:
A causal approach. Technical report, Technical report, Technical Report R-23, Purdue Al Lab,
2016.
[42] Sixian Zhang, Weijie Li, Xinhang Song, Yubing Bai, and Shuqiang Jiang. Generative meta- adversarial network for unseen object navigation. In Computer Vision ECCV 2022 17th European Conference, Tel Aviv, Israel October 23-27, 2022, Proceedings, Part XXXIX, volume 13699 of Lecture Notes in Computer Science, pages 301-320. [43] Sixian Zhang, Xinhang Song. Yubing Bai, Weijie Li, Yakui Chu, and Shuqiang Jiang. Hierarchi- cal object-to-zong graph for object navigation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 15130-15140, 2021.
[44] Sixian Zhang, Xinhang Song. Weijie Li, Yubing Bai, Xinyao Yu, and Shugiang Jiang. Layout- based causal inference for object navigation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 10792-10802, June 2023.
13

---

A Appendix
A.1 Proof of equation (3)
For state 9t and intent & at step t. the optimal counterfactual policy Torf = argmax may (s;, it). and the optimal standard policy πO = argmaxV= we have V=z,, (st, de) 2 (st 41). For standard optimal policy, since x² executes its intent in an optimal action of without recoonsierrtin,nnn we have:
dj argmar.k
rr++++1 arl
(8)
Va: (se, bn) 2 Vx. (82, it) = V (st, (() cs
E
k M 1 df]
k=U
(9)
2 Σ x (a)l())
E
7 7 ad)
a.
k=0
V (8+)
Thus considering the information its own intent, the agent can learn a counterfactual policy gaining more value.
A.2 Impact of different intents on policy exploration
We train our model with PPO [28] which encourages the exploration with an entropy loss L e25[=] c2 Ca.ea P( MoggP(a)) where c2 < is the coefficient. Since our agent does trials con- ditioned on its intent, maximizes the conditional entropy H(A!) We are curious about how this conditional exploration can make the new policy distribution different from the conditional distribution (intent). To study the question and also how the similarity between intent and action affects policy exploration, we evaluate the mean KL. divergence between the current and old policy distributions under several settings throughout the training. In particular, frequency synchronizing the action policy and the intent policy as 1 epoch, 3 epochs, 6 epochs, epochs, 12 epochs, 15 epochs, and without intent. In Figure 6, we report the mean KL divergence between the distributions of current policy and different old policy (from epoch behind to 15 epochs behind). The trend of each curve shows that the counterfactual information of intent encourages the agent to explore new strategies. The comparison between curves shows that the more different actions and intents, the more the model tends to explore new policies rather than exploit the old policy. Thus the frequency of synchronization can be utilized to adjust how much the agent explores new policies under counterfactual situations.
-E.,sA P(ayléc)oog (ogg) (ic)
vary the
lepoch
Sepochs
Gepochs
/9epechs
/ 12cpochs
/ Sepochs
wofintent
0.0084
KL Divergence
0.0074
0.0069
0.0064
0.0054
7
10
Gap number of epochs between the current policy and old policy
Figure 6: Mean KL divergence of the current and old policy distributions.
14


Reviews:
[{"review_id": "t7dJ2NDOfC", "reviewer": "Reviewer_HVkt", "summary": "This paper tackles the problem of interactive visual navigation; i.e., an agent navigating in an enviornment where it is allowed to affect the configuration of the environment (e.g., by moving objects around or picking them up), to improve navigation performance. The key idea is to learn a hierarchical policy that factors in agent intent to propose an action that either results in navigation towards the goal, or interaction with an obstacle. A dataset based on the PROC-THOR simulation environment is also introduced, to facilitate evaluation.", "questions": "Overall, I like where this paper is going. The hierarchical policy presented herein is sound. My concerns are not with the method; rather with the choice of evaluation metrics. In my review (see weaknesses above), I have attempted to also provide a few mitigation strategies (these are by no means extensive -- merely initial thoughts on how newer evaluation metrics may be designed for the task at hand).\n\nI would like to see this discussed further in the author response phase. If addressed adequately, I have no hesitation in bumping my score up.", "limitations": "Adequately addressed", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "[S1] The proposed method is sound; and for the most part, well-defined. The problem formulation is easy to follow, and the description of the method is clear.\n\n[S2] Interactive navigation is a challenging and relevant robotics/embodied-AI problem to the Neurips community. While a large fraction of existing approaches focues on non-interactive visual navigation, this paper explicitly considers affecting state changes by manipulating obstacles, resulting in a novel problem setting where there isn't much prior work.\n\n[S3] I find the positioning of this paper w.r.t. existing literature fair. The baseline methods considered for evaluation are representative of the various flavors of non-interactive visual navigation approaches that have, over the years, been proposed for PointGoal navigation.\n\n[S4] The paper discusses enough implementation details that a reasonable practitioner may be able to replicate the key aspects of the model architecture and the state, action, reward structures.\n", "weaknesses": "Meta-comment: I have one major concern with the experiment design and evaluation setup; which unfortunately results in the key claims of the paper not being substantiated. I have tried to elaborate the issue, and also the rationale behind it. I also suggest a few mitigation strategies (note: these aren't the only possible ones; other strategies welcome too). While I am unsure whether these may be addressed in the short author response window, if these are adequately addressed, I would have no reservations in bumping my score up.\n\n\n[W1] **[Major] Evaluation metrics and experiment design**:\n\nInteractive navigation is very tricky to evaluate. In (non-interactive) navigation scenarios, the path length metrics used in literature are often highly correlated to execution time. Assuming that each atomic action executed by the agent takes nearly the same time to execute, a longer path length would mean that the agent takes longer to reach the goal. However, in the interactive navigation scenario, typical path length metrics like SPL are no longer good indicators of the amount of time it would take to complete the task. This is because, the time spent in picking up or moving objects will also count towards the overall \"time-to-reach-goal-state\"; and picking up or moving objects is heavily dependent on object states, which voids the assumption that each action takes roughly the same amout of time.\n\nI understand the rationale behind using typical metrics like SR (success rate), SPL (success-weighted inverse path length), and FDT (final distance to target) -- these metrics allow for easy benchmarking with existing approaches (esp. non-interactive navigation techniques); these are also easily accessible via modern simulators that support PointGoal navigation.\n\nThat said, reporting only these above metrics only portrays the benefits of an interactive navigation strategy, while masking away the disadvantages. Execution time is often lost when interacting with objects, which goes unaccounted for. This becomes apparent when you consider the following counterexample: assume that an agent spends more than half its time moving (or picking up) every object encountered in its way; but ends up taking the shortest possible (i.e., optimal) path to goal. It will then end up with a success rate and SPL of 1 and an FDT value of 0. However, in reality, an agent that takes a twice as long path to the goal (but does not interact with any object) will end up reaching the goal at about the same time, and will have the same SPL, SR, and FDT values (recall that, in this hypothetical scenario, the interactive agent spends half its time interacting with obstacles in its path). This calls for a different approach to evaluate interactive navigation approaches; ideally the SPL metric should also account for time lost due to interaction. Measuring this time could be challenging (or impossible) to do in the first place, so here are a few potential mitigation strategies to consider.\n\n* While episode lengths are fixed, to ensure the agent does not take an indefinite amount of time, to compare fairly against non-interactive agents, each episode length must be capped to the number of timesteps in which a non-interactive agent completes the task. This will ensure that a fair comparison is possible, when a reasonably accurate time estimate is available for each interaction.\n* An alternative would be to consider evaluating solely in scenarios where it is impossible to reach the goal without interacting with objects. (more nuance on this follows in [W2] below).\n\n[W2] **Eval on data subsets**: I would have liked to seen more granularity in terms of the quantitative results presented in Table 3. It would, for instance, be useful to split the dataset into a variety of categories, depending on whether or not interaction is required to solve an episode, and the level of difficulty of the episode. This becomes more important, because the dataset is claimed as a contribution. Table 2 goes a bit along this direction, indicating the number of obstacles per room; however, it is not clear how many of these episodes absolutely require interactive navigation (i.e., cannot be solved by a non-interactive optimal agent). (The dataset construction seems to ensure a 50% likelihood that a path is unavailable -- I would argue in favor of forming two splits of the dataset, one where no non-interactive agent trajectory exists; and one where an optimal non-interactive agent will need to take a longer path).\n\n\n[W3] **[Minor]** The paper, in its current form, falls short in terms of technical rigor when discussing aspects of causal inference. E.g., lines 134-135 \"However, without considering the causalities from obstacles, it’s hard to generalize to unseen environments or large-scale datasets.\"; line 38 \"... , learning causality through RL training is challenging due to the existence of unobserved...\"; line 137 \"The above confounding bias can be tackled...\". Neither the \"causalities stemming from obstacles\" nor the \"confounding bias\" have been clearly been defined in the paper. It is also important to clearly define the type of causal relationships that are being learned by design (perhaps section 4.2 is a good place to do so).\nRevising these aspects of the paper will make the descriptions of the approach more accurate.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive concerns and address them in detail in the following lines.\n\nQ1. Evaluation metrics and experiment design.\n\nThe reviewer is concerned about the evaluation of InterNav that “in the interactive navigation scenario, typical path length metrics like SPL are no longer good indicators of the amount of time it would take to complete the task.” and “That said, reporting only these above metrics only portrays the benefits of an interactive navigation strategy, while masking away the disadvantages.” and provides a few potential mitigation strategies of evaluation.\n\nA1. We agree that “time of completing the task” is also an important evaluation indicator for InterNav. In our scenario, by calculating the standard time cost of each action (including movement and manipulation), a time cost metric can also be evaluated in our experiments, which can be regarded as a time-measurement variant of the SPL metric. Also, considering that most previous InterNav works [1,2] report the SPL, in order to compare with those works directly, we’ll report both time and path length metrics in the following response and final version.\n\nFirst, we evaluate an additional metric STS (short for Success rate weighted by Time Steps) to measure the time cost of task completion: $STS=\\frac{1}{N}\\sum_{n=1}^{N}Suc_{n}\\frac{L_{n}/grid}{TS_{n}}$, where $L_{n}$ is the shortest path length, $TS_{n}$ is the timesteps agent takes to complete the task, and $grid=0.25m$ is the unit distance of agent moving forward in one step. Thus $L_{n}/grid$ represents the number of timesteps it takes to navigate to the goal by merely moving forward (without spawned obstacles). Since the agent takes atomic actions in AI2-THOR simulator and each of them shares the same amount of time to execute (a timestep), we measure the time cost with the number of timesteps. STS is higher when the agent accomplishes the task with less time and the ideal situation is that the goal is directly ahead and there is no need for interaction where STS is 1 (it's the most ideal situation for all navigation scenarios so that it's computable for InterNav). As a matter of fact, we train our model following that idea, since our reward shaping $r=r_{success}+\\Delta_{dis}-r_{tp}$ encourages interaction that efficiently reduces the goal distance with fewer timesteps, rather than pursuing a shorter trajectory. Thus, both strategies of efficient bypass and effective interaction are rewarded. We report the performance of several models in the table below, and the result shows that STS is a more stringent measure and is able to reflex the performance difference between models. \n\n|Methods|STS (all)|STS (N$\\geq$4)|\n|:----:|:----:|:----:|\n|PPO|0.134|0.086|\n|NIE|0.155|0.102|\n|HER|0.135|0.084|\n|PPO+intent|0.163|0.109|\n|CaMP|**0.177**|**0.121**|\n\nSecond, in InterNav scenarios, SPL is an indicator of agents' ability of object interaction, which is crucial especially in cluttered environments. Fei Xia et al. [2] have introduced SPL to interactive navigation as the measurement of path efficiency that “Path Efficiency: how efficient the path taken by the agent is to achieve its goal. The most efficient path is the shortest path assuming no interactable obstacles are in the way.” Although the most efficient path may not correspond to the least time cost, it indicates the most effective interaction. For InterNav in complex multi-room environments, we aim at improving agent's interactive ability so it can proactively change the environment for better navigation, rather than limiting its strategic choices to its capacity.\n\nQ2. Eval on data subsets. The reviewer suggests that \"I would argue in favor of forming two splits of the dataset, one where no non-interactive agent trajectory exists; and one where an optimal non-interactive agent will need to take a longer path\".\n\nA2. We appreciate the suggestion and form a split of dataset where non-interactive trajectories (longer than the shortest path) exist to enrich the evaluation. We first calculate the ratio of that split in the whole dataset: 20.5% (overall), 27.4% (1 $\\sim$ 2 rooms), 18.6% (3 $\\sim$ 5 rooms), 12.4% (6 $\\sim$ 10 rooms). Then we report the performance of models (a non-interactive PPO trained on ProcTHOR is included) on the **new split** in the table below. It's interesting to find that PPO without interaction achieves better STS (0.201) compared with PPO (0.181), although it obtains lower SR on the non-interactive set (47.4%) and the whole set (21.5%). It indicates that the strategy of object interaction may cost unnecessary time in uncrowded environments and the agent needs to balance the efficiency and efficacy during the task.\n\n|Methods|SR (%)|SPL|STS|FDT|\n|:----:|:----:|:----:|:----:|:----:|\n|PPO (non-inter)|47.4|0.309|0.201|4.62|\n|PPO|51.5|0.306|0.181|3.44|\n|NIE|58.8|0.345|0.188|3.01|\n|HER|51.9|0.316|0.176|3.40|\n|PPO+intent|70.4|0.390|0.222|2.38|\n|CaMP|**72.3**|**0.407**|**0.236**|**2.05**|\n\nQ3.  Concept clarification.\n\nA3. Thanks for the concern. By \"causalities from obstacles\" we refer to the causal relationships from the obstacle (O) to other causal factors (i.e. O $\\rightarrow$ A, O $\\rightarrow$ R in Figure 2). By \"confounding bias\" we refer to the negative phenomenon of $P(R|do(A))\\neq P(R|A)$ caused by the confounder (i.e. obstacles in InterNav). And our method is designed to better learn the causality from action to reward (A $\\rightarrow$ R). We will revise the concept clarification of our paper for better understanding.\n\n**References**\n\n[1] Zeng, Kuo-Hao, et al. \"Pushing it out of the way: Interactive visual navigation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n\n[2] Xia, Fei, et al. \"Interactive gibson benchmark: A benchmark for interactive navigation in cluttered environments.\" IEEE Robotics and Automation Letters 5.2 (2020): 713-720.\n"}, {"review_id": "abFZjW39GG", "reviewer": "Reviewer_7DX2", "summary": "This paper introduces a causally-inspired hierarchical policy framework for the interactive navigation task in the AI2THOR environment. The framework consists of a master policy, intent policy, and three sub-control policies. The intent policy embeds intuitive intents from the sub-control policies into the master policy, enabling it to make counterfactual decisions. The proposed approach, named CaMP, is evaluated on a newly collected dataset in the ProcTHOR multi-room scenes. The experimental results show that CaMP outperforms the baselines, achieving the best performance on the interactive task.", "questions": "Please address following points for a more comprehensive understanding of the proposed approach.\n\nClarifications:\n\no) Please provide further clarification on the connection between the confounding bias resulting from unmeasurable obstacles and the CaMP model design, specifically the relationship between Sec. 3 and Sec. 4.\n\no) Clarify why using the weighted-sum of action logits is an effective way to represent intent and its relationship to the confounding bias.\n\nMissing details:\n\no) How to determine which object to interact with when using the Push/Pick actions?\n\no) What is the amount of force applied during the Push action, and is it correlated with the object's mass?\n\no) Provide the exact dimension of the intent embedding $P(i_t)$. Is it equivalent to the size of the total action space?\n\no) Provide architectural details of the policy, such as whether the CNN is a simple CNN or a CLIP pretrained ResNet, and the number of layers in the GRU.\n\no) Clarify the meaning of \"epochs with a rollout of data\" at L191. Does it refer to the number of update iterations using a rollout of data in the PPO implementation in AllenAct?\n\no) Missing implementation details of the HRL baseline. Does the HRL apply all outputs from the sub-control policies in a single execution, or does it follow a traditional HRL policy where the master policy calls one of the sub-control policies at each step?\n\nConfusions:\n\no) Clarify whether $r_{tp}$ at L227 is the same as $r_{sp}$ at L229.\n\no) At L235, it is mentioned that $r_{nav}$ is used to learn the push and pick sub-control policies. This appears to contradict the context provided at L235. Please clarify the requirements for task success, specifically if the agent needs to reach the target position.", "limitations": "-) While the proposed CaMP is interesting and evaluated in a complex environment with non-trivial tasks, it would be beneficial to see more results in a simpler environment. For example, evaluating the model on a 2D environment like PettingZoo, where a dot agent performs the interactive navigation task by interacting with obstacles, could provide insights into how the model disentangles intertwined factors by the causally-inspired design in a more controlled setting.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "+) The interactive navigation (IN) task is an interesting and challenging task in the field of embodied AI. This paper contributes to the progress in this area by introducing a causally-inspired hierarchical reinforcement learning (HRL) policy. The use of a newer and larger dataset with complex scene layouts and diverse objects enhances the realism of the task. The results demonstrate the effectiveness of the proposed HRL policy, highlighting the advancements in tackling the challenges of interactive navigation.\n\n+) The decomposition of the embodied policy into a master policy and sub-control policies is a reasonable and effective approach for the IN task. This work demonstrates the potential of HRL in tasks where interactions with the environment are crucial for achieving goals.\n\n+) The explanation of confounding bias and the causal diagram provided in Fig. 2(a) help clarify the concept and its relevance to the IN task. However, the discussion of heavy obstacles as an example may need further refinement.\n\n+) The experimental results presented in the paper provide strong evidence for the effectiveness of CaMP. It outperforms several baselines, including PPO, NIE, and HRL, on the IN task using the newly collected large-scale dataset in ProcTHOR multi-room scenes.", "weaknesses": "-) The connection between confounding bias resulting from unmeasurable obstacles and the counterfactual policy design is not adequately explained. It remains unclear how the counterfactual policy, through the intent policy, effectively addresses the bias caused by obstacles. The paper lacks a convincing explanation for the direct line from $O$ to $A$ in Fig. 2(b). It is not clear how the weighted-sum of action logits from the sub-control policies can fully capture the policies' intents and uncover the causality depicted in Fig. 2(a) ($O$ -> $A$ and $O$ -> $R$).\n\n-) The model design lacks intuition. Instead of introducing an additional Intent Policy to generate intents, a more straightforward approach would be to recursively use the Master Policy to obtain intents $P^j(i_t)$ and provide feedback to the Master Policy. This recursive feedback mechanism could be extended to multiple levels, denoted by $j$ ∈ $J$, with $P^0(i_t)$ representing a void intent. Additionally, rather than using the action logits from the sub-control policies to represent intents, exploring the utilization of hidden features returned from the GRU in each sub-policy would offer a more intuitive approach.\n\n-) Several implementation details are missing, such as determining which object to interact with when using Push/Pick actions, the force applied during the Push action, and whether the force is correlated to the object's mass. Further clarification on these aspects would greatly enhance the understanding of the proposed approach. More questions regarding missing details can be found in the Question.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive criticism. We address the concerns in detail in the following lines.\n\nQ1. Model design explanation. The reviewer wonders “the connection between confounding bias resulting from unmeasurable obstacles and the counterfactual policy design”, “how the weighted-sum of action logits from the sub-control policies can fully capture the policies' intents and uncover the causality depicted in Fig. 2(a)”, and argues that “a more straightforward approach would be to recursively use the Master Policy to obtain intents and provide feedback to the Master Policy”.\n\nA1. We appreciate the questions and further clarification is detailed as follows.\n\nFirst, counterfactual decision-making theoretically addresses the problem of learning sub-optimal policy caused by confounding bias from unmeasurable obstacles. Obstacles in InterNav scenario can be regarded as an unobserved confounder (UC) since they influence both the decision-making of actions and the generation of rewards. For instance, the agent may decide to take object interactions when encountering an obstacle (O $\\rightarrow$ A). Here, obstacles (O) serve as the mediator from state to action (S $\\rightarrow$ A). The causality from action to reward (A $\\rightarrow$ R) is confounded by UC, leading to poor estimation of value (accumulated reward) which likely results in sub-optimal policy in RL training. In the theorem of causality inference, unlike observable confounders, UCs can hardly be addressed directly with methods like *Intervention* or *Back-door adjustment*. However, it can be theoretically proved that a counterfactual policy considering intent obtains more value than a standard policy when there exists UC [1]. \n\nIn Section 4, we apply counterfactual policy to a hierarchical decision framework. In addition to addressing UC, learning counterfactual policy also addresses the indirect feedback problem of master policy by providing it with information about the low-level decision-making through integrated intent.\n\nSecond, we believe the sum of actions from sub-control policies weighted by master policy’s decision can fully represent agent’s hierarchical intent, since it contains agent’s intent on each atomic action and the full distributional information of four policies. Since the primary definition of intent is “action before execution” that $I=i_t=f_i(s_t,o_t)$, we don’t see the rationale of implementing intent with hidden features from GRU instead of action logits.\n\nThird, we find the idea of extending the recursive feedback to multiple levels interesting since it may explore the effect of \"recursive intent\", namely the intent generated based on a priori intent. And we are training new models based on the baseline of PPO to study how a \"recursive intent\" may help policy learning and will report the results in our final paper (given the limited time for model training). Nonetheless, we believe implementing agent’s intent with an intent policy is reasonable and it’s flexible for us to utilize old intent from iterations behind to balance the policy exploration.\n\nQ2. Missing details and confusions.\n\nA2. Thanks for pointing them out and we will reply to them item by item as follows.\n\n(1) **Q**: \"How to determine which object to interact with when using the Push/Pick actions?\"\n\n**A**: When taking Push/Pick actions, the object to interact with would be the closest pushable/pickable (predefined according to category) and observable (within 1.25m) object.\n\n(2) **Q**: \"What is the amount of force applied during the Push action, and is it correlated with the object's mass?\"\n\n**A**: The amount of force applied on the object during the Push action is 100 Newton.\n\n(3) **Q**: \"Provide the exact dimension of the intent embedding $P(i_t)$. Is it equivalent to the size of the total action space?\"\n\n**A**: The dimension of intent embedding is 12, equivalent to the size of our action space.\n\n4) **Q**: \"Provide architectural details of the policy, such as whether the CNN is a simple CNN or a CLIP pretrained ResNet, and the number of layers in the GRU.\"\n\n**A**: The CNN is implemented as a simple CNN and the number of GRU layers is 1, which is in line with prior work [2] and the default setting of AllenAct.\n\n5) **Q**: \"Clarify the meaning of \"epochs with a rollout of data\" at L191. Does it refer to the number of update iterations using a rollout of data in the PPO implementation in AllenAct?\"\n\n**A**: “Epochs with a rollout of data” refers to the number of update iterations using a rollout of data in PPO.\n\n6) **Q**: \"Missing implementation details of the HRL baseline. Does the HRL apply all outputs from the sub-control policies in a single execution, or does it follow a traditional HRL policy where the master policy calls one of the sub-control policies at each step?\" \n\n**A**: In the HRL baseline, the master policy calls one of the sub-policies (with the same splits of action space of our model) at each step and the action is output by the sub-policy.\n\n7) **Q**: \"Clarify whether $r_{tp}$ at L227 is the same as $r_{sp}$ at L229.\" \n\n**A**: It’s a typo and both should be step penalty $r_{sp}=0.01$.\n\n8) **Q**: \"At L235, it is mentioned that $r_{nav}$ is used to learn the push and pick sub-control policies. Please clarify the requirements for task success, specifically if the agent needs to reach the target position.\"\n\n**A**: The reward for interactive auxiliary tasks should be $r_{inter}=r+r_{as}-r{af}$, where $r=r_{success}+\\Delta_{dis}-r_{sp}$ and $r_{success}$ is obtained when the goal of interactive task is achieved (taking Done when the obstacle is cleared).\n\n**References**\n\n[1] Zhang, Junzhe, and Elias Bareinboim. Markov decision processes with unobserved confounders: A causal approach. Technical report, Technical Report R-23, Purdue AI Lab, 2016.\n\n[2] Zeng, Kuo-Hao, et al. \"Pushing it out of the way: Interactive visual navigation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n"}, {"review_id": "mrg589ny0g", "reviewer": "Reviewer_KGF8", "summary": "Broadly, the paper tackles the Interactive Navigation task: navigating to a goal and interacting with obstacles as necessary, e.g. pushing a chair out of the way. \n\nThey use the ProcTHOR simulator with 12k multi-room scenes and generate navigation episodes that are suitably cluttered with obstacles. Their embodiment is abstract, with a discrete high-level action space (e.g. PushLeft, PickUp, Drop, RotateLeft).\n\nTheir approach is a hierarchical model, with three pre-trained low-level \"action policies\" (skills): navigate, push, and pick. A master policy sequences the three. Specifically, the navigation policy is run by default; the master policy can interrupt it by invoking an interaction policy (push or pick); the interaction policy then runs exclusively until it self-terminates, returning control to the master policy.\n\nTheir main contribution is the addition of an intent policy; they claim this helps the master policy make better decisions in the presence of obstacles. I'll describe this more below.\n\nThey evaluate their approach against several baselines:\n* random actions\n* a monolithic (non-hierarchical) sensors-to-actions policy trained with end-to-end RL, labeled as PPO\n* a prior approach labeled as NIE that aims to predict state change of observed objects\n* a hierarchical policy that uses the same nav/push/pick action subspaces as CaMP but differs in how they are combined\n* a monolithic sensors-to-actions policy with the addition of CaMP's intent policy\n\nThey also study ablations related to the intent policy.\n\nI'll now describe the intent policy in detail, with the caveat that I'm looking for further clarification from the authors here (see Questions section).\n\nThe architecture of the intent and master policy are mostly identical; their inputs include goal embedding and extracted visual features and they feature recurrent units.\n\nAt timestep t, the intent policy outputs w'.t, a distribution over the three action policies, essentially choosing between push/pick/navigate. Meanwhile, each action policy j ouputs a'.t^j, a distribution over discrete actions. Finally, the intent is the sum of all a'.t weighted by w'.t. For example, if the intent policy on a given step is biased towards push and the push action policy is biased towards the PushLeft action, the overall intent will be biased towards PushLeft.\n\nThis intent is fed to the master policy alongside observations (together, \"intent-specific state\"), which chooses an action policy (either allowing the nav policy to continue, or interrupting by invoking push or pick).\n\nThe intent policy and master policy generally share parameters and generally behave similarly, except the intent policy is synced from the master policy only periodically during training, such that in practice the intent will differ from the taken action some of the time.\n\nThey describe this intent-informed policy as \"exploring counterfactuals\", and a component here is learning a value function for the above-mentioned intent-specific state.", "questions": "In Figure 5 and in your video at 0:33, we see the agent making only 90-degree turns. However, the PPO baseline visualized in Figure 5 shows other turn angles. Can you explain this? Is your CaMP agent trained using a more restricted action space (versus action space in baselines)? If so, does this make your comparison results to baselines less clear? I would speculate that restriction to 90-degree orientations would simplify learning particularly in your scenes, which appear to be mostly axis-aligned (Figure 4; walls and most furniture). And a related question: 5.1 mentions \"adjacent nodes\". Is there a discrete structure to the position of objects, e.g. a grid?\n\nCan you explain a bit about the object dynamics of ProcTHOR, specifically, how does the simulator decide which objects are nonmovable, pushable, or pickable? And can you speculate on how the trained CaMP policy is able to predict this? I.e. is it overfitting to a specific object set, generalizing based on apparent object size in the image sensors, or something else?\n\n>Once an interaction policy is called, it does not return the control until the sub-task termination (output Done).\n\nCan you elaborate on how this works? For example, does the master policy include a recurrent unit, and does this unit receive continuous observations even while an interaction sub-task is running? Related: it would help to modify Figure 3. As-is, Figure 3 suggests that the master policy is free to select any action policy on any step.\n\n>To ensure that the intent represents the decision of the agent, we synchronize the parameters of two networks (π'Ω ← πΩ) during training.\n\nIn between these syncs, is the intent policy being updated during the learning phase of PPO, or is it frozen?\n\nCan you discuss sim-to-real? How might CaMP be used in a real-world setting?\n\nRelated to sim2real, can you also address one specific concern? First, a few assumptions:\n\n= you wish to keep the same hierarchical architecture\n\n= you could build low-level push/pick/nav policies that work on a real robot\n\n= you require most training of the master policy to happen in simulation due to the required scale of experience-collection\n\nMy specific sim2real concern: from 0:29 in the video, it appears that the push subtask doesn't move the agent or change its view (it's a \"magic push\"). The agent isn't required to approach the object in a certain way or move itself to effectively push the object. So, I speculate that this makes it remarkably easy for the master policy to chain navigation and pushing: the navigation policy essentially resumes after push completion from exactly its last pose, almost as if the obstacle were teleported away. Chaining real-world nav and push subtasks in this way would likely fail due to the push task moving the agent.\n\nI'd like greater clarity in section 3.2, on counterfactual decision-making and the basis for your intent policy and overall causal policy. A few specific questions: \n\n>Trained based on its own intent, the agent can obtain both experimental experiences (when a.t = i.t) and counterfactual experiences \n(when a/t != i.t), boosting the exploration of new strategies\n\nCan you elaborate? How does the causal policy architecture \"boost\" this exploration? How does this produce better exploration than, say, simply tuning temperature on a stochastic policy?\n\n>The intent also provides context about the obstacles due to their causal relation\n\nCan you elaborate? Can you experimentally verify/quantify what context the intent is providing about obstacles?\n\n>Intuitively, an agent that explores different counterfactual situations (e.g. \"what if I push/pick up the box instead of bypassing it?\") has a better understanding of the expected value of the current state, compared to the agent taking actions out of intuition.\n\nIt seems to me that a stochastic policy will try pushing, picking, and bypassing the box many times over the course of large-scale training and thus get this understanding of the expected value. What is the essential difference with your causal policy?", "limitations": "No concerns here", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The InterNav task is a challenging, relevant task for Embodied AI.\n\nThe task, scene/episode dataset, and model architecture are clearly communicated in figures and text, with minor exceptions noted below.\n\nThe scenes and episodes appear diverse and high-quality.\n\nBaselines and ablation study are rigorous.\n\nThe authors show strong performance against their baselines.", "weaknesses": "There's ambiguity on some paper details. I'll list bullets here; see my questions for more details:\n\n= action space and positioning of objects\n\n= object dynamics in ProcTHOR\n\n= details of invoking and terminating interaction subtasks\n\n= learning for the intent policy parameters\n\nThe paper doesn't discuss sim-to-real transfer or otherwise discuss how CaMP might be applied in a real-world setting. \n\nThe biggest weakness is that the intuition and conceptual value of the intent component aren't clearly articulated. For details, see my questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the detailed questions and reply to them in the following lines.\n\nQ1. Action space and positioning of objects.\n\nA1. Our CaMP agent is trained using exactly the same action space as other baselines. In particular, the agent makes only 90-degree turns when taking *RotateRight* and *RotateLeft*. The turning angles shown in Figure 5 are just for visualization by combining the trajectories of several adjacent steps. In section 5.1, we mention \"adjacent nodes\" referring to different doors on the path of cross-room navigation. As for the position of objects, all obstacles are spawned on the reachable points of agents, which are distributed on a grid network with a grid size of 0.25m.\n\nQ2. Object dynamics in ProcTHOR.\n\nA2. The interactive attributes of objects are decided according to their object categories (e.g. all tables are unmovable) in the ProcTHOR simulator. We speculate that CaMP policy learns about those attributes by generalizing based on both visual appearance from image and depth sensors and prior experience of interaction.\n\nQ3. Details of invoking and terminating interaction subtasks.\n\nA3. The master policy consists of a GRU and a MLP, and keeps receiving observations when an interaction sub-task is running, yet the output of the master policy will not influence the behavior of the agent. In Figure 3 we use green arrows to denote the call and control procedure of multi-policy and we will modify it to further clarify the transition of control.\n\nQ4. Learning for the intent policy parameters.\n\nA4. We keep the parameters of the intent policy frozen in between syncs since the intent produced by the intent network is just for replicating the decision of the master policy.\n\nQ5. Sim-to-real transfer and application in a real-world setting.\n\nA5. We appreciate the decent concern and agree with the assumptions. It's true that the sim-to-real gap of object interaction supported by ProcTHOR is quite big compared with that of navigation. However, we speculate that the main challenge of applying our method lies in interaction rather than navigation, since object manipulation and interaction dynamics are more complex to learn with limited training data in the real world. Meanwhile, in our hierarchical framework, each call of the nav policy can be regarded as a solo navigation heading toward the target with different starting positions, which reduces the complexity of long-term navigation. Recent development of simulators [1,2] makes the large-scale training for object manipulation with robotic arms possible. Thus we believe it's practical to transfer our method first to manipulation-included, simulated tasks and then to real-world settings to overcome the sim-to-real challenge.\n\n**Q6. Intuition and conceptual value of the intent component**. The reviewer questions about \"How does the causal policy architecture \"boost\" this exploration\", \"What is the essential difference with your causal policy\" compared with a stochastic policy, and \"what context the intent is providing about obstacles\".\n\nA6. We elaborate our idea of \"intent-aware counterfactual exploration\" as follows. \n\n* First, counterfactual policy boosts the exploration of new strategies by accumulating new experiences posterior to that from the original policy. During RL training, an experience used for loss calculation is commonly denoted as a tuple $e_t=\\left \\langle s_t,a_t,r_t,s_{t+1} \\right \\rangle \\in E$, where $r_t,s_{t+1}$ are determined by $s_t$ and $a_t$. Since the counterfactual policy is aware of intent that $s_t^{\\prime}=(s_t,i_t)$, experiences from it $E^{\\prime}$ include more information than experiences from a standard policy $E$. Moreover, the action distribution follows $P(a_t|i_t)=\\pi(s_t,i_t)$, which is posterior to the intent distribution $P(i_t)=\\pi(s_t)$. Thus $E^{\\prime}$ contains two sets: experimental ones $E_{exp}|\\_{a_t=i_t}$ that the original policy \"would have collected\", and counterfactual ones $E_{ctf}|\\_{a_t \\neq i_t}$ that are new experiences exploring other strategies. \n\n* Therefore, second, there are three advantages of counterfactual exploration compared with stochastic exploration:1) $s_t^{\\prime}$ provides more knowledge about the environment (discussed below). 2) While stochastic experiences are independently distributed, $E_{ctf}$ are collected based on the original strategy, making them more valuable. For example, an agent, who tries to push a box knowing its original intent is to bypass it, may learn that pushing is more effective in this situation since it receives a higher reward. Yet random trials can be hard to correlate and result in inefficient learning. 3) Stochastic experiences are not applicable to on-policy algorithms like PPO since random policy differs too much from the policy we are training. In contrast, counterfactual policy explores new experiences by merely changing the input (add intent) while maintaining the same policy function $\\pi$.\n\n* Third, the intent encodes knowledge about obstacles through agent's understanding of the environment. Since the intent shares the same causal parents of action (S $\\rightarrow$ I $\\leftarrow$ O in Figure 2), the policy network can be regarded as an encoder leveraging its prior experience. Thus intent is a proxy for influencing UC [3].\n\n**References**\n\n[1] Ehsani, Kiana, et al. \"Manipulathor: A framework for visual object manipulation.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.\n\n[2] Xiang, Fanbo, et al. \"Sapien: A simulated part-based interactive environment.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.\n\n[3] Forney, Andrew, Judea Pearl, and Elias Bareinboim. \"Counterfactual data-fusion for online reinforcement learners.\" International Conference on Machine Learning. PMLR, 2017."}, {"review_id": "EF4xGh1BOA", "reviewer": "Reviewer_6UQq", "summary": "This paper introduces the multi-room interactive navigation problem and proposes a novel model that is motivated by counterfactual reasoning. In particular, the paper posits that obstacle objects serve as a confounding factor when understanding the relationship between actions taken and the outcomes observed / reward received. To address this, a counterfactual reasoning based model is proposed, which explicitly encourages exploring actions outside the distribution predicted by the policy (i.e., what would happen if I did action X instead of Y?). The proposed model is hierarchical in nature, with low-level policies for navigation, picking, and pushing skills, and higher-level policies for selecting a skill to execute. The higher-level policy is conditioned on an intent, specifying where the policy is likely navigating to next. This is expected to allow the policy to account for the intent and explore actions that go against the intent.  Results on the ProcThor simulated dataset demonstrate the superiority of the proposed policy over alternative baselines.", "questions": "Kindly address the weaknesses stated above.", "limitations": "No limitations have been discussed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "* The idea of performing counterfactual reasoning is interesting and novel in the context of embodied AI.\n* The problem setting proposed is a good extension to prior work on single-room interactive navigation and is valuable for the community to work on.\n* The paper clarity is good, but it makes assumption about how knowledgable the reader is with causal inference (see weaknesses). \n* The experiments are well designed and ablation studies convey useful information to understand the overall model. The proposed model also performs much better than reasonable baselines.", "weaknesses": "# Post-rebuttal comments\n* The authors have sufficiently addressed my concerns and provided new experiments to quantify the improvement in interaction-ability. I'm happy to raise my rating to accept.\n\n--------------------------------------------------------------------------------------------------\n# Paper writing clarity\n* The paper writing clarity can be improved quite a bit with regards to causal inference. Since this is a relatively new topic to the embodied AI space, most readers may be unfamiliar with the topic and jargon like \"counterfactual reasoning\", \"confounding factors\", \"structural causal model\", \"structural functions\", etc. also, L129 was not obvious from my first reading (i.e., the difference between do(A) and A). \n* More clarity can be provided about what the counterfactual situations here are (e.g., more examples like L131-133). \n* In Figure 3, is the entire model differentiable? For example, are gradients from the loss propagated through to intent predictions w_t^{'} and a_t^{'}?\n* In the experiments, an explicit connection should be made to how the proposed model is using counterfactual reasoning and how it addresses the issue of confounding factors. \n\n# State vs. obstacles as confounding factors \nThe idea of treating obstacles as confounding variables makes sense. But at a high level, isn't the state variable itself a confounding factor? What is the value is isolating only obstacles here?\n\n# Experiment section can be improved\n* Error bars are missing in Tables 3 and 4. It will be useful to have results from training and evaluating on multiple seeds, especially for the top-3 methods (NIE, PPO+intent, CaMP). Similar issue for Table 4.\n* In Table 4, why is \"sync. /3*rollout\" worse than \"integrated intent\"? Aren't they both the same models?\n* In Table 4, why is \"sync. /epoch\" worse than \"wo/intent\"? If CaMP is synchronized frequently, I would expect the intent to be selected as the action itself, leaving little to no room for counterfactual exploration. So \"sync. /epoch\" should match \"wo/intent\", right?\n* There is no analysis on how well each models interact with the \"relevant\" obstacles, i.e., obstacles that lie along the shortest path and can be moved. This can be measured via precision and recall metrics. \n    * Precision = fraction of interacted objects that are \"relevant\" obstacles\n    * Recall = fraction of \"relevant\" obstacles interacted with", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks to the reviewer for the appreciation and suggestions for our work. We address the concerns in the following lines.\n\nQ1. Paper writing clarity. The reviewer questions about \"the difference between do(A) and A\", \"what the counterfactual situations here are (e.g., more examples like L131-133)\", and \"In Figure 3, is the entire model differentiable?\"\n\nA1. Thanks for the suggestion and we will revise our paper to further elaborate on the causal preliminaries. Our replies to the questions are as below:\n\n* Do-calculus $do(A=a)$ is an intervention operation that actively sets the variable $A$ as $a$ in the system while keeping the rest of the system unchanged, and the postintervention distribution $P(R|do(A=a))$ is obtained. On the contrary, the conditional distribution $P(R|A=a)$ is observed passively in the condition when $A$ appears to be $a$. Thus the actions executed by embodied agents can naturally be regarded as interventions do(A) to the environment. Formally in our case (Figure 2(a)), distribution $P(R=r|do(a))=\\sum_{x} P(r|a,s)P(s)$, while $P(R=r|a)=\\sum_{x} P(r|a,s)P(s|a)$. Since the agent decides the action $a$ based on the observation of state $s$, the distributions of $a$ and $s$ are not independent, which results in $P(s)\\neq P(s|a)$ and $P(R=r|do(A=a))\\neq P(R=r|A=a)$.\n* The counterfactual situation we discuss $R_{A=a}|A=i$ can be understood as the result the agent would obtain had it taken action $a$, given that it intends to take $i$. For example, in the initial stage of training, when encountering a table the agent may intend to bypass the obstacle since it lacks interaction skills and interactions empirically lead to low rewards. With the help of counterfactual policy, the agent may explore the counterfactual situation when it tries to push the table aside given its intent is to rotate right. Once the interaction successfully clears the path, the agent will obtain a relatively higher reward than navigation, which becomes a valuable experience for the agent to learn interactive strategy.\n* The entire model is differentiable, because the integrated intent $i_t$ is obtained with\nthe weighted-sum of low-level action distribution. However, we do not update the intent network with gradients from the loss. Instead the parameters of the intent network are updated by synchronization with the master policy.\n\nQ2. State vs. obstacles as confounding factors. The reviewer questions that \"But at a high level, isn't the state variable itself a confounding factor? What is the value is isolating only obstacles here?\"\n\nA2. We address obstacles in isolation since they are crucial Unobserved Confounders (UC) with negative impact while the distribution of state is observable through visual observations, although state $S$ indeed is a confounder of causality $A \\rightarrow R$. In essence, the agent aims to learn and optimize the state-specific causal effect $E[R|do(A=a),S=s]$ (L127), which represents the reward expectation when the agent takes action $a$ observing state as $s$. Therefore the causality $S \\rightarrow R$ in Figure 2 is also useful for policy learning and should be considered. However, the distribution of obstacles can not be measured by policy $\\pi(s)$, which is harmful to the learning of effective policy.\n\nQ3. Experiment issues. The reviewer argues that \"Error bars are missing in Tables 3 and 4.\", \"There is no analysis on how well each models interact with the \"relevant\" obstacles\", and questions about the model performances.\n\nA3. Thanks for the concerns and we address them by items as follows:\n* We report the performances of models (in Table 3,4) by averaging their scores on 5 tests with the same random seed. We now evaluate several methods on **multiple seeds** in the table below (in the form of *mean* $\\pm$ *variance*). The results show that the methods obtain stable performances on multiple seeds.\n* In the ablation study, we vary the sync frequency on model \"ppo+intent\" (L264) in order to study the effect on counterfactual policy and remove the influence of applying multi-policies. Thus in Table 4, \"sync./3*rollout\" corresponds to \"ppo+intent\" while \"integrated intent\" is identical to CaMP.\n* According to the clarification above, the \"sync. variants\" are modified based on \"ppo+intent\" while the \"intent variants\" are modified based on CaMP in Table 4, which makes the performance of \"sync./epoch\" relatively worse than \"wo/intent\".\n* In InterNav, the agent attempts to interact with the \"relevant\" obstacles for the purpose of navigating more efficiently along a shorter path. Thus how well the model interacts with obstacles actually lies in how much the interactions shorten the path. To that end, we apply SPL metric to measure the length of the path the agent takes to complete the task. \n\nFurthermore, to answer the question \"Does the shortest path mean the most efficient navigation?\", we consider an example where a large-size table lies in front of the agent on the shortest path, with an apple on the right side of the table. Obviously, moving the table aside is harder than picking up the apple and may be less efficient. Thus we additionally evaluate a metric STS (Success rate weighted by Time Steps, see our reply to *reviewer HVkt* for more details) to analyze the time efficiency of task completion in the table below. The results on SPL and STS show that our method obtains better path and time efficiency compared with the baselines.\n\n|Methods|SR (%)|SPL($_{\\pm e-5}$)|FDT($_{\\pm e-3}$)|STS($_{\\pm e-5}$)|\n|:----:|:----:|:----:|:----:|:----:|\n|PPO|42.3$_{\\pm 0.57}$|0.249$_{\\pm 1.98}$|4.86$_{\\pm 1.56}$|0.139$_{\\pm 0.87}$|\n|NIE|51.3$_{\\pm 1.01}$|0.290$_{\\pm 2.45}$|3.86$_{\\pm 4.22}$|0.157$_{\\pm 2.22}$|\n|PPO+intent|53.0$_{\\pm 1.93}$|0.296$_{\\pm 2.20}$|3.80$_{\\pm 1.71}$|0.163$_{\\pm 1.14}$|\n|CaMP|**56.7**$_{\\pm 1.19}$|**0.309**$_{\\pm 2.75}$|**3.71**$_{\\pm 1.97}$|**0.176**$_{\\pm 1.02}$|"}]

Author rebuttal:

Meta-review: This work introduces a new dataset of multi-room interactive navigation tasks, along with a causal, hierarchical, multi-policy framework for addressing these tasks. The framework consists of a high-level policy, an intent policy, and sub-control policies. The authors conduct extensive benchmarking and analysis of their framework on the ProcTHOR platform. 

The reviewers highlight several strengths of this work, including:
* The dataset is timely, challenging, and interactive, and provides a standardized platform for the embodied AI community.
* The causal, hierarchical, multi-policy framework is empirically effective, as demonstrated by the extensive benchmarking.
* The motivation and presentation of the approach are clear and instructive.

The AC concurs with the unanimous recommendation of accepting this submission. 

The authors have constructively discussed several writing and technical clarifications, new results, new data splits, etc. with the reviewers. These changes would improve the value and impact of the paper. A minor edit, that was missed in the reviews, could be a more inclusive re-naming of the current high-level ("master") policy.
Decision: Accept (poster)