SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Environment settings. Following the standard settings, we let MoveAhead move the agent ahead by 0.25 meters, RotateRight and RotateLeft change the agent's azimuth angle by  $\pm 90$  degrees, LookUp and LookDown rotate the agent's camera elevation angle by  $\pm 30$  degrees. The DirectionalPushs let the agent push (along  $\pm z$  and  $\pm x$  axis) the closest visible object with a constant force. The agent takes the END to indicate that it has completed an episode.
Evaluation metrics. We adopt Success Rate (SR), Final Distance to Target (FDT), and Success weighted by Path Length (SPL). SR is the ratio of successful episodes in total episodes, FDT is the average geodesic distance between the agent and the goal when the episode is finished, and SPL is calculated as  $\frac{1}{N}\sum_{n=1}^{N} \text{Suc}_n \frac{L_n}{\max(P_n, L_n)}$ , where  $N$  is the total episodes number,  $\text{Suc}_n$  is the successful indicator of  $n$ -th episode,  $L_n$  is the shortest path length, and  $P_n$  is the length of the real path.
In non-interactive navigation, the metric SPL that measures path efficiency is also a usable indicator of time efficiency, since the length of a path is usually proportional to the time it takes to execute the corresponding navigation actions (e.g. move ahead, rotate right). However, a shorter path does not guarantee less time consumption for task completion in InterNav due to the object interaction. For instance, the agent may take the shortest path to the target through extensive interactions which cost a large amount of time and end up less effective than taking a detour path. Thus we evaluate an additional metric STS (Success rate weighted by Time Steps) to measure the time cost of task completion:  $STS = \frac{1}{N} \sum_{n=1}^{N} \text{Suc}_n \frac{L_n / grid}{TS_n}$ , where  $L_n$  is the shortest path length,  $TS_n$  is the timesteps the agent takes to complete the task, and  $grid = 0.25m$  is the unit distance of a step. Thus  $L_n / grid$  represents the number of timesteps it takes to navigate to the target by merely moving forward. STS can be regarded as a time-measurement variant of SPL and the score is higher when the agent accomplishes the task with less time. The ideal situation is that the target is directly ahead and there is no need for interaction where STS is 1.
5.2 Implementation Details
Our method is implemented and evaluated using the AllenAct [34] framework. The egocentric observation is set as  $300 * 300$  RGB and depth images. We first train three action policies then the whole model, utilizing Adam with an initial learning rate of  $3 \cdot 10^{-4}$  for policy training and  $1 \cdot 10^{-4}$  for joint model training. The hidden size of CNN, GRU is set as 512. The intent embedding size is set as 12 (the same as the intent size and the length of action space). The frequency of synchronization  $K$  and epoch number  $k$  are set as 3 and 1. All models are trained for 10 million steps with 2 million warm-up steps in single-room scenes (1m for action policies and 1m for joint training) and 8 million in all scenes<sup>1</sup>.
Reward shaping. We set the reward for our task as three parts:  $r = r_{success} + \Delta_{dis} - r_{sp}$ , where  $r_{success} = 10$  is provided if the agent takes action  $Done$  meanwhile achieving the goal.  $\Delta_{dis}$  represents the difference of geodesic distance between the agent and the goal position, and  $r_{sp} = 0.01$  denotes the step penalty.
Three auxiliary tasks are designed as: (1) The Navigate policy is pretrained with the PointNav task where the agent moves to the goal point in an obstacle-free environment. The task shares the same reward  $r_{nav} = r$  as above. (2) To pretrain the policy of Push and Pick, the agent is placed aside an obstacle opposite to the goal. It achieves success by taking Done when the obstacle is cleared. The reward  $r_{inter} = r_{nav} + r_{as} - r_{af}$ , where  $r_{as}, r_{af}$  denote the successful reward and failed penalty of chosen action (e.g. fail to conduct Drop before picking up something). Since effective pushes or picks change the length of the shortest path,  $\Delta_{dis}$  also applies to interaction reward.
5.3Baselines
Random: This baseline randomly takes actions with the same probability until the agent reaches the target or reaches the maximum number of steps.
PPO: We set a common baseline trained with PPO algorithm according to [35]. Similar to our method, it extracts visual features with an encoder and reads the goal position with an embedding layer, then a single policy network predicts actions based on fused features.
NIE: We implement the method NIE [39] which predicts the state change of observed objects conditioned on actions with a module in addition to the PPO baseline. The representation of object change is fused with visual features and goal embedding.
HRL: To study the effect of our multi-policy framework, we implement a common hierarchical policy supported by AllenAct [34] that shares the same three sub-spaces of action. The difference is that the low-level policies are called at each step, not able to plan actions in a period.
PPO+intent: We develop a counterfactual policy based on the PPO baseline according to Figure 2(c) to study the effect of intent-aware decision-making.
<sup>1</sup>The project is available at: https://github.com/polkalian/InterNav.
Table 3: Quantitative results.
Methods	all	N≥4	non-interactive
SR(%)↑	FDT(m)↓	SPL↑	STS↑	SR	FDT	SPL	STS	SR	FDT	SPL	STS
Random	1.32	7.68	0.002	0.001	0	12.29	0	0	2.42	7.35	0.003	0.001
PPO [35]	42.7	4.84	0.252	0.134	23.3	7.79	0.139	0.086	51.5	3.44	0.306	0.181
NIE [39]	52.0	3.94	0.287	0.155	37.3	6.54	0.195	0.102	58.8	3.01	0.345	0.188
HRL	43.4	4.80	0.253	0.135	24.3	7.01	0.169	0.084	51.9	3.40	0.316	0.176
PPO+intent	54.7	3.84	0.305	0.163	38.0	6.29	0.198	0.109	70.4	2.38	0.390	0.222
CaMP	56.3	3.67	0.327	0.177	41.4	5.76	0.231	0.121	72.3	2.05	0.407	0.236
Table 4: Ablation results.
Methods	all	N≥4	non-interactive
SR(%)↑	FDT(m)↓	SPL↑	STS	SR	FDT	SPL	STS	SR	FDT	SPL	STS
Sync. variants:
sync. /epoch	53.0	4.16	0.316	0.160	36.7	6.95	0.196	0.107	63.4	2.53	0.375	0.219
sync. /rollout	50.7	4.22	0.277	0.145	36.0	6.82	0.175	0.105	57.9	3.10	0.345	0.191
sync. /3*rollout	54.7	3.84	0.305	0.163	37.3	6.54	0.195	0.109	70.4	2.38	0.390	0.222
sync. /5*rollout	50.3	4.25	0.279	0.150	30.7	6.85	0.147	0.098	55.9	3.21	0.337	0.185
intent variants:
wo/intent	50.8	4.03	0.294	0.151	31.0	6.79	0.151	0.095	59.2	2.98	0.347	0.202
test wo/intent	41.7	4.95	0.202	0.111	24.7	7.72	0.110	0.085	50.5	3.56	0.296	0.168
recursive intent	51.4	4.06	0.292	0.156	32.7	7.04	0.159	0.096	62.8	2.61	0.353	0.215
policy intent	52.5	4.32	0.304	0.155	35.8	7.09	0.191	0.100	62.6	2.91	0.357	0.216
integrated intent	56.3	3.67	0.327	0.177	41.4	5.76	0.231	0.121	72.3	2.05	0.407	0.236
5.4 Results Analysis
Comparison with baselines. We pick model parameters on the validation set and report their performance on the whole testing set, hard scenes with more rooms ( $N \geq 4$ ), and the splits of episodes that exist clear paths (non-interactive). As quantitatively shown in Table 3, CaMP outperforms all baselines and obtains  $+13.6 / +7.5$ ,  $+18.1 / +9.2$ , and  $+4.3 / +3.5$  improvement in SR, SPL, and STS ( $\text{all}/N \geq 4$ , %) respectively compared to the model PPO. Compared with PPO+intent, CaMP obtains more improvement in hard scenes ( $+3.4$  and  $+3.3$  in SR, SPL) which shows the robustness of multi-policy facing complex environment. Besides, the relatively poorer performances of HRL and Random model reflect the challenge of our dataset. The limited performance in hard scenes shows that effective methods for navigation in plenty of rooms are still in need. The contrary trends across STS and SPL when comparing PPO and HRL on the hard split and the non-interactive split indicate that both path and time efficiency should be measured considering InterNav scenarios.
Ablation study. Our ablation study aims to explore the effect of two model designs. First, we change the synchronization frequency of model PPO+intent as 1 epoch, 1 rollout, 3 rollouts, and 5 rollouts. As shown in Table 4, the variant with  $K = 3$  gains the relatively best performance, which reflects how the balanced exploration-exploitation benefits policy learning.
Second, we study how the model intent influences performance: We implement (1) CaMP without the intent network as wo/intent, (2) CaMP that inferences without the intent network as test wo/intent, (3) CaMP provided with the intent recursively generated based on the prior intent as recursive intent, and (4) CaMP provided with the intent of the global policy as policy intent. Here our model CaMP is referred to as integrated intent. As shown in Table 4, the integrated intent effectively improves the decision-making ability of the global policy compared with wo/intent  $(+5.5 / + 3.3$  and  $+10.4 / + 8.0$  in SR, SPL). Without the information of the intent of action policies, policy intent shows less gain effect compared with integrated intent (-3.8/-2.3 and -5.6/-4.0 in SR, SPL). The limited performance of test wo/intent suggests that the inference process of our model also relies on the intent. The results of recursive intent suggest that although the secondary intent still provides incremental information about the system, the counterfactual situation based on the "counterfact" may become less meaningful for learning causality.
Top view (scene_1999)
Figure 5: Qualitative cases. The starting and target points are marked by blue circles and obstacles are marked by yellow boxes. The blue arrows and red arrows denote the trajectories from CaMP and PPO, respectively. The nodes and crosses denote successful object interactions and failed ones. Markings in colors are invisible to agents.
Case study. We qualitatively compare our model with a baseline PPO on a typical instance in a 4-room scene in Figure 5. The blue trajectory of CaMP shows that the agent successfully removes obstacles of tomato and pillow, and attempts to bypass an immovable sofa to reach the target. While the red trajectory shows that PPO intends to interact with a table and a bed which are too heavy to be pushed aside and ends up getting stuck in a corner. The comparison suggests that our model can make better decisions on whether and how to interact with obstacles.
Table 5: Obstacle interaction results.
Methods	IR(m)	ISR(%)	PuSR(%)	PiSR(%)
PPO [35]	0.271	23.2	23.8	21.9
PPO+intent	0.371	27.0	25.1	28.6
CaMP wo/intent	0.287	20.4	19.9	23.5
CaMP	0.402	27.6	27.1	29.8
Study of obstacle interaction. To study how intent-aware policy learns the obstacle interaction and its outcome, we make a direct evaluation on obstacle interaction of several models in Table 5: (1) Interaction Reward (IR): the reduction of geodesic distance  $\Delta_{dis}$  caused by interactions. (2) Interaction Success Rate (ISR): the ratio of taking interactions successfully (a more detailed measurement compared to the overall metric SR measuring task completion). A failed case of interaction is when the agent tries to pick up a heavy table or attempts to push a chair out of sight. We also report the success rate of Push and Pick interactions separately as PuSR (Push Success Rate) and PiSR (Pick Success Rate). The results (comparison between PPO and PPO+intent, CaMP and CaMP wo/intent) show the outcome of our causal method in pursuing the interaction efficacy of reducing the distance to the target (IR) and avoiding invalid actions (ISR). The results on PuSR and PiSR also suggest that successfully pushing an obstacle is more challenging since the obstacle may get stuck and is not movable in all directions.
In this paper, we tackle the task of Interactive Navigation (InterNav) which requires mastery of long-horizon navigation and object interaction in complex environments. We take a causal view of InterNav and clarify the challenge caused by the confounding bias of obstacles. A Causal Multi-policy Planning (CaMP) model is proposed following the main idea of reducing inefficient exploration with multi-policy and exploring counterfactual interaction by considering the primary intent of model's decision-making. We construct a large-scale dataset of interactive navigation with various obstacles spawned across multiple rooms. Experiments on our dataset illustrate the effectiveness of our method and support our causal modeling of InterNav.
This work is supported by the National Key Research and Development Project of New Generation Artificial Intelligence of China, Grant Number: 2018AAA0102504. This work is in part by Beijing Natural Science Foundation under Grant JQ22012, and in part by the National Natural Science Foundation of China under Grant 62125207.
[39] Kuo-Hao Zeng, Luca Weihs, Ali Farhadi, and Roozbeh Mottaghi. Pushing it out of the way: Interactive visual navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9868-9877, 2021.
[40] Dong Zhang, Hanwang Zhang, Jinhui Tang, Xian-Sheng Hua, and Qianru Sun. Causal intervention for weakly-supervised semantic segmentation. Advances in Neural Information Processing Systems, 33:655-666, 2020.
[41] Junzhe Zhang and Elias Bareinboim. Markov decision processes with unobserved confounders: A causal approach. Technical report, Technical report, Technical Report R-23, Purdue AI Lab, 2016.
[42] Sixian Zhang, Weijie Li, Xinhang Song, Yubing Bai, and Shuqiang Jiang. Generative meta-adversarial network for unseen object navigation. In Computer Vision - ECCV 2022 - 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIX, volume 13699 of Lecture Notes in Computer Science, pages 301-320.
[43] Sixian Zhang, Xinhang Song, Yubing Bai, Weijie Li, Yakui Chu, and Shuqiang Jiang. Hierarchical object-to-zone graph for object navigation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 15130-15140, 2021.
[44] Sixian Zhang, Xinhang Song, Weijie Li, Yubing Bai, Xinyao Yu, and Shuqiang Jiang. Layout-based causal inference for object navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10792-10802, June 2023.
A.1 Proof of equation (3)
For a state  $s_t$  and intent  $i_t$  at step  $t$ , the optimal counterfactual policy  $\pi_{ctf}^* = \arg\max V_{\pi_{ctf}}(s_t, i_t)$  and the optimal standard policy  $\pi^* = \arg\max V_{\pi}(s_t)$ , we have  $V_{\pi_{ctf}^*}(s_t, i_t) \geq V_{\pi^*}(s_t, i_t)$ . For a standard optimal policy, since  $\pi^*$  executes its intent  $i_t$  as an optimal action  $a_t'$  without re-consideration, we have:
$$
\begin{array}{l} a _ {t} ^ {\prime} = \operatorname {a r g m a x} \mathbb {E} \left[ \sum_ {k = 0} ^ {\infty} \gamma^ {k} r _ {t + k + 1} \mid s _ {t}, a _ {t} \right] (8) \\ V _ {\pi_ {c t f} ^ {*}} (s _ {t}, i _ {t}) \geq V _ {\pi^ {*}} (s _ {t}, i _ {t}) = V _ {\pi^ {*}} (s _ {t}, a _ {t} ^ {\prime}) \\ = \mathbb {E} \left[ \sum_ {k = 0} ^ {\infty} \gamma^ {k} r _ {t + k + 1} \mid s _ {t}, a _ {t} ^ {\prime} \right] (9) \\ \geq \sum_ {a _ {t} \in A} \pi^ {*} (a _ {t} | s _ {t}) \mathbb {E} [ \sum_ {k = 0} ^ {\infty} \gamma^ {k} r _ {t + k + 1} | s _ {t}, a _ {t} ] \\ = V _ {\pi^ {*}} \left(s _ {t}\right) \\ \end{array}
$$
Thus considering the information of its own intent, the agent can learn a counterfactual policy gaining more value.
A.2 Impact of different intents on policy exploration
We train our model with PPO [28] which encourages the exploration with an entropy loss  $L_{t}^{S} = c_{2}S[\pi ] = -c_{2}\sum_{a_{t}\in A}P(a_{t})\log P(a_{t})$  where  $c_{2} < 0$  is the coefficient. Since our agent does trials conditioned on its intent, it maximizes the conditional entropy  $H(A|I) = -\sum_{a_t\in A}P(a_t|i_t)\log P(a_t|i_t)$ . We are curious about how this conditional exploration can make the new policy distribution different from the conditional distribution (intent). To study the question and also how the similarity between intent and action affects policy exploration, we evaluate the mean KL divergence between the current and old policy distributions under several settings throughout the training. In particular, we vary the frequency of synchronizing the action policy and the intent policy as 1 epoch, 3 epochs, 6 epochs, 9 epochs, 12 epochs, 15 epochs, and without intent. In Figure 6, we report the mean KL divergence between the distributions of current policy and different old policy (from 1 epoch behind to 15 epochs behind). The trend of each curve shows that the counterfactual information of intent encourages the agent to explore new strategies. The comparison between curves shows that the more different actions and intents, the more the model tends to explore new policies rather than exploit the old policy. Thus the frequency of synchronization can be utilized to adjust how much the agent explores new policies under counterfactual situations.
Figure 6: Mean KL divergence of the current and old policy distributions.