Learning Active Camera  for Multi-Object Navigation
Authors: Peihao Chen
Dongyu Ji
Kunyang Lin
Weiwen Hu
Wenbing Huang
Thomas H. Li
Mingkui Tan
Chuang Gan
Venue: NeurIPS 2022 Accept (2022)
Keywords: Multi-object navigation
Visual indoor navigation
Active perception

Abstract: Getting robots to navigate to multiple objects autonomously is essential yet difficult in robot applications. One of the key challenges is how to explore environments efficiently with camera sensors only. Existing navigation methods mainly focus on fixed cameras and few attempts have been made to navigate with active cameras. As a result, the agent may take a very long time to perceive the environment due to limited camera scope. In contrast, humans typically gain a larger field of view by looking around for a better perception of the environment. How to make robots perceive the environment as efficiently as humans is a fundamental problem in robotics. In this paper, we consider navigating to multiple objects more efficiently with active cameras. Specifically, we cast moving camera to a Markov Decision Process and reformulate the active camera problem as a reinforcement learning problem. However, we have to address two new challenges: 1) how to learn a good camera policy in complex environments and 2) how to coordinate it with the navigation policy. To address these, we carefully design a reward function to encourage the agent to explore more areas by moving camera actively. Moreover, we exploit human experience to infer a rule-based camera action to guide the learning process. Last, to better coordinate two kinds of policies, the camera policy takes navigation actions into account when making camera moving decisions. Experimental results show our camera policy consistently improves the performance of multi-object navigation over four baselines on two datasets.

Learning Active Camera for Multi-Object Navigation
Peihao Chen 1.6 Dongyu Ji' Kunyang Lin Weiwen Hu' Wenbing Huang
Thomas H. Li Mingkui Tan Chuang Gan
South China University of Technology,
Pazhou Laboratory, MIITIBBM Watson AI Lab, 4UMass Amherst,
$ Gaoling School of Artificial Intelligence, Renmin University of China, Information Technology R&D Innovation Center of Peking University. Key Laboratory of Big Data and Intelligent Robot, Ministry of Education, p gmail.com, mingkuitan @ @scut.edu.cn
Abstract
Getting robots to navigate to multiple objects autonomously is essential yet difficult in robot applications. One of the key challenges is how explore environments efficiently with camera sensors only. Existing navigation methods mainly focus on fixed cameras and few attempts have been made to navigate with active cameras. As result, the agent may take very long time to perceive the environment due to limited camera scope. In contrast, humans typically gain larger field of view by looking around for a better perception of the environment. How make robots perceive the environment as efficiently as humans is a fundamental problem in robotics. In this paper, consider navigating to multiple objects more efficiently with active cameras. Specifically, we cast moving camera to a Markov Decision Process and reformulate the active camera problem learning problem. However, we have to address two new challenges: 1) how to learn good camera policy in complex environments and 2) how to coordinate it with the navigation policy. To address these, we carefully design a reward function encourage the agent explore Moreover, we exploit human experience to infer a rule-based camera action to guide the learning process. Last, to better coordinate camera policy takes navigation actions into account when making camera moving decisions. Experimental results show camera policy consistently improves the performance of multi-object navigation over four baselines on two datasets.
reinforcement
areas by moving camera actively.
kinds of policies, the
Introduction
In the multi-object navigation task, intelligent embodied agent needs navigate to multiple goal objects in a 3D environment. Typically, no pre-computed map is available and the agent needs to use stream of egocentric observations ability is the basis for indoor robots and embodied AI. Significant recent progress on this problem be attributed the availability of large-scale visually rich 3D datasets [71, 7, 66, 69, 32, 29], developments in high-quality 3D simulators [71, 61, 46, 2, 30), and research on deep memory- based architectures that combine geometry and semanties for learning representations of the 3D environment [8, 68, 58, 31].
perceive the environment. This navigation
Despite these advances, how to efficiently perceive the environment and locate goal objects is still an unsolved problem. Agents in current research [9, 12, 49] perceive the environment via an RGB-D camera. However, the camera is set look forward and the range of its view is often limited. As
*Coresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Active- Camera Agent
things
Update map
Current goal is front of me. Unexplored area on
What should the action
Existing Agent
Goal-1 Goal-2
Agent
Occupled Free
Go look forward Perceive redundant informatiin
Update map
Unexplored Camera direction Move direction
Figure 1: Illustration of an active-camera agent. The active-camera agent turns its camera the left actively to look for novel things, i.e., Goal-2, when walking towards Goal-1.
result, these agents can only perceive the environment in front of themselves within or slightly beyond the view range [58]. As shown at the bottom-right in Figure 1, the agent keeps looking forward and perceives redundant information which is nearly identical the previous observations. we human beings may be attracted by surrounding things so that we turn our heads actively to receive more information when keeping walking straight. this way, we can not only reach the position in front of ourselves efficiently but also get familiar with room arrangement and object location.
contrast,
Motivated by the above observations, design agent which seeks to coordinate both camera and navigation actions. The agent, called active-camera agent, employs two kinds of policies, namely the navigation policy which determines where to go and a new camera policy which determines where to look. As shown at the top-right in Figure I, the agent has found an object, i.e., blue Goal-1, located the end of a corridor. Two kinds of policies cooperate and actuate the agent turn its camera to the left. As result, the agent walks closer to the Goal-1 and locates another goal, i.e. yellow Goal-2, for the following navigation process. However, is nontrivial to learn good camera policy because of sophisticated RGB-D observations and complex indoor layouts. How to extract useful information from observations to judge which direction is worth being explored is difficult. Besides, the agent and the camera are moving simultaneously. How to coordinate these two actions for both the navigation and exploration processes still unknown.
forward and
In this paper, we propose an Explloratio--Oriente (EXPO) camera policy to determine camera actions. Specifically, to better understand the sophisticated RGB-D observations, transform them to top-down occupancy map. Each pixel indicates whether it is an unexplored, free or occupied area. Such a map helps to simplify unstructured RGB-D observations and provides the necessary information, e.g., explored areas and room layouts, for determining camera actions. Besides, to reduce the learning difficulty, we propose a heuristic module infer an expected camera direction according to heuristic rules. We consider the heuristic module an expert and exploit it to guide the learning process. We then feed three types of information, i.e., the progressively built map, the expected camera direction, and the upcoming navigation action, into neural network to predict camera action. The neural network considers both the heuristic rules and the navigation intention. We use reinforcement learning algorithm to train the neural network by awarding camera actions that maximize the exploration area. We incorporate the EXPO camera policy with existing navigation policies [58, 68] and train them in end-to-end manner for the multi-object navigation task. Extensive experiments on two benchmarks demonstrate the effectiveness of the proposed methods.
Our main contributions are threefold: 1) Unlike existing agents that are set to look forward, we propose a navigation paradigm that an agent coordinates camera and navigation actions for efficiently perceiving environments to solve the multi-object navigation task. 2) We propose to learn an EXPO camera policy to determine camera actions. The camera policy leverages heuristic rules to reduce the learning difficulty and takes into account the navigation intention to coordinate with the navigation policy. Such camera policy can incorporated with most existing navigation methods. 3) Extensive experiments demonstrate consistent improvement over four navigation methods on MatterPort3D [7] and Gibson [71] datasets. More critically, the camera policy also exhibits promising transferability to unseen scenes.

---

2
Related Work
Visual indoor navigation. According to the types of goal, visual navigation tasks can be cate- gorized into different classes such as PointGoal [33, 9] task where the goal is given coordinate, ObjectGool [33, 68] task where the agent needs to navigate to an object given by language and ImageGoal [12, 74] task where the object is specified by an image. The location of the goal is not explicitly known in the last two categories of the above tasks and the agent is expected to explore the environment to find the goals. Classical approaches [28, 67, 34, 63, 45] solve the navigation problem via path planning [11, 34] on a constructed map, which usually requires handcraft design. Recent works [74, 53, 73, 65, 68, 15, 64, 50, 56, 38] aim to use learning- based methods in end-to-ent manner to learn a policy for navigation. Other works [9. 12, 58, 43, 47, 5, 48, 18, 52, 17, 27, 60, 23] combine the classic and learning- baasing-basedd methods, which use a learned SLAM module with spatial map or topological map. We propose a camera policy, which seeks to move the camera actively and can be incorporated with existing navigation methods, to improve the navigation performance.
Exploration for navigation. Common methods explore the environment based on heuristics like the frontier-based exploration algorithm [72, 35, 24J. which chooses a frontier point between explored and unexplored areas as the exploration goal. Recent works tackle the exploration problem via learning [9, 21, 6, 25, 22, 57, 36, 44, 54J, which allows the agent to incorporate knowledge from previously seen environments and generalize to novel environments. Specially, Chen et al. [21] and Jayaraman al. [40] use end-to-end reinforcement learning policy maximize external reward (i.e., exploration area). Burda et al. [6] and Dean et al. [22] consider intrinsic rewards such as curiosity for efficient exploration, which performs better when external rewards are sparse. Chaplot et al. [9], Ramakrishnan [58] and Chen el al. [16] infer an exploration goal by a learned policy and navigate it using path planner, which avoids sample complexity problem in end-to-end training. Elhafsi et al. [26] predict the map out of view range for better path planning. Unlike existing methods, we try actively control the camera direction for efficient exploration.
Active perception. Active perception [3] aims to change the state parameters of sensors according to intelligent control strategies and gain more information about the environment. Common methods guide the view selection using information terooottttterticccc approaches, such as mutual information [42, 13, 14, 37J. Recent work applies this strategy different tasks such object recognition [41, 1, 39], object localization [10, 74, 51, 55] and scene completion [40, 59]. We refer readers to [20] and [4] for detailed review. In this paper, we propose a camera control strategy for active camera moving to help multi-object navigation.
3 Multi-object Navigation with Active Camera
3.1 Problem Formulation
Considering an agent equipped with one RGB-D camera in a novel environment, the multi-object navigation task asks the agent to explore the environment to find multiple goal objects and then navigate to them. During these processes, existing methods [68] design a navigation policy to process the observations o from the environment and infer navigation action a" (i.e., FORWARD, TURN-LEFT, RIGHT, and FOUND). The navigation action is responsible for moving toward goal objects and indicates whether agents have found goals. However, these methods do not consider moving the camera direction actively during the navigation process. Thus, agents can only perceive the field along their navigation direction, which causes low efficiency in finding goal objects [58].
To resolve the above problem, we propose a new navigation paradigm that consists of a navigation policy (-) and a camera policy (-). We reformulate both the navigation and camera moving process Partially Observable Markov Decision Process, where the observation o received by the agent does not fully specify the state of the environment. At each time step, for given state estimated from o, these two policies predict navigation action a" T T"(+) together with camera action a ~ (-) for active camera moving. The action space A for a" is similar to the navigation action space. The possible camera action includes [TURN-CAMERAL --LAAA.EERREEETT....... TURN-CAMERA --AMEAA-RIGHT,,
and KEEP], where KEEP indicates the camera direction remains unchanged. After performing the action, the policies will receive a reward r whose details can found in Section 3.3. We call the agent using this paradigm active-camera agent and the general scheme is shown in Figure 2.
3

---

Environment
Observation
Exploration-Orientee Camera Policy
RGB
Map
P Policy Network
Heuristic Direction
Actor
Camera Action
Mapper
Heuristic Module
GRU
Critic
"look left" 0
Depth
Navigation
Existing Navigation Policy ()))
Action
at forward" X
@
Ocuuped
Unexplored
Free
Agent
Move Direction
Camera Direction
Encoder
Figure 2: General scheme of an active-camera agent. The agent consists of two policies, namely the exploration-oriented camera policy determining where to look and existing navigation policy [58, 68] determining where to go.
3.2 Learning Determine Camera Action
Learning a camera policy for determining camera actions is challenging because 1) it is difficult to understand complex room structure from RGB-D observations; 2) the position of agents is changing all the time so camera policies must coordinate with navigation policies. In this paper, we design an EXPO camera policy that consists of three components, L.e., mapper, heuristic module, and policy network. The mapper transforms the RGB-D image to an occupancy top-down map, which reveals the room layout and location of unexplored areas straightforwardly, The heuristic module infers a heuristic direction that is worth exploring. The inferred direction can be served as reference for camera policy and reduce the learning difficulty. Then, a policy network predicts camera actions considering the encoded features of the map, heuristic direction, and upcoming navigation action. The upcoming navigation action informs the policy network of the next location of agents. The overview of the camera policy is shown in Figure 2. Next, we introduce each component in detail.
Mapper. We follow existing work [58] build an occupancy map by mapper. Each map pixel has three possible values, i.e., 0, 1, indicating an unexplored, free or occupied area, respectively. Specifically, the mapper takes RGB-D as input and predicts a local map M'. This local map represents small area in front of the camera. Then we integrate the local map into a global map M via registration function R(-), Lee, M = R(M', M9, q), where & is the camera's 's pose representing its position and orientation. The global map covers the whole explored environment.
Heuristic module. We seek to find a heuristic direction indicating which area is worth being explored by using heuristic rules. The principle is to find direction where the camera sight is not blocked by occupancy (e.g., wall). Following this principle, we draw K points
1 uniformly around the agent. Each point is located at angle 0; from the current navigation direction and r meters away from the agent. We calculate geodesic distance D (p(p:) ())))), K from agent to these points on the progressively built global map M via A path planning algorithm D(-) [34]. During path planning, the geodesic distance is smaller than threshold 7 (7 > r) and the point pt locates at unexplored areas in the global map, we denote these points explorable because there is no obstacle locating between the agent and point p;- Moving the camera to these explorable directions allows the agent to see unexplored parts of the environment. If there exist such explorable points, we select one point closest to the camera direction and take its angle 0, as the heuristic direction 0* Otherwise, set 0* 0, i.e., facing the navigation direction.
we consider both the free and unexplored areas navigable.
Policy network. With the progressively built map and inferred heuristic direction, we use recurrent network to aggregate this information at each time step. The network takes as input map features, heuristic direction features, and navigation action features. Specifically, a convolutional neural network is exploited to extract map features from an egocentric map which is cropped from the global map. The other two types of features are encoded from the heuristic direction 0* and navigation action a", respectively by using a learned embedding layer. The output state features from the recurrent network are fed into an actor-critic network. The actor predicts a camera action and the critic predicts the value of the agent's 's current state.

---

Algorithm Training method for active-camers agent.
Require:
The parameters of the navigation policy x" r-(-) and camera policy x x²()), mapper m(-)). the number of points K, the radius r, the distance threshold y, map registration function R(-), A algorithm D(-),
angle selection function S(.).
1: Initialize the parameters of πr" (-) and x (-) randomly.
2: Initialize an occupancy global map M 0.
3: while not convergent do
4: Collect observation o from environment.
5: // Update the global map
6: Let local map M' = m(o) and M = R(M', M2, where g is agent's pose.
7:
Crop an egocentric map M" from M
8: Il Infer heuristic direction
9:
Obtain points P= {p:) being meters away from agent's 's position angles θ= {0:|0;- 2mi ) Calculate geodesic distance from the agent these points D (d. on M
11:
Select index set of explorable point {i [= d, < 7))
12:
Obtain heuristic direction SS(6s, E Z}), where S returns the closest direction to current camera direction.
13:
I/ Sample action and update policies
Sample navigation action a" n²" (-lo) and camera action a x (- ,0" ,*"."").
15:
Compute reward via Eq. (I).
16:
Update the navigation and camera policies via PPO.
17:
end while
3.3 Reward Function for Camera Policy
We expect the camera policy to help agents explore environments more efficiently. We achieve this objective in two ways. On the one hand, we encourage the camera policy to follow the heuristic rules by awarding actions that move cameras toward the heuristic direction. This heuristic reward is defined the reduction of in successive time steps, i.e., Theuristic 8t-1 1 8y. The E [0?; 180%] represents the angle between the camera direction and the heuristic direction. On the other hand, because is impractical design heuristic rule cover all situations,
explicitly encourage the camera policy to explore more areas by an area reward Farea- G. C--1. which indicates the increase of explored area in successive time steps. The camera agent makes its own decision according recommended direction and occupancy map information. In addition, to avoid constant camera moving and reduce energy consumption, we expect agents to execute camera motor for camera moving as less frequently as possible by introducing a turn -ennalyy reward l'urn-penaltyy = 1l [ttum-cammraa, where 1 |fturn-cameri equals when the agent actuates camera motor move a camera and otherwise equals to O. The detail about camera motor execution will be described in Section 3.4. To sum up, the reward for camera policy as follows:
a Theuristic B Faren Trumm-pnnltt:
where and | are two scaling apprrpparamttrs.. We train the camera policy to maximize such exploration reward using PPO [62]. In this way, the EXPO camera policy is encouraged to take into account heuristic rule-based decisions and to explore more areas by executing the camera motor as less frequently as possible.
3.4 Combination of Navigation and Camera Policies
Navigation and camera actions execution. We assume the camera is attached to mobile robot platform using a revolute joint. The robot can move to a location using a wheel actuator and turn the camera using camera motor simultancously. The action space in this paper is based coordinates. The rotation angles for navigation and camera actions are set to the same. Thus, if both navigation and camera actions tend to move to the same direction, e,g., turning the platform and camera to the left, we only need to actuate the robot wheel and do not need to actuate the camera motor.
real-world
Incorporating camera policy into existing navigation methods. Existing navigation methods can be mainly categorized into two types, namely modular SLAM-based [58, 72] and end-to-end learning-basse [68, 70]. For the first type of navigation method, we use the existing navigation policy to infer a navigation action, which is then fed to the proposed camera policy to infer camera
5

---

action. The paradigm is shown in Figure 2. In this case. it is worth mentioning that the camera policy coordinates with the navigation policy by conditioning the output of the navigation policy. The navigation policy decides navigation action by exploiting a progressively built map, which is built from historical camera policy output. For the end-to-end learning-basee the existing navigation policy typically contains the same recurrent network as our camera policy and uses it to predict navigation action. To better coordinate two kinds of policies, we use one recurrent network to predict both navigation action and camera action. Specifically, we feed this recurrent network both camera policy input (i.e., map and heuristic direction) and navigation policy input (typically RGB-D images). Unlike the paradigm in Figure 2. the recurrent network in this case does not need the features of navigation action anymore because it predicts both types of actions at the same time. We use the summation of exploration reward in Equation and navigation reward in existing methods (typically the distance reduced to goals) to evaluate the goodness of predicted navigation-camera actions. The paradigm is shown in Figure ?? in Supplementary Materials.
navigation method,
4 Experiments
4.1 Experimental Setup
Task details. We follow MultiON [68] to place multiple goal objects in the environment randomly. The objects are embodied with cylinders in different colors. In this way, We are free to decide the location and the number of goal objects to adjust the task difficulty. The agent succeeds in an episode if it calls FOUND action within a 1.5 meters radius of all objects in order. We call FOUND automatically when the agent near the current goal object because we focus on evaluating the effectiveness of the agent finding objects and navigating to them. By default, we place three goal objects and denote 3-ON task. We also show results of 2-ON and 1-ON in Supplementary Materials. We perform experiments on two photorealistic 3D indoor environments, i.e., Matterport3D [7] and Gibson [71].
We follow the existing work [68] to evaluate the navigation success rate using Success and Progress metrics. Success indicates the percentage successfully finished episodes, while Progress indicates the percentage of objects being successfully found. We also evaluate the navigation efficiency using SPL and PPL, which are short for Success weighted by Path Length and Progress weighted by Path Length, respectively. The weight is proportional to the navigation efficiency and is defined as d/d, where d is the length of the ground-truth path and d is the path length traveled by an agent.
Implementation details. For navigation and camera actions, a FORWARD action moves the agent forward by 0.25 meters and TURN action turns by 30". The maximum episode time step is 500 because an agent with a global ground-truth map (i.e. oracle agent) finishes more than 97% of episodes within 500 steps. Our camera policy tries to narrow the performance gap between such an oracle agent and the agent with a progressively built map. We set K = 8, r = 2.4, y = r 1.2 2.88 in heuristic module empirically. We use mapper that outputs the ground-truth of occupancy anticipation [58]. because how to train good mapper is orthogonal to our work. Reward scaling factors a and B3 are set to 10 and respectively such that three reward terms are in the same order of magnitude at initialization. We evaluate models for five runs using the same set of random seeds and report the mean results and standard deviation. More details are shown in Supplementary Materials.
4.2 Baselines
Mapping + FBE [72]: This SLAM-based navigation method breaks the problem into mapping and path planning. We use depth projection [21] for mapping and trontier-bounday--exppee (FBE) method [72] to select an exploration point. Once the built map covers goal objects, the agent navigates to them by taking deterministic actions [8] along the path planned by A algorithm [34].
OccAnt [S8]: This baseline is the same as the previous one except that we replace the depth projection with an occupancy anticipation neural network [58]. Such network can infer the occupancy state beyond the visible regions. We assume the neural network is well-traines so that we use the ground- truth occupancy state within the field of view for experiments.

---

Table 1: Multi-object navigation results (%) for incorporating camera policy into different baselines on Matterport3D and Gibson datasets.
MatterPori3D
Gihson
Method
SPL
PPL
Success
Progress
SPL
PPL
Success
Progress
OceAnt
53.0+0.0
57.7+0.1
72.0=0.1
80.2 #0.1
16.1
77.8+0.1
89.0+0.1
91.8+0.1
Naive Camera Action
48.7
53.1
69.120.6
76,820.2
69.920.6
71.920.4
84,8
88.3
Our Camera Policy
27.9
62.120.2
75.6-0.5
82.620.2
78.
80.7
89.9
92.4+0.5
Mapping + FBE
40.1
45.4
62.3_0.0
12.5
62.1
64.9 64.910.2
80.910.4
86.210.4
+ Naive Camera Action
3
41.2+1.0
55.3-1.4
66.9+1.3
55.8+0.5
58.5+0.4
77.5+0.6
83.1+0.3
Camera Policy
44.6
19.8
6..2.0.9
74.1 10.3
68.7
71.2
84.410.1
88.9
MultiON
33.0+0.5
43.8+0.6
-0.6
60.5+00.
56.5+0.3
62.2+0.2
68.4+0.1
77.3++0.0
Naive Camera Action
32.422..1
25.1
41.9=0.1
60.2
5..1
61.620.0
64.5
75.240.2
Our Camera Policy
38.7=007
49.5=00.
51.1=0.0
67.3+0.3
59.620.3
66.8+0.2
69.1=0.3
79.020.1
DD-PPO
16.7.2.22
29.2:0.2
22.2-001
40.9
30.1
41.210.4
39.4
54.810.5
Naive Camera Action
16.7+o.1
30.4+0.1
20.8
39.2
31.3+00
43.4+0.2
38. 38.8+0.6
54.5
Our Camera Policy
19.1
34.0.004
24.0
43.9
33.9
45.310.3
40.5.0..
55.310.3
DD-PPO [70]:
This end-to-end learning- based baseline performs navigation using RL algorithm. It consists of a recurrent policy network, which takes as input RGB-D images, goal object categories and previous actions for predicting navigation action.
MultiON [68]; This is the variant of DD-PPO, with progressively built object map as an extra input. Each cell of the object map is one-hot vector indicating the existence of the goal objects. We store an object on the map once it is within the field of view of agents. We encode the egocentric cropped object map and feed it to the policy network. This bascline is the same as OracleEgoMap variant presented by [68].
4.3 Multi-object Navigation Results
Results on Matterport3D dataset. In Table 1, the agent with our EXPO camera policy performs better on multi-object navigation task upon four baselines. Specifically, our agent increases Success and Progress metrics for large margin, indicating incorporating our camera policy helps agents successfully navigate to more goal objects. We attribute the improvement to a better exploration ability of our agent. With such ability, the agent finds more goal objects in limited time step and then navigates to them. Besides, the improvement on SPL and PPI. indicates our agent navigates to goal objects along shorter path. The agent does not need to walk inside all rooms. Instead, the actively moving camera allows the agent to perceive what is inside a room when passes by. The above results show our agent navigates goal objects more efficiently with higher success rate. We encourage readers to see the visualization in Figure 4 and watch supplementary videos.
The camera action provides more movement freedom for the agent. We are interested in the question that whether the improvement comes from simply extending the action space. To this end, we remove our camera policy. The agent determines camera actions naively. Specifically, for ILAA-based baselines, the agent chooses camera action randomly. For learning-based baselines, the agent learns to predict both camera and navigation actions using the original navigation input and rewards. In Table 1, using these naive camera actions brings little improvement or even negative influence. This is not surprising because it is nontrivial to coordinate the camera and navigation actions. Also, larger action space may increase the learning difficulty. These results further suggest the importance of the proposed camera policy for determining reasonable camera action.
Transferability of camera policy Gibson dataset. We evaluate the transferability of the learned camera policy on Gibson dataset. In Table 1, a similar trend is observed on all baselines, i.e., using naive camera actions does not help for navigation while our EXPO camera policy performs better than baselines. It is worth noting that the EXPO camera policy is trained on Matterport3D and has not been fine-tuned on Gibson dataset. There are significant differences between these two datasets in scene style and layout distribution [7, 71]. The consistent improvement demonstrates that our active-camera agent has learned general exploration and navigation skills for the multi-object navigation task. also shows the possibility of transferring the agent to a real-world scene.
7

---

Table 2: Ablation study input information and reward types of camera policy based on OccAnt baseline.
Table 3: Comparison between rule- camera actions and learned camera policy based on two types of baselines.
Method
SPL.
PPL.
Success
Progress
Method
SPI.
PPL.
Success
Progress
OceAnt
52.940.0
57.7
72.026.1
80.2
woo Map Input
55.910.7
60.1
73.2.07
80.3.03
Random Action
48.7+0.5
53.140.4
69. 69.1+06
76,8+022
w/o Heuristic Ingut
56.510.6
60.410.1
74.0.0.
80.7.00
Swing Action
55.6101
60.1400
13.1
81,4
Ileuristic Action
56.3101
60.316.1
74.0161
80.91o1
woo NavAction Input
16.9
60.7,0a
74.7.00
81.2 _ea
Learned Policy
57.920.
62.1402
75.610.
82.6102
who Houristic Reward
53,8+0..
58.120.
72.75077
80,3g04
MaliiO
33.010.
43.8
44.1
60.5105
weo Aren Reward
16.910.a
61.020a
74.3
81.4g05
Random Action
3.1
10.2.9.
4.8.u.
1981
we Tum Reward
57.2
61.1205
74.6
81.7.04
Heuristic Action
Swing Action
13.020.
19.9+0.7
25.4
32.2+6.4
19.7:1.
27.6408
41.1214
46,3404
Ours
57.9
62.1+0.a
75.6-03
82.6-02
Learned Policy
38.7
49.5.0.
51.1c0
67.3-oa
4.4 Further Analysis
Rule-based camera actions vs. learned camera policy. In contrast to learning a neural network, one may use handcrafted rules to decide camera actions. We compare the learned camera policy with three types of rule-based camera actions, i.e., 1) selecting a random camera action; 2) forcing the agent to look forward and swing within 90 around the navigation direction; 3) following the heuristic direction inferred from the heuristic module.
The results upon a SLAM-based baseline, i.e., OccAnt, are shown in Table 3. Exploiting random camera actions drops the performance because the agent often looks backward and captures redundant useless information. The other two types of rule-based camera actions improve the performance slightly. These camera actions help the agent build map covering more areas. Consequently, the SLAM-based navigation policy can plan a better path for navigation using a path-planning algorithm. However, it is hard for us design a robust rule covering all situations. For example, the swing camera actions may miss some explorable areas because the agent has passed by these areas before the camera swing to the direction pointing to them. Also, there exist false positive areas in the occupancy map (e.g., free space behind a table is predicted as an obstacle). These areas may mislead the heuristic module to consider an unexplored area as an explored one. Compared with these camera actions, our learned camera policy brings more significant improvement. We attribute the improvement to the exploration reward in Equation 1. With such a reward, the agent is encouraged to take into account not only the handeraft rules but also the noisy occupancy map to predict better camera action.
As for learning- based baseline, i.e., MultiON, using these three types of rule-based camera actions significantly drops the performance. In these experiments, it is worth noting that we have fed previous camera action to inform the policy in which direction the observation is taken from. We suspect the poor performance is because the camera movement decided by rules is uncontrollable by the navigation policy. As a result, the navigation policy can not get desired observations for predicting the next navigation action. In contrast, the learned camera policy, which is trained together with the navigation policy, allows the agent to determine how to move its camera by itself. The above results further demonstrate the importance of the learned EXPO camera policy.
Ablation study on camera policy inputs and rewards. We conduct this ablation study by removing one of the inputs and rewards of a camera policy upon OccAnt baseline on MatterPort3D dataset. In Table 2. removing any input or reward will drop the performance. We note that human knowledge (ie., heuristic input and reward) is important for the camera policy. Awarding the policy to follow this knowledge can be considered as form of regularization and guidance for learning. The egocentric occupancy map input and area reward are also critical. They encourage the agent to explore more areas. With a better exploration ability, the agent can find goal objects and navigate to them more efficiently. The turn-penalty reward has little influence on navigation performance. However, it helps to reduce the frequency of actuating camera motor described in Section 3.4. Experimental results show that without this reward, agents actuate the camera motor for 12.41% % of total time steps. Adding turn-penalty reward decreases the number to 5.70%.
Does camera policy work with imperfect mapper? Our EXPO camera policy obtains indoor layout information mainly from the progressively built map. In this subsection, we would like to evaluate whether the proposed camera policy works with a noisy map. To this end, we use a learned neural network [58] to predict the occupancy map from RGB-D images. The predicted map contains many false positive points (e.g., predicting the free space occupancy) and false negative points (e.g..
8

---

40
1-26
Sucves Kate (%)
Explored Arta (ta))
Ours Baseline
30
-0
20
40
60
80
100
20
403
64
50
100
Step
Step
Figure 3: Exploration results. Left: success rate of finding three objects. Right: explored areas within limited time step. Standard deviation are negligibly small. - w/o camera action. - Random camera action. - Our camera policy.
Figure Visualization of multi-object naviga-
tion. The start position: O. Goal order:
E, E. Camera direction: *. Navigation di-
rection: When only is displayed, the camera and navigation directions are the same.
predicting a wall as free space). These noisy points may mislead the camera policy to make a wrong camera action. Experimental results on the OccAnt baseline show that incorporating camera policy brings consistent improvement. Specifically, SPI. and PPL increase from 13.6 to 18.1 and from 20.5 to 25.4, respectively. Success and Progress increase from 18.39 to 24.0 and from 28.4 to 34.2, respectively. These results demonstrate that the proposed EXPO camera policy is robust to the map noise and can be deployed a robot with no need for a ground truth map.
Does improvement come from extra information? Compared with baselines, our active-camera agent leverages extra information (i.e., occupancy map, heuristic direction and exploration reward) for determining camera actions. For fair comparison, we add this extra information into MultiON to build an enhanced baseline. Results on MatterPort3D dataset demonstrate that the enhanced baseline performs slightly better than MultiON baseline but worse than our active-camera agent. Specifically, SPI., PPI., Success and Progress are 35.1, 45.9, 48.0, and 63.2, respectively. We suspect the improvement of the enhanced baseline comes from the fact that heuristic direction provides location information about unexplored arcas. Also, the extra rewards encourage agents to explore these areas. However, due to the limited action space, the agent in enhanced baseline can not coordinate their camera and navigation actions well, which limits the performance.
Exploration performance. One of the critical abilities for the multi-object navigation task is exploring the environment efficiently to locate goal objects. To evaluate the exploration ability, we place the agent in a novel environment with three goal objects located in different places. We follow FBE method [72] to explore the environment. Given a limited time step budget, we evaluate the success rate of finding all three goal objects and explored areas in Figures 3. Compared with the baselines that the agent is always looking forward or moving its camera randomly, the agent with our EXPO camera policy finds more goal objects and explores more areas. These results have the same trend of navigation performance in Table 1, suggesting that our EXPO camera policy helps to explore the environment more efficiently and consequently boosts the multi-object navigation performance.
Visualization. In Figure 4, both baseline (i.e., OccAnt) and our agent are navigating to Goal-I at the beginning. During this process, our agent moves its camera actively at time step t = 16 and t = 26, finding Goal-2 and Goal-3 respectively. Knowing the location of goal objects, our agent plans the shortest path for navigation. In contrast, the baseline agent goes straight to Goal-1, with the camera looking forward constantly. Consequently, after it navigates to Goal-1, cannot find other goal objects and has to waste time exploring the environment again. Failure case analysis can be found in Supplementary Materials.
5 Conclusion
In order to solve the uncoordinated camera-navigation actions problem of existing agents, we propose navigation paradigm in which agents can dynamically move their cameras for perceiving environments more efficiently. Such exploration ability important for multi-object navigation. To determine the camera actions, we learn a camera policy via reinforcement learning by awarding it to explore more areas. Also, we use heuristic rules to guide the learning process and reduce the learning difficulty. The proposed camera policy can be incorporated into most existing navigation methods. Experimental results show that our camera policy consistently improves the multi-object navigation performance of multiple existing methods on two benchmark datasets.

---

Acknowledgments
This work was partially supported by National Key R&D Program of China (No. 2020AAA00069000, National Natural Science Foundation of China (No. 62072190 and No. 62006137), Program for Guangdong Introducing Innovative and Enterpreneurial Teams (No. 2017ZT07X183),
References
[1] P. Ammirato, P. Poirson, E. Park, Kosecka, and C. Berg. dataset developing and benchmarking active vision. In ICRA, pages 1378-1385, 2017. 3
[2] P. Anderson. Wu, Teney, J. Bruce, M. Johnson. N. Sünderhauf, D. Reid, Gould, and A. den
Hengel. vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. CVPR, pages 3674-3683, 2018,
[3] R. Bajesy. Active perception. Proceedings of the IEEE, 76(8):6661005, 1988.
[4] R. Bajesy, Aloimonos, and K. Tsotsos. Revisiting active perception. Autonomous Robots, 42(2):177-
196, 2018.
[5] E. Beeching, J. Dibangoye, O. Simonin, and C. Wolf. Learning plan with uncertain topological maps. In ECCV, pages 473-490, 2020.
[6] Y. Burda, H. Edwards, D. Pathak, A. J. Storkey, T. Darrell, and A. A. Efros. Large-scale study of vuriosity-drive learning. In ICLR, 2019. 3
A. X. Chang, A. Dai, T. A. Funkhouser, M. Halber, M. Nießner, M. Savva, S. Song, A. Zeng, and Y. Zhang. Matterport3d: Learning from RGB-D data indoor environments. In 3DV, pages 667-676, 2017. 1, 2,
[8] D. Chaplot, D. Gandhi, A. Gupta, and R. R. Salakhutdinov. Object goal navigation using goil-oriented semantic exploration. In NeurlPS, 2020. 6,
[9] D. S. Chaplot, D. Gandhi, S. Gupta, A. Gupta, and R. Salakhutdinov. Learning to explore using active neural SLAM. In ICLR, 2020. 1,3
[10] D. Chaplot, Jiang, Gupta, and A. Gupta. Semantic curiosity active visual learning. ECCV, pages 309-326, 2020. 3
[11] D. S. Chaplot, D. Pathak, and J. Malik. Differentiable spatial planning using transformers. In JCML, pages 1484-1495, 2021.
[12] D. S. Chaplot, R. Salakhutdinov. A. Gupta, and S. Gupta. Neural topological SLAM for visual navigation. In CVPR, pages 12872-12881, 2020. 1,
[13] B. Charrow, G. Kahn, Patil. Liu. K. Goldberg, P. Abbeel. N. Michael. and V. Kumar. Information- theoretic planning with trajectory optimization for dense mapping. RSS, 2015. 3
[14] B. Charrow, S. Liu, V. Kumar, and N. Michael. quadratic mutual information. In ICRA, pages 4791-4799777 14998, 2015. 3
cnormatiionthhoorete mapping using cauchy-schwar
[15] C. Chen, 7. Al-Halah, and K. Grauman. Semantic audio-visual navigation. CVPR, pages 15516-15525,
2021. 3
[16] C. Chen, S. Majumder, Z. Al-Halah, R. Gao, S. K. Ramakrishnan, and K. Grauman. Audio-visual waypoints for navigation. CoRR, abs/2008.0966 2020. 3
[17] K. Chen, J. K. Chen, J. Chuang, M. Vázquez, and S. Savarese. Topological planning with transformers for vision-an -ad---nnd-llngugee navigation. In CVPR, pages 11276-11286, 2021.
[18] K. Chen, P. de Vicente, G. Sepulveda, F. Xia, Soto, Vázquez, and Savarese. A behavioral approach to visual navigation with graph localization networks. In RSS, 2019. 3
[19] P. Chen, D. Ji, K. Lin, R. Zeng. T. H. Li, M. Tan1, and C. Gan. Weakly-supervised multi-granularity map learning for vision- and language navigation. In NeurlPS, 2022.
[20] S. Chen, Y. Li, and N. M. Kwok. Active vision in robotic systems: A survey of recent developments. International Journal of Robotics Research, (O(11):1343-137) 2011.
[21] T. Chen, S. Gupta, and A. Gupta. Leariing exploration policies for navigation. In ICLR, 2019. 6,6
[22] V. Dean, S. Tulsiani, and A. Gupta. See, hear, explore: Curiosity via audio-visual association. In NewrlPS, 2020. 3
[23] Z. Deng, K. Narasimhan, and O. Russakovsky. Evolving graphical planner: Contextual global planning for vision-and-language navigation. NeurlPS, pages 20660-20672, 2020.
[24] C. Dornhege and A. Kleiner. A frontier-void based approach for autonomous exploration in 3d. Advanced Robotics, 27(6):459 468, 2013.
10

---

[25] Y. Du, C. Gan. and P. Isola. Curious representation learning for embodied intelligence. In ICCV, pages 10408-10411 2021.
[26] A. Elhafsi, B. Ivanovic, L. Janson, and M. Pavone. Map-predictive motion planning in unknown environ-
ments, In ICRA, pages 8552 8558, 2020. 3
[27] K. Fang, A. Toshev, I. Fei-Fei, and S. Savarese. Scene memory transformer for embodied agents in long-horizon tasks. In CVPR, pages 538-547, 2019. 3
[28] J. Fuentes- Pacheeo J. R. Ascencio, and J. M. Gendón-Mancha, Visual simultaneous localization and mapping: survey. Artificial Intelligence Review, 43(1):55-81) 2015. 3
[29] C. Gan, Y. Gu, S. Zhou, J. Schwartz, S. Alter, J. Traer, D. Gutfreund, J. B. Tenenbaum, J. H. McDermott, and A. Torralba. Finding fallen objects via asynchronous audio-visual integration. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10523-10533, 2022.
[30] C. Gan, J. Schwartz, S. Alter, M. Schrimpf, J. Traer, J. De Freitas, J. Kubilius, A. Bhandwaldar, N. Haber, M. Sano, et al. Threedworld: A platform for interactive multi-modal physicalll simulation. NeurIPS, 2021.
[31] C. Gan. Y. Zhang, J. Wu, B. Gong, and J. B. Tenenbaum. Look, listen, and act: Towards audio-visual embodied navigation. In ICRA, pages 9701-9707, 2020.
[32] C. Gan, S. Zhou, J. Schwartz, S. Alter, A. Bhandwaldar, D. Gutfreund, D. L. Yamins, J. J. DiCarlo, J. McDermott, A. Torralba, et al. The threedworld transport challenge: A visually guided task-and-motion planning benchmark towards physically realistic embodied ai. In 2022 International Conference on Robotics and Automation (ICRA), pages 8847-8854, 2022.
[33] S. Gupta, J. Davidson, S. Levine, R. Sukthankar, and J. Malik. Cognitive mapping and planning for visual navigation. In CVPR, pages 1772, 2017.
[34] P. E. Hart, N. J. Nilsson, and B. Raphael. A formal basis for the heuristic determination of minimum cost paths. IEFF Transactions on Systems Science and Cyhernetics, 4(2):100-1077, 1968. 3,
[35] D. Holz, N. Basilico, F. Amigoni, and S. Behnke. Evaluating the efficiency of fronticr-based exploration strategies. In ISR, pages 1-8, 2010.
[36] R. Houthooft, X. Chen, Y. Duan, J. Schulman, D. Turck, and P. Abbeel. VIME: variational information maximizing exploration. In NewrlPS, pages 1109 1117, 2016. 3
[37] M. G. Jadidi, J. V. Miró, and G. Dissanayake. Mutual information based exploration on continuous occupancy maps. In IROS, pages 6086 6092, 2015.
[38] U. Jain, I. Liu, S. Lazebnik, A. Kembhavi, I.. Weihs, and A. G. Schwing. Gridtopix: Training embodied agents with minimal supervision. In ICCV, 2021. 3
[39] D. Jayaraman and K. Grauman. Look-ahead before you leap: End-to-end active recognition by forecasting the effect of motion. In ECCV. pages 489-505, 2016. 3
[40] D. Jayaraman and K. Grauman. Learning to look around: Intelligently exploring unseen environments for unknown tasks. In CVPR, pages 1238-1247, 2018. 3
[41] D. Jayaraman and K. Grauman. End-to-end policy learning for active visual categorization. JEEE Transactions on Pottern Analysis and Machine Intelligence, 41 417))1101-1114 2019.
[42] B. J. Julian, Karaman, and D. Rus. On mutual information-based control of range sensing robots for mapping applications. International Journal of Robotics Research, 33(10))1375-1333 2014. 3
[43] P. Karkus, Cai,, and D. Hsu. Differentiable slam-net: Learning particle SLAM for visual navigation. In CVPR, pages 2-255, 2021.
[44] K. D. Katyal, A. Polevoy, J. L.. Moore, Knuth, and K. M. Popek. High-speed robot navigation using predicted occupancy maps. ICRA, pages 5476-5482, 2021.
[45] J. Kim, E. Lee, M. Lee, Zhang, and Y. M. Kim. Sgolam: Simultaneous goal localization and mapping for multi-object goal navigation. CORR, abs/2110.07171, 2021. 3
[46] E. Kolve, R. Mottaghi, D. Gordon, Y. Zhu, A. Gupta, and A. Farhadi. AI2-THOR: interactive 3d environment for visual AI. CoRR, abs/1712.05474, 2017.
[47] E. S. Lee, J. Kim, and Y. M. Kim. Self-supervised domain adaptation for visual navigation with global map consistency. In WACV, 2022. 3
[48] Y. Liang, B. Chen, and S. Song. Sscnav: Confidence-awwre semantic scene completion for visual semantic navigation. In ICRA, pages 13194-13200, 2021.
[49] Q Luo, M. Sorokin, and S. Ha. A few shot adaptation of visual navigation skills to new observations using meta- leannng. In ICRA, pages
2021. 1

---

[50] P. Marza, L. Matignon, O. Simonin, and C. Wolf. Teaching agents how map: Spatial reasoning for multi-object navigation. CoRR, 0bs/2107.00011. 2021. 3
[51] S. Mathe, A. Pirinen, and C. Sminchisescu. Reinforcement learning for visual object detection. In CVPR, pages 2894-2902, 2016. 3
[52] X. Meng, N. D. Ratliff, Y. Xiang, and D. Fox. Scaling local control to large-scale topological navigation. In ICRA, pages 672-678, 2020. 3
[53] A. Mousavian, A. Toshev, M. Fiser, J. Kosecká, A. Wahid. and J. Davidson. Visual representations for semantic target driven navigation. In ICRA, pages 8846 8852, 2019, 3
[54] T. Nagarajan and K. Grauman. Learning affordance landscapes for interaction exploration in 3d environ- ments, In NeurIPS, 2020.
[55] D. Nilsson, A. Pirinen, E. Gärtner, and C. Sminchisescu. Embodied visual active lcarning for semantic segmentation. In AAAI, pages 2373-22--2 2373, 2021. 3
[56] S. Patel, S. Wani, U. Jain, A. G. Schwing, S. Lazebnik, M. Savva, and A. X. Chang. Interpretation of
emergent communication in heterogeneous collaborative embodied agents. In ICCV, 2021. 3
[57] D. Pathak, D. Gandhi, and A. Gupta. Self-supervised exploration via disagreement. In ICML, pages 5062-5071, 2019.
[58] S. K. Ramakrishnan, Z. Al-Halah, and K. Grauman, Occupancy anticipation for efficient exploration and navigation. In ECCV, pages 400-418, 2020. 1, 2, 3, 5, 6,
[59] S. K. Ramakrishnan and K. Grauman. Sidekick policy learning for active visual exploration. In ECCV, pages 424 442, 2018. 3
[60] N. Savinov, A. Dosovitskiy, and V. Koltun. Semi-parametre topological memory for navigation. In ICLR, 2018. 3
[61] M. Savva, J. Malik, D. Parikh, D. Batra, A. Kadian, O. Maksymets, Y. Zhao, Wijmans, B. Jain, J. Straub, J. Liu, and V. Koltun. Habitat: A platform for embodied AI research. In ICCV, pages 9338-9346, 2019.
[62] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. CoRR, abs//707 707.006377 017770033,,7777 2017. 5
[63] J. A. Sethian. A fast marching level set method for monotonically advancing fronts. Proceedings of the National Academy of Sciences of the United States of America, 93(4):1591 1996. 3
[64] Z. Seymour, K. Thopalli, N. C. Mithun, H. Chiu, Samarasekera, and Kumar. Maast: Map attention with semantic transformers for efficient visual navigation. In ICRA, pages 13223- 13230, 2021. 3
[65] W. B. Shen, D. Xu, Y. Zhu, F. Li, J. Guibas, and S. Savarese. Situational fusion of visual representation for visual navigation. In ICCV, pages 2881-2899 2019. 3
[66] J. Straub, T. Whelan, L. Ma. Y. Chen, E. Wijmans, S. Green, J. J. Engel, R. Mur-Artal, C. Ren, S. Verma, A. Clarkson, M. Yan, Budge, Yan, X. Pan, J. Yon. Zou, Leon, N. Carter, Briales, T. Gillingham, E. Mueggler, Pesqueira, M. Savva, D. Batra, H. M. Strasdat, R. Nardi, M. Goesele, S. Lovegrove, and
R. A. Newcomhe. The replica dataset: A digital replica of indoor spaces. CoRR, abs/190600579,, 2019.
[67] S. Thrun. Probabilistic robotics. Communications of the ACM, 45(3):52 27, 2002. 3
[68] S. Wani, S. Patel, U. Jain, A. X. Chang, and M. Savva. Multion: Benchmarking semantic map memory using multi-object navigation. In NewrlPS, 2020. 1, 3, 4, 6,
[69] L. Weihs, M. Deitke, A. Kembhavi, and R. Mottaghi. Visual room rearrangement. In CVPR, pages 5922-5931, 2021.
[70] E. Wijmans, A. Kadian, A. Morcos. S. Lee, I. Essa, D. Parikh, M. Savva, and D. Batra. DD-PPO: learning near-perfect pointgoal navigators from 2.5 billion frames. In ICLR, 2020. 5,7
[71] F. Xia. A. R. Zamir. Z. He, A. Sax. J. Malik, and S. Savarese. Gibson env: weal-world perception for embodied agents. In CVPR, pages 9068 9078,9999,9908 2018. 2, 6,7
[72] B. Yamauchi. frontier-based approach for autonomous exploration. In CIRA, pages 146-151, 1997. 3,5,
6.9
[73] Yang, Wang, A. Farhadi, A. Gupta, and Mottaghi. Visual semantic navigation using scene priors. In ICLR, 2019. 3
[74] Y. Zhu, R. Mottaghi, E. Kolve, J. J. Lim, A. Gupta, L. Fei-Fei, and A. Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In ICRA, pages 3357-3364, 2017. 3
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Please see Section ?? in the supplemental materials.
(c) Did you discuss any potential negative societal impacts your work? [Yes] Please see Section ?? in the supplemental materials.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [No] will publish
code upon acceptance.
(b) Did you specify all the training details (c.g., data splits, hyperparameters, how they were chosen)? [Yes] We have included the training details in the supplemental materials. (c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Ye]] We have run evaluation using 5 seeds and reported error bars.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We have included the details the computational resources in the supplemental material.
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] We have added the license in the supplemental materials.
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [No]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "bgkTbZHXX8m", "reviewer": "Reviewer_uqJh", "summary": "In this paper, the authors propose to learn the active camera policy for multi-object robot navigation. Specifically, they show how to use camera rotations effectively to increase the information while in motion, and how to integrate the active camera policy with the navigation policy. Authors train active camera policy using Reinforcement Learning (RL) and carefully design a reward function that encourages robots to explore more areas. Through multiple baseline methods composed of SLAM-based, and end-to-end learning-based, the authors show that integrating active camera policy with the base navigation policy improves object navigation accuracy and efficiency on multiple metrics.", "questions": "1. What is the egocentric map size and how did you choose it? How does the policy performance change with increasing / decreasing this map size? I believe increasing cropped egocentric map would increase % of an unexplored area of the input map, and it would be good to see how it affects the policy. \n2. For the experiment with an imperfect mapper, can the authors please also add what was the accuracy of such a trained mapper? \n", "limitations": "The authors have mentioned very generic limitations. Few assumptions of the paper should be part of the limitations as well.\n1. Along with testing the effectiveness of such a method in the real-world environment, it is also useful to mention the limitation that different types of robots might also have to experiment with environment representations. \n2. This active camera policy limits the use of a learning-based locomotion policy for high DoF robots such as Spot, and ANYMAL. Since the camera is not facing the robots' traversal path continuously. \n\n", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Paper is well written. The authors show clear motivation for the need for active cameras to aid exploration while navigating. They formulate the problem, propose a reinforcement-learning-based solution, and show the effectiveness of active cameras through extensive experiments.\n2. Trained Active Camera Policy (EXPO) generalizes to other datasets than the one it’s been trained on (MatterPort3d → Gibson)\n3. Authors show ablation study to show the usefulness of Active Camera Policy design components\n4. Authors show improvements on 4 different baselines applied to 2 datasets. Baseline methods consist of SLAM-based, end-to-end learning-based methods, and hybrid methods.\n\nWeakness:\n1. Since the method is primarily trying to increase the scan horizon while moving, the authors should provide some details on the camera field of view. While I understand this is standard for the datasets like Mattterport3D, and Gibson, to be comprehensive it is necessary to include the field of view in the paper.\n2. While an active camera is a great idea to scan an unexplored environment map while moving, the same can be achieved by fitting multiple cameras and ensuring we have a large field of view. Ideally, having multiple cameras covering a larger field of view will serve as an upper bound of what can be achieved.\n3. I encourage authors to add more limitations of the method as described in the limitations review section\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "RQba-I2AUB4", "reviewer": "Reviewer_iN1f", "summary": "This paper enables efficient multi-goal navigation through active perception.\nThe camera of the agent, which determines its limited field of view, is actively controlled to turn towards scene regions that are still unexplored.\nCamera rotation is decoupled from agent movement, i.e. the agent can still move in directions other than the facing direction.\nThe camera control problem is framed as an MDP with rewards proportional to spatial exploration of the environment (measured in uncovered area, based on a maintained 2D occupancy map).\nCamera actions are produced by a parametric policy optimized through PPO for the aforementioned MDP (with an explicit critic that approximates state values).\nThe policy is informed through a very strong heuristic for how the camera should turn, in the form of an extra objective term, but also seems to improve upon it (due to the refinement through the exploration objective from the described MDP).\nIt is demonstrated that actively controlling the camera leads to improved efficiency and success rate for the multi-goal navigation task on the MatterPort3D and Gibson data sets.\nBoth SLAM-based and end-to-end RL navigation baselines are augmented with the active camera policy in the experiments.\nGeneralization from MatterPort3D (used for training) to Gibson is demonstrated.\n\n**Technical details**\nThe proposed scheme is used to augment existing navigation methods that are already tailored to the multi-goal navigation task.\nThe camera policy is recurrent and is conditioned on features from a top-down 2D map. \nIt also takes in the hand-crafted heuristic camera control and the action produced by the movement policy as input (the latter is necesary only when a SLAM model is used, as camera and movement controllers are decoupled).\nWhen an end-to-end RL model is considered, the camera and movement policy are modelled by the same recurrent neural network, jointly optimizing an MDP with rewards for both navigation and exploration.", "questions": "**Questions**\n- The proposed method maintains a 2D occupancy map of the environment, which is necessary to compute the exploration heuristic. Where do the camera poses come from in the end-to-end learned RL experiments (for the SLAM-based navigation task I assume the SLAM estimates are used)? These poses are needed to update the occupancy map with the RGB-D observations.\n- Have you considered quantifying the exact way in which the trained camera policy deviates from the heuristic?\n", "limitations": "I don't see any major negative societal impact. There is a short section on limitations right before the conclusion that mentions the lack of real-world experiments. It could be extended based on the remarks above.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The approach appears straightforward & functional. The design choices are pragmatic.\n- The motivation is easy to understand and follow.\n- The fact that camera and motion control needs to be considered jointly has been accounted for, albeit only for the end-to-end learned systems.\n- Ablations of the method are appreciated, they indicate that the different modelling choices were necessary.\n- It is nice to see the policy generalizes from one data set to the other.\n\n**Weaknesses**\n- The authors discuss the camera policy in the context of an MDP, but are the agent states actually observed? My impression was that the multi-goal navigation task is a partially-observed problem, where the observations are only RGB-D images. In that case the ideal decision problem would technically be a POMDP, and this should be discussed in the manuscript. \n- Optimizing the camera policy in isolation of the overarching movement control problem can be problematic, as the authors have also noted in the experimental section. This has been accounted for in the end-to-end learned experiments, where a joint control task is formulated. But for the SLAM-based controllers, I am not convinced conditioning on the movement controls is enough. Particularly if you consider the ideal POMDP formulation, the movement controller should be aware of the images that will be perceived in the future (due to camera control), as this will affect the quality of state estimation (which both policies should account for). What is the reason to keep the controllers separated in the SLAM case?\n- In a POMDP the uncertainty of the state estimates normally plays a role, and I believe the agent states used to update the occupancy map are deterministic.\n- The method is evaluated in only two environments in simulation, real-world experiments have remained out of scope.\n\nOverall, I find that the proposed solution has value, the proposed idea is straightforward and seems to work as intended.\nThe solution is very specific in scope, tailored to the problem of actively controlling a camera for exploration, but on the other hand spatial exploration is an important topic and spatial agents are ubiquitous.\nThe other reservations I have are related to the missing discussion of how the method fits into the POMDP paradigm; I do not necessarily expect the method to change, but being explicit about its limitations would be appreciated.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "BIqCjZUEBoq", "reviewer": "Reviewer_pNpw", "summary": "The paper tackles the challenge of SLAM-based multi-object navigation for indoor\nrobots. This task is typically approached with an algorithmic or RL method which\ndrives a robot equipped with an RGB-D sensor through an unknown finite\nenvironment. The robot's goal is to find a series of predefined objects under a\nlimited step budget. \n\nThe paper makes the observation that prior approaches, such as Active Neural\nSLAM, assume a fixed camera robot configuration with a limited forward-facing\nFoV. The paper then proposes decoupling the robot motion from the camera yaw.\nThe robot can thus choose to, e.g., move in a direction while looking to the\nside if it decides this provides more information. Two policies are learned,\none for navigation, one for looking around, with the navigation decision\nfeeding into the camera policy. The navigation policy is also responsible for\ndetecting the goal.\n\nThe paper focuses on the camera policy, which is designed to work both on top\nof SLAM-based, as well as end-to-end navigation policies.\n\nThe method is trained on-policy using the photorealistic 3D environment\nMatterport3D. The camera policy can generalize to the Gibson environment without\nfine-tuning. In general, allowing the robot to move its camera independent of\nits movement direction improves over baselines like not moving the camera,\nmoving it randomly, or moving it according to predefined patterns. This\nimprovement is consistent when evaluated on top of multiple navigation methods.\nThe paper also includes ablation studies for some of the key design decisions.  \n", "questions": "- [Q1] What is the actuation noise in the environment? If the policy always expects\n  turning itself or its camera in exact 30° increments, suddenly rolling out in an\n  environment with noisy actuation may cause serious problems.\n", "limitations": "Please see \"Weaknesses\".", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- [S1] The paper demonstrates cross-domain generalization by training on\n  Matterport3D and successfully transferring the policies to the Gibson Env\n  without any fine-tuning. \n- [S2] The proposed method is also shown to help improve both a SLAM-based, as\n  well as a learning-based baseline.\n- [S3] The paper defines an interesting new direction in active sensing.\n\n## Weaknesses\n- [W1] Clarity: It took me a while to realize that the method is actually fully\n  compatible with both SLAM-based and end-to-end navigation systems. Would it be\n  possible to add a diagram, maybe in the supmat if the main paper can't fit it,\n  showing the specific information flow used when the end-to-end paradigm is\n  used in navigation?\n  - Either way, it would be nice to clarify this earlier in the paper, as it\n    makes it easier to understand how the camera policy fits into the existing\n    navigation pipeline.\n- [W2] Multi-Camera Baseline: While I do understand the point brought up in\n  Appendix A, I am not fully convinced by the argument.\n  - A multi-pinhole-camera robot could make for an interesting comparison, in\n    spite of the increased costs. A complex panoramic camera (with the\n    associated geometric and hardware complexity) is not strictly required. \n  - An additional advantage of multiple fixed camera is eliminating the need for\n    the moving parts of the revolute joint and its associated servo. \n  - Second, I wouldn't call the computational overhead of, say, four cameras\n    \"huge\", since naive baselines like concatenating the images before feeding\n    them to an end-to-end navigation policy would probably work pretty well\n    already.\n  - Multi-camera systems, albeit not depth-based, are already ubiquitous in mobile\n    phones, drones, and (even non-autonomous) vehicles.\n  - That being said, I still think it makes sense for this problem setting to\n    exist, as coordinating between motion and active sensing is generally an\n    important problem in robotics, which hasn't received as much attention as it\n    should have. \n\n\n## Suggestions\n- L194: \"module SLAM-based\" -> \"modular, SLAM-based\"?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept