CAtNIPP: Context-Aware Attention-based Network for Informative Path Planning
Authors: Yuhong Cao
Yizhuo Wang
Apoorva Vashisth
Haolin Fan
Guillaume Adrien Sartoretti
Venue: CoRL 2022 Poster (2022)
Keywords: deep RL
informative path planning
context-aware decision-making

Abstract: Informative path planning (IPP) is an NP-hard problem, which aims at planning a path allowing an agent to build an accurate belief about a quantity of interest throughout a given search domain, within constraints on resource budget (e.g., path length for robots with limited battery life). IPP requires frequent online replanning as this belief is updated with every new measurement (i.e., adaptive IPP), while balancing short-term exploitation and longer-term exploration to avoid suboptimal, myopic behaviors. Encouraged by the recent developments in deep reinforcement learning, we introduce CAtNIPP, a fully reactive, neural approach to the adaptive IPP problem. CAtNIPP relies on self-attention for its powerful ability to capture dependencies in data at multiple spatial scales. Specifically, our agent learns to form a context of its belief over the entire domain, which it uses to sequence local movement decisions that optimize short- and longer-term search objectives. We experimentally demonstrate that CAtNIPP significantly outperforms state-of-the-art non-learning IPP solvers in terms of solution quality and computing time once trained, and present experimental results on hardware.

CAINIPP: Context-Aware Attention-based Network for Informative Path Planning
Yuhong Cao
Yizhuo Wang National University of Singapore wy980u nus edu
National University of Singapore caoyuhongêu. nus. edu
nus.
Apoorva Vashisth
Haolin Fan
Indian Institute of Technology. Kharagpur av19996iitkgn ic
National University Singapore e08162650u. nus. edu
Guillaume Sartoretti
National University of Singapore
mpegasunus.. eeweenuuueeduusggeee..
Abstract:
Informative path planning (IPP) is an NP-hard problem, which aims at planning path allowing an agent to build accurate belief about quantity of interest throughout a given search domain, within constraints on resource budget (e.g.. path length for robots with limited battery life). IPP requires frequent online planning as this belief is updated with every new measurement (i.e., adaptive IPP), while balancing short-term exploitation and longer-term exploration avoid sub- optimal, myopic behaviors. Encouraged by the recent developments in deep re- inforcement learning, introduce CAtNIPP, fully reactive, neural approach to the adaptive IPP problem. CAINIPP relies on self-attention for its powerful ability capture dependencies in data multiple spatial scales. Specifically, our agent learns to form a context of its belief over the entire domain, which it uses sequence local movement decisions that optimize short- and longer-term search objectives. We experimentally demonstrate that CAtNIPP significantly outper- forms -aate-ff-the-ar non-learning IPP solvers in terms of solution quality and computing time once trained, and present experimental results on hardware.
Keywords: deep RL, informative path planning, context-aware decision-making
Introduction
In many real-life robotic deployments that involve data acquisition, such as mapping/exploration of unknown areas for inspection or search-and-rescue applications, environmental monitoring, and surface inspection/reconstrutii [1, 2, 3], an autonomous robot needs to plan path to visit given domain and obtain measurements about sealar field of interest, without a priori knowledge of the true underlying distribution of this information. That is, starting from uniform distribution with high uncertainty, the agent must construct a belief over the distribution of interest throughout the domain (e.g., target likelihood, temperature, surface roughness) based on successive measurements along its path. This problem known as the informative path planning (IPP) problem. Specifically, IPP aims to plan path that maximizes information gain, while satisfying budget constraint (e.g., path length for robots with limited battery life). IPP problems can be further classified as either non-udaptive or adaptive. Non-adaptive solvers pre-plan complete path offline and execute this pre-determined path, without any replanning upon obtaining new measurements online [4, 5, 6). On the other hand, adaptive solvers replan the search path frequently as the agent's belief is updated based on new measurements [1, 2, 7]. While our approach can also be used for non-adaptive IPP, we focus on the more general adaptive IPP problem for its wider applicability to real-life robotic tasks.
Differently from general path planning problems, where the agent is often assigned a goal position, IPP requires the agent identify and visit all potential interesting areas throughout the environment.
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

Therefore, efficient IPP solvers must reason about the entire agent's belief to make non-myopic decisions [8], which balance short-term exploitation of known interesting areas with longer-tern exploration of unknown areas in the domain. Many IPP solvers rely on computationally expensive means to optimize long-horizon trajectories [4, 5, 7]. Trading off solution quality in favor of lower computing times, more recent approaches have embraced sampling- based planning [7, 9, 10].
To further improve computing time and solution quality, we in- troduce CAINIPP, a deep rein- forcement learning (dRL) based framework for 2D adaptive IPP. We first decrease the complexity of our continuous-space search domain by generating proba- bilistic roadmap [11], i.e., ran- dom sparse graph that covers the domain. We then associate this roadmap with the agent's belief, and formulate adaptive IPP as a sequential decision-making prob- lem on this graph. We propose and train aitentioo-bbsed neural network that outputs policy to select which neighboring node to visit next, thus allowing us to iteratively plan an (adaptive) informative path. There, self-attention over the graph nodes allows the agent to construct global context, by embedding its entire belief into local decision features while identifying dependencies between nodes/areas at different spatial scales. In doing so, the neural-based, reactive nature of our approach drastically improves planning time, compared to planners that optimize full trajectories, while context-awarenes, helps improve solution quality by allowing our agent to identify and sequence non-myopic decisions that near- optimally balance short- and long-term objectives. The main contributions of this work are:
(d) Figure 1: CAINIPP's trajectory sampling variant for adaptive IPP, showing the path executed by the agent so far (black), and the long-horizon trajectory that was just (re-) rreanplanned (white), of which the red portion will be executed until the next replanning step. a) True interest map (unknown to the agent), b) Predicted interest map from all measurements so far, c) Associated pre- dicted standard deviation, and d) Predicted high-interest areas.
(a)
(b)
(c)
We propose new fully reactive, policy-based dRL framework for adaptive IPP, which signifi- cantly outperforms sfate-of-the-art IPP solvers in terms of both solution quality and computing time. In CAINIPP, the agent learns subtle global representation of its entire belief over the do- main, that allows it to sequence non-myopic decisions that can achieve longer-term objectives.
We further propose to rely on receding-horizon, sampling massli based trajectory optimization (Fig. 1) to output higher-quality, longer-horizon trajectories by leveraging the nature of our stochastic policies, while remaining tractable and usable in real time. We demonstrate these variants of CAINIPP simulation as well as on hardware on a light-intensity-banl IPP task.
2 Related Work
To reduce the computing time and improve solution quality, most recent IPP methods have relied on randomized, sampling-based methods. For non-adaptive IPP solvers, Hollinger et al. [9] introduced the RIG-tree algorithm, which utilizes RRT-star (a Rapidly-exploring Random Tree variant [12]) to randomly build tree structure used to explore the environment and maximize information gain. Arora et al. [5] combined Constraint Satisfaction and Travelling Salesman Problems to introduce Randomized Anytime Orienteering (RAOr). RAOr iteratively samples the search space and solves a TSP instance to visit all sampled locations within the given budget constraints. According to Popović el al. [1], despite RIG-tree and RAOr not having been initially designed for adaptive IPP, they only require seconds to plan, which allows them to be generalized to adaptive online replanning scheme in a traditional receding-horizon manner. Regarding solvers initially designed for adaptive IPP, Hitz et al. [7] proposed an evolutionary strategy to achieve state-of-the-at performance. They applied CMA-ES to generate candidate solutions from multi-variate Gaussian distribution, where the mean and covariance matrices are adaptively updated according to the evaluation of the candi- dates. Apart from these conventional solvers, there has been a couple of recent, value-basee RL solvers for IPP. Wei et al. [6] proposed an Rase- solver, which reasons about the positions of measurement (for their predicted reduction uncertainty) but without considering measurement values, thus limiting its use to non-adaptive IPP problems. Ruckin et al. [13] proposed specialized CNN-based solver for 3D IPP, developed for a specific problem statement that assumes image-like measurements and relies on Kalman Filtering for belief update.

---

3 Background
In this section, we first introduce Gaussian Processes (GPs), which are used to model the agent's belief (i.e., predicted interest map). We then formulate the general definition of our IPP problem based on such a GP. Finally, we describe the adaptive replanning requirement for adaptive IPP.
Gaussian Process In IPP, interest (e.g., target distribution, temperature, or radiation level), is asso- ciated with the 2D environment & C R² and modeled as continuous function Ç ع - IR. Gaussian Processes have been widely used to represent such a continuous interest distribution, by providing a natural means to interpolate between discrete measurements [1, 6, 7]. so that Ç ~ G P). Specifically, given a set of n' locations X C at which interest is to be inferred, a set of n observed locations X' C & and the corresponding mcasurements set V, the mean and covariance of the GP are regressed as: / 1(2*)) + K(X", ( X) + T] - (((x)), P K ((X*, xx*) K(X* , X²) + - x K(X", X)) where (--) is pre-trained/selected kernel function, of is a hyperparameter describing the measurement noise, and is the n x n identity matrix. In this work, following [1, 10), we use the Matérn 3/2 kernel function.
Informative Path Planning The general IPP problem aims to find an optimal trajectory a in the space of all available trajectories for maximum gain in some information-theoreiee measures:
v = argmax I(1b), s.t. C(t) S B,
NEW
where 1 3 + + is the information gained from the measurements obtained along the trajectory d C 3 - Rt maps trajectory v to its associated execution cost, and B E Fº is the given path-length budget. Following [5, 6, 7]. the trajectory & is given a start and a destination but we note that our method can be easily extended to remove the need for destination. To evaluate the information gained from measurements, following [1, 4], we use the variance reduction of the GP to represent information gain: I(2² = Tr(P-)= - Tr(P.), where Tr(-) denotes the trace of matrix, P and p+ are the prior and posterior covariances, which are obtained before and after taking measurements along the trajectory v.. In this work, model the data collection of common sensors, we let the agent take a measurement every time it has traveled a fixed distance from the previous measurement, thus the number of measurement is only determined by the path length budget B.
Adaptive Replanning If the information gain only depends the covariance P, i.e., the location of measurements, the objective is considered non-adaptive since the trajectory could be entirely planned offline ahead of time, based on the agent's initial (often uniform) belief. However, in real- world applications such as search-and-rescue, we usually aim to discover regions of high interest and further cover (exploit) them. To this end, following [1, 7], rely on the upper confidence bound to define high-interest areas Xy: X2 = {z; E 1M, + 3P,, 2 Pub ). where 14 and P, are the prior mean and variance of the GP at the measurement location T₁. Mah and 3 E R+ are used to control the threshold and confidence interval respectively (Junh 0.4, = 1, in practice). By replacing X' with X; in the covariance calculation, we restrict the information gain in the objective function Eq. (1) to the high-value areas predicted by the GP. This formulation makes the IPP objec- tive dependent on the measurement values in addition to their location, making the problem truly adaptive. Therefore, frequent online replanning of the trajectory is required to minimize uncertainty in the (now dynamically defined) high-interest areas X7.
4 Method
In this section, we cast adaptive IPP as an RL problem and detail our attention-based neural network, as well our Long-horizon planning strategy to further boost the performance of a learned policy.
4.1 IPP as a RL Problem
Sequential Decision-making Problem First, to avoid the complexity associated with a continuous search domain, we rely on probabilistic roadmaps (PRM) [11] to build a route graph G = (V, E), nodes over the domain, and E set of edges. Each node e (zi, v:) E V is connected to its h nearest neighboring nodes and up is the destination. Then, to solve the adaptive IPP using RI, we formulate it as a sequential decision-making problem on this graph. That is, we let agent interact with the environment by choosing which node to move to from
with V a set of cintoommly-andoo--ampppl.
3

---

amongst the neighbors of its current node. Movement between nodes happens as straight line. As a result, the agent's 's trajectory & can be represented as an ordered set (4),, ui...., va), Vi; V. where ys and the denote the start and destination nodes respectively. As result, the trajectory is adaptively planned, since it is constructed from sequential movements, each depending on the agent's global belief, which gets updated online based on new measurements.
Observation The observation S; = {G', Ve: Be: th M) of our IPP agent consists of three parts: the augmented graph, the planning state, and the budget mask.
The augmented graph G' = (V', EE is used to describe the environment modeled by the GP. is combination of the route graph G = (V. E) and the GP ()( P). where each node of (ui, u(v;); P(u;)) E V' This augmented graph stores the information about the agent's s global belief and determines the agent's local action space. The planning state is defined by {re, Be, Vs.c). where Ve E V is the current position of the agent, B. = B - C(ee,,) is the remaining budget, and Vg.cc = (Wa: (i....,,",. V.) is the executed trajectory so far. The budget mask M is a binary vector containing one element for each node in the route graph, stating whether selecting this node at the current step would result in violating the budget constraint. To obtain this mask,
pre-solve the shortest path problem using Dijkstra [14] to compute the minimal cost to the current node and to the destination from each node et- We then compute virtual budget B = B min C(ve, 22) min C(vs, sha) for each node based on the planning state. Finally, according to B", we compute each entry of M if B <0 0
as M, = 0 otherwise. That is, actions (i.c., neighboring nodes) that would inevitably result
in budget overruns are specifically filtered out by the budget mask M,, which iteratively guarantees the completeness of CAINIPP by forcing the agent towards the destination via the shortest possible path when the budget is about to run out.
Action Each time the agent reaches a node, the GP is updated based on all measurements obtained so far, and the agent immediately selects its next action. Specifically, at each such decision step t, given the agent's observation, our attention- based neural network outputs stochastic policy to select the next node to visit out of all neighboring nodes. The policy is parameterized by the set of weights 0: ne( = Us (Ve, 2))= E E st), where E is the edge set of the underlying graph.
Reward At each decision step, to maximize information gain, the agent is given a positive reward based the reduction in uncertainty associated with its most recent action: ry (Tr(Pt-1) Tr(P*)) ( (Tr((t-1)). where we experimentally found that scaling the reward by Tr(P(-l) helped stabilize training by keeping the rewards consistent in magnitude. However, this normalization in- troduces a deviation between the training objective and the IPP objective. Therefore, at the last decision step of each episode, we introduce a negative correction reward ra = aa (r(Pd), where pd is the covariance after executing the whole trajectory v) and is scaling factor (1 in prac- tice). We empirically observed that this correction reward helps mitigate the bias introduced by the (dynamic) normalization used in our dense rewards, towards the true IPP reward.
4.2 Neural Network Structure
The proposed attention-based neural network consists of an encoder and a decoder modules (see Fig, 2). We use the encoder to model the observed environment by learning the dependencies be- tween nodes in the augmented graph G', i.e., the context. Based on the features extracted by the encoder, the planning state {Ve, Be, $e,c]. and the budget mask M, the decoder then outputs the policy over which neighboring node to visit next. To handle graphs with arbitrary topologies, our encoder uses standard Transformer attention layer with graph Positional Encoding (PE) based on the graph Laplacian's 's eigenvector [15], thus providing the neural network with the ability to reason about node connectivity. While general policy-based based RI. agents have fixed action space, our de- coder is inspired by the Pointer Network [16] to allow the dimension of the final policy to depend on the number of neighboring nodes, allowing our network generalize to arbitrary graphs
Attention Layer The Transformer attention layer [17] is used as the fundamental building block in our model. The input of such an attention layer consists of the query source h and the key- and-value source h".". The attention layer updates the query source using the weighted sum of the value vector, where the attention weight depends on the similarity between key and query. We com-
pute the updated feature h4 as: q wohg, k; = WKK, 0s = WV h .uy va ai " h12- Σj= Ui where Wº, W V are d d learnable matrices. The updated fea-
q. -k;
4

---

Node Input
Current Node Feature
Enhanced Current Node Feature
LSTM
Decoder
Graph
Mask
Policy
Gaussian Process
Planning state
Construct Augmented Graph
Positional
Embedding
Encoder
Action
Value
Node Features
Context-aware Node Features
Neighhoring Features Selecting Neighboring Nodes
Figure 2: CAINIPP's attention- neural network. The encoder module relies on self- attention identify and represent the global dependencies between nodes in the agent's belief (i.e., augmented graph) as context-awen aware node features (grey dashed circle), the planning state, and the mask, the decoder relies on cross-attention to output the final, context-aware policy (and value estimate during training).
node features. Relying on the current and neighboring context-
tures are then passed through the feed forward sublayer, composed two linear layers and a ReLU activation. As in [17], we use layer normalization and residual connections in these two sublayers.
Encoder The encoder is used to model the observed environment by learning dependen- cies between nodes in the augmented graph G'. We first embed the node inputs V into d-dimensional node features hy and add Laplacian positional embeddings: h² o + b24 + P + bPE WD bD + WPE Ao + bPE
i> 0
==0 where A; the pre-computed k-dimensional Laphlcii
eigenvector, and WT, WD E Rdx4, WPF E Rd b",,b",..... P R are learnable parameters. Note that the destination node vó is embedded another linear layer. The node features then passed to attention layer, where h2 AR. = h". is commonly done in self-attention mecha- nisms. We term the output the encoder, hT the context-aware node features, since each of these updated node features heTh contains the dependencies of v with all other nodes.
Decoder The decoder is used to output a policy based on the context-aware node features, the planning state {ve, Be, ). and the budget mask M. We first merge the information about the budget and the high-interest areas to the -wwre node features: h = [h , + 8 where W E (dx()) and B E IRd are learnable parameters. Then, according to the current position ve and the edge set E, we select the current node feature hen the neighboring features hen from hen and the neighbor mask M,, from M. After that, the current node feature h are passed to a LSTT block, where the hidden state and cell state are input from previous current node feature along the executed trajectory v2.co The LSTM output hen is merged with the destination feature hg compute the enhanced current node feature hec, h w Agr)) + 6C, where WC E dxddd and bC E Rd are learnable parameters. We feed the enhanced node current feature and the neighboring features to an attention layer, where 1 = // and k. = h". We denote the output of this attention layer se which is simultancously passed to linear layer to output the state value V(sz), and to the final attention layer with the neighboring features, where h = he and h = h" the attention weights a; as the final policy u; for the IPP agent, where invalid nodes explicitly masked using M", The masked is finally normalized to yield the probability distribution for the next node to visit: n; Te( wg(oe) e" />
For this final attention layer, we directly
4.3 Training
Our model is trained using PPO [18]. At the beginning of each training episode, we average 8 12 random 2-dimensional Gaussian distributions the unit square (0, 1] construct the true interest map. The robot's belief starts as a uniform distribution GP Po)) Pa 1. The start and destination positions are randomly generated in [0, 1 During training, the number of nodes for our
5

---

Table I: Comparison with SOTA IPP solvers (10 trials 30 instances for each budget). Tr(P) is the average covariance matrix trace after running out of budget (standard deviation in parentheses). T(s) is the average total planning time in seconds.
Budget 6
Budget 8
Budget 10
Budget 12
Method
Tr(P)
T(s)
Tr(P)
T(s)
Tr(P)
T(s)
Tr(P)
T(s)
RIG-Tree
12.89)+
132.36
15.44(=5.49)
192.74
7.74(+3.01)
240.58
4.801+22211)
291.31
RAOr
16.80(+15.16)
17.12
11.17(=3.87)
40.13
6.28(12.25)
73.60
4.71(=1.13)
127.44
CMA-ES
17.44(16.09)
124.23
10.48(=5.38)
181.41
6.77(13.74)
241.47
4.51(122422
268.69
g. g.(800)
22.86(116442)
1.23
7.72(12.777)
1.68
3,97(11.46)
2.20
2.70(1:1.18)
2.52
(4)
20.19(+33888)
90.31
7.04(#1.44)
123.56
3.82(=0.61)
158.44
2.52(=0041)
194.97
graph is randomized within [200, 400) for each episode, the number of neighhoring nodes is fixed to k = 20, and the budget is randomized within [6, 8). A measurement is obtained every time the agent has traveled 0.2 from the previous measurement. We set the max episode length to 256 time steps, and the batch size 1024. We use the Adam optimizer with learning rate 10-4, which decays every 32 steps by a factor of 0.96. For each training episode, PPO runs 8 iterations. Our model is trained on workstation equipped with a 19-10980XE CPU and four NVIDIA RTX 3090 GPUs. We train our model utilizing Ray, a distributed framework for machine learning [19]. We run 32 IPP instances in parallel accelerate the data collection and training, and need around 24h converge." !2
4.4 Trajectory Sampling
Until now, we discussed solving the IPP in the standard RL manner, i.e., iteratively selecting the next node to visit each time the agent reaches node. Inspired by conventional non-learning IPP solvers, we further propose receding-horizon strategy for our RI. agent, where an m-step trajectory is output at each (re)plannnng step but only a portion of it is executed before the next replanning step (see Fig. 1). We utilize the learned policy for further optimization by sampling, which has been shown to be a reliable optimization strategy for learning-based routing planner [20, 21J. That is, at each planning step, based on the learned policy, our trajectory sampling method parallely plans number s of m-step trajectories, and then selects the trajectory that maximizes the information gain as the final trajectory "²" During the m-step planning process, only the covariance of the GP can be predicted, since no measurement is actually taken before executing the trajectory.
5 Experiments
In this section, we compare CAINIPP with ofate-of-the-aat (SOTA) baselines IPP solvers on a fixed set of randomly generated environments with identical randomized conditions. We also present numerical and experimental validation of CAtNIPP on an fight-intensity-bate based adaptive IPP task. In our supplemental material, also tested a number of variants of our model and its generalizability.
5.1 Comparison Results
We compare CAtNIPP against a number of state-of-the-art IPP solvers: (a) CMA-ES [7] (we use linear B-spline for the CMA-ES solver for fair comparison), (b) RAOr [5], and (c) RIG-tree [9]. Following [1]. we implement RAOr and RIG-tree in receding-horizon manner to make them adap- tive. All considered solvers (except our fully reactive, greedy variants) replan paths after executing 0.4 of their previously planned trajectory. Starting from hyperparameters suggested by their original papers, we tuned these solvers to output highest-quality solutions, while keeping the total planning time similar to our trajectory sampling variants. This enables us to offer a fair comparison between our methods and these baselines. CAtNIPP's greedy and trajectory sampling variant are denoted by (n( and (m)) respectively, where n is the number of nodes for the route graph and m is the number of trajectories sampled at each planning step (n is fixed to 400 for ts.). Greedy variants work in the standard RL. manner, i.e., the agent always selects the action with highest activation in its policy. Trajectory sampling variants plan a 15-step trajectory and execute the first steps before replanning (in practice, 0.4 of traveled distance), following the receding- horizon setup in [7].
I 'Actually, hours of training on one NVIDIA RTX 2080 (batch size 256) can yield similar performance. "Our full code and trained model can be found https htt///gttttutb/ http///ammma/b///mmmammommma ab/CCABIIPP

---

103
0.3
TH(P
102
1s.
RMSE
g.(80))
10
CMA-ES
RIG-tree
RAO:
Attention weights visualization from the trained encoder. The query source is the node at the current position (black) and the keys source (blue). One attention head and three attention heads pay attention longer- and shorter-term high-interest areas respectively.
Figure 4:
Used budget
Used budget
Figure 3: Comparison with SOTA IPP solvers for fixed budget of 10 (10 trials on 30 in-
nodes in the augmented graph
stances). Our two CAINIPP variants significantly outperform all solvers in reducing uncertainty (left) as well as root mean square eπrоr (right).
We report the uncertainty remaining (covariance matrix trace Tr(P) in the iigh-inteeest areas X,, lower is better) after finishing the mission, as well as the total planning time in Table 1. The evolution of the remaining uncertainty and root mean square error compared to the ground truth || - <lla plotted Fig. 3. Upon closer inspection, we note that RAOr tends to plan trajectories that exploit nearby high-interest areas, thus reducing the uncertainty slowly the early stage and more rapidly later on. RIG-tree, on the other hand, has a strong tendency for exploration, which leads to a fast uncertainty reduction earlier on, but slower one in the later stages of an episode. As a meta- heuristic solver, CMA-ES finds a good trade-off between exploration and exploitation, resulting in the best overall performance among non-learning solvers (even better than CAtNIPP with budget 6).
Fig. 3 shows that CAtNIPP maintains best overall performance with respect to all metrics throughout the whole budget span. We performed paired t-tests between the covariance reduction results of our variants and each of the baselines (6 tests in total), which all yielded p-values lower than 1.06 10-1 Using Bonferroni correction for these multiple tests, we find the final significance threshold p 1.67 10 (for standard, original p = 0.01 threshold), indicating that CAINIPP (most likely) significantly outperforms all other baselines in terms of covariance reduction. We further note that our ts. variants exhibit improved solution quality over our greedy variants (8% better in average) by further refining these solutions, e.g., in some extreme cases where greedy variants do particularly poorly. However, greedy variants plan up to around 100x faster than ts. variants and other SOTA IPP solvers, making it the most time-efficient CAtNIPP version.
5.2 Generalization Results
In our supplemental material, we include detailed testings of CAINIPP's generalization capabilities. We demonstrate that our model, trained on randomly generated ground truths, can generalize to completely different environments never seen during training, such as different Gaussian mixture models or even handcrafted, non-Gausee oon-Gussiia distributions, while still outperforming baselines. Our results suggest that CAtNIPP endows the agent with a general strategy for IPP, rather than overfitting to particular class of ground truths.
5.3 Attention Visualization: Learning Be Context-aware
We believe that the superior solution quality of CAINIPP mainly comes from its ability to be context- aware, and thus to avoid the type of hight-sightesnnes usually associated with local, reactive IPP planners. We investigated the learned attention mechanism at the core of CAtNIPP by visualizing learned attention weights (larger dots means higher weight) at representative time steps. In par- ticular, Fig. 4 shows that the left attention head and the right three attention heads of the encoder have learned focus on the longer- and shorter-term high-interest regions respectively. Given these context-awee aware node features, the agent finally learns make local decisions that can optimize ob- jectives at the different scales identified by the encoder. Our ablation results in the supplemental material further confirm that the presence of the encoder is critical.
5.4 Numerical and Experimental Validation
We carried out experiments to validate CAINIPP's performance on TurtleBot3 robot, over a printed grayscale image of 2.38 x 2.38m² representing the ground truth interest map (see Fig. 5). The robot
7

---

is equipped with an on-board camera used to measure the ground light (grayscale) intensity (as an example of a simple onboard intensity-level sensor, e.g., tenmprratur/rradioctttvvvvyg levels), while its position is obtained by a downward-facing, overhead camera.
In this experiment, trained CAt- NIPP model adaptively outputs the next node location visit based on the agent's current be- lief, using a 400-nodes graph, and the agent only takes measurements when reaching node. This exper- iment confirms that CAtNIPP is easily deployable on robot for on- line, reactive planning, and high- lights its low computational cost (S 0.1s per decision CPU). Our
(a) True interest map (b) Early agent belief (c) Final agent belief Figure 5: Experimental validation of CAINIPP on Turtle- Bot3. Selected nodes are shown in green, and robot trajectory in red (intermediate path in (b), and full final path in (c)). supplemental material includes simulation videos in environments of up to 8 x 8m2.
Limitations
We believe that the limitations of CAINIPP lie at three different levels: generalizability, area dis- cretization, and implementation readiness:
Our proposed method is sensitive changes in the hyperparameters of the GP's kernel func- tion and the upper bound confidence that defines high-interest areas. Currently, CAINIPP would require retraining any of these parameters were changed significantly. In practice, the hyperpa- rameters of the GP's kernel are decided by key characteristics of the actual sensor equipped on the robot (e.g., range or resolution of camera). Nevertheless, based on our experience, for given type of real-world task (e.g., search and rescue, fire detection, agriculture monitoring), the defini- tion of thigh-interest" usually rather static. Thus we would expect that, after an initial training based on these parameters, our learned model will likely not need any further training in practice, since retraining would only be needed if the task or onboard sensor are drastically changed.
We currently assume uniform sampling of the route graph, which may prevent the agent from reaching an interesting area due to insufficient graph coverage. However, CAINIPP is already able handle arbitrary graphs. To address this issue, especially in later stages of the planning, will further investigate online re-sampling of graph nodes, c.g., according to the current belief.
We currently plan paths in simplified graph, where paths between nodes are straight lines, ignor- ing most real-life robot motion constraints (i.e., holonomic robot assumption). Future work will explicitly consider the robot's motion model, e.g., in the state representation (velocity, heading. kinematic/dynamics constraints), to better trade-off robot-speciffcity with ease of implementation.
Conclusion
In this paper, we introduce CAtNIPP, a sniicy-gradient-barr dRL method for adaptive IPP that re- lies on self-attention endow the agent with the ability to sequence local decisions, informed by its global context over the search domain to avoid short-sightenedness. In addition to solving adaptive IPP by simply greedily exploiting our learned policy, which can be done at very low computational cost (~ 0.1s per decision once trained), we propose a sampling baaelngg- strategy that utilizes the learned policy more efficiently to output higher-quality solutions, while keeping the computing time on par with existing IPP solvers. We experimentally demonstrate that both variants of CAINIPP significantly outperform state-of-the-an IPP solvers in terms of solution quality and planning time, with strong generalization to classes of distributions never seen during training. Finally, present experimental results on physical and simulated robots in a representative online, adaptive IPP task, showing promises for robotic deployments in real-life monitoring, inspection, mapping scenarios.
Future work will mainly focus on extending our model multi-agent IPP, where robots need to rea- son about each other to cooperatively plan informative paths, by leveraging synergies and avoiding redundant work. We also plan to investigate the use CAtNIPP for robot exploration tasks, where more real-world object such as obstacles and sensors need to be considered in the planning process.
8

---

Acknowledgments
This work was supported by Temasek Laboratories (TL@@UUS) under grant TL/SRP/21/19
References
[1] M. Popović, Vidal-Calleja, G. Hitz, Chung, Sa, R. Siegwart, and J. Nicto. An infor- mative path planning framework for uav-based terrain monitoring. Autonomous Robots, 44(6): 889-911 911, 2020.
[2] Z. W. Lim, D. Hsu, and W. Lee. Adaptive informative path planning in metric spaces. The International Journal of Robotics Research, 35(5):585-595, 2016.
[3] A. Bircher, M. Kamel, K. Alexis, H. Oleynikova, and R. Siegwart. Receding horizon path planning for 3d exploration and surface inspection. Autonomous Robots, 42, 02 2018. doi: 10.1007/s105144
[4] Binney and G. Sukhatme. Branch and bound for informative path planning. In 2012 IEEE international conference robotics and automation, pages 2147-2154. IEEE, 2012.
[5] S. Arora and S. Scherer. Randomized algorithm for informative path planning with budget constraints. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 4997-5004. IEEE, 2017.
[6] Y. Wei and R. Zheng. Informative path planning for mobile sensing with reinforcement learn- ing. In IEEE INFOCOM 2020-IEEE Conference Computer Communications, pages 864 873. IEEE, 2020.
[7] G. Hitz, E. Galceran, M.-B. Garneau, F. Pomerleau, and R. Siegwart. Adaptive continuous- space informative path planning for online environmental monitoring. Journal of Field Robotics, 34(8):1427-1449 2017.
[8] A. Meliou, A. Krause, C. Guestrin, and J. M. Hellerstein. Nonmyopic informative path plan- ning in spatio-temporal models. In AAAI, volume 10, pages 16-7, 2007.
[9] G. A. Hollinger and G. S. Sukhatme. Sampling-base robotic information gathering algo- rithms. The International Journal of Robotics Research, 33(9):1271-1287. 2014.
[10] M. Ghaffari Jadidi, J. Valls Miro, and Dissanayake. Sampling-based incremental infor- mation gathering with applications to robotic exploration and environmental monitoring. The International Journal of Robotics Research, 38(6):658-66 685, 2019.
[11] R. Geraerts and M. H. Overmars. A comparative study of probabilistic roadmap planners. In Algorithmic foundations of robotics pages 43-57. Springer, 2004.
[12] Karaman and E. Frazzoli. Sampling- algorithms for optimal motion planning. The international journal of robotics research, 30(7):846-88) 894, 2011.
[13] J. Rückin, L. Jin, and M. Popović. Adaptive informative path planning using deep reinforce- ment learning for uav-based active sensing. arXiv preprint arXiv:2109.13570 2021.
[14] E. W. Dijkstra. A note on two problems in connexion with graphs. Numerische mathematik, (1):269-22)) 271,, 1959.
[15] V. P. Dwivedi and X. Bresson. A generalization transformer networks to graphs. arXiv preprint arXiv:2012.09699 2020.
[16] O. Vinyals, M. Fortunato, and N. Jaitly. Pointer networks. arXiv preprint arXiv: 2015.
[17] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, Jones, A. N. Gomez, F.. Kaiser, and I. Polo- sukhin. Attention all you need. In Advances neural information processing systems, pages 5998-6008 2017.

---

[18] L. Schmid, M. Pantic, R. Khanna, L. Ott, R. Siegwart, and J. Nieto. An efficient sampling based method for online informative path planning in unknown environments, IEEE Robotics and Automation Letters, 5(2): 2020.
[19] P. Moritz, R. Nishihara, S. Wang, A. Tumanov, R. Liaw, E. Liang, M. Elibol, Z. Yang, W. Paul, M. I. Jordan, et al. Ray: A distributed framework for emerging ai applications. In Proceedings of OSDI, pages 561-577, 2018.
[20] W. Kool, H. Van Hoof, and M. Welling. Attention, learn to solve routing problems! arXiv preprint arXiv: 0100180030055 2018.
[21] Y. Cao, Z. Sun, and G. Sartoretti. DAN: Decentralized attention- neural network to solve the minmax multiple traveling salesman problem. arXiv preprint arXiv:2109.00 04205, 2021.
10


Reviews:
[{"review_id": "q9FA-4SCJga", "reviewer": "Reviewer_QZo2", "summary": "This paper introduces a deep reinforcement learning (dRL) method for solving the adaptive IPP problem. The proposed dRL architecture leverages a self-attention layer to inform local decisions with global information, allowing for non-myopic decision making. The authors provide thorough experimental and simulated demonstrations of strengths of their architecture and comparisons to state of the art IPP solutions. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- Well written.\n- Well placed in existing literature\n- Provides a novel solution to a well researched problem.\n- Proposed solution has benefits outside of only solution quality/efficiency (provides attention map)\n- Supplementary videos and gifs are good\n- Experimental validation is strong, particularly in demonstrating the benefit of the attention layer.\n\nWeaknesses:\n\nCertain technical details that should be emphasised are skimmed over, for example:\n- As with any learning method, issues arise with generalisability. WIth a training time of 24 hours with a substantially powerful workstation, - This seems like a major limitation of the proposed method:\n    - e.g. what if the true function of interest is not well represented by the learned kernel? Retraining would be required.\n    - During training, the ground truth function of interest must be available. If for example the function of interest in Fig. 4 were inverted (low interest where high interest areas once were), the attention layer learned would be obsolete. Retraining would be required.\n- Training reward is not equivalent to the IPP objective, and it is unclear how the negative correction reward proposed recovers equality.\n\nThis approach requires prior knowledge of the ground truth function to learn a good policy, which diminishes its utility for field-based/realistic scenarios greatly. If ground truth is available for training, no policy needs to be learned at all.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Weak accept, provided the authors add significant justification for their method given the limitations regarding generalisation/requirement for ground truth during training. More emphasis should be placed on the proposal and benefits of the attention layer, as this is (in my opinion) the main strength of the paper.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {"issues": "- Strong justification of using RL for the IPP problem is required: especially in the proposed approach where training requires ground truth to be available\n- Authors could benefit from emphasising the relatively small improvement gained from trajectory sampling rather than employing the greedy selection method: it would emphasise the naturally non-myopic nature of their policy. Other IPP solutions only embed non-myopic behaviour from trajectory sampling. \n- Caption of Table 2 has typo: should say “WITHOUT” \n- In subsection: Adaptive Replanning, authors should make it clear that the mean and variance in the UCB is prior to fusing measurements along the future trajectory.\n- Authors should justify why the negative correction reward proposed solves the discrepancy between training and IPP objectives. \n"}}, {"review_id": "iYYSwoXZtXg", "reviewer": "Reviewer_6v6E", "summary": "This paper presents a deep reinforcement learning approach (CAtNIPP) for informative path planning (IPP), where the goal is to gain information by navigating a new environment while satisfying some resource budget. Notably, CAtNIPP appears to be the first RL based adaptive IPP solver. The proposed approach outperforms prior non-learning based IPP solvers both in terms of information gain and computing time on a set of randomly generated 2D navigation problems. CAtNIPP is also evaluated on a simple 2D navigation task on a physical robotic platform (TurtleBot3).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n(1) CAtNIPP is very clearly described and represents a novel application of recent advances in machine learning, notably RL and attention mechanisms, to an important problem in robotics (information path planning).\n\n(2) CAtNIPP is very clearly positioned with respective to prior work. Namely, prior RL based IPP approaches exist, but only work for non-adaptive settings, while prior conventional IPP solvers are shown to achieve worse solution quality in a higher amount of time than CAtNIPP.\n\n(3) The simulation experiments are very well designed and reported, with comparisons to relevant baselines and clear demonstration that CAtNIPP achieves generally higher solution quality in general than conventional approaches while being significantly faster (at least the greedy variant is).\n\nWeaknesses:\n\n(1) The main weakness of this work is that the experimental evaluation is very simplistic, both in simulation and in physical experiments. While the experiments do illustrate the benefit of CAtNIPP over prior approaches, this benefit is restricted solely to simple 2D navigation tasks in free space. Tasks with greater complexity involving realistic robot motion constraints and more complex specifications (eg. trying to gain information about different rooms in a house from image observations) would make the utility of CAtNIPP for practical robotic problems much more clear.  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "CAtNIPP is very well motivated and well described, and the simulation experiments do thoroughly evaluate and demonstrate the benefits of CAtNIPP with respect to prior IPP solvers. However, the main uncertainty I have with this paper is that the experiments are so simple that the relevance of CAtNIPP to practical robotic problems is not clearly demonstrated. I am still leaning towards an accept decision at the moment, but would be more confident in this decision if a stronger case can be made regarding the relevance of this paper for practical robotic problems.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "(1) Simulation and physical experiments are so simple that the relevance of CAtNIPP to practical robotic problems is not clearly demonstrated. Adding more complex robotic control domains (or at least a more complex task specification) would significantly strengthen the contribution of the paper for CoRL."}}, {"review_id": "M4NQOHmngtI", "reviewer": "Reviewer_HgVR", "summary": "This paper proposes a method for informative path planning (IPP). In IPP, the task is to design a trajectory for sampling locations of a Gaussian process with maximum reduction in uncertainty, while respecting a budget constraint. \nThe paper proposes to remap the IPP problem as a Markov decision process via a probabilistic roadmap graph, and applies reinforcement learning (RL) with neural network (NN) based function approximation to solve for a policy. Besides the remapping of the problem, the main innovations of the paper are the NN structure using attention/transformers and adapting it to handle arbitrary graph topologies.\nThe proposed method is compared to three IPP algorithms from the literature. The proposed method can perform very well, that the attention mechanism is crucial for the good performance, and that the method can also to some degree generalize beyond its training data. A proof-of-concept demonstration with real robot hardware is also given.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\n- The empirical performance compared to previous state-of-the-art is strong. The ablation studies also confirm that the attention mechanism proposed in the encoder is crucial for good performance, comparing the significance of the papers main contribution.\n\n- The paper is technically sound, and clearly describes its methodology. The RL reformulation seems particularly useful, and the idea of discretizing the task using a probabilistic roadmap is quite nice. This could be of interest to practicioners in other tasks requiring discretization as well. It also seems like this approach could easily be extended to environments with more complex topologies.\n\n- The section on limitations was very clear and comprehensive. Well done!\n\n- The generalization analysis in the supplementary material is quite interesting, indicating that the proposed method can also work for field intensity distributions that strongly differ from the Gaussian mixtures used during training.\n\n# Weaknesses\n\n- My main concern is how applicable this method would be when the process to be identified is not that well known, or the ability to generate training data for the method is limited or cannot match the actual target process. The generalization study does give some positive indications, but a more thorough analysis and discussion of this would be good.\n\n- I suppose the times in Table 1 do not include the training times for the proposed method. It seems that the comparison as presented is not entirely fair, and in particular RAOr seems a strong competitor.\n\n- It was not clear what the plus/minus values in parentheses in Table 1 or the shading in Figure 3 correspond to. Depending on this, it is possible that the observed improvements are not statistically significant.\n\n- The environments are \"empty squares\" without features such as obstacles, or dead-ends, characteristic of real-world environments. These sorts of environments might be a very interesting benchmark for the long-term planning capabilities of the proposed method. I suppose there is nothing preventing using the PRM method for such environments as well?\n\n# Minor comments\n- Section 2: there are several misplaced/misnamed citations here. Please check and revise. Examples: Line 73: [9] refers to Hollinger et al., but in main text referred to as Karaman et al. - Line 76: Hitz et al. [5] does not match the corresponding item in bibliography\n- Line 123: there is something missing here from the definition of $\\mu_{th}$, the preceding sentence ends abruptly.\n- Lines 185-186: please use consistent notation for $W^K$, $W^V$\n- Table 2 heading: \"WTIHOUT\" --> \"WITHOUT\".\n- Figure and table captions use bold face somewhat arbitrarily, a consistent style would be preferable.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper proposes an interesting new view of informative path planning as a reinforcement learning problem, and proposes a practical method for taking advantage of the reformulation, with good empirical performance. There are some concerns about generalization capability and applicability beyond simple topologies or well-known target processes. The authors have addressed these concerns and my comments in their rebuttal, and I recommend accepting the paper.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "Could you comment or on address the weaknessess listed above, and correct the minor issues?", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "3rMFWjH40Vx", "reviewer": "Reviewer_8gWj", "summary": "The paper studies the informative path planning problem (IPP) in a Gaussian random process/field (GP). The objective is to maximize the variance reduction at points where the field values exceed the GP upper confidence bound, while respecting the path length constraints. The sets determined by the upper confidence bound change with each additional measurement motivating an online adaptive algorithm.\n\nThe main contribution of the paper is an algorithm based on deep reinforcement learning (DRL) to compute approximate solutions to the IPP problem. An extension of the DRL algorithm based on receding horizon planning is also provided which provides higher quality solutions. In simulations, the proposed algorithms (DRL and DRL+receding horizon) yield more informative paths when compared to previously proposed algorithms. In addition, the computation time of DRL (excluding training time) is lower. The idea to use reinforcement learning for IPP is quite interesting!", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe biggest strength of the paper lies in the formulation of the IPP problem as a reinforcement learning problem. The parts of the formulation that I found interesting are discussed below:\n\n1. Construction of the state/observation\n       At each step of an online adaptive algorithm for IPP, the robot needs to make a decision on which node to visit next based on the information collected. The paper constructs a state vector which encodes the current position of the robot, the state of the environment (as predicted by the GP), and the remaining budget (as a binary mask).\n\n2. Action/Control space:\n       Since the robot is operating on a graph, its action space (which node to visit next?) is determined by the degree of its current vertex. However, most RL methods have fixed-dimensional action spaces. The paper circumvents this issue by using Pointer Networks, which allows the algorithm output to depend on the degree of the current vertex. This is a neat idea!\n\n\nWeaknesses:\n\n1. Since the output of the proposed approach is a stochastic policy, it is not clear whether the algorithm will always reach the goal/destination vertex. If this is not guaranteed, then the algorithm does not always provide a feasible solution.\n\n2. While the proposed approach works very well in simulation, the limited generalizability is concerning. The authors mention that the model requires retraining if parameters, such as sensor type or task specification, change. However, the issue could extend to other parameter changes. For example, environment dimensions, kernel hyperparameters, change of kernel, connection factor in the PRM, number of nodes in the PRM, etc. Since the computational resources required for training is significant (24 hours training time, distributed training on 4 GPUs), the generalizability is a serious question since the other baselines do not require retraining.\n\n3. The experimental section is quite extensive (10 trials on 30 instances for each budget). However, there are a few missing details:\n   - What types of functions are considered? It would be interesting to see how the policy works on functions that are not samples from a GP.\n   - What is the environment size considered? Is it the same considered for training the policy i.e., the unit square?\n   - What is the source of randomness between the experiments? Is it the probabilistic roadmap that changes between trials while keeping the number of nodes fixed?\n\n4. The chosen architecture has yielded good results for the IPP problem. However, it would be nice to get more insight into the design process:\n   a. Is the selection of \\mu=0.4, \\beta=1 fixed for all environment types? What if the GP has non-zero mean and a priori variance > 1?\n   b. It seems that the attention module is crucial to the network construction. Does the same hold for the other modules of the architecture? For example, is the LSTM module required for the decoder?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper proposes a novel approach to solve the IPP problem using deep reinforcement learning. While the generalization/applicability of the algorithm to environments with different parameters is limited, the idea is quite interesting. With some more experiments and clarifications, I propose a weak accept.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "1. Since the generalization capability is limited, I am not convinced that it is correct to say the proposed approach outperforms baselines in terms of compute time. It would be more appropriate to mention the conditions under which the proposed approach is preferable over the baselines i.e., environments with fixed parameters.\n\n2. If the experiments section only considers functions that are samples from a GP, it would be more thorough if other types of functions are considered as well. For example, piecewise constant, Holder smooth functions, etc.\n\n3. The discussion on the completeness of the algorithm should be included.\n\n4. The paper mentions the metric used for comparison is the uncertainty remaining after finishing the mission. However, it is not clear whether this is over all n' locations where the quantity is to be inferred or if it is over the upper confidence bound sets determined at the end of each algorithm."}}]

Author rebuttal:

Decision: Accept (poster)