Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper introduces a deep reinforcement learning (dRL) method for solving the adaptive IPP problem. The proposed dRL architecture leverages a self-attention layer to inform local decisions with global information, allowing for non-myopic decision making. The authors provide thorough experimental and simulated demonstrations of strengths of their architecture and comparisons to state of the art IPP solutions. 
strengths_and_weaknesses: Strengths:

- Well written.
- Well placed in existing literature
- Provides a novel solution to a well researched problem.
- Proposed solution has benefits outside of only solution quality/efficiency (provides attention map)
- Supplementary videos and gifs are good
- Experimental validation is strong, particularly in demonstrating the benefit of the attention layer.

Weaknesses:

Certain technical details that should be emphasised are skimmed over, for example:
- As with any learning method, issues arise with generalisability. WIth a training time of 24 hours with a substantially powerful workstation, - This seems like a major limitation of the proposed method:
    - e.g. what if the true function of interest is not well represented by the learned kernel? Retraining would be required.
    - During training, the ground truth function of interest must be available. If for example the function of interest in Fig. 4 were inverted (low interest where high interest areas once were), the attention layer learned would be obsolete. Retraining would be required.
- Training reward is not equivalent to the IPP objective, and it is unclear how the negative correction reward proposed recovers equality.

This approach requires prior knowledge of the ground truth function to learn a good policy, which diminishes its utility for field-based/realistic scenarios greatly. If ground truth is available for training, no policy needs to be learned at all.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Weak accept, provided the authors add significant justification for their method given the limitations regarding generalisation/requirement for ground truth during training. More emphasis should be placed on the proposal and benefits of the attention layer, as this is (in my opinion) the main strength of the paper.
issues: - Strong justification of using RL for the IPP problem is required: especially in the proposed approach where training requires ground truth to be available
- Authors could benefit from emphasising the relatively small improvement gained from trajectory sampling rather than employing the greedy selection method: it would emphasise the naturally non-myopic nature of their policy. Other IPP solutions only embed non-myopic behaviour from trajectory sampling. 
- Caption of Table 2 has typo: should say “WITHOUT” 
- In subsection: Adaptive Replanning, authors should make it clear that the mean and variance in the UCB is prior to fusing measurements along the future trajectory.
- Authors should justify why the negative correction reward proposed solves the discrepancy between training and IPP objectives.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper studies the informative path planning problem (IPP) in a Gaussian random process/field (GP). The objective is to maximize the variance reduction at points where the field values exceed the GP upper confidence bound, while respecting the path length constraints. The sets determined by the upper confidence bound change with each additional measurement motivating an online adaptive algorithm.

The main contribution of the paper is an algorithm based on deep reinforcement learning (DRL) to compute approximate solutions to the IPP problem. An extension of the DRL algorithm based on receding horizon planning is also provided which provides higher quality solutions. In simulations, the proposed algorithms (DRL and DRL+receding horizon) yield more informative paths when compared to previously proposed algorithms. In addition, the computation time of DRL (excluding training time) is lower. The idea to use reinforcement learning for IPP is quite interesting!
strengths_and_weaknesses: Strengths:

The biggest strength of the paper lies in the formulation of the IPP problem as a reinforcement learning problem. The parts of the formulation that I found interesting are discussed below:

1. Construction of the state/observation
       At each step of an online adaptive algorithm for IPP, the robot needs to make a decision on which node to visit next based on the information collected. The paper constructs a state vector which encodes the current position of the robot, the state of the environment (as predicted by the GP), and the remaining budget (as a binary mask).

2. Action/Control space:
       Since the robot is operating on a graph, its action space (which node to visit next?) is determined by the degree of its current vertex. However, most RL methods have fixed-dimensional action spaces. The paper circumvents this issue by using Pointer Networks, which allows the algorithm output to depend on the degree of the current vertex. This is a neat idea!


Weaknesses:

1. Since the output of the proposed approach is a stochastic policy, it is not clear whether the algorithm will always reach the goal/destination vertex. If this is not guaranteed, then the algorithm does not always provide a feasible solution.

2. While the proposed approach works very well in simulation, the limited generalizability is concerning. The authors mention that the model requires retraining if parameters, such as sensor type or task specification, change. However, the issue could extend to other parameter changes. For example, environment dimensions, kernel hyperparameters, change of kernel, connection factor in the PRM, number of nodes in the PRM, etc. Since the computational resources required for training is significant (24 hours training time, distributed training on 4 GPUs), the generalizability is a serious question since the other baselines do not require retraining.

3. The experimental section is quite extensive (10 trials on 30 instances for each budget). However, there are a few missing details:
   - What types of functions are considered? It would be interesting to see how the policy works on functions that are not samples from a GP.
   - What is the environment size considered? Is it the same considered for training the policy i.e., the unit square?
   - What is the source of randomness between the experiments? Is it the probabilistic roadmap that changes between trials while keeping the number of nodes fixed?

4. The chosen architecture has yielded good results for the IPP problem. However, it would be nice to get more insight into the design process:
   a. Is the selection of \mu=0.4, \beta=1 fixed for all environment types? What if the GP has non-zero mean and a priori variance > 1?
   b. It seems that the attention module is crucial to the network construction. Does the same hold for the other modules of the architecture? For example, is the LSTM module required for the decoder?
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The paper proposes a novel approach to solve the IPP problem using deep reinforcement learning. While the generalization/applicability of the algorithm to environments with different parameters is limited, the idea is quite interesting. With some more experiments and clarifications, I propose a weak accept.
issues: 1. Since the generalization capability is limited, I am not convinced that it is correct to say the proposed approach outperforms baselines in terms of compute time. It would be more appropriate to mention the conditions under which the proposed approach is preferable over the baselines i.e., environments with fixed parameters.

2. If the experiments section only considers functions that are samples from a GP, it would be more thorough if other types of functions are considered as well. For example, piecewise constant, Holder smooth functions, etc.

3. The discussion on the completeness of the algorithm should be included.

4. The paper mentions the metric used for comparison is the uncertainty remaining after finishing the mission. However, it is not clear whether this is over all n' locations where the quantity is to be inferred or if it is over the upper confidence bound sets determined at the end of each algorithm.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents a deep reinforcement learning approach (CAtNIPP) for informative path planning (IPP), where the goal is to gain information by navigating a new environment while satisfying some resource budget. Notably, CAtNIPP appears to be the first RL based adaptive IPP solver. The proposed approach outperforms prior non-learning based IPP solvers both in terms of information gain and computing time on a set of randomly generated 2D navigation problems. CAtNIPP is also evaluated on a simple 2D navigation task on a physical robotic platform (TurtleBot3).
strengths_and_weaknesses: Strengths:

(1) CAtNIPP is very clearly described and represents a novel application of recent advances in machine learning, notably RL and attention mechanisms, to an important problem in robotics (information path planning).

(2) CAtNIPP is very clearly positioned with respective to prior work. Namely, prior RL based IPP approaches exist, but only work for non-adaptive settings, while prior conventional IPP solvers are shown to achieve worse solution quality in a higher amount of time than CAtNIPP.

(3) The simulation experiments are very well designed and reported, with comparisons to relevant baselines and clear demonstration that CAtNIPP achieves generally higher solution quality in general than conventional approaches while being significantly faster (at least the greedy variant is).

Weaknesses:

(1) The main weakness of this work is that the experimental evaluation is very simplistic, both in simulation and in physical experiments. While the experiments do illustrate the benefit of CAtNIPP over prior approaches, this benefit is restricted solely to simple 2D navigation tasks in free space. Tasks with greater complexity involving realistic robot motion constraints and more complex specifications (eg. trying to gain information about different rooms in a house from image observations) would make the utility of CAtNIPP for practical robotic problems much more clear.  
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: CAtNIPP is very well motivated and well described, and the simulation experiments do thoroughly evaluate and demonstrate the benefits of CAtNIPP with respect to prior IPP solvers. However, the main uncertainty I have with this paper is that the experiments are so simple that the relevance of CAtNIPP to practical robotic problems is not clearly demonstrated. I am still leaning towards an accept decision at the moment, but would be more confident in this decision if a stronger case can be made regarding the relevance of this paper for practical robotic problems.
issues: (1) Simulation and physical experiments are so simple that the relevance of CAtNIPP to practical robotic problems is not clearly demonstrated. Adding more complex robotic control domains (or at least a more complex task specification) would significantly strengthen the contribution of the paper for CoRL.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes a method for informative path planning (IPP). In IPP, the task is to design a trajectory for sampling locations of a Gaussian process with maximum reduction in uncertainty, while respecting a budget constraint. 
The paper proposes to remap the IPP problem as a Markov decision process via a probabilistic roadmap graph, and applies reinforcement learning (RL) with neural network (NN) based function approximation to solve for a policy. Besides the remapping of the problem, the main innovations of the paper are the NN structure using attention/transformers and adapting it to handle arbitrary graph topologies.
The proposed method is compared to three IPP algorithms from the literature. The proposed method can perform very well, that the attention mechanism is crucial for the good performance, and that the method can also to some degree generalize beyond its training data. A proof-of-concept demonstration with real robot hardware is also given.
strengths_and_weaknesses: # Strengths

- The empirical performance compared to previous state-of-the-art is strong. The ablation studies also confirm that the attention mechanism proposed in the encoder is crucial for good performance, comparing the significance of the papers main contribution.

- The paper is technically sound, and clearly describes its methodology. The RL reformulation seems particularly useful, and the idea of discretizing the task using a probabilistic roadmap is quite nice. This could be of interest to practicioners in other tasks requiring discretization as well. It also seems like this approach could easily be extended to environments with more complex topologies.

- The section on limitations was very clear and comprehensive. Well done!

- The generalization analysis in the supplementary material is quite interesting, indicating that the proposed method can also work for field intensity distributions that strongly differ from the Gaussian mixtures used during training.

# Weaknesses

- My main concern is how applicable this method would be when the process to be identified is not that well known, or the ability to generate training data for the method is limited or cannot match the actual target process. The generalization study does give some positive indications, but a more thorough analysis and discussion of this would be good.

- I suppose the times in Table 1 do not include the training times for the proposed method. It seems that the comparison as presented is not entirely fair, and in particular RAOr seems a strong competitor.

- It was not clear what the plus/minus values in parentheses in Table 1 or the shading in Figure 3 correspond to. Depending on this, it is possible that the observed improvements are not statistically significant.

- The environments are "empty squares" without features such as obstacles, or dead-ends, characteristic of real-world environments. These sorts of environments might be a very interesting benchmark for the long-term planning capabilities of the proposed method. I suppose there is nothing preventing using the PRM method for such environments as well?

# Minor comments
- Section 2: there are several misplaced/misnamed citations here. Please check and revise. Examples: Line 73: [9] refers to Hollinger et al., but in main text referred to as Karaman et al. - Line 76: Hitz et al. [5] does not match the corresponding item in bibliography
- Line 123: there is something missing here from the definition of $\mu_{th}$, the preceding sentence ends abruptly.
- Lines 185-186: please use consistent notation for $W^K$, $W^V$
- Table 2 heading: "WTIHOUT" --> "WITHOUT".
- Figure and table captions use bold face somewhat arbitrarily, a consistent style would be preferable.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: This paper proposes an interesting new view of informative path planning as a reinforcement learning problem, and proposes a practical method for taking advantage of the reformulation, with good empirical performance. There are some concerns about generalization capability and applicability beyond simple topologies or well-known target processes. The authors have addressed these concerns and my comments in their rebuttal, and I recommend accepting the paper.
issues: Could you comment or on address the weaknessess listed above, and correct the minor issues?
review_title: null

Meta-review: metareview: The 4 reviewers all recommended a weak accept rating for this work. The work makes a relevant contribution in the area of informative path planning.
Overall some of the key strengths and weaknesses include:

Strengths:

-  a well written work that is easy to follow
-  benefits of attention-based neural network is shown in experiments
-  a novel solution with interesting approach where RL and attention mechanisms are applied to IPP

Weaknesses:

- questions around generalization and retraining are raised by reviewers QZo2, 8gWj and not fully resolved
- benefits of the approach are only evaluated on relatively simple environments, expensive compute requirements
- negative correction reward serves only as a heuristic to approximate IPP objective

A concerning weakness of the approach (shared by several IPP algorithms) is the limited complexity of the presented robotic experiments, featuring simple 2D navigation environments. This may be a concern for the CoRL audience and may limit the significance of the proposed approach.

I follow the four reviewers' assessment and agree with the "weak accept" rating. 




confidence: 3: The area chair is somewhat confident
Decision: Accept (Poster)