Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: ### Summary

Motivated by the fact that RL algorithms are notoriously sample inefficient from pixels and that humans use attention, the authors propose Attention-driven Robotic Manipulation (ARM), which they claim can be applied to several robotics tasks without prior task knowledge. Compared to current methods which fail to train, ARM trains in only a few hours and is successful at the end manipulation tasks. The authors’ main algorithmic improvements with ARM include 1) a Q-attention agent that extracts interesting pixel locations with an explicit attention module, 2) a confidence-aware critic that leads to increased stability of training, and 3) a data augmentation strategy around expert demonstrations. 

---

### Strengths

- The paper is well motivated, and the authors recognize a salient problem with most RL systems failing with sparse rewards and being sample inefficient.
- The authors propose several potentially novel contributions, including their Q-attention agent with an explicit attention module, the confidence-aware critic that improves performance, the keyframe discovery strategy, and the demo augmentation method.
- The authors’ demo augmentation strategy is clever in that the required pose transformations are easy to collect and calculate, and this strategy clearly helps performance.
- The authors’ experiments clearly answer the questions they posed around how well this technique learns and how sensitive it is to the number of expert demonstrations. In addition their ablation experiments around which components contribute to the overall pipeline’s success are useful.
- The experimental results show a clear improvement over prior techniques, working with fewer demonstrations and fewer environment steps. The authors’ intuitive explanations for why their method works when others fail are also well-reasoned.

---

### Weaknesses

- The authors claim that their technique can be applied to several tasks without prior task knowledge. This claim only seems weakly supported in the paper. Specifically, to make this claim I would expect to see results across multiple robot platforms. The keyframe selection and demo augmentation strategy does seem specific to the current platform and set of tasks and may not work as well in stochastic environments.
- The authors replace the hand-engineering of reward functions with several hand-engineered modules, as such the theoretical motivation for these modules is less strong. Although I expected to see each module ablated and tested in isolation, these results were lacking in the experiments.
- For the Q-attention agent, the authors claim that a L2 loss prevents overestimation of the Q-values. Because the authors claim that this results was achieved experimentally, such an ablation or analysis should be present in the experiments or supplementary figures.
- The motivation for the Q-attention agent is not made immediately clear. Specifically, the Q-attention agent seems rather similar to performing hard attention with REINFORCE or other policy gradient methods, except here the hard attention is performed with Q-learning. As such, I’m unsure of the actual novelty of this technique.
- The next-best pose agent is particularly confusing since it mentions per-pixel Q-values and per-pixel Bellman losses. This formulation makes sense in the Q-attention, since the action is the pixel being chosen, but makes less sense in the next-best pose agent, since presumably the state is the entire 16x16 window of pixels, and the action is the pose. An ablation on this component’s design would be extremely helpful. It’s not clear to me that the performance gain comes from using 16x16 q-values as opposed to 1x1 q-values, or whether it comes from using confidence values attached to the state action pairs.
- The sparse reward structure being used is missing from the discussion.
- The authors point out that the keyframe selection method likely needs to become more complex with more sophisticated tasks. This again reduces the motivation and impact of the algorithm, if the hand-engineering of the reward function must now be replaced with hand-engineering of the keyframe selection method.
- I would like to see some qualitative analyses of the keyframe selection method and how successful it is. Specifically, what do the sample demonstrations look like, and what percent of frames are keyframes? Intuitively, are there some failure modes of the keyframe selection method where some interesting keyframes are not included?
- I have some confusion around the replay buffers. The pseudocode in algorithm 1 makes it seem that there is a single buffer that stores the final action (pose) taken, but the description claims that the projected pixel coordinates act as optimal actions for the Q-attention agent. Are these projected pixel coordinates also used in a separate replay buffer to warmstart the Q-attention agent?
- It sounds like the proposed methods around data augmentations and discovery strategies are only necessary because the method has multiple agents as opposed to a single end-to-end agent. 
- Although none of the compared baselines work on the RLBench tasks, do the authors have some intuition on what would be required to make them successful? Is this due to using a sparse reward structure instead of a dense one? Or not having enough demonstrations? Or not having enough samples to interact with the environment? How does the authors’ method’s performance compare to these baselines in these other regimes? Including these would make the comparison much stronger.
- The ablations present in Figure 6 study whether specific components are necessary (for example whether Q-attention is useful), but not on whether the formulation of that component is the correct formulation (for example, whether Q-attention is better than hard attention). Because the proposed modules are relatively complex and unintuitive, these ablations are critical.
- I would recommend including ablations across all 10 tasks, potentially in the supplementary pages. The current evidence is less convincing that the augmentation strategy or confidence methods are necessary for success, due to the limited number of tasks profiled.

---

### Recommendation

Overall, I vote for rejecting. I think the pipeline proposed by the authors is overly complex, and I am not yet convinced about the motivation for each individual module the authors include. The experimental results are currently lacking in terms of providing this motivation, and missing crucial ablations.

If the authors respond to my comments above with stronger motivations, clearer explanations for how the modules work, and more detailed ablative experiments, their submission will be stronger and many of my concerns would be resolved.

---

### Minor Feedback

- Figure 2 and 3 are unclear since multiple pixels are highlighted, but earlier it is implied that only a single pixel is selected by the Q-attention agent.
- Including videos of demonstrations would be potentially helpful as a qualitative comparison of the behaviors learned between the authors’ proposal and the baselines.
- A qualitative analysis of the failure modes of the authors’ method would also be helpful, but not required.
- Is there a way to visualize the behavior of the next-best pose agent? For instance, how does the pose vary depending on the crop passed in to the agent?
- I don’t understand the reasoning in the appendix (section 7). Specifically, the authors claim that “as in hard-attention style approaches, we are unable to propagate gradients [...] by instead cropping the observation, we alleviate this problem.” The authors’ method still performs non-differentiable cropping, so I don’t see how they are able to propagate gradients. Likewise, cropping observations is commonly done with hard attention, so I don’t see how this method is particularly novel.
review_title: Official Blind Review #1

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary: 

This work focuses on sparse-reward robotic manipulation tasks from image and point cloud inputs, given a few demonstrations, and proposes an algorithm that consists of a Q-attention module and a confidence-aware critic. The Q-attention module is an RL agent, which takes image and point cloud inputs with pixel positions as actions. The image and point cloud observations are then cropped, centered at this pixel position output, and passed down the pipeline to the critic. To make the best use of the demonstrations, the replay buffer is initialized with the demonstrations and further augmented with transitions from demo states, chosen through a keyframe selection process. 



Strengths: 

- The results demonstrate how, in the difficult setting of learning from sparse rewards and image inputs, only the proposed approach can learn meaningful behavior while the other RL approaches achieve basically zero return. Though, it would be good to include the returns of the expert agent in the plot as well to show the upper bound performance.
- Through an ablation study, it’s clear that the Q-attention agent is important for its ability to reduce the input dimension to the continuous control agent. The demo augmentation process is also quite critical to ensure that the Q-attention and continuous control agents can correctly learn together, since the success of one agent relies on the success of the other agent. 
- The paper is overall well-organized and written, and the figures and illustrations are helpful for the reader’s understanding.



Weaknesses:

- It is not completely clear what advantages the Q-attention agent brings over standard attention modules. While a discussion is included, a plot or some other form of quantitative comparison would be more illustrative. 
- It’s also not clear why an image crop is the best choice of input for the control agent. For instance, an alternative might be the pixel position itself. This is also highly related to keypoint representations, and there isn’t a discussion of or comparison to them.
- There’s also a key assumption made here that only a fixed region of the image input is relevant to the task at any given timestep. I’m not sure if this is entirely reasonable for environments with different camera viewpoints and different sized objects to manipulate.
- The keyframe selection is based on heuristics: it looks at frames where (1) the gripper state changes as it corresponds to the agent grasping or releasing an object and (2) the agent velocity is near zero as it often corresponds to a new stage of the task.
- The choice of baselines also seems quite arbitrary.



Recommendation:

I am recommending to reject this paper. There are a number of choices in this approach that could be further justified, especially with regards to the Q-attention agent.



Questions:

- Is the pixel position outputted by the Q-attention agent also passed into the continuous control agent? If not, could doing so speed up learning since the cropped image doesn’t provide information about where objects are?

- Instead of a cropped image, is it sufficient to just pass the pixel position, or perhaps the top k pixel positions, given by the Q-attention agent to the continuous control agent? On that note, this is highly related to keypoint representations and it would be good to include a discussion of this in the related work section. 

- How well does the Q-attention agent scale to higher-dimensional image inputs? Additionally, how does the crop size affect performance and efficiency of the continuous control agent? That is, with larger objects and a different viewpoint, a much larger crop may be necessary to capture enough context to solve the task. 


Minor Comments:

-The labels of Fig. 6 need to be bigger.



Update after Reading Authors' Response:

I appreciate the authors’ thorough response and the new results for different crop sizes. It’s not too surprising that the performance varies with the crop size and that the best crop size depends on the task and environment (among other variables such as camera resolution, camera viewpoint, and object sizes). And so, I share the concern of the other reviewers: it’s unclear whether this approach would be similarly successful in other manipulation settings without prior knowledge of the task outside of RLBench.
review_title: Interesting approach to attention in robotic manipulation but requires further clarification

final_score: rating: 7: Good paper, accept
review_content: The paper tackles the problem of training a robotic manipulation model, starting with a set of examples of successful expert trajectories. Training starts with a select set of samples from the expert episodes as the seed of an experience buffer and proceeds to generate new episodes to train on through an RL procedure.

The robotic manipulation model model first applies an attention mechanism in the form of choosing an area of the input images to crop around (though use of a cropping Q function). It then selects the next desired robotic pose from the cropped input using a version of SAC that uses a confidence-augmented Q-function loss and finally applies a motion control mechanism to translate the desired end effector pose into robotic actuation in the simulated setting. 

The method makes use of some assumptions regarding what would be the important states and makes use of a selection mechanism to populate an experience buffer. Specifically, it assumes that points in time where the gripper is actuated or velocities approach zero represent necessary steps ("keyframes") in the completion of a task. It then generates examples where the desired robotic action for a state along an example trajectory is the end effector pose of the next keyframe and the desired crop is around the projected location of the desired end-effector pose in the camera image.

Results show that whereas vanilla implementations of SAC, TD3 and QT-Opt with a standard replay buffer can not learn the tasks, the proposed method succeeds. A further ablation study shows the crucialness of the attention-driven cropping step in this setting and the importance of the sample selection strategy and the confidence layer of the SAC Q function.

I find the combination of these applied characteristics of the method to be sensible and powerful:
 - Starting with expert-provided trajectories (and breaking them down into steps / example-poses using the proposed example sampling method)
 - Focusing of the input signal towards the region of interest (bootstrapping with end effector locations) as a better state representation.
 - Using high level control as the action space (next desired pose as opposed to immediate joint actuations)

The method is presented as a general manipulation algorithm. While it does seem to generalize to a rich set of examples in RLBench, it makes use of some properties that are limited to a subset of manipulation tasks. Specifically, tasks where the important activity is limited to a narrow region of the observed field of view and is comprised of steps that can be treated as series of intermediate goals (poses in which velocities are small or the gripper is actuated). It would have been interesting to see whether there are RLBench tasks that should be solvable using a single camera view on which this method fails and to understand the failure reasons.

The methods compared against feel like straw-men. Surely there must be some off-the-shelf methods targeted at learning from expert examples that do not completely fail on these tasks?

There are multiple methods these days that start by transforming the observations into task-oriented state representations (certainly a focus of interest for ICLR). It would have been interesting to see comparisons with alternatives to image crop as a representation (https://arxiv.org/abs/2007.07170 presents one such example).

The paper does not provide code to replicate the experiments. Seeing as it aims to address a known application, this is unfortunate as it would help clarify implementation details and support further research.

Nits:

I found the representation of the point cloud unclear. It seems like it is treated as another RGB from a different view (e.g. cropped to 16x16 pixels?) but if that is true, why call it a point cloud?

The use of the per-pixel Q values (the downstream Q function that has confidence values) is a bit unclear. It might be a no-brainer to someone more familiar with SAC but I found its use to be unclear just from reading this paper. Same is true for the use of 2x Q-functions (referred to as "the double Q trick").

It was unclear to me whether during use, the next iteration of pose selection is performed once the robot has reached the previously selected pose (though the low-level motion control agent) or whether the pose selection occurs at every simulation step regardless of distance from the previously generated pose.

review_title: An interesting solution to training a robotic manipulator using expert examples in sparse-reward  tasks.

Meta-review: This paper proposes  a general manipulation algorithm for tasks that have sparse rewards. The algorithm uses a Q-attention to extract interesting pixel locations with an explicit attention module. A data augmentation method is also proposed to generalize expert demonstrations. 
While the proposed method and experiments seem interesting, two out of the three reviewers had several issues regarding the clarity of the paper. The main issue is that a better ablation study needs to be performed to assess the proposed system. For instance, the reviewers question the advantages the Q-attention agent brings over a standard attention module. The authors provided detailed explanations and a video showing examples of the expert demonstrations. It is not clear however if the clarity issues are completely resolved. 
Decision: Reject