Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents a method to end-to-end learn manipulation tasks from raw multi-modal sensory data. Multimodality refers to the use of both propriocetion and RGB-images. The proposed RL method is model-free and off-policy. It relies on image augmentation to learn a multimodal representations fed to an actor and a critic network. The proposed methodology is tested on three robot manipulation tasks from the Deepmind Control suite. Comparisons with state-of-the-art alternatives (i.e. DrQ-v2, DDPG from state and SAC from state). Remarkably the proposed approach does better than all other approaches, including DDPG from state. Ablations studies show: (1) the importance of decoupled representations for the actor and the critic networks, (2) the importance of choosing the representation dimensions, (3) the importance of layer normalization and (4) the minor benefits of scheduled exploration noise. 
strength_and_weaknesses: The paper present results which are beyond the state of the art. However, presented results are limited to three relatively simple tasks for which a staged reward was needed. The proposed approach is relatively standard and the major theoretical contribution seems to be just a marginal improvement with respect to existing alternatives.
clarity,_quality,_novelty_and_reproducibility: The paper is well written and easy to follow. Its novelty is marginal. Reproducibility is conditional on authors open-sourcing the code and authors don't mention this possibility. 
summary_of_the_review: I think the paper is relatively well executed and the published results are a significant step with respect to the state of the art. However, nothing really new is presented in the paper to the point that it becomes surprising to notice that these significant improvements could be achieved with minor adjustments to the network architecture (see ablation studies). Also, the paper is limited to 3 simple tasks which limits a lot the soundness of the proposed approach.

I usually try to be more verbose and provide some useful feedback in my reviews but I don't have much to say on this specific paper. My final score is "below threshold" but I am struggling with finding the right evaluation; it would be great if the community could have the code and test the approach on more numerous and more complicated manipulation tasks.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents an algorithm called Multimodal End-to-end Reinforcement Learning (MERL) that integrates visual and proprioceptive observations in learning model-free reinforcement learning policies. Specifically, the method builds on SAC and DrQv2, and adds encoders to handle both proprioceptive and image encoders into the policy and value function architectures. The method is evaluated on simulated 3D robotic manipulation tasks from DMC (Jaco tasks), and is compared to SAC from state and DrQv2. The authors also perform an ablation analysis of their method, in terms of how multimodal fusion is performed, the size of the multimodal representation, use of layer normalization, and exploration noise. 
strength_and_weaknesses: Strengths: 
- The proposed method is conceptually simple and addresses an important problem of how to perform multimodal fusion for policy learning. 
- The empirical results on the Jaco tasks are strong.
- The paper includes detailed motivation and description of the design decisions for the method, which is quite important for model-free deep RL methods.
- The paper is well-written and clear.

Weaknesses:
- Overall, I feel that the paper is too narrow in its scope for the claims that it makes. While the method is proposed to be a general multimodal framework, it is only demonstrated in a single setting with proprioceptive and image data, rather than consider other modalities like touch, sound, etc.
- Furthermore, the evaluation is only performed on three DMC tasks, and there are many specific design decisions that are made that could impact performance on these tasks in particular.
- According to the ablation study, the performance improvements of the method seem to be largely attributable to the use of layer normalization in the architecture. It would be helpful to understand if layer norm only helps MERL or if the other baselines also improve on this task with the use of layer norm.

clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is quite well-written and clear. However, one point which I think would benefit from additional explanation is the ablation of the shared vs. decoupled architecture ablation study. It is not obvious how the shared architecture performs multimodal fusion; a more specific description or diagram would be helpful.

Quality: It’s not noted how many random seeds the experiments are conducted over. This would be helpful for understanding the significance of the results.

Novelty: The idea of having separate encoders for different modalities has been previously explored (Lee et. al, 2019), but not in the same setting as this work. The novelty is primarily in the application of this type of method to this particular setting. 

The authors have thoroughly provided implementation details such that this work seems reproducible.

summary_of_the_review: Overall, the paper is clear, detailed, and well-executed, but I think that in its current form, there are three main shortcomings: 1) only proprioception + vision are tested, 2) the ablation studies don’t completely demonstrate why MERL is helpful (see layer norm comment above), and 3) there are a limited number of tasks and only from the DMC suite.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper introduces MERL, Multimodal End-to-end Reinforcement Learning, a framework that combines multimodal (namely vision and proprioception) representation learning and RL.
strength_and_weaknesses: The paper is well written, clear and structured in a nice way. The method is introduced in a clear way, as well as the experimental setup.
Some quite important pieces of the approach adopted are not described in the main paper, such as the formulation of the reward stages used to train the agents, which actually can play a very important role.
The empirical evaluation includes interesting ablations of the method, examining the impact of various design choices.
Some important information are missing or unclear. What did you mean with the sentence "Unlike in ... state-based RL such as DDPG and SAC, our method stores transition ... in the replay buffer,"? Both DDPG and SAC use a replay buffer.
My main concern is that, in my opinion (unless I missed something important from the paper), there is no novelty. Vision+proprioception based agents have been trained successfully in several robotics domains, including manipulation (eg https://openreview.net/forum?id=U0Q8CrtBJxJ, http://proceedings.mlr.press/v87/kalashnikov18a.html, etc). The two modalities considered, that is proprioception and vision, have been explored in several works, and most recent works look at other challenging modalities in the context of multimodal representations especially in robot manipulation, in particular using tactile or audio signals, for example, in addition to vision and proprioception.
Other considerations, like the observation that having gradients from the agents going through the representation module during training hinders the performance, are well-known as well.
What differentiate your approach for existing vision+proprioception RL approaches, especially in the domain of robot manipulation?
Training directly from vision is in fact a challenging task, in particular it can be very data-inefficient; how did you approach this problem and what made your approach overcome it?
clarity,_quality,_novelty_and_reproducibility: The clarity of the paper is good, however I think there is no novelty in it (unless I have completely missed some important thing of this work).
summary_of_the_review: The paper is well written, but lacks novelty.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a new reinforcement learning architecture for learning robot behaviors from pixels.
strength_and_weaknesses: Strenghs:
- very important and broad problem to be tackling
- well-motivated, general approach
- detailed description of the algorithm, including hyperparameters
- excellent set of ablations

Weaknesses:
- Some of the prior art is somewhat misrepresented. Many of the works cited in the section which ends with 'Such methods are commonly referred to as state-based RL.' are not state-based, but learn directly end-to-end policies from pixels (Kalashnikov et al, Ibarz et al for instance). [ADDRESSED]
- The paper compares against a single baseline (DrQ-v2), but only on a narrow subset of the tasks that DrQ-v2 was able to perform. In particular, being able to train humanoid from pixels was one of the strong claims from DrQ-v2, and experiments in this setting are notably absent from this paper, and (somewhat conveniently) excluded by making this a manipulation-centric paper. This makes this reviewer very suspicious that maybe the authors tried their approach on humanoid, it didn't work well, and they maybe opted to not show these results. If that were the case, I would insist on these results to be incorporated in the paper, so that a reader would have a complete picture of the behavior of the approach. If there are other reasons why focussing on the three Jaco tasks made sense, I would love to see it being made explicit in the paper. [ADDRESSED]
- The paper does a good job at contrasting the architecture choices made against DrQ-v2. It would have been interesting to 1) further contrast with other pixel-to-action approaches that have been explored in the past, and 2) try to provide some intuition as to why this alternative architecture was interesting to consider beyond merely the empirical results. [ADDRESSED]
- No real-world results. I am very suspicious of any result which only evaluates algorithms on 'simulated pixels'. There is little evidence for a strong correlation between relative performance in these settings and real-world settings. [Not addressed, but OK to not have in scope of this paper.]
- No discussion of systems aspects: training step and inference speed, memory in contrast to other approaches. [Not addressed, but OK to not have in scope of this paper.]

clarity,_quality,_novelty_and_reproducibility: Approach is clear and novel. High degree of reproducibility.
summary_of_the_review: Solid contribution, however validated against a single baseline, on a narrow set of evaluations, only in simulation. To make this paper truly convincing, I would want to see 1) a broader palette of pixel-to-actions tasks being explored or 2) justification as to why only manipulation ought to be in scope, or 3) real-world results.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper presents a new RL approach that combines visual and proprioceptive observation to learn policies. The reviewers appreciate the strong experimental results and comparisons with DrQv2 along with the ablation analysis. The multimodal fusion idea is conceptually simple and easy to understand. However, there is a major concern shared by the reviewers on the novelty of the work. The paper is quite narrow in scope as design decisions in 3 DMC tasks may not translate to real-world improvements on robots. The author feedback was helpful to get additional perspective. For example the 3 Humanoid tasks improved the scope. Nevertheless during the discussion phase, the reviewers found that the novelty concern still present. In the words of one of the reviewers : "I find it difficult to identify the true novelty in it. Maybe the novelty is the use of a decoupled representation, one for the actor and one for the critic? In any case, why (what is the reason behind) the method outperforming the other variants is still unclear to me."

I highly encourage the authors to strengthen the writing of the paper vis-a-vis the motivation and novelty of the method. Perhaps extending the experimental evaluation would also convince the community that the algorithm generalizes beyond the existing tasks.

justification_for_why_not_higher_score: Novelty is unclear. Experiments are quite narrow on DMC tasks,
Decision: Reject