Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper introduces a contrastive learning objective to learn representations for an RL agent. The basic objective is taken from ReLIC, but instead of augmenting states (which requires prior knowledge of the problem domain) for each timestep the authors use timesteps of the other trajectories in a minibatch as negative examples.
main_review: The proposed approach seems like a novel combination of existing methods with the advantage of requiring less prior knowledge and higher sample efficiency.
Using time steps from other trajectories in the buffer seems well motivated and avoids the need for designing augmentations.
Experiments are comprehensive and the ablation study is appreciated.
A minor gripe is the use of Y for input and X for output of the transformer.
summary_of_the_review: The proposed approach is a novel combination of existing methods that is well motivated. Experiments are comprehensive, including an ablation study.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper introduces a new transformer architecture, called CoBERL, targeting reinforcement learning. The key ingredients of the proposed architectures are the use of LSTMs to reduce the need of large transformers and a contrastive learning procedure that doesn't require of human data augmentation.
main_review: Strengths

* The problem of the growing size of transformers as we require larger time dependencies is of great concern, the trick of forwarding these representations to an LSTM layer is simple yet effective from the results provided
* Avoiding the need of hand-engineered data augmentations is a big plus for representation learning subobjectives
* The paper evaluates the proposed solution in multiple hard and well known benchmarks, authors also provide relevant ablation studies

---
Weaknesses 

* General comments:
   * My biggest concerns are about clarity and presentation, the paper relies heavily on readers previous knowledge, which requires combined combined intuition from multiple disciplines. Additionally, several points of the architecture are never explained (neither given a intuition to the reader). Below I provide specific details
   *  The number of random seeds in the empirical evaluation, only 3, is below current accepted standard for accepted RL works (which is a minimum of 5).
---
Below I detail specific points that I believe need further clarification (in order of appearance):

* I missed a brief background section explaining how representation learning is used in RL, also, this paper gets and improves ideas from ReLIC, we should be able to understand this work without requiring go to ReLIC's
* Section 2.1, 3rd paragraph, you mention that 15% of the embedding is masked, it seems a very specific value, which ones did you explore? what happened with those different values? (some intuition)
* Also there, and in section 2.1 in general, I missed to be given intuitions of why each component was designed in that specific way. E.g., starts directly explaining the masking, I would try to build into the reader the intuition of  why are we masking? before entering in the details. This is much better done in section 2.2 for instance.
* What is exactly doing the "Gate" function? it is mentioned multiple times, and it is evaluated in the ablations, but I couldn't find where is it explained what is this function. It is mentioned that helps to skip the transformed output until the weights are warmed-up, how does it knows this?. I believe that the answer to this would also fit well in a background section.
* Related work section, paragraph 2, says "Second, CoBERL combines the transformer architecture with GRUs to..." until this point GRUs are never mentioned, only LSTMs, so it is not evident if you use both indistinctively (if that is the case I would only say recurrent layer) or if not, where are the GRUs in the architecture and what is their role.
* First paragraph page 6, second to last line, you say "We use Peng's Q($\lambda$) " this is missing a proper citation.
* Small typo, two lines below what I mentioned above, you are missing an space after V-MPO

----------
### Post Author's response and updated version

I think that authors did a good work in the updated version, the work is much better self-contained now and easier to follow. Importantly, now all the experiments are the result of 5 independent runs instead of 3, which raises the reader's confidence in the empirical results.

Given the strengths I reported in my original review, I updated my scores.




summary_of_the_review: The new architecture is interesting, proposes novel ideas that solve important issues of previous solutions and shows good empirical results. However, the paper has clarity issues and doesn't seem completely self-contained. Also the reduced number of random initializations goes below usual quality standards in published RL work. 

Still, I believe these points could be fixed through the rebuttal, so I am open to revising my score if the authors submit an updated version that tackles these points.

----------
Authors correctly addressed my concerns in the updated version
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper replaces the recurrent architecture of R2D2 with a Transformer-LSTM hybrid, and introduces a bi-directional contrastive loss that will be combined with the RL loss for overall optimization. The architecture is supplemented with a learnable gate between the Transformer and LSTM layers that should learn to regulate whether Transformer's output at time t should be fed into LSTM or it is not necessary, making the overall learning easier and leading to higher data efficiency and performance than the baseline (GTrXL).
main_review: Normally when I hear "LSTM" I think "hard to train, requires many samples". When I hear "Transformer" I think the same thing. Yet in this work it is being claimed that a combination of these data-hungry methods results in higher data efficiency that with other methods. At which point I naturally get suspicious :) and want to investigate more.

Major concerns / criticisms / discussion
----------------------------------------

* The abstract promises that "COBERL consistently improves data efficiency across the full Atari suite, a set of control tasks and a challenging 3D environment", however most of the reported results are about performance improvement (after 200M steps (on Atari)). The results on improved data efficiency come in a for of AUC curves as compared to GTrXL. My understanding is that GTrXL was chosen as baseline because it demonstrated the highest human normalized score, but if the main question is about data efficiency and AUC, then a method that rises quickly (Rainbow?) and plateaus later would have a higher AUC, especially at an earlier cut-off points (for example 50M instead of 200M). What if we change the metric from "AUC at 200M steps" to "AUC at the moment when an algorithms achieves 100% human performance"? The motivation behind this line of thinking comes from domains where data samples are expensive (real robots, medical observations, etc) where what we care about is getting to good performance fast, and not so much about "being more data efficient by the time we have collected 200M data samples". Do we know that GTrXL is a good baseline for data efficiency even if we discard the arbitrarily chosen (by the community) mark of 200M samples?
 
* The concept of "self-attention consistency" that seems to be a crucial idea driving this contribution seems underexplained to me: what do authors mean by self-attention being consistent, consistent across what, and why that consistency is a good thing?

* A subjective note on readability: This paper was quite a hard read, perhaps because the explanation of the introduced concept and main ideas is mixed with implementational details and minor notes throughout the text. This makes it a bit hard to distill the core idea and important parts of the contribution from non-essential implementational details. 


Questions
---------

* In the Rainbow paper (https://arxiv.org/pdf/1710.02298.pdf) the reported improvement over human is ~200%+ (Figure 1), in the current work it is reported as 874%, where does this difference come from?

Minor remarks
-------------

page 3 "augmentations make other methods less data efficient" -- unclear what is meant by this statement, isn't the point of augmentations the opposite?

page 7 par 1: typo "Figure 2 presentt this analysis"

Figure 2 caption "els.(higher is better)" -- some formatting issue with spaces and the dot
summary_of_the_review: My evaluation largely depends on whether the chosen beaseline is consistent with the stated main contribution of this work: would AUC of other methods for RL would also be lower at earlier cut-off points. I see two potential possibilities:
(a) The chosen baseline is actually good for data efficiency -> the main claim of the paper holds, all's good
(b) The baseline was not good for data efficiency -> the paper should not claim data efficiency, but rather just go with performance increase (1425%, while baseline is 1202%).
I am looking forward to the discussion to gain clarity on this point.
review_title: null

Meta-review: This paper introduces a new transformer architecture for representation learning in RL. The key ingredients of the proposed architectures are a novel combination of existing methods: (1) the use of LSTMs to reduce the need for large transformers and (2) a contrastive learning procedure that doesn't require human data augmentation.  The resulting approach requires less prior knowledge and provides higher sample efficiency. The paper is convincing, with comprehensive experiments on multiple challenging and well-known benchmarks and an ablation study. The reviewers did expressed concerns that parts of the paper are a very difficult read and could use improvement, especially those relying on substantial external background. The intuition behind several components could be improved, and there are some clarity issues, as detailed in the individual reviews.
Decision: Accept (Spotlight)