Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: Image augmentations have recently become a standard component of deep RL algorithms.

Previous work has enforced consistencies at a sample-level.

This paper proposes to look at the distribution of statistics at a minibatch-level in order to enforce consistencies.

Paper shows results on standard benchmarks in discrete and continuous control (Atari and DMC).
main_review: Strengths:
Simple Idea
Well explained
Standard benchmarks

Weaknesses:
Not a significant improvement compared to prior work.
summary_of_the_review: The idea is simple and easy to implement and people can try to fork this in whatever they're doing. However, the improvements shown aren't significant to strongly push for accepting it.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a regularization method for reinforcement learning that encourages the Q-value of the original image (i.e., original state) and the Q-value of the transformed image (new state) to be the same. This method enhances the robustness of RL methods against environment variation. This paper introduces the background and the motivation of the proposed method. Discussion and comparison of the difference between related works, such as SAC and DrQ, is also provided. Experiments show that the proposed method can improve the performance of image-based methods, even outperforming several state-based methods.
main_review: On the one hand, the proposed method is very simple. It can be added to many RL methods as a regularization term. Experiments show the validity of this method. 

On the other hand, however, the method that pulls the output of an image and its transformed counterpart close to each other is already commonly used in computer vision methods, thus, the novelty of this paper is limited. More study and analysis on the regularization is expected, such as different functions for loss computation besides MSE and KL, how does regularizing Q-value differ from other methods.

Additionally, there are several small issues:
1)	Figures in this paper are far from elegant. Symbols in Fig.1 is too large, while text in Fig.3 is too small.
2)	I suggest reorganizing Sec.4. Some discussion can be set in a single subsection.
3)	In Sec.4, “Equation 2” looks like a typo.
4)	$n$ used in Eq.1 denotes the number of actions, not the batch size.

summary_of_the_review: This paper proposes a simple method to improve the performance of image-based RL. My main concern is about the novelty as proposed method is commonly used in CV methods. More further discussion and study will improve the quality of this paper.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a very simple auxiliary loss to regularize Q-values given mini-batches of actions between images and their affine augmentations.

This approach is then validated on continuous control and ALE environments (standard benchmark).
main_review: The method is very simple and effective but there has been a lot of prior work, some of which is not cited or compared. For example, RAD (Laskin et al. Reinforcement Learning with Augmented Data) is a simpler method which is extremely relevant but there is no mention of it. 

Table 5 in the RAD paper referred above is inconsistent with Table 1 highlighted in this paper. They both refer to the same 500k and 100k scores in the DeepMind Continuous Control Suite. Scores in RAD are sometimes similar or better than the proposed method. But perhaps more surprisingly, the CURL scores are different in these two tables. I am not sure why this would be the case. This is a significant issue that needs to be addressed by the authors. 
summary_of_the_review: The authors should provide an explanation regarding the discrepancies in results highlighted in this paper vs the RAD paper for the same benchmark tasks. The experimental validation and baselines need more work before this paper is ready for publication.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to use data augmentation to regularize the Q distribution by matching the Q values between augmented and unaugmented states (Alg 1). The results are competitive on the DeepMind control suite tasks (Tab 1) and Atari (Tab 2).
main_review: Data-augmented methods for reinforcement learning have been increasing in popularity and this paper demonstrates another way of using augmentation to regularize the Q values that empirically performs well.

The paper is generally well-written and easy to understand the contribution. The comparisons to the baselines of DrQ/CURL/PlaNet/SAC-AE in Tables 1 and CURL/Eff Rainbow are, to the best of my knowledge, the relevant baselines on these challenging tasks. To the best of my knowledge, the data augmentations are similar to the ones used in DrQ/CURL, but used to regularize the Q function on many actions sampled rather than in the standard updates.

I found two minor parts confusing:
1. The definition of the Q-value distribution at the bottom of page 1 is not a proper statistical distribution. It instead appears to be pointing out that for a fixed state, we can consider a collection of values of applying different actions from that state.
2. After eq (1), the paper says the actions $a_i$ are selected from the minibatch for a fixed state $s_t$. This also seems consistent in Alg 1. Do I understand correctly that this uses the actions from irrelevant states in the minibatch to use for this regularization term? If so, is it interesting to consider other distributions over the actions to be sampled here?
summary_of_the_review: It's an interesting new usage of data augmentations that is nicely demonstrated
review_title: null

Meta-review: The paper proposes a simple modification to how data augmentation is done in image-based RL. This results in some improvements on benchmark tasks. The change essentially amounts to adapting data-augmentation strategies that are already understood in other fields to deep RL. However, the effect of data augmentation in simple image-based deep RL tasks is already known. As such, I think the contribution in this paper is quite incremental -- the notion that data augmentation in deep RL helps is already known, and the particular augmentation strategy proposed here is not especially novel. So while it's good in terms of producing improved results on some benchmark tasks, it doesn't seem to be of high significance to the study of reinforcement learning or machine learning more broadly. As such, I think it could be a valuable contribution to a more narrow venue, or as a technical report, but is too incremental and narrow in scope for ICLR.

A note to the authors (this did not impact the paper decision): due to the unfortunately lackluster quality of the reviews, I read and reviewed the paper myself as well to be able to produce a more accurate meta-review. In the balance, I see the point the authors make in the response that some of the results in prior work (e.g., CURL) are unfortunately unreliable. That's not the fault of the authors, it's the fault of the prior works. I took this into account in my assessment. In this sense, I do think the comparison to prior work is sensible. On the other hand, I think the practice of reporting only very specific checkpoints (e.g., 100k and 500k), though borrowed from prior work, is not a good way to report results, as it hides the real performance of the methods.
Decision: Reject