Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a learnable data augmentation procedure for speeding up reinforcement learning from pixels. In particular in applies a transformation to the input images, which is parameterized by a Gaussain distribution. The proposed method combines three main components:

1. Standard Q-learning approach with data augmentation, similar to DrQ
2. Contrastive learning loss for the encoder
3. Smoothing objective for the transformation parameters

The paper proposes a theoretical justification for this approach and shows state-of-the-art results on DM Control and Atari. 
strength_and_weaknesses: Strengths:

1. While practical the proposed method is also justified from a theoretical perspective as well.
2. The approach is intuitive and straightforward.
3. Produces state-of-the art results on some standard benchmarks. 

Weaknesses:

Only wandering what results are on the Humanoid task, which has now been successfully solved from images. 
clarity,_quality,_novelty_and_reproducibility: The paper is well written and easy to follow. The supplementary material has ample implementation details for reproducible. While the core idea of data augmentation for visual RL is not new, the proposed method takes that further and produces state of the art results. 
summary_of_the_review: Well written, intuitive paper with good justification and results.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a simple yet effective learning module upon the framework of contrastive reinforcement learning. The module could automatically learn a optimal transformation for data augmentation. The algorithm is theoretically founded and also demonstrated by expeiments on DMControl and Atari 100k.
strength_and_weaknesses: Strength:

- The idea is simple but effective, which might be easy to implement and follow.

- The authors make theoretical efforts in justifying the framework.
- The experiments are extensive, conducted on DMControl and Atari 100k. Also the baselines (DrQ v2 and CURL) are decent.

Weakness:

- The theoretical analysis in this paper uses several inequalities to achieve the final objective, while some reasoning looks not very clear to me. See following comments for more details.
- In Equation 5,  the authors use the triangular inequality $d(g(x),g'(x'))\leq d(g(x),g'(x))+d(g'(x),g'(x'))$ so that they could minimize the upper bound. However, the LHS $d(g(x),g'(x'))$ could be optimized directly. So what is the necessarity of using the upper bound? And it is better that the necessarity be illustrated  from both the theoretical perspective and the empirical perspective. I am very willing to take feedbacks from authors.
- In Equation 6, the authors still try to prove that minimizing the LHS means minimizing the RHS. However, the LHS is the lower bound and thus it seems not meaningful to minimize the lower bound. If I do not misunderstand, the main theorem might not be able to motivate the CoIT algorithm. There is a chance I misundertand and there could be more efforts in detailing the theorem.
- The writing could be improved. There are many variables not well stated. For example,  in Theorem 4.1, the variables L_g and L_f are not stated in the main paper. 

- The authors propose a multi-task objective that combine three loss functions in total. Though it is common that different objectives are used with different weights, the hyperparameters given in appendix seem to show that the weight for objectives must be carefully choosen for each tasks to make it work. More explanations about the hyperparameter selection and the experiment observation about the hyperparameter might be necessary.



Typos:

- related work: DRQ -> DrQ
- main results: Table 2 -> Table 1
- Some theorem links are not correct. One careful check might be great.

note: this is an emergency review
clarity,_quality,_novelty_and_reproducibility: The clarity of the main theorem is not clear to me, and I expect more explanations from authors . Other parts are good.

There are several typos and wrong links in the paper, which might hurt the paper quality. 

The novelty is limited but I appreciate the efforts that make it work.

The reproducibility is good since the authors provide the source code.
summary_of_the_review: I like the proposed algorithm for its simplicity and the theoretical analysis is also somewhat interesting. The main concern for me is (a) whether the main theorem is constructive for the algorithm and (b) whether the objectives really help generally for different tasks or it is carefully tuned for each single task to make it work. The detailed questions are mentioned before, which I hope the authors could address.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a learnable data augmentation framework for visual RL, which learn the parameterized image augmentation together with model-free RL. Some theoretical interpretations are provided to show the rationale of learned augmentation. Experiments on DMC and Atari100 benchmarks demonstrate that the proposed method CoIT achieves better performance than SOTA augmentation methods for model-free RL.
strength_and_weaknesses: ### Strengths

- The idea of learnable augmentation is relatively novel.
- The authors provide some theoretical interpretation for why learnable augmentation is desired.
- The proposed method achieves relatively promising results on several benchmark environments. 

### Weaknesses

- I think the proposed algorithm seems simple yet effective. I like the idea of learning augmentations. However, the theoretical analysis sections are a little confusing and not very to the point. First, why is the replay buffer fixed? During training of model-free RL, I would expect the replay buffer to be consistantly updated, with imperfect data coverage. It is weird to me that the paper motivates by "dynamic distribution" and "training-friendly" but does not discuss much on the distribution shift problem during training. Second, I don't think Section 4.1 and 4.2 have provided sufficient evidence for the "optimality" of the proposed metric and the algorithm. Some claims are not very convincing. For example, why does Theorem 4.1 suggest that "incorporating augmentation directly cannot well meet the basic stationary environment"? 
- There exist a bunch of grammar errors and typos, which may cause confusion. Some notations are not clearly defined in the context. For example, in Theorem 4.1, what does "Lemma 4.2 holds for functions $f_{\xi}$ and $f_{\bar{\xi}}$ respectively" mean? Lemma 4.2 is a lemma, not a condition for $f$. Also, $L_g$ and $L_f$ appear in the theorem without definitions. 
- The ablation study also confuses me. Why does the convergence of $\mu$ and $\sigma$ mean that TRANSFORM successfully smooths the distribution shifts? Can the authors provide more explanation?

Questions:
- Why is Gaussian distribution used to generate augmentations? Would a deconvolutional network work here?
- How is "tasks solved" defined in Figure 1?
clarity,_quality,_novelty_and_reproducibility: The novelty and quality of the paper are relatively good. But the clarity needs to be improved, as summarized in "Strength And Weaknesses".

For reproducibility, a Github repo is provided, although I have not tried to run it yet.
summary_of_the_review: This paper presents a relatively novel idea and show good empirical success of the method. But there are several clarity issues of the theory and critical claims, making it hard to justify the quality of the paper. I would like to increase my rating if the authors can address my confusion with convincing evidence.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new framework to improve the sample efficiency of model-free RL algorithms through combining them with a learnable data augmentation approach. The paper provides a theoretical analysis of how their method works to improve the representation learning and reports results on DMControl and Atari100K.
strength_and_weaknesses: Strength:
1. The idea proposed in this paper is novel.
2. The proposed idea is presented clearly and the paper is globally well-written.
3. The theoretical analysis is provided.
4. The proposed framework is easy to implement.

Weakness:
1. Although the idea seems to be new, the contribution of this paper seems to be limited. According to Table 1, the improvement resulted from the proposed framwork is quite limited. Second, it is not discussed whether the proposed method can also contribute to improved performance in addition to sample efficiency. Besides, the improvement on stability is not very obvious, more explainations should be provided.
2. The paper lacks comparison with more recent methods, such as soda, svea and so on.
3. The paper lacks parameter analysis. For instance, how are hyperparameters α and λ determined?

Minor:
1. The details of how to adjust to the optimal Gt(µt, σt) can be mentioned in Algorithm 1.
2. Lf and Lg should also be mentioned in the paper.
clarity,_quality,_novelty_and_reproducibility: Clarity: The proposed idea is presented clearly.

Quality and Novelty: The idea is novel and the theoretical analysis is also provided. However, the contribution and improvement seem to be limited.

Reproducibility: The code is provided.
summary_of_the_review: This paper proposes a new solution for the problem, and experiments on two benchmarks demonstrates its effectiveness. The major concerns are limited contributions and the lack of comparison with more recent approaches.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper presents a new data augmentation approach, which is learnable, to improve visual reinforcement learning. Reviewers generally find the approach intuitive, effective and novel. There are some concerns on paper presentation and comparisons, but mostly got resolved during rebuttal. The AC agrees with the majority of the reviews and recommend to accept the paper.
justification_for_why_not_higher_score: There are still some unresolved questions from the reviewers.
Decision: Accept: poster