Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies reinforcement learning across domains in the presence of visual distractions. A visual distraction is a part of the underlying state that is uncontrollable, i.e., its change is independent of agent's action, and does not affect the Q values. The proposed approach uses domain randomization in a Deep Q-learning setup. At each time step $t$ in an episode, one assumes access to two observations $o^1_t$ and $o^2_{t+1}$ which denote the same controllable state but different uncontrollable state. Here the superscripts 1 and 2 are used to denote the two different domains. The proposed approach includes two additional loss:

- a domain transfer loss: $L(o^1_t, a_t, r_t, o^2_{t+1}) + L(o^2_t, a_t, r_t, o^1_{t+1})$, where $L$ is the temporal difference loss defined by $L(o, a, r, o') = (Q(o, a) - r - \gamma \max_{a'} Q_{target}(o', a'))$. This is done to ensure that Q functions remain invariant to the uncontrollable noise. This claim is not proven.
 
- a variance loss that computes variance of $\{L(o^1_t, a_t, r_t, o^2_{t+1}), L(o^2_t, a_t, r_t, o^1_{t+1})\}$. This is used so that losses for both domains are roughly of the same scale. It is not explained why this would happen but a reference to Krueger et al., 2021 is made.

Experiments on Distracting Control Suite (Stone et al, 2021) are presented and show promise of the approach.

strength_and_weaknesses: Strength:

1. Robust reinforcement learning (RL) is an important topic and visual distractions are one of the important things an RL agent needs to be robust to. 
2. Proposed approach is simple to implement and test

Weakness:

Two main concerns are:

1. The proposed approach requires access to a pair of observations at each time step that have the same controllable state $(s)$ but different uncontrollable state $(\bar{s})$. If I understand correctly, experiments use the knowledge of how visual distractions are added to generate these observations. How will this be accomplished in practice? E.g., consider a driving scenario where the agent needs to drive. The observation space is an image that shows what is ahead of the car. On the side of the car, there can be noise from people walking or varying weather conditions. How will one generate this pair of observation that only changes these distractions but not the controllable state? This may require domain knowledge almost equivalent to knowing the state. In contrast, the domain knowledge required in computer vision pipeline, as abstracted in a data augmentation setting, is very basic and includes simple operations such as rotation and flipping. 

It is also assumed that different domains differ only in the uncontrollable state noise but otherwise have the same state. This is a particular kind of domain generalization. For example, if one considers navigation in two different buildings, then even the underlying state is different.

2. The writing in this paper is very low quality. This includes mathematical notations used without definition, using words without definition, etc. See clarity section below for examples. Even though the main ideas are easy to understand for an RL audience, the paper needs serious editing for readability.
clarity,_quality,_novelty_and_reproducibility: Clarity issues:

1. What is $S_t, A_t, R_t, T_{t+1}, t$ when transition is defined? You want to specify that $t$ denotes a timestep.

2. what is $r(T)$? Do you mean the reward of transition $T$?

3. It says in the line below equation 3 that $L_Q$ is defined in Equation 1, however, that is not the case. You want to instead define $L_Q$ separately as:

$L_Q(s, a, r, s') = (Q_\theta(s, a) - r - \gamma \max_{a'} Q(s', a'))^2$

and then use it directly in Equation 1. 
 
4. What is $R^i$ in Equation 2? Do you mean $R(D^i)$?

5. DQN equations are defined in terms of state but later the state is considered latent. Consider defining observations and latent state from the very beginning in Section 2.

6. How is a domain defined? Do domains differ only in how the style variable $\bar{s}$ is defined and evolves? 

7. What does $X \sim \mathcal{D}$ denote in definition of $R_l(D, \theta)$? What is this distribution over the domain? Similarly, what is meant by _"training distribution is partitioned into multiple domains"_? Are we referring to distribution over transitions? While it is intuitively clear what is happening in Section 2, sometimes (as above) the exact meaning of each line is hard to understand since terms are not clearly defined. 

Questions

1. For domain translation loss, why not directly consider minimizing $(Q(o^{1}_t, a-t) - Q(o^2, a_t))^2$ instead? This will ensure that the Q values agree on these pair of observations.


Related Work

Following are a few related works that address learning distractor-free representations and are not discussed:

1. _Denoised MDPs: Learning World Models Better Than the World Itself_, Wang et al., 2022
2. _Provably Filtering Exogenous Distractors using Multistep Inverse Dynamics_, Efroni et al., ICLR 2022

Minor Points:

1. _"heavy visual distractions"_: instead of saying heavy, give explanations for why they are hard

2. _"mixing..."_  -> drop "..." and start the sentence with such as or e.g., 

3. - _(Figure 1_ missing a closing bracket

4. - "(DIQL)for" lacking space

summary_of_the_review: I currently lean towards weak reject based on two concerns: (i) strong assumptions on generating a pair of observations with the same state but different style variable, and (ii) issues with writing. For (i), I would like to understand how to satisfy this assumption in practice outside simulations. I'll change my score based on author response (and paper revision) to these concerns.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper presents an approach for augmenting Q-learning to take advantage of domain randomization, name DIQL.  By producing observations from two different domains (here defined as visual observations with different distracting perturbations, as in the distracting control suite), DIQL can train its q-network to be invariant to the sorts of perturbations that do not affect the task.  
strength_and_weaknesses: The paper makes an interesting comparison between data augmentation and domain randomization, and suggests that RL methods can take advantage of invariance built in to different domain randomization techniques.  The method is described well and the experiments show empirical advantages of the DIQL approach.

The paper currently has the following weak points, but these may be a result of a misunderstanding on my part: 
- DIQL depends on having $o^1_t$ and $o^2_t$ where each comes from a different domain, but where the underlying state $s_t$ is the same.  How can this correspondence be found?  If this is just assumed known (as it must be to generate different domains), why not give direct access to the state $s_t$ to the learning algorithm rather than the more roundabout domain-translation terms in the loss?  Many other task-relevant state representation methods I am aware of do not assume this correspondence is known, but rather must select positive and negative examples for, say, contrastive learning.
- I would have liked to see augmentation-based representation learning methods, since the paper makes a point to contrast domain randomization with data augmentation.  Additionally, some selection of bisimulation-based and contrastive methods (like the ones cited in the paper) would be good to compare against empirically.
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written, and aside from some missing experimental comparisons, the experiments are well designed and present interesting results.  The different loss terms introduced in the paper for augmenting Q-learning based on different domains seem to be novel.    
summary_of_the_review: The paper presents an interesting method for learning task-relevant representations of high dimensional state.  However, I am not convinced that the method is feasible, since it requires knowledge of the ground truth state to generate pairs of observations, and this knowledge seems like it should be off-limits to any approach that attempts to learn task-relevant features.  I'm hoping I've misunderstood a main point in the paper, but if not I do not think it passes the bar for acceptance.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper tackles the problem of distraction robustness and out-of-distribution (OOD) generalization in model-free RL. To this end, the paper proposes to use domain randomization combined with some modification to the Q-learning objective. Specifically, the model takes two views, one clean and one noisy. The noisy view is generated by adding random noise to the groundtruth state (as opposed to pixel-space augmentation). For each view, there are two loss terms, regressing the Q-network toward the two Q-targets computed from both views (as opposed to only a single Q-target computed from the same view). To further encourage invariance of the Q-network, the paper proposes to also minimize the variance among these loss terms, following a previous work on OOD generalization. The experiments focus on the Distracting Control Suite, with SAC being the backbone RL algorithm and main baseline. It is shown that the proposed method outperforms SAC in distracting environments, and can generalize better than SAC to larger noise during evaluation. Ablation shows that all loss terms have effect.
strength_and_weaknesses: - Strengths
    - The paper is generally well written and easy to follow.
    - Ablation study shows the effectiveness of each loss term.
- Main Weaknesses
    - It seems unclear whether the problem setting is interesting or meaningful to tackle. This paper assumes access to the clean state in order to be robust to distractions. This sounds impractical, and most work in this field (e.g., [DBC](https://openreview.net/forum?id=-2FCwDKRREu), [TIA](https://proceedings.mlr.press/v139/fu21b.html), [TPC](https://proceedings.mlr.press/v139/nguyen21h.html), [Dreaming](https://arxiv.org/abs/2007.14535), [DreamerPro](https://proceedings.mlr.press/v162/deng22a.html)) does not have access to clean states when dealing with distractions. It is even unclear whether the access to clean states is necessary, as the paper does not compare to these methods.
    - The baseline is SAC with no access to the clean state. This seems to be a weak baseline, and it is not surprising that the proposed method, with access to the clean state, can outperform this baseline. A reasonable baseline could be [DrQ](https://arxiv.org/abs/2004.13649), with the augmented views replaced by the clean and noisy views.
- Questions and Minor Issues
    - Why use one clean view and one noisy view, not two noisy views? What happens when you use many noisy views?
    - It is a bit confusing to put models trained on different noise levels into one figure (e.g., Figure 3).
clarity,_quality,_novelty_and_reproducibility: - Clarity
    - The paper is generally well written and easy to follow. It introduces reasonable background knowledge to understand the method.
- Quality
    - The results do not seem significant enough due to reasons explained in Main Weaknesses.
- Novelty
    - There is reasonable novelty to adapt previous work on OOD generalization to domain-invariant Q-learning.
- Reproducibility
    - Experiments are run with 4 seeds. Architecture details and hyperparameters are provided in Appendix.
summary_of_the_review: I recommend reject for the paper in its current form, due to potentially limited significance and weak baselines.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a Q-learning algorithm that combines domain randomization to achieve distraction robustness and out-of-distribution (OOD) generalization. To encourage invariance, an auxiliary loss for variance minimization is introduced from a previous work. The experiments are performed on Distracting Control Suite and SAC is chosen as the baseline. It is shown that the proposed method performs better than SAC under visual distraction.

Strength. The paper is easy to follow. The ablation study is reasonable. 

Weakness. The main weakness is the assumption that the ground truth state is available. All three reviewers raised a concern about this. The authors answered that it's possible because they assume that the access to a simulator where the ground truth state is accessible. But, the compared baseline SAC does not use this clean state. So, the comparison is unfair. In addition, it is clearly an evidence that the paper is poorly written, making all three reviewers confused. 

justification_for_why_not_higher_score: All reviewers agree that the assumption is too strong or unrealistic.
Decision: Reject