Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a representation learning method for image-based state observations for RL learning algorithms. This is proposed as an auxiliary task built around self-supervision between successive observations, with a distinct focus on disentangling the learned representations to differentiate between relevant and irrelevant aspects of the environment when learning a policy. The proposed temporal disentanglement (TED) augmentation is then shown to improve performance to a handful of selected RL algorithms. 
strength_and_weaknesses: ## Strengths

- The paper is well written and is very clear through the motivation and development of the proposed TED auxiliary task. The clarity falters somewhat in the experimental setup and results sections (specifically addressed in the next section). For the most part, I found the paper easy to understand and could follow the conceptual build-up arguing for the use of SSL techniques when considering representation learning for RL. 
- I appreciate the depth of discussion about the conceptual framework built through related work. This is a great example of good scholarship on the behalf of the authors. I feel confident about where to place the paper's significance as a result of the detailed overview and technical support derived from previous work. There are some missing references however. In particular, there has been a collection of papers looking into SSL within Deep RL in the recent few years that should be cited (more on this below).
- The construction of TED as an auxiliary task is straightforward and seems like it would be straightforward to incorporate into future research. The major challenge is the engineering aspect of appropriately constructing the batch for SSL. It's promising that the authors intend to release their code to smooth over this barrier. 
- The TED augmented algorithms are quite thoroughly compared to relevant baselines on a variety of environments demonstrating the utility of the approach. It was really nice to see the proposed TED applied to different algorithms to demonstrate that the proposed approach is more flexible rather than taking advantage of unique qualities of a single approach/problem domain. The rigor of the experimental analysis is completed with an ablation study showing that the inclusion of the different components in the SSL batch help contribute to the overall performance.

## Weaknesses
- The deviations considered as unrelated/related between episodes are quite narrow. It seems that only color is considered where there are potentially more impactful deviations that are not included (e.g. viewing angle, perturbations in the dynamics, occluded aspects of the scene, etc.). Aside from Figure 1 and a limited collection in the Appendix, there are no visual demonstrations of what the environment variations are. This would be a great inclusion in Section 5.1 to solidify what exactly is meant by distractor and relevant/irrelevant variations. So much of the discussion and explanation of these terms (throughout the whole paper) relies perhaps too much on these specifics being completely understood by the reader.
- The paper doesn't completely outline how $\mathcal{L}_{TED}$ is integrated into the RL loss. It is mentioned that the representation and policy learning are performed hand-in-hand. In algorithm 1 there's no indication of what is done to accumulate the loss with the different samples $x\in\{X', X'', X'''\}$ for every transition in B. 
- The clarity of the experimental evaluation is much worse than the preceding sections (specific details in the next section) which casts some doubt on the overall utility of the the approach. 
- Figure 1 could probably be better detailed in the text of Section 4.1. There are some missing points of notation (e.g. $z_{\tilde{t}}^{\tilde{e}}$ as well as an unclear differentiation between where the representation of the next observation $z_{t+1}^e$ is used in the RL algorithm. Is it assumed that the RL algorithm is multistep? Or is this an implied use of the next observation within the "target" of a TD-error update? 
- The disentanglement metric comes across as an afterthought and is not very clear how and where it's used to evaluate the performance. E.g. how significant is a score of 0 vs. a score of 1? Without this insight, the tables in "Figure" 4 are relatively meaningless...
- There is inconsistency between experiments where the same algorithm is used in the same domain (e.g. CURL/DBC/DrQ on cartpole swingup and panda reach). There is no explanation of where the different experiments differ. Are the results of provided in "Figure" 4 a single run of these algorithms? As far as I can see, the only change between these experiments is the base algorithm that TED was applied to (RAD --> SVEA). Additionally, it seems that the experiments are set-up much differently since there are different time scales (and performances of these baselines) between Figure 2 and Figure 3. *This is a major concern.*
- The ablations are only performed on one algorithm-TED combo. How generalizable are the insights seen in Section 5.3?
- There is a claim that TED enables continual learning without catastrophic forgetting. There are no experiments to test this claim and no other discussion about this term/concept throughout the paper. 

### Missing papers 

```Mazoure, B., Tachet des Combes, R., Doan, T. L., Bachman, P., & Hjelm, R. D. (2020). Deep reinforcement and infomax learning. Advances in Neural Information Processing Systems, 33, 3686-3698.```

```Schwarzer, M., Rajkumar, N., Noukhovitch, M., Anand, A., Charlin, L., Hjelm, R. D., ... & Courville, A. C. (2021). Pretraining representations for data-efficient reinforcement learning. Advances in Neural Information Processing Systems, 34, 12686-12699.```


clarity,_quality,_novelty_and_reproducibility: ## Clarity

In Section 4.2 it's not clear how the stacks of image observations differ between timesteps. Is it that just a single frame is replaced? This seems pretty limiting in terms of assessing the temporal correlation between observations since a significant amount of information is the same between the observations... How is the encoder prevented from mode collapse? If this is the desired component of disentangled representation learning, wouldn't it be easier to just frame difference the observations and learn from that input representation? I fear that the approach is too focused on a complex solution to perhaps consider more simple alternatives that provide the same benefit.

As discussed above, the overall clarity of the paper is good. I think that Sections 1-4 are well written and easy to follow. The overall construction of Section 5 is too but suffers from perhaps trying to do too much with limited space. As a result there are a lot of important details omitted. The primary detail is that it's unclear what the performance curves are in Figures 2, 3 and 5. Are these the held out test colors? Are these training curves with a disruption of the "test" settings being introduced? 

There are no real reasons why the specified domains were selected. Why were the baselines selected? Why were the specific base RL algorithms chosen? Were there any motivating characteristics that led to their selection?

## Quality

Along the lines of the concerns about the clarity of the experiments. It's unclear whether the baselines are properly executed. For example DBC is shown to hardly learn anything in these environments while in the original paper the algorithm does well. An egregious example of this is on the finger/spin and walker/walk domains. Without explanation of how the experimental domains differ from those executed in Zhang, et al. this leads one to believe that the baselines are not fairly compared to the proposed TED auxiliary task. **This is a major concern**.

## Novelty/Significance

As stated above, I fear that the approach is so focused on complex solutions to a relatively simple question (e.g. we want observations from consecutive time points to be closer in representation space, agnostic of distractors) that there hasn't been enough attention placed on more straightforward questions. This also raise some questions about the significance/novelty of the proposed TED module... I wonder if focusing the representation learning to only consider the single next observation as "similar" is a significant drawback as observations within the same trajectory but at a longer temporal horizon may be as equivalently informative when constructing a representation. I wish there was some thought/discussion in the paper about *why* the focus is solely on single-step transition correspondence. As such, I have a hard time really ascribing much significance to the conceptual grounding for the setting of this paper.

As a recommendation/suggestion. It would've been really interesting to see what a TED-augmented version of the baseline comparisons would achieve on these domains. e.g. would CURL+TED improve upon CURL? 

## Reproducibility

The concepts surrounding the development of the proposed TED are pretty straightforward that it wouldn't be too difficult to reproduce. However, as described above, the construction of the SSL batch for each transition could stand to be better outlined as well as the overall experimental procedure (e.g. how are the TED losses combined across the different samples for each transition in the batch?). 
summary_of_the_review: I enjoyed this paper. I am intrigued by the apparent simplicity of the proposed TED auxiliary task. I am however worried about the limited scope of the variations tested on in the construction and execution of the experiments. I am also fairly concerned about whether the experimental evaluation was properly executed in comparison to baselines across the different domains. There are also concerns about the clarity of how the proposed approach is actually implemented. In light of these concerns I have currently rated this paper as a "borderline reject". I would be interested in raising my score if the authors were able to adequately clarify my questions and some of the concerns raised in my review. I am however not entirely confident that this can be done without substantial revision of the experiments since it's not clear that they were executed fairly.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Thank you for your submission. I mentioned the main points in each section, and I elaborated on the points in the Detailed Comments.

## Summary

The paper proposes TED, an auxiliary task for deep RL agents that encourages learning disentangled representations. TED is shown to improve generalization across color variations of training domains, and is evaluated across different task suites and in combination with different "base" RL agents.
strength_and_weaknesses: ## Strengths

1. To me, one of the most interesting findings in the paper is that TED improves how fast the agent can adapt to new tasks, and how different the ability to adapt can be for auxiliary tasks that look equally good in terms of training performance.
2. The design that incorporates disentanglement into the predictor is a core algorithmic strength.
3. The empirical study in the paper has wide coverage in terms of algorithms and task suites.
4. The evidence provided adequately supports the claims being made.

## Weakness

1. The temporal sampling is a limitation of this particular method. How limiting is a matter of opinion, but in my opinion it is a strong limitation. 
2. In my opinion the focus on demonstrating that TED outperforms previous baselines limits the impact of the paper.
3. Some choices in the empirical study make the results harder to immediately compare to previous work.
clarity,_quality,_novelty_and_reproducibility: ## Novelty and Placement

The idea is novel to the best of my knowledge, but it bears a lot of similarity with [CURL](https://arxiv.org/pdf/2004.04136.pdf). I believe comparing TED to CURL would better emphasize the novel components in TED.

## Support

The claims are mostly adequately supported, but some changes in the training regime can help the comparison to previous work.

Moreover, the paper argues that disentangled representations should help with generalization, and I think it would be useful to have a discussion on this specific point. For example, consider a study where you ablate the classification loss with a more common not-disentangled predictor. The ablation on the loss weight is not too informative here because it is ablating the whole auxiliary task, not really disentanglement.

I am somewhat concerned about the generality of the conclusions about generalization, given the "separation by color" between training and test tasks. However, maybe even this setting is a good starting point, since it's enough to expose some degradation in performance of the baselines.

## Clarity

The paper is clearly written but some improvements can be made:
* clarify the classification objective
* highlight Strengths 1 and 2
* clarify the connections to negative results (related to stationarity and nonstationarity)

## Reproducibility

The paper explains the idea, evaluation and implementation well enough to reproduce. The bands in the plots could be improved from 1 standard deviation to confidence bands to help understand what to expect from similar runs.
summary_of_the_review: ## Concerns

My main concern is the impact/relevance of the results for the broader community. Modulo any concerns raised by other reviewers, I will raise my score to accept (8) if:
* the authors improve the clarity of the paper (the three points above)
* can demonstrate that, in the setting considered, TED's disentanglement component makes a big contribution to generalization.

## Detailed Comments

**Strength 1**. I would like the paper to emphasize this a bit more. It helps make a stronger argument for why we should care about the evaluation setting being considered in the paper. Plus, I think this is quite a surprising fact, because we should care about generalization to tasks that are slightly outside our training set, but we rarely do, and as researchers we may inadvertently discarding/deprioritizing research ideas that fail to outperform existing baselines in training, whereas they could be better than the baselines in ways that training performance does not measure, but that we could also care about (i.e., in this paper, adaptation to new tasks).

**Strength 2**. This could also be investigated further. There's a chance that conclusions about disentangled representations generalizing better carry over to other settings, so the conclusions in this work can be useful for other work too.

**Weakness 1**. This is my only concern about the algorithmic contribution. The method is highly dependent on the negative-example distribution "making sense" or making the classification problem challenging enough to drive representation learning. A failure case would be if the time index $t$ can easily be inferred from $o_t$. In that case a representation that discards everything else and retains only information about `t` can classify optimally. This can therefore be an issue with recurrent representations that retain some amount of information about `t` in them. TED may not immediately carry over to settings where we can't design a suitable negative example distribution, but some empirical conclusions of this paper can still be useful more broadly (namely the conclusions from Strengths 1 and 2).

**Weakness 2**. This is related to my request for more emphasis to Strengths 1 and 2, and it's about making explicit some of the conclusions that go beyond the settings where Weakness 2 is not a problem.

**Strength 4 and Weakness 3**. The choice of evaluation (train for N steps, then change tasks and train for N more steps). This is quite different from many previous works on generalization, in particular, previous work that has discussed zero-shot generalization (ZSG). I think this paper made the right decision to look at adaptation because ZSG would not give a full picture of TED's usefulness. However, certain changes could have helped compare the results against the ZSG in previous work. For example, reporting ZSG performances in control tasks at 100k and 500k.

**Comparison to CURL**. I recommend comparing TED to CURL in the presentation because many components are similar. The comparison will help emphasize the main impactful idea in TED, which I believe to be the classification loss that encourages disentanglement. In my opinion the choice of negative example distribution makes sense and is adequate, but is domain dependent (Weakness 1).

**Investigating good ZSG cases**. I noticed that in Figure 3 SVEA-TED is generalizing quite well _zero-shot_. It seems essentially unaffected by the change at the vertical dotted line, and continues to improve the agent in some cases. If this is not an artifact of smoothing, I think it can be quite an interesting finding. For future work, I suggest the authors reflect and investigate what it is about the SVEA+domain combination that is causing this effect, and whether it can be incorporated into the auxiliary task to bear in other domains. Figure 3(b) is a really interesting example of this phenomenon, where the baselines degrade a lot at 1e6 steps, but SVEA-TED and SVEA-DR continue to improve "almost as though" there has been no change in the training distribution.

**Relation to theoretical results**. The paper draws some connections to negative results about disentanglement, but only in Sections 1, 2.2 and 6. Without additional context or specific knowledge of the results, the comments in this paper are vague to the reader. Moreover, the "decomposition" in terms of stationary and non-stationary factors does not seem to have any bearing in the discussion of the results. It would be good to see this clarified, especially if the authors have intuition on the subject that is guiding how they designed or interpret TED.

**Related work**. It may be worth citing [CURL](https://arxiv.org/pdf/2004.04136.pdf) or [MoCo](https://openaccess.thecvf.com/content_CVPR_2020/papers/He_Momentum_Contrast_for_Unsupervised_Visual_Representation_Learning_CVPR_2020_paper.pdf) in reference to the target encoder.
There are some other papers with auxiliary tasks that can be relevant to current and future work, for example: [UNREAL](https://arxiv.org/pdf/1611.05397.pdf), [SimCore](https://proceedings.neurips.cc/paper/2019/file/2c048d74b3410237704eb7f93a10c9d7-Paper.pdf), [PBL](http://proceedings.mlr.press/v119/guo20g/guo20g.pdf) and [SPR](https://arxiv.org/pdf/2007.05929.pdf).

**Figures**. I suggest placing the figures slightly below where they are first mentioned. For example, Figure 2 is a page before the results discussion. Also, should "Figure 4" be "Table 4"? Finally, the curve colors used may be difficult for colorblind readers to interpret. Please consider whether the color scheme in use is accessible or if it can be improved.

**Hyperparameters** While selecting $\alpha$ will often require domain-specific experimentation, I have found that a good rule of thumb for auxiliary tasks with RL is to make the scaled auxiliary loss of the same order of magnitude as the RL loss.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to improve the generalization of RL through a self-supervised auxiliary task which aims to learn disentangled representations. The auxiliary loss is used to train a classifier to discriminate between temporal and non-temporal pairs so that the encoder is encouraged to disentangle temporal structure in the observations. The proposed auxiliary loss can be easily combined to existing RL algorithms and experimental results on three benchmarks are reported.
strength_and_weaknesses: Strength:
1. Generalization of RL algorithms is an important research area, and this paper proposed a self-supervised auxiliary loss to achieve improved generalization in RL.
2. The idea of combining the TED auxiliary loss with RL is novel.
3. The paper is clearly presented and well-written.
4. The proposed method is easy to implement and the experiments are conducted on different benchmarks.

Weakness:
1. TED requires non-temporal samples from both the same episode and different episodes. However, the necessity of using non-temporal samples from the same episode is not convincing. Why not just use samples from different episodes or mix the two sources during sampling? The ablation study also does not include experiments that use non-temporal samples from different episodes only.
2. The parameter alpha seems to be a sensitive parameter. However, experiments are only implemented on the cartpole swingup task with three different values. More analysis should be added. Is the optimal value of alpha different in different tasks?
3. In many experiments, the training does not seem to have converged yet. More training steps are suggested. Also, the number of DMC tasks are quite limited in this paper.
4. In Figure 4, what if the TED loss is combined with CURL in the cartpole swingup task.
clarity,_quality,_novelty_and_reproducibility: The proposed TED auxiliary loss is novel for solving RL problems. The contributions are presented clearly and the code is also provided.
summary_of_the_review: Generalization of RL algorithms is an important topic and the motivation of this paper is good. To solve the generalization problem of RL, the paper proposes a novel disentangled representation learning auxiliary task and experiments on three benchmarks demonstrate its effectiveness. The major concern is the completeness and quality of the experiments.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: ### Summary

The paper proposes a method called Temporal Disentanglement (TED) for RL problems where the state is represented in terms of image-based observations. To achieve this the paper proposes an auxiliary loss that does self-supervision between observations based on what is irrelevant and relevant for the task. The experiments focus on Panda Gym and DM Control suite environments and the paper shows improvements over base RL agents.

Below I will summarise some of the  strengths and the weaknesses pointed out by the reviewers:

### Strengths

- The paper is well-written and clear. The TED approach is well-motivated.
- The approach is simple and straightforward. It is relatively easy-to-implement.
- Thorough experiments and comparisons. 
- The claims that are being made are sufficiently supported by the experiments.
- The idea of combining the TED auxiliary loss with RL is novel.

### Weaknesses

- The deviations considered as unrelated/related between episodes are narrow. 
- The lack of details on how the TED loss is incorporated to RL.
- The temporal sampling can be a limitation of TED. 
- In many experiments, the training does not seem to have converged yet.

### Decision
Improving generalization of RL algorithms is an important under-studied area. This paper proposes a novel auxilary loss that seems to be effective in terms of improving the generalization of the RL algorithms. The proposed approach is simple and easy-to-implement. The claims are well-supported by the experimental evidence. The authors did a good job in terms of addressing some of concerns raised by the reviewers. Thus, I am recommending this paper for acceptance.
justification_for_why_not_higher_score: Although the paper is well-written and experiments look promising. There is further evidence needed to confirm that the proposed approach is generalizable to other RL tasks. The approach has some limitations (such as the temporal sampling) as pointed out by the reviewers. 
justification_for_why_not_lower_score: As mentioned in the decision, I think this paper is a valuable contribution to the ICLR community that would be interested in RL.
Decision: Accept: poster