Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper presents a method named TCSR which is based on EfficientZero. TCSR helps a reinforcement learning agent to learn better latent representations in a self supervised manner.
strength_and_weaknesses: Weaknesses: 
The paper has several weakness in writing, results, explanation of the approach. See below: 

Results: 
- The paper does not evaluate the proposed method with significant experiments. Only Table 1 shows results on Atart100K which shows improvement in 11 out of 26 games as compared to the author's claim of 17 out of 26. These results are also not consistent with Figure 2. The table results are also not explained. 
- The visualization shown Figure 1 is highly basic and any encoder should be able to encode the representations. Why many method are failing to encode it is not explained anywhere in the paper. 
- Training details are not provided, the results seems not reproducible with any implementation details. 
- The author's claim self-supervision. I didn't find any information which help TCSR to train in a self-supervised manner. 

Method: 
- The approach is not detailed. The reader cannot understand the proposed method as many details are missing or not explained. For example, Figure 3 is never explained and it is the training pipeline. I am not able to infer what is the training part in this figure. 
- The notations are not clear. In s_t^k, s is state at time t but what is k?
- The authors gave a good emphasis on consistent loss but it is not explained. an equation mentioned it in sec 3.2 but what is the loss? The equations are not numbered. 
- The TCSR loss in sec 4.1 is not clear. Why t+k time is the target for state at time t?

Writing: 
- Sec 3.1, first paragraph third line, encodes a observation
- Sec 3.2, represnetation
- Full stop in last line of page 5, many places in sec 4.1. 
- Table:1 can be written as Table 1.
clarity,_quality,_novelty_and_reproducibility: The paper is not clear at many places and the proposed method is not detailed enough. No implementation details are provided which makes the paper not reproducible. 
summary_of_the_review: The paper has several weaknesses and it is not ready for publication. See weaknesses for justification.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes Temporal Change Sensitive Representation, a self-supervised representation algorithm that seeks to capture the temporal representation changes in RL. Experiments on different datasets suggest that the proposed algorithm does outperform benchmark SOTAs in the Atari100k environment. 
strength_and_weaknesses: Strength:

The paper is well-organized. Each section does what it aims to do. The algorithms are described very clearly.

Weaknesses:

1. The extent of novelty is relatively low. On a high level, the paper appears to be a combination of the existing EfficientZero algorithm(a temporal-aware variant of the MuZero algorithm over non supervised contents, with data augmentation and minor adjustments over similarity losses. More elaboration on novelty would be helpful.

2. The motivation should be stated more clearly. While the experiments are adequate, there should be more explanation as to how/why each set of experiments correspond to the context/setting of the problem. Additionally, it would be helpful to provide some visual illustrations(or other qualitative demonstrations) as to why the algorithm works, as opposed to simply listing the performances SOTA and conclude accordingly. 

3. Would it be possible to incorporate experiments on other datasets? Empirical results solely on Atari-based datasets seem a bit inadequate, especially when the algorithm seeks to address representational learning for reinforcement learning problems in general.   
clarity,_quality,_novelty_and_reproducibility: The paper is written clearly, and the results/experiments contain adequate details to be convincing.  
summary_of_the_review: My rating would be a 3, though I'm happy to adjust my score if the authors address my concern.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a new auxiliary loss for learning representations for reinforcement learning from images. The loss penalizes the difference between the change of two subsequent latent states for one image augmentation and the change of two subsequent states for a different augmentation. By integrating this loss into "EfficientZero" (Ye et al. 2021) the performance on Atari benchmarks good be improved.

strength_and_weaknesses: Strength:
- The paper presents a novel and reasonable auxiliary, that achieves good empirical results.

Weaknesses:
- The contribution is rather small, as the modification of the previous method is only incremental and not theoretically justified.
- The presentation is sloppy.
clarity,_quality,_novelty_and_reproducibility: Clarity:
- Although the content of the paper is relatively small, it is difficult to understand the proposed method without knowing the related work in detail, since the paper not at all self-contained. For example, not even the actual loss (the main contribution) is shown in the paper; instead, the reader is assumed to know $\mathcal{L}_{\text{SimSiam}}$. Furthermore, the difference between the policy $\pi$ and $p$ is not clear as they have not been properly defined. The paper should discuss EfficientZero and the SimSiam loss in more detail, which could have been done in the current submission, given that it only used a bit more than seven pages.
- The paper is not well-structured. For example, Fig. 1 is mentioned in the Appendix, but shows a comparison with EfficientZero which has not been mentioned at that point. The actual Figure is only described in the experiments section at the end of the paper.
- Also the writing is quite sloppy and even contains errors that could have been automatically found with a spell check ("represnetation"). 
Several sentences even have multiple mistakes.

Quality and Reproducibility:
- The paper contains little information on how the experimental study has been conducted. The paper does not come with any appendix nor source code and is therefore not at all reproducible. It is not reported which hyperparameters and network architectures have been used and how they have been tuned.
- For many environments, the proposed method performed worse than the previously reported results of EfficientZero. While the paper argues that the comparison is fair by using the same implementation (of the original authors of EfficientZero) for evaluating the new method and for rerunning EfficientZero, it seems like the authors of the current submission did not even try to reproduce the reported EfficientZero results beyond running the code on their own machine. 

Novelty:
- The novelty of the paper lies given only by a new loss term, which was not well motivated.
 
summary_of_the_review: The contribution of the paper is incremental, but could be somewhat interesting.
However, the communication of the work is substandard, and the paper would require a major revision by:
- describing the proposed method in a more self-contained way,
- providing an in-depth description on the experimental procedure,
- improving the structure of the paper,
- and, ideally, investigating the discrepancy between the reported results of EfficientZero, and the results obtained in the current experiment (I recommend contacting the authors).
Furthermore, providing the source code for reproducing the experiments would be highly appreciated.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors propose TCSR; they perform experiments to show that it improves upon EfficientZero and achieves SOTA performance on the Atari100K benchmark.

strength_and_weaknesses: This is an interesting and well-motivated paper studying an important topic.

However, the contribution is entirely empirical (the contribution boils down to the proposal of TCSR, and the claim that TCSR “achieves SoTA performance”), and the experiments are deeply flawed.  The results are based on 3 runs, which is unacceptable in RL, due to the high variance between runs.  This topic has been repeatedly covered in publications, talks, etc. in recent years, so I will not elaborate further.  It is interesting to notice, however, the fact that the “reported EfficientZero” score greatly outperforms the TCSR score.  The large discrepancy between the “reported EfficientZero” score and the “source code re-run EfficientZero” score likely comes from the low number of runs used.  This discrepancy is an excellent illustration of the problem with only using 3 runs.

Clarity issues:
- The loss functions that are in the definitions of the MuZero loss are not defined (nor is it easy to infer what they are).
- Similarly, the SimSiam loss is not defined.
- g_{state} is not defined.
- The difference between s^ and s~ (defined at the end of section 3) is not clear.  They “each represent an augmentation of a parameter”.  So there are two parameters to augment?  (I do not think that’s correct, but I do not know how else to read it.)

Minor edits:
- “reward is sparse(Shelhamer et al., 2016)” (missing a space)
- “The training pipeline is as shown in Figure 3” (missing period)
clarity,_quality,_novelty_and_reproducibility: The quality is poor, primarily due to the flaw with the main contribution discussed above.  The novelty seems to be slightly limited, but is adequate.  Regarding clarity: the paper is well-written in general, but has some severe clarity issues in a few places, including undefined terms (see above).
summary_of_the_review: This is an interesting and well-motivated paper studying an important topic.  However, the contribution is entirely empirical, and the experiments are deeply flawed.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper presents a new self-supervised RL algorithms called TCSR. It builds on top of efficientNet for the Atari100k benchmark suite. The experimental results in this paper on Atari100k are quite strong. However, there are several concerns from the reviewers. The novelty of this work is limited as it is a combination of EfficientZero and data augmentation-based similarity. Hence we would need to see strong experimental results. However, on the experimental front, it is quite unclear on what is the context of the experiments. The writing and discussion needs to be significantly improved to conveys the key insights and methodology. Finally, Atari100k is quite a narrow set of tasks. We recommend experimenting with a broader set of tasks, perhaps including the DM Control tasks.
justification_for_why_not_higher_score: Writing is poor. Experimental analysis is narrow in scope. Novelty is limited.
Decision: Reject