Venue: ICLR 2023 TinyPapers

Reviews:

final_score: rating: Great Start (GS): a submission which meets some of the reviewing criteria but has room for improvement
review_content: summary_of_contributions: The paper is concerned with finding representations of states in reinforcement learning. The goal is have a metric that approximately  two nearby states are also nearby in the value functions space. The paper proposes a new metric that overcomes certain limitations (theoretical and practical) of prior works 
strengths_and_weaknesses: **Strengths**

- The paper states the problem that needs to be solved in a clear manner. This allows even a generalist machine learning researcher to get a sense for the problem being solved
- Empirical evidence is provided to back the theory developed in the paper that clarifies the advantages of proposed approach

**Weaknesses**

- The paper uses the term `pseudo-metric` which is a fairly technical term. Could the authors clarify what is the difference between `pseudo-metric` and a `metric`?
- Please consider using text in the figure caption as opposed to symbol to simplify Figure 1's interpretation
- The paper claims that the proposed metric is lower than `MICo` but two other metrics in Figure 1 are lower. Does this suggest that the other metrics are better for practical use? Some discussion may help with the presentation
- Consider including code that accompanies your work
suggested_changes: Please check **Weaknessess** for suggested changes
review_title: null

final_score: rating: Clear, Correct, and Reproducible (CCR): a submission which meets the reviewing criteria
review_content: summary_of_contributions: A new metric with on-policy guarantees but need further improvement in writing 
strengths_and_weaknesses: Summary:

The authors propose a a new (pseudo) metric, inspired by the MICo distance, which is called PMiCo (Pseudo-metric Matching under independent Couplings). This metric is desirable because it can be learnt from transition samples and yields on-policy guarantees.

Strength:

It seems the proposed sampling-based bisimulation metric outperforms the baseline and has on-policy guarantees. The proof in the appendix looks correct.

Weakness:

1. The authors should double-check the notations. For example, when the authors introduce the definition of the bi-simulation metric d~ , the notation X is different from \mathcal{X}. Does X also denote the state space as \mathcal{X} does? And there are two wasserstein distance in the paper, W in d~  and \mathcal{W} in π-bisimulation. Do them denote the same distance?
2. How the bisimulation metric is defined is not clear to me. Without further explanations to fixed point of an operator, it’s hard to understand what is the bisimulation metric indeed. What’s more, how do you confirm the fixed point is unique? Any references?
3. The motivation to propose the new bisimulation metric is not strong enough. I would recommend to add several sentences in the introduction and background section to explain what is sampling-based method and why we need that.
4. Figure 1 need a better legend, it's hard to tell which one is the proposed method and how it shows that PMiCo metric yields a tighter bound.
suggested_changes: Please take a loot at the weakness
review_title: null

Decision: Invite to revise