Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper contributes a) a continuous version of the MDP homomorphism and b) a homomorphic policy gradient. An approach based on lax bisimulation metrics learns a policy and a continuous MDP homomorphism simultaneously. The paper evaluates on DM control tasks from pixel observations and shows improvements compared to reasonable baselines (DrQ-v2, DBC, DeepMDP, etc). 
strengths_and_weaknesses: # Strengths
- The paper tackles an important problem: learning representations for reinforcement learning problems, especially those that recover structure and/or symmetry in the input MDP.
- The approach, of mapping into an abstract space and learning a policy + abstraction simultaneously is reasonable and has precedent. 
- According to the paper, the approach is able to learn in smaller latent dimensions and recover symmetric representations, which is a nice result. Discovering symmetries in MDPs is a largely unsolved problem, and the paper takes a step in that direction.
- It's useful to have an extension to continuous MDP homomorphisms and to have the homomorphic policy gradient theorem. The paper could be of interest to the field of state representation learning and to the field of symmetries in RL. 

# Weaknesses
- Usually in MDP Homomorphisms, $g_s$ is a surjection. In the paper, it is required to be a bijection. This comes with the disadvantage that equivalent actions belonging to a single state can no longer be aggregated/mapped onto the same abstract action. For symmetries this should not be an issue, but making $g_s$ bijective limits the levels of aggregation in action space.
- The work is evaluated on DMControl tasks. One of the baselines is DBC. In the original DBC paper and follow-up work (e.g. [42]), there are experiments with background distraction videos. Including such experiments in this work, to show robustness to distractors that are irrelevant to the task, would make the paper stronger. 
- The potential of this approach to find symmetries in MDPs is part of its appeal. The paper does allude to this by including plots that seem to show some degree of symmetry recovery. However, the direction is not explored in-depth in the paper. 
- Missing references:
    - Mahajan, Anuj, and Theja Tulabandhula. “Symmetry learning for function approximation in reinforcement learning.” arXiv preprint arXiv:1706.02999 (2017).
    - Biza, Ondrej, and Robert Platt. “Online abstraction with MDP homomorphisms for deep learning” AAMAS 2019

questions: - Could you elaborate on what the effect is of requiring $g_s$ to be a bijection?
- In Figure 3a, the learned policy looks approximately, but not exactly symmetric. The original problem does have exact symmetry, and in Fig 3b the action representations look like they may have recovered the symmetry. Why is the policy less symmetric than the representations are? (This Figure is hard to read. It would be improved by clarifying which of the plotted values represent the chosen actions)
- The paper claims that legs are mapped onto each other in Figure 6a/b. Could you elaborate on how one can read this in the Figure and how you know that legs are represented in the abstract/latent state space structure?
- Suggested references:
    - Park, Jung Yeon, et al. “Learning Symmetric Embeddings for Equivariant World Models.” ICML 2020 
    - Narayanamurthy, Shravan Matthur, and Balaraman Ravindran. “On the hardness of finding symmetries in Markov decision processes.” ICML 2008.
- Appendix C.1 says that $R$ and $\bar{R}$ are both bounded. Where does this assumption come from? 
- Page 2, line 50-51 claims that the proposed approach leads to more robust solutions. The approach seems similar to an MDP homomorphic version of DBC. Why does the approach outperform DBC?

limitations: - The checklist says empirical limitations are discussed in Section 7, but I could not find this. Could you point me towards the paragraphs?
- Section 4 discusses part of the theoretical assumptions/limitations.
- Code was included with submission.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes MDP homomorphisms and homomorphic policy gradient for continuous control tasks. The contributions are threefold: 1. introducing the continuous MDP homomorphisms, 2. deriving the homomorphic policy gradient theorem that ties the induced abstract MDP into the policy gradient, 3. developing the deep homomorphic policy gradient algorithm which simutaneously learn the policy and homomorphism.
strengths_and_weaknesses: 1. The introduction of MDP homomorphisms to continuous control tasks is interesting and significant.
2. The paper is overall well written, with clear motivations and theoretical explainations. 
3. The authors agree to make their code publicly available and has provided the source code in the appendix.

questions: 1. In Line 165, the authors mentioned "we assume having access to an MDP homomorphism map ... . The problem of learning such mapping from samples is addressed in Section 6". Would the derived policy gradient and the policy optimisation algorithm depend on an accurate MDP homomorphism?
2. In addition to the first question, what happens to Theorem 4 and 5 if the MDP homomorphism is not an exact equality (as in Equations (6) and (7) where there is an epsilon difference) [1]?

[1] Abel, David, David Hershkowitz, and Michael Littman. "Near optimal behavior via approximate state abstraction." International Conference on Machine Learning. PMLR, 2016.

limitations: The authors clearly discussed the limitation of assuming deterministic policies in Section 4.2.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper extends the theory of MDP homomorphism abstractions to continuous state and action spaces, and includes a theorem that motivates the use of MDP homomorphisms in actor-critic reinforcement learning agents. The paper proposes an agent that learns an abstraction of the ground environment while learning to solve a particular task. The agent is evaluated on continuous control tasks.

strengths_and_weaknesses: # Strengths:

1. The paper extends the theory of MDP homomorphisms to continuous action spaces (in both the ground and the abstract MDP), which I consider to be a significant contribution.
2. The paper aligns with prior literature on using bisimulation metrics in model-free learning by using the lax bisimulation metric, which includes action abstractions. I find this to be well-motivated.
3. The proposed method is evaluated on many continuous control tasks, the high variance in the results of deep reinforcement learning agents is controlled for and the source code for the method is included.

# Weaknesses:

1. There is a possibility that the quantitative benefits we see in the proposed method are due to increased model capacity only. I could not find a discussion of the sizes of the evaluated models.
2. Having an abstraction component should provide additional benefits, such as compactness of representations and robustness to distracting features. These benefits are demonstrated to a limited extent, and some details of the experiments are unclear (please see my detailed comments).

# Detailed comments:

## Writing

– This paper is close to previous works on using bisimulation metrics in model-free reinforcement learning. I am missing a discussion of the similarities between the proposed model and prior works (like DBC [10]) and a discussion of the benefits of adding action abstraction to this framework.

– The literature review is insufficient, many prior works on MDP homomorphisms are missing [1, 2, 3, 4, 5, 6]. Additionally, there are a few more works on bisimulations that could be of interest [7, 8, 9]. This list of papers I provided is non-exhaustive.

– Minor comment: This might come down to personal preference, but I do not like the way the introduction is written. The introduction should tell a story and build interest in the rest of the paper. Instead, the introduction in this paper reads like an extended list of contributions.

– Minor comment: The authors cite one of Ravindran’s original papers on MDP homomorphisms, but not his thesis, which I would consider to be the definitive source [11]. 

## Experiments

– Rather than an increase in agent performance, which could be attributed to the increase in its capacity, I see the main benefit of state and action abstractions in (a) learning compact representations, (b) ignoring distractions and (c) facilitating better transfer of the learned representation. (a) is shown in the cartpole environment and I find this experiment very compelling. My only complaint is that I could not find what the size of the state-action encoding is (i.e. we could make the state encoding small, but capture all information in the state-action encoding). (b) Is not included in the paper and I find this to be a missed opportunity. For example, DBC [10], which is a baseline in this paper, shows convincing experiments with distracting backgrounds in their Figures 3 and 4. (c) is shown in the appendix, but I am not sure how to interpret the results of the experiment.

– Lines 305 - 307: Perhaps I misunderstand, but the claim that DHPG “mapped all legs to the same abstract latent state” is problematic. Firstly, I do not know if Figure 6b supports this claim. There are four clusters in Figures 6a and 6b, and they look similar. Secondly, it seems unlikely that having a single abstract latent state to track the state of all four legs is desirable – the legs have to move independently to facilitate locomotion. I suppose there is some division of what is represented in the state and the state-action encoder, but that is not explained in the paper.

## References

[1] Vishal Soni and Satinder Singh. 2006. Using Homomorphisms to Transfer Options Across Continuous Reinforcement Learning Domains. In Proceedings of the 21st National Conference on Artificial Intelligence - Volume 1 (AAAI’06). AAAI Press, 494–499.

[2] Alicia P. Wolfe. 2006. Defining Object Types and Options Using MDP Homomorphisms.

[3] Alicia Peregrin Wolfe and Andrew G. Barto. 2006. Decision Tree Methods for Finding 
Reusable MDP Homomorphisms. In Proceedings of the 21st National Conference on Artificial Intelligence - Volume 1 (AAAI’06). AAAI Press, 530–535.

[4] Jonathan Sorg and Satinder Singh. 2009. Transfer via Soft Homomorphisms. In Proceedings of The 8th International Conference on Autonomous Agents and Multiagent Systems - Volume 2 (AAMAS ’09). International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 741–748.

[5] S. Rajendran and M. Huber. 2009. Learning to generalize and reuse skills using
approximate partial policy homomorphisms. In 2009 IEEE International Conference
on Systems, Man and Cybernetics. 2239–2244. 

[6] Ondrej Biza and Robert Platt. 2019. Online abstraction with MDP homomorphisms for deep learning. In International Conference on Autonomous Agents and MultiAgent Systems.

[7] Abel, D.; Hershkowitz, D. E.; and Littman, M. L. 2016. Near Optimal Behavior via Approximate State Abstraction. In Proceedings of the International Conference on Machine Learning, 2915–2923.

[8] Abel, D.; Arumugam, D.; Asadi, K.; Jinnai, Y.; Littman, M. L.; and Wong, L. L. S. 2019. State Abstraction as Compression in Apprenticeship Learning. In AAAI.

[9] Ondrej Biza, Robert Platt, Jan-Willem van de Meent, and Lawson LS Wong. 2020. Learning Discrete State Abstractions with Deep Variational Inference. In Proceedings of the 3rd Symposium on Advances in Approximate Bayesian Inference.

[10] https://arxiv.org/abs/2006.10742 

[11] https://all.cs.umass.edu/pubs/2004/ravindran_thesis04.pdf 
questions: 1. The abstraction components (the state and state-action encoders f and g, the abstract critic 
\bar{Q}, the abstract transition predictor \tau and the reward predictor \bar{R}) add additional capacity to your model . What is the number of FLOPS (or the number of parameters if it is difficult to measure FLOPS) in these components compared to a basic actor-critic model? Could the model size be controlled for in the experiments to make the comparison against the baseline fairer?
2. The abstract encoders f(s) and g(s, a) could capture similar information, since they both have access to the state. In the extreme, the model could use f(s) for reward prediction only and g(s, a) for everything else, since all functions except for the reward predictor are conditioned on the outputs of both f and g. Do f and g output similar features?
3. Could you clarify lines 305-307 and Figure 6? Please see my comment above.
4. What is the size of the state and the state-action encoding in Figure 5.

limitations: Limitations are not explicitly discussed, but shoud be similar to previous works such as DBC and DDPG.
review_title: null

Meta-review: metareview: The paper a new algorithm for representation learning in RL based on MDP homomorphism. The paper presents a nice theory, the algorithm is well motivated and the experiments are convincing. The concerns of the reviewers such as comparisons with DBC or DM control suite with distractions have been addressed adequately by the authors. All reviewers evaluated the paper very positively and I join their decision. 
confidence: Certain
Decision: Accept