Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper considers the HiP-MDPs where a set of low-dimensional hidden parameters models the variations in tasks, this setting is naturally applicable to settings with transfers. The main contribution of this paper is the theoretical analysis on the regret upper bounds of model and policy transfer algorithms, as well as multi-step prediction error. These bounds are proven to be tight in linear deterministic settings. The theoretical analysis in this paper provides insights on the performance of model/policy transfer algorithms and its connection with Lipschitz constants.

strength_and_weaknesses: 
The strength of this paper is mainly theoretical, it rigorously characterizes the robustness of model/policy transfer methods by hidden parameter estimation error. Thus, it provides practical perspectives on choosing between model or policy transfer methods. For example, policy transfer is likely to outperform model transfer in the sense of robustness when the differences between actions are small. On the other hand, model transfer is likely to outperform policy transfer in terms of robustness when the different between neighboring states are small. Empirical results further validates the theoretical findings.

The main weaknesses of this work are listed as follows:
- It is unclear how the analysis presented in this work is related with previous theories. In many places the authors have mentioned that their results share similarity with some of the previous results, for example Theorem 3.1 is a generalization of theorems in a bunch of papers, similar paradigm of the algorithm framework has been widely used in complex continuous control tasks. The authors emphasis the major significance of their work, but it seems that it fails to provide a more detailed comparison on how much it improves upon previous results on HiP-MDPs.
- There is little intuition on how the results are derived, a brief outline of proof on at least one of the main theorems would help readers to better understand the technical novelty of this work.
clarity,_quality,_novelty_and_reproducibility: The main concepts, methodology and the interpretations of the Theorems are clearly explained. The paper provides a novel theoretical analysis that helps to choose between different models in practice.
summary_of_the_review: Overall, this paper is strong in theory and the experimental results support the main theoretical claim. The presentation can be improved at some point.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this paper, the authors studied the Hidden-Parameter MDP (HiP-MDP) framework, which is kind of non-stationary MDP. There is a set of parameters controlling the dynamics and reward, which may vary by tasks. This setting is very important since it can be applied in meta reinforcement learning and lifelong reinforcement learning. The authors characterized the robustness of both model transfer and policy transfer algorithms with respect to the hidden parameter estimation error, which I believe is a meaningful research. In this paper, the authors not only obtained the Lipschitz continuity, but also the regret bound under both settings. 
strength_and_weaknesses: Strength:
(1) Very clear paper writing and very high paper quality. The literature review is very complete. 
(2) Proofs in this paper are very clear and easy to follow. 
(3) The experiments are very complete. I believe they are reproducible. 

Weakness:
The context width is quite tricky in the supplement. 
clarity,_quality,_novelty_and_reproducibility: This paper has very clear writing and very high quality. The results of this paper are very solid, and I believe the experiments in this paper are reproducible. To the best of my knowledge, the novelty of the results is high. 
summary_of_the_review: To sum up, I would like to give this paper an "accept" mainly because of its theoretical significance and novelty.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studied the performance bound when transferring model or policy between HiP-MDPs. The authors introduced some assumptions on the model/policy Lipschitz continuty and established the Lipschitz continuty property of (optimal) value functions in HiP-MDPs. In Sec. 4 & 5, they derived the upper bound when transferring the optimal policy in learned Lipschitz model and pretrained Lipschitz optimal joint policy, repsectively. Finally, in Sec. 6, they conducted some evaluation experiments to verify the theoretical finding.
strength_and_weaknesses: ### Strength

The Lipschitz continuty assumptions about the model/policy are reasonable, from theoretical perspective. It's also good to see the theoretical results are verified by empirical evaluation in Sec. 6.

### Weakness

Since the experiment part is just some evaluation about the theoretical results, instead of proposing some novel and effective empirical algorithms inspired from theory, I would believe the novelty of theory part is what to be evaluated the most.

However, it seems to me the theory part is not significant enough, and the results, although somewhat new, are kind of trivial and not surprising. Theoretical results about value difference bounds w.r.t. Lipschitz constants had been established in previous works (e.g. [1]), and the authors main contributions seems just analyzed the extended setting called HiP-MDPs, where the model/reward functions will take an additional ``hidden parameter'' $\theta$ as input which capturing the variation across different models. I found the techniques are similar and there is not much novelty.

Besides, the theory also has some restriction in practice. Although similar assumptions occurs in previous papers, the requirement that $\gamma$ times (combination of) Lipschitz constants is less than 1 occurs in almost all the main theorems (seems because of the $1-\gamma K$ occurs in the denominator). However, in practice, it is hard to verify whether such condition holds, or give any prediction about what will happen if such conditions do not hold. I think the authors should provide at least some examples about when such assumptions can be true in practice.


[1] DeepMDP: Learning Continuous Latent Space Models for Representation Learning. Carles Gelada, Saurabh Kumar, Jacob Buckman, Ofir Nachum, Marc G. Bellemare
clarity,_quality,_novelty_and_reproducibility: The paper writing is clear, the proof looks correct and the experiments seems easy to produce. My main concern is that the novelty of this paper is not enough.
summary_of_the_review: This paper studied an interesting setting. However, as I discussed in ``Weakness'' section, I think this paper is lack of novelty and the contribution is not enough for ICLR.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper provides theoretical justification for two families of methods, namely model-transfer and policy-transfer, that perform transfer learning in hidden-parameter MDPs. All reviewers agree that the theoretical results are significant and sound. It's also encouraging to see theory papers having preliminary experiments to justify the algorithm and settings. I vote for acceptance of this paper.
justification_for_why_not_higher_score: The theoreticals result while reassuring do not provide any new theoretical or empirical insight, and the experiments are also preliminary. 
justification_for_why_not_lower_score: NA
Decision: Accept: poster