A General Theory of Relativity in Reinforcement Learning
Authors: Lei Han
Cheng Zhou
Yizheng Zhang
Venue: ICLR 2022 Submitted (2022)
Keywords: Reinforcement Learning
General RL Theory
Policy Transfer
Dynamics Modeling

Abstract: We propose a new general theory measuring the relativity between two arbitrary Markov Decision Processes (MDPs) from the perspective of reinforcement learning (RL). Considering two MDPs, tasks such as policy transfer, dynamics modeling, environment design, and simulation to reality (sim2real), etc., are all closely related. The proposed theory deeply investigates the connection between any two cumulative expected returns defined on different policies and environment dynamics, and the theoretical results suggest two new general algorithms referred to as Relative Policy Optimization (RPO) and Relative Transition Optimization (RTO), which can offer fast policy transfer and dynamics modeling. RPO updates the policy using the \emph{relative policy gradient} to transfer the policy evaluated in one environment to maximize the return in another, while RTO updates the parameterized dynamics model (if there exists) using the \emph{relative transition gradient} to reduce the gap between the dynamics of the two environments. Then, integrating the two algorithms offers the complete algorithm Relative Policy-Transition Optimization (RPTO), in which the policy interacts with the two environments simultaneously, such that data collections from the two environments, policy and transition updates are all completed in a closed loop to form a principled learning framework for policy transfer. We demonstrate the effectiveness of RPO, RTO and RPTO in the OpenAI gym's classic control tasks by creating policy transfer problems.


Reviews:
[{"review_id": "tOZpsRaNDCw", "reviewer": "Reviewer_G2Gj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Since my knowledge of empirical RL is very limited, below I will mostly comment from a theoretical perspective. \n\nThe first main theorem (Theorem 1) in this paper is already well-known and rather basic. It's a simple combination of performance difference lemma and model difference lemma. \n\nThe second main theorem (Theorem 2) provides the key theoretical justification for the two algorithms. Briefly speaking, it provides a low bound for the model-difference-induced gap that is relatively easier to optimize in practice. Its derivation is rather standard. And by only looking at the statement of this Theorem, this lower bound looks not very tight. The quantity on the LHS is at most $O(\\delta_1)$ but the second term in the RHS is order $\\Omega(\\delta_1)$. Also, it's hard to tell whether the RHS could be negative in many cases. I am wondering in what cases this low bound would be a good approximation (either in the additive or multiplicative sense) to the quantity we want to maximize? Is there any theoretical justification?\n\nIn short, I am not well convinced by the theoretical justification in this paper. \n\nNevertheless, It's likely that it provides two really nice empirical algorithms that work pretty well in practice since I can see from the experimental section that the performance seems to be very nice. However, because I am not familiar with the standard benchmark tasks and algorithms in this field, I would like to hear more from other reviewers on this aspect.", "summary_of_the_paper": "This paper studies how to fast adapt an old policy to a new but similar environment. It proposes two new algorithms RPO and RTO complemented with some theoretical justification and empirical performance demonstration.", "summary_of_the_review": "See above.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "aH_lSSv6eO4", "reviewer": "Reviewer_gbpy", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n    1. The idea is novel and inspiring.\n    2. The proposed algorithms are practical.\n    3. The overall storyline is clear.\n     \nWeaknesses:\n    1. Lot of confusions\n        a) The main merit of RPO seems to be that RPO can collect infinite data from one MDP1 with which it only needs a few data from MDP2 to get a good policy in MDP2. However, it is not clear to me whether the merit was demonstrated from the experiments. How much data was actually used in each of the two MDPs? What is the meaning of \"training step\"? Do baseline algorithms use the same amount of data as RPO? \n        b) I can not see how equation 7 is obtained from equation 6 in the deterministic case. Could you explain it?\n        c) Could you explain the obtained bounds?\n        d) The proposed algorithms (RPO and RTO) are derived from only parts of the bounds and ignore other parts of the bounds even though they are also relevant. And there is no explanation about why they can be ignored.\n        e) pi_old and phi_old do not appear in the pseudo-code while they appear in the equations.\n    2. Lots of experimental details are missing\n        a) what is the optimizer?\n        b) what is the initialization?\n        c) what is PPO-zero?\n        ", "summary_of_the_paper": "This paper proposed a way to decompose the difference between values of two policies in two MDPs respectively. Such a decomposition results in two parts, the first one is the difference between values of one policy under two MDPs; the second part is the difference between values of two policies under the same MDP. Using this decomposition, the paper then proposed three algorithms. \n\nThe first algorithm, called RPO, is used when there are two MDPs that the agent can interact with. The agent uses data from both two MDPs to get a good policy for one of the two MDPs. Such an algorithm is expected to be useful when gathering data from one MDP is costly while it from the other MDP is much cheaper. \n\nThe second algorithm is a model learning algorithm. In this setting, the agent only interacts with one MDP and learns a model to approximate the MDP. The RTO algorithm is different from the classic model learning algorithm (regression) in that the model is learned to achieve some consistency between the predicted values from the model and from the MDP.\n\nThe third algorithm combines the first two algorithms and is a full model-based algorithm. Specifically, now the agent only interacts with one MDP and learns a model of that MDP using RTO. Meanwhile, it also maintains and updates a policy that is expected to perform well in the MDP, using data from both the MDP and the model.\n\n", "summary_of_the_review": "Overall I really like the main idea of this paper. But there are just so many things, including those that are key to their main results, that confuse me. I believe that this paper would become an important work to the field once it is clear.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "_d-r9YU7yd8", "reviewer": "Reviewer_U1Ad", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper presents a grand-sounding “general theory of relativity in RL”, but both components of the theory are already known from prior work. The authors acknowledge that the explicit form of the policy-induced gap is given by the policy improvement theorem of Kakade and Langford, but they seem unaware that the explicit form of the dynamics-induced gap (Eq. 3) appeared in a 2018 paper [1] under the name “telescoping lemma” (Lemma 4.1). Considering that Eq. 3 is arguably the main novel (according to the authors) theoretical contribution of this submission, I think this is a serious issue, and the paper should not be accepted in its current form.\n\nThe algorithms are novel to my knowledge, however. I think a rewrite of the paper which just states Theorem 1 (acknowledging the prior work!) and then moves on to Theorems 2/3 and RPO/RTO/RPTO could still be a useful contribution. It would be stronger still if RPTO were shown to be effective for model-based RL, a setting which is frequently mentioned for motivation but not explored empirically.\n\nThe experimental results could be better. RPTO's performance looks good, but RPO fails to outperform PPO-warm on 2 out of 4 environments, and RTO’s performance is not even plotted.\n\nOther comments on the writing:\n* I feel pretty strongly that the phrase “general theory of relativity” should not be used because it already has a widely accepted usage in physics, leading to confusion from readers who go in expecting an interesting connection to or application of Einstein’s theory. A simple fix would be to replace all instances of “relativity” with “transfer”.\n* “Instead, with much deeper investigation of the fundamental dynamics-induced value gap, we can finally get the explicit identity equation in Eq. (3), instead of a bound. As we will show later, Theorem 1 suggests two thoroughly new algorithms for policy transfer and transition update. Details of the superiority of Theorem 1 over MBPO can be found in the proofs in appendix.“ I understand that some degree of salesmanship is necessary in ML research these days, but I felt it should be toned down a bit in this passage.\n* In Proposition 1, it is not correct to refer to $C$ as a constant, as it depends (via $\\delta_2$) on the policy which you are trying to optimize.\n* I do not understand why Eq. (6) follows from the previous equation. If you are using a non-trivial result from previous work, please cite it.\n\n[1] Yuping Luo, Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, Tengyu Ma. Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees.", "summary_of_the_paper": "The paper studies transfer in reinforcement learning (RL), beginning with a theorem that relates the performance of one policy under a particular dynamics to another policy under different dynamics. This is broken down into a “dynamics-induced gap” and a “policy-induced gap”, for which explicit expressions are given. Optimizing a bound on the policy-induced gap w.r.t. the policy leads to an algorithm they call Relative Policy Optimization (RPO), and similarly optimizing a bound on the dynamics-induced gap w.r.t. the dynamics leads to an algorithm they call Relative Transition Optimization (RTO). The two algorithms can be combined into a single algorithm, Relative Policy-Transition Optimization (RPTO), which optimizes both the policy and the dynamics. Experiments indicate that RPTO achieves better transfer performance, both in terms of sample efficiency and asymptotic performance, than RPO and PPO warm-started from an expert policy from the source task.", "summary_of_the_review": "The submission fails to cite a previous paper which proved their main theoretical result, and the experiments leave room for improvement.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "G3apA7TLzsG", "reviewer": "Reviewer_vBa4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. Equation (1),(4) is called the Performance Difference Lemma. Equation (3) is called the Simulation Lemma. See e.g. Lemma 1.16 and Lemma 2.2 of this textbook [https://rltheorybook.github.io/rltheorybook_AJKS.pdf]. These are standard tools for analyzing model-based RL, and you can also find them in almost every theoretical model-based RL paper, so I don't know what's surprising or new about Theorem 1...\n\n2. Regarding the 1st remark below Theorem 1, the easiest way to estimate $J(P',\\pi)-J(P,\\pi)$ is to just run $\\pi$ in both $P$ and $P'$... The variance of $J(P',\\pi)$, thus the amount of data needed to estimate it accurately, is strictly smaller than the variance of the quantity inside the expectation on the RHS of equation (3).\n\n3. The experimental results show only marginal improvement over PPO, especially since each iteration of RPO and RPTO collect twice the amount of data than PPO (one trajectory in $P$ and one trajectory $P'$).", "summary_of_the_paper": "This paper proposes a model-based policy gradient algorithm that performs gradient updates on both the policy and a differentiable environment simmulator. ", "summary_of_the_review": "Overall, there is no theoretical justification of why the proposed method is better than vanilla TRPO/PPO, and the empirical advantage is also limited. The so-called \\textit{theory of relativity} for RL is nothing but basic regret decompositions well-known in the RL community.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper did not strike any reviewer as a critical addition to the literature, including various concerns regarding (1) the use of the general theory of relativity, (2) some components are well known in the past works.
Decision: Reject