Title: Occupancy-based Policy Gradient: Estimation, Convergence, and Optimality
Venue: NeurIPS 2024
Accepted: yes
Keywords: policy gradient

offline reinforcement learning

offline policy gradient
Abstract: Occupancy functions play an instrumental role in reinforcement learning (RL) for guiding exploration, handling distribution shift, and optimizing general objectives beyond the expected return. Yet, computationally efficient policy optimization methods that use (only) occupancy functions are virtually non-existent. In this paper, we establish the theoretical foundations of model-free policy gradient (PG) methods that compute the gradient through the occupancy for both online and offline RL, without modeling value functions. Our algorithms reduce gradient estimation to squared-loss regression and are computationally oracle-efficient. We characterize the sample complexities of both local and global convergence, accounting for both finite-sample estimation error and the roles of exploration (online) and data coverage (offline). Occupancy-based PG naturally handles arbitrary offline data distributions, and, with one-line algorithmic changes, can be adapted to optimize any differentiable objective functional.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nWith a focus on \"occupancy-based\" methods for RL, the authors propose a \nmodel-free, policy gradient method for policy optimization in finite-horizon \nMDPs via occupancy estimation. Guided by the representation of return in terms \nof the state-visitation probability and reward, they express the gradient of \nthe return in terms of gradient of the log of the state-visitation probability. \nThey propose to estimate the latter term via squared loss regression (see \neq 2). This estimate is then used to approximate the gradient of the return \nwhich is then plugged in to the usual policy gradient objective. In the online \nRL setting, they provide gradient estimation guarantees and convergence to a \nnear-optimal policy. For offline RL, due to insufficient coverage of the \noffline dataset, they provide guarantees on the estimate of the gradient of the \nclipped return.\n\n**Update**: I have revised my score based on the following considerations:\n1. The authors satisfactory response to the rebuttal,\n2. The unique approach to policy gradients, which may be insightful in advancing research on policy gradient methods,\n3. While rigidity of their assumption in the offline setting may limit its broader applicability, it also leaves room for further improvement.\n4. The authors willingness to address the other reviewers suggestions on notation and presentation.\n\nSTRENGTHS:\n1. To the best of my knowledge, the authors propose a novel perspective to policy gradients in online and offline RL based on computing the return gradient by first estimating the gradient of occupancy measures.\n\n2. Their method for the online setting is technically sound with local and global convergence claims supported by theoretical analysis and relevant assumptions.\n\nWEAKNESSES:\nFor the offline setting, the authors introduce new assumptions in the appendix which appear to influence their main result in Theorem 16. Precisely, in the proof of Theorem 16, the authors make use of Lemmas which refer to assumption 7 on page 37 and assumption 8 on page 38 to respectively control the MLE estimation error and density ratio estimation error. As such the paper is not self-contained and the results are rather difficult to verify.\n\nQUESTIONS:\nJust some additional comments:\n\n1. Line 70: should be $h'\\geq h$.\n2. In Line 151, do you mean $\\psi^{\\pi}$ instead of $\\psi$? Also, the first expression $d^{\\pi}=\\mu(s)^{T}\\psi$ seems inconsistent.\n\nLIMITATIONS:\nNone. This is a purely theory paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces policy gradient algorithms that focus on\nestimating the gradient of the *occupancy measure* with respect to the\npolicy parameters. In this framework, one can easily extend convergence\nanalysis beyond the standard RL objective (e.g., maximization of\nexpected return) to a much broader class of occupancy objectives, such\nas imitation learning or pure exploration. In addition to an online\nalgorithm, the authors design an algorithm for the offline setting based\non pessimistic corrections to density ratios, and provide bounds on the\npolicy gradient error for both under function approximation. In the case\nof the offline algorithm, this paper demonstrates convergence bounds\nunder weaker assumptions than existing methods. In the case of the\nonline algorithm, the paper further provides conditions under which\ntheir policy gradient algorithm achieves global optimality.\n\nSTRENGTHS:\nThis paper is very interesting and extemely thorough. I really like the idea of\nestimating the occupancy gradient in order to generalize over objectives for\nsequential decision-making; these seems quite powerful, and has the potential of\nbeing very impactful. Effectively, here the authors have provided a\nmeta-algorithm (with convergence bounds) for a much broader class of algorithms\nthan standard policy gradient methods, which can unify several fields of research.\nThe Offline OccuPG method was really neat, and makes great\nuse of the existing FORC method for estimating clipped density ratios.\n\nWEAKNESSES:\nThe biggest weakness for the paper, in my opinion, is that it could\nbenefit from extra clarification at many points. Realistically, this was\nprobably omitted in favor of making room for more content (the paper is\nvery dense), but I suggest the authors add some guiding discussions.\n\nParticularly, I think the following should be clarified:\n\n- **Motivation**: My first instinct after having read the draft was\n  \"this was very interesting to read, but how is the field of RL better\n  off in light of these results?\". The conclusion and introductory\n  sections place a lot of emphasis on the fact that this paper presents\n  the first algorithms/convergence results based on estimating the\n  occupancy measure, but it's not immediately clear why this is so\n  impactful. Ultimately, my impression is that the following two points\n  (especially the first) are the real winners:\n  1.  The ability to optimize a general class of functionals;\n  2.  Weaker assumptions for convergence with the offline algorithm.\n\n  I believe emphasizing these points more, and motivating them, can\n  really help the reader appreciate the results, but I found these\n  points to get a little lost in the sea of math.\n- **Comparison to existing work**: While I understand that this paper is\n  the first to analyze convergence of PG methods based on occupancy\n  measures, the paper focuses largely on the application of optimizing\n  expected return. As such, as someone that is not intimately familiar\n  with convergence results for PG methods, it would have been helpful to\n  have a more concrete comparison between the bounds presented in this\n  work and existing ones.\n- **Misc. writing/clarity issues**: Some math and/or algorithmic details\n  were ambiguous and/or unclear, listed below. This made the logic a\n  little difficult to follow at times.\n\n## Minor issues\n\nOn line 28, \"In answer\" -\\> \"In response\".\n\nIn Algorithm 1, $\\mathcal{D}^{\\mathrm{grad}}$ is not properly defined. On\nline 2 of the algorithm, it is implied that it is seeded the same way as\n$\\mathcal{D}^{\\rm reg}$, but on line 7 it appears that\n$\\mathcal{D}^{\\rm grad}$ is supposed to contain reward data (unlike\n$\\mathcal{D}^{\\rm reg}$).\n\nIn Algorithm 1, I think the definition of $\\mathcal{D}^{\\mathrm{reg}}$ has a\ntypo. It says $\\mathcal{D}_h^{\\mathrm{reg}} = \\{(s\\_h, a\\_h,\ns\\_{h+1})\\}\\_{i=1}^n$, but the index $i$ does not appear anywhere. My\nguess is that it should really be something like\n$\\mathcal{D}\\_h^{\\mathrm{reg}} = \\{(s\\_h^{(i)}, a\\_h^{(i)},\ns\\_{h+1}^{(i)})\\}\\_{i=1}^n$, where e.g. $\\{s^{(i)}\\_h\\}$ is the sequence of\nstates encounted over the course of rollout $i$.\n\nIn Theorem 2, $\\mathsf{pd}\\_{\\mathcal{G}}$ looks confusing (not clear\nthat it is $\\mathsf{p}$ times $\\mathsf{d}\\_{\\mathcal{G}}$ as opposed to a\nvariable called $\\mathsf{pd}\\_{\\mathcal{G}}$). Is it necessary to use\nsans serif font for $\\mathsf{p}$?\n\nIn definition 46, I believe there is a notational issue. It says\n$\\mathcal{F}\\subset\\mathbb{R}^{\\mathcal{X}}$, but then you have a term\n$\\mathrm{sign}(f(x_1^n - c))$ for $f\\in\\mathcal{F}$. But $x_1^n -\nc\\not\\in\\mathcal{X}$. Are you actually mapping $f$ over the dimensions\nof $x_1^n - c$? Also, there is a typo here, $c$ should be $v$.\n\nQUESTIONS:\nIn Corollary 6, what does it mean to run OccuPG with initial\ndistribution $\\nu$?\n\nOn line 238, you defined $\\bar{\\pi}_h = (\\pi\\land\nC^\\mathbf{a}_h\\pi^D_h)$. I am having some trouble interpreting this.\nFirstly, should it not be $(\\pi_h\\land C^\\mathbf{a}_h\\pi^D_h)$? Then, if\nthat's the case, is this equivalent to\n\n$$\\begin{align*}\n\\bar{\\pi}_h(a\\mid s) = \\min\\{\\pi_h(a\\mid s), C^a_h\\pi^D_h(a\\mid s)\\}?\n\\end{align*}$$\n\nGiven that the text focuses almost entirely on the problem of maximizing\nexpected return, how do the bounds given in this paper compare to those\nof existing results? Is there any benefit to running OccuPG if you only\ncare about expected return (especially in the online setting)?\n\nHow tight are your convergence bounds, particularly in the case of\noptimizing general functionals?\n\nIn Lemma 5, when should one expect to have $\\mathcal{C}^{\\pi^*}<\\infty$?\n\nLIMITATIONS:\nNo major issues in this regard. Two limitations come to mind that were not\nthoroughly addressed:\n1. For optimization of general functionals, convergence results depend on a\n   Lipschitz assumption on the functional that is only discussed on the\n   appendix. How generally is this assumption satisfied?\n2. For the global convergence result (Lemma 5), there is a very strict\n   assumption on the initial state distribution, which would not often be\n   satisfied in practice (to my understanding). That said, other theoretical\n   works have made similar assumptions.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

comments: SUMMARY:
This paper investigates the convergence of model-free policy gradient (PG) methods that compute the gradient through occupancy.

STRENGTHS:
The convergence analysis and the theorem appear to be sound.

WEAKNESSES:
In my opinion, the paper makes overclaims. The convergence depends on a coverage coefficient, which removes the need for exploration. Therefore, the obtained bound does not indicate the number of samples needed for exploration when the initial policy is not good. Additionally, the proposed algorithm does not address the problem of exploration.

QUESTIONS:
See the 'Weakness' part.

LIMITATIONS:
See the 'Weakness' part.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-17