Title: Inverse Q-Learning Done Right: Offline Imitation Learning in $Q^\pi$-Realizable MDPs
Venue: NeurIPS 2025
Accepted: yes
Keywords: imitation learning

primal dual methods

sample complexity guarantees
Abstract: We study the problem of offline imitation learning in Markov decision processes (MDPs), where the goal is to learn a well-performing policy given a dataset of state-action pairs generated by an expert policy. Complementing a recent line of work on this topic that assumes the expert belongs to a tractable class of known policies, we approach this problem from a new angle and leverage a different type of structural assumption about the environment. Specifically, for the class of linear $Q^\pi$-realizable MDPs, we introduce a new algorithm called saddle-point offline imitation learning (\SPOIL), which is guaranteed to match the performance of any expert up to an additive error $\varepsilon$ with access to $\mathcal{O}(\varepsilon^{-2})$ samples. Moreover, we extend this result to possibly nonlinear $Q^\pi$-realizable MDPs at the cost of a worse sample complexity of order $\mathcal{O}(\varepsilon^{-4})$. Finally, our analysis suggests a new loss function for training critic networks from expert data in deep imitation learning. Empirical evaluations on standard benchmarks demonstrate that the neural net implementation of \SPOIL is superior to behavior cloning and competitive with state-of-the-art algorithms.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe RL imitation learning problem is considered where a dataset is provided based on an expert policy and the objective is to output a policy whose expected return is nearly as large. Previous works assumed that the expert policy belonged to a policy class which was given to the learner and provided bounds based on the complexity of the policy class. This work instead considers an assumption on the structure of the MDP, namely that the state-action value function of all policies belong to a function class ($Q^\\pi$-realizability) given to the learner. The authors prove that their algorithm (SPOIL) requires samples from the expert policy that scales polynomially in the quantities of interest. This result is interesting because there exist MDPs that satisfy the $Q^\\pi$-realizability assumption; however, a complex policy class can be defined, which implies that the SPOIL algorithm will require less samples than previous policy based algorithms. The theoretical results are supported with experiments.\n\nSTRENGTHS AND WEAKNESSES:\n$\\textbf{Strengths:}$\n1. The contribution is clearly stated and compared to previous results.\n2. The work provides the first positive result under the $Q^\\pi$-realizability assumption, which is a weaker structural assumption than previous works.\n3. Experiments are provided supporting the theory.\n\n$\\textbf{Weaknesses:}$\n1. I think the performance difference lemma (Lemma 1) should have a factor of $1/(1-\\gamma)$ multiplying the right side of the equality based on the current definition of $\\rho^\\pi$. However, I think that $\\rho^\\pi$ was meant to be defined with a factor of $(1-\\gamma)$ in front since that is how it is used throughout (for instance see the equation below line 148). I did not go through all the proof details in the appendix; however, if $\\rho^\\pi$ is indeed incorrectly defined then the authors should make sure this does not affect their final result, or update it appropriately.\n2. Does Assumption 1 need realizability of $\\textit{all}$ policies, or is it enough to have it for just stationary policies, which I think is more commonly assumed? \n3. The norm notation $||\\cdot||$ is used in Assumption 1 but I do not think it is defined anywhere. I suspect it is the 2-norm, but it should be made explicit.\n4. The objective (eq 2) is very similar to that of offline policy optimization. It seems that a result from the offline policy optimization setting would imply a result in the imitation learning setting since there the policy $\\pi$ that is output satisfies $|\\max_{\\pi'}\\rho^{\\pi'} - \\rho^\\pi| \\le \\epsilon$. It would be useful if a discussion could be added (at least in the appendix) about how the offline imitation learning setting relates to the offline policy optimization setting.\n5. There are multiple typos that should be fixed. For example: Line 287, Line 327, Line 336\n6. The conclusion states the work does not require trajectory access while some other works do. I think this is the first time trajectory access is discussed in the paper and it is challenging to understand what is meant by trajectory access and why previous works needed this since it is not mentioned in the body. It would be helpful if trajectory access was discussed in more detail in the body when comparing to previous results.\n\nQUESTIONS:\nPlease see the Weaknesses section above.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper a core challenge in imitation learning is addressed, inconsistent expert behavior. Rather than assuming the expert policy lies within a known class of optimal policies, a common but often unrealistic assumption especially in scenarios where expert behavior may be irrational, or influenced by external factors (as with human experts). This issue is known as misspecification. To address this, the authors propose a formulation that avoids step-by-step behavioral imitation and instead aims to match the expert's performance outcome. This frees the learning agent from inconsistent behaviors that may be apparent in the expert policy and focus instead on learning policies that achieve comparable returns.\n\nThe proposed algorithm, SPOIL (Saddle-Point Offline Imitation Learning), is based on an actor-critic model structured as a min-max optimization of an empirically defined loss function to that fits the previous narrative. The authors implement this approach for both linear Q-functions, which are straightforward to analyze and allow efficient closed form updates; and for general function approximators, such as neural networks, which are more flexible and realistic but require optimization and introduce additional computational complexity.\nThey provide experimental results, first comparing both variants of SPOIL to BC in a controlled MDP environment to showcase its training and final converging superiority, and then proceed to standard gym environments, showcasing their performance with different sizes of expert data in contrast to other high-performing approaches.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. The approach has an interesting premise, especially as the community strives to get closer to a more realistic representation of imitation learning, where there is a plethora of ways for policies to get inconsistent or unstructured in general.\n\n2. Solid theoretical work, supported by well-structured theoretical analysis, including performance guarantees\n\n3. The SPOIL algorithm is appropriate for the presented problem, including both a linear version that is easier to implement and observe results, as well as a more general version for modern deep learning approaches. \n\n4. The structure and presentation of the paper is clear and overall straight forward.\n\nWeaknesses\n\n1.\tThe authors present experimental results, but there’s little explanation of what are the main differences between SPOIL and the baseline methods used.\n\n2.\tThe chosen environments are all toy environments, simple by their nature and low dimensional. These are fine for initial testing but not enough to support broader claims, especially when making state-of-the-art comparisons. \n\n3.\tSince the authors compare their work to BC, it would be fitting to also discuss how SPOIL handles or is affected by covariate shift.\n\n4.\tThe introduction packs in a lot of background without examples or intuitive explanations.\n\nQUESTIONS:\n1. Can you elaborate on your experimental results? \n\n2. What are the key differences with each one of the baselines?\n\n3. In which ways are the observed improvement of your algorithm significant?\n\n4. Can you comment on the lack of experiments in other, less toy, domains?\n\nSuggestions:\n\nMore challenging environments like Mujoco or Atari games would better fit your narrative. The authors could also take a look at this survey paper for environments used in imitation learning: https://arxiv.org/abs/2404.19456\n\nA simple example could have helped clarify the core idea early in the introduction.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-26"}

comments: SUMMARY:
This paper studies the problem of offline imitation learning in MDP given a dataset of state action pairs of expert policy. This paper relaxes the assumption of expert policy belonging to a known class to the learner and introduces a new algorithm for the class of linear $Q^{\pi}$-realizable MDPs that guarantees to match expert performance up to $\epsilon$ with access to $\mathcal{O}(\epsilon^{-2})$. Furthermore, they extend their results to non linear setting but with a worse sample complexity. Finally, they conduct empirical evaluations on their algorithms.

STRENGTHS AND WEAKNESSES:
Strengths:
- Quality: this paper is technically sound with support of theory and experiments.
- Clarity: paper is well presented with assumptions, theorems, algorithms and experiments.
- Significance: This paper relaxes a common assumption that expert policy is known to the learner and provides algorithms in both linear and non lineaner $Q^{\pi}$-realizable MDPs with theoretical guarantees.

Weaknesses:
- Originality: the novely mainly lies in the proof, the design of the algorithm is primal-dual style, which is similar to previous work of "Iq-learn: Inverse soft-q learning for imitation"

QUESTIONS:
Questions:
- why on Figure 1, the BC agents doesn't improve much with complex policy class?
- can you run IQ-Learn in linear Q environments as well to see its performance against proposed algorithms?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the problem of imitation learning when the expert policy may not be realizable by the policy class at hand. The authors propose an alternative view via Q-functions, where instead the core structural assumption made is that any policy's corresponding Q function is realizable. Under this structural hypothesis, imitation learning can be cast as a Q-learning-like problem without directly imposing policy realizability. In particular, successfully solving the saddle-point problem of minimizing a policy that achieves low worst-case Q-error with respect to the expert. Sample complexity bounds are provided for this alternate procedure and simple numerical experiments demonstrate it can outperform naive behavior cloning.\n\nSTRENGTHS AND WEAKNESSES:\nOverall, this paper introduces an interesting alternative perspective to imitation learning, where direct realizability of the expert policy can be avoided by instead considering the induced Q-functions of the policies. The analysis and interpretations in this paper make me lean toward acceptance.\n\n### Strengths\n\nThe paper tackles a tenuous aspect of imitation learning theory in policy realizability. It is demonstrated that Q-realizability can be somewhat more natural of an assumption. Given the imitation learning objective is to achieve low reward suboptimality, the performance difference lemma demonstrates it suffices to mimic the Q function of the expert. The resulting objective function and ensuing analysis are to my knowledge novel and somewhat appealingly straightforward, which indicates that the ideas in this paper may be useful downstream.\n\n### Weaknesses\n\nThe overall observation that imitation learning can be reduced to a fitting Q-functions is not necessarily new, see e.g. [1] which is discussed in the appendix related work. However, considering the similarities in the key object of study, these works should be explicitly discussed in the main paper.\n\nThough the proposed method presents various advantages to other offline imitation learning methods, it is mentioned that policy-based methods may see advantages in other settings where the policy class is simple and the environment is difficult. It may be pedagogical for the camera-ready to provide a simple example where this is the case. Orthogonal to this nuance, a rather trivial observation of when policy-based methods such as BC are advantageous is when expert demonstrations do not carry reward labels, which deserves a bit of discussion, as a potential advantage of IL over RL is that it can be agnostic to rewards (and as such reward optimality).\n\nAs a bibliographic note, the paper [2] mentioned in the appendix related work, which considers imitation learning over continuous action spaces, demonstrates exponential-in-horizon error cannot be improved in general, actually includes any algorithm including reward-based ones, as long as the demonstration distribution is unchanged (i.e. offline IL).\n\n[1] Swamy et al. \"Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gap\"\n\n[2] Simchowitz et al. \"The Pitfalls of Imitation Learning when Actions are Continuous\"\n\nQUESTIONS:\nIt would be helpful to clarify some of the above points under Weaknesses. Besides that, I have a question out of curiosity: when lacking reward labels, is this method compatible with inverse reinforcement learning? If not, what are the potential theoretical barriers?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}