Title: IOSTOM: Offline Imitation Learning from Observations via State Transition Occupancy Matching
Venue: NeurIPS 2025
Accepted: yes
Keywords: Learning from Observations

Imitation Learning
Abstract: Offline Learning from Observations (LfO) focuses on enabling agents to imitate expert behavior using datasets that contain only expert state trajectories and separate transition data with suboptimal actions. This setting is both practical and critical in real-world scenarios where direct environment interaction or access to expert action labels is costly, risky, or infeasible. Most existing LfO methods attempt to solve this problem through state  or state-action occupancy matching. They typically rely on pretraining a discriminator to differentiate between expert and non-expert states, which could introduce errors and instability—especially when the discriminator is poorly trained. While recent discriminator-free methods have emerged, they generally require substantially more data, limiting their practicality in low-data regimes. In this paper, we propose IOSTOM ($\textit{Imitation from Observation via State Transition Occupancy Matching}$), a novel offline LfO algorithm designed to overcome these limitations. Our approach formulates a learning objective based on the joint state visitation distribution. A key distinction of IOSTOM is that it first excludes actions entirely from the training objective. Instead, we learn an $\textit{implicit policy}$ that models transition probabilities between states, resulting in a more compact and stable optimization problem. To recover the expert policy, we introduce an efficient action inference mechanism that $\textit{avoids training an inverse dynamics model}$. Extensive empirical evaluations across diverse offline LfO benchmarks show that IOSTOM substantially outperforms state-of-the-art methods, demonstrating both improved performance and data efficiency.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents imitation learning from observation via state transition occupancy matching (IOSTOM), a new offline LfO algorithm that excludes actions entirely from the training objective. The proposed algorithm learn an implicit policy that models transition probabilities between states, rather than leveraging action information of non-expert demonstrations. In the experiments, the authors demonstrate that IOSTOM outperforms other baseline algorithms on diverse sets of environments and tasks from D4RL benchmark.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- The proposed method is theoretically well-grounded, with clearly derived formulations based on flow constraints, Lagrangian duality, and Q-weighted behavior cloning.\n- The method is evaluated across a wide range of environments and consistently achieves strong performance.\n\n**Weaknesses**\n\n- There is no experimental comparison with LobsDICE, one of the major baseline algorithms, considering the same problem setting.\n- Since hyperparameter selection for the real environment is not possible in offline settings, analysis of hyperparameter alpha is thought to be necessary.\n- It requires an additional hyperparameter beta, and its performance is very sensitive to $\\beta$.\n- It seems necessary to check whether fair experiments and comparisons were conducted in an offline setting. (ex. when determining hyperparameter $\\alpha$ and $\\beta$, whether the settings with the best performance in the test environment were selected and reported as the final results.)\n\nQUESTIONS:\n- What is the main reason for considering mixed state visitation disruption instead of starting from directly matching the objective with $d^E$? (What is the main problem if we derive it in the same way as direct matching for $d^E$?) It would be better if the reason for this part was explicitly explained in the paper.\n- How was the value of hyperparameter $\\alpha$ selected? How sensitive is IOSTOM's performance to hyperparameter $\\alpha$?\n- The optimal value of the hyperparameter alpha seems to be different for each environment. Is there no way to automatically determine $\\alpha$ (also for $\\beta$)?\n- LobsDICE is also a representative comparison algorithm that can be considered in the same setting. Why didn't you compare it? Can you add a comparison experiment?\n- In the paper, it was explained that the absence of discriminator learning is an advantage, but instead, $V$ learning was introduced. Isn't it similar? What makes it better?\n- Is it true that all the experimental learning was done with only offline datasets? Is it true that hyperparameter search and final model selection were also done without testing the environment?\n- In all experiments, were you only training using the offline dataset? Was there no access to the test environment even during hyperparameter search and final model selection?\n\nLIMITATIONS:\nYes, the authors adequately addressed the limitations and potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes IOSTOM, an offline imitation learning from observation method based on state transition occupancy matching that is discriminator-free and data-efficient. The proposed method operates on a state-only expert dataset and state-action suboptimal dataset. The proposed method consists of several steps. First, IOSTOM recovers a state-transition probability distribution (\"implicit policy\") by minimizing $\\chi^2$-divergence between the learned state-transition and expert state-transition (regularized by suboptimal dataset policy), which can be converted to an unconstrained bilevel Q-value optimization via Lagrange multipliers and convex conjugates. Soft value estimates are adopted to avoid Out-Of-Distribution (OOD) samples. Finally, the policy is retrieved by advantage-weighted behavior cloning from the suboptimal state-action dataset (as a lower bound surrogate for optimizing implicit policy with Inverse-Dynamic Model (IDM)). The proposed algorithm is evaluated on multiple environments which outperforms prior methods.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The paper is well-written and easy to follow; though the paper is math-heavy, the high-level idea behind the formulae (first optimize state-pair divergence, then use Lagrange multiplier & convex conjugate to convert to unconstrained problems, then soft values to avoid OOD issues, and finally extract policy via lower bound of implicit policy with IDM) is very clearly conveyed.\n\n2. The idea of this paper is interesting and well-supported by mathematical derivations: bypassing discriminators by optimizing \"implicit policy\", which is a novel notion.\n\n3. The evaluation results look solid with sufficient amount of ablations. I particularly appreciate the video demonstrations in the supplementary material.\n\n**Weaknesses**\n\n1. the training objective in Eq. 9 seems potentially unstable, as there is a term $(Q(s,s')-\\frac{2}{1-\\gamma})^2$. When $\\gamma$ is close to 1 with long horizon tasks, this can lead to a very large value of $Q$ which causes numerical issues during optimization.\n\n2. The number of expert dataset used in the experiment is quite large, especially for the MuJoCo datasets. In the appendix, the authors stated that they use 30 expert trajectories for \"few-expert\" scenario. In contrast, both SMODICE (https://arxiv.org/pdf/2202.02433, Sec. 6.1) and PW-DICE (https://arxiv.org/pdf/2311.01331, Appendix D.2) uses **single** expert trajectory. I would like to see how the proposed algorithm work with even less amount of expert trajectories than the current setting.\n\n**Minor Weaknesses**\n\n1. Some reference format are incorrect. For example: \"[34] introduce ReCOIL\" -> \"Sikchi et al. [34] introduce ReCOIL\"; \"Following [40]\" -> \"Following Xu et al. [40]\".\n\n2. The period in Tab. 4, Tab. 5 and Fig. 3 caption is missing.\n\n3. The main paper uses \"inverse dynamic model\", but the appendix line 648 uses \"inverse kinematics model\".\n\nQUESTIONS:\nI have several questions:\n\n1. Is it possible to add a teaser figure for this paper? I feel people who are less familiar with the state occupancy matching framework can understand the idea more easily with an illustration.\n\n2. Is there a more direct ablation / empirical evidence (e.g. visualization) to show in some cases discriminators fail other than final performance difference?\n\n3. Why do the authors choose $\\alpha=0.5$ for distribution mixture with suboptimal dataset, which seems quite large? Can the authors provide an ablation on different values of $\\alpha$?\n\n4. Why do the authors choose to update policy during the training loop of other components (as shown in Alg. 1) instead of finishing the other parts of training first, which could potentially cause instability? $\\theta$ is not involved in other parts of training after all.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

comments: SUMMARY:
This paper introduces a novel approach (IOSTOM) for offline LfO problem by joint state visitation distribution matching. This method specifically avoids using discriminators or inverse dynamics models, and instead learns an implicit policy via state-to-state transition probabilities without using action label. Empirical results on D4RL datasets demonstrate the efficacy of the method.

STRENGTHS AND WEAKNESSES:
**Strengths:**

- The paper presents a novel LfO approach grounded in occupancy matching.

- The theoretical framework and derivation of the algorithm is very solid.

- The method is proved to be  practical and efficient with its discriminator-free and IDM-free design.

- Demonstrates better performance on diverse and challenging benchmarks, outperforming all baselines, especially inclusing an interesting real-world case study on marine navigation.

- Demonstrates robustness to low-data settings and sub-sampling.

  

**Weaknesses:**

* Access to fully observable actions in sub-optimal dataset is hard to guarantee. It would be helpful to discuss how realistic it is.

QUESTIONS:
What is the intuition of using a mixture of joint state visitation distributions $d_\mathrm{mix}^I, d_\mathrm{mix}^{E,I}$? How does the mixing parameter $\alpha$ influences the training? It would be helpful to introduce the intuition before deriving the full equation.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-29

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces ISOTOM, a framework for offline imitation learning from observation (IfO). ISOTOM aims to address the offline IfO problem while improving training stability by eliminating the need for a discriminator and an inverse dynamics model. Specifically, ISOTOM relies on state-transition distribution matching to remove dependence on action information. The paper first presents a joint-state Q-learning formulation to recover the implicit policy, followed by policy learning through advantage-weighted behavioral cloning. Empirical results demonstrate that ISOTOM outperforms baseline methods in MuJoCo environments and a marine navigation scenario.\n\nSTRENGTHS AND WEAKNESSES:\n[Strengths]\n\n(1) Quality\n- The paper provides both theoretical analyses and empirical results demonstrating the effectiveness of the proposed method. However, I did not examine the proofs in the appendix in detail.\n- The experiment section includes numerical results not only for MuJoCo simulated environments but also for a real-world marine navigation dataset.\n\n(3) Significance\n- The proposed method demonstrates superior performance compared to the baselines in both MuJoCo simulation environments and a real-world marine navigation scenario.\n\n(4) Originality\n- The paper introduces an objective function that excludes actions, which appears to be a novel contribution in the context of offline imitation from observation.\n\n[Weaknesses]\n\n(2) Clarity\n- I believe the primary issue that makes me hesitant to recommend solid acceptance is the lack of clarity in the paper. While this may partly reflect my own limitations in expertise, the presentation feels unstructured and densely written, which hinders comprehension. For example:\n    - In Section 3, the definitions of $d^E(s)$ and $d^I(s)$ are missing or unclear.\n    - In Section 4.1, $d_0(s)$ is not properly defined.\n    - The transition from Equation (3) to (4) is not clearly explained. I was able to derive (4) from (3), but it required effort that could have been avoided with a brief clarification.\n    - The paper does not specify where the proofs of the main theorem can be found, even though they are in the appendix.\n    - Additionally, several parts of the text (e.g., lines 121, 124, 146, 148, 150, 225, 235, 244) are written in a way that makes them difficult to parse.\n\nThese issues in notation, definition, and mathematical transitions collectively hindered my ability to follow the technical development of the paper. As a result, it was difficult to clearly identify the main contributions and understand the reasons behind the strengths of the proposed method compared to prior work. Improving the logical flow between sections and organizing the key concepts more clearly would significantly enhance readability and overall comprehension.\n\nQUESTIONS:\n- Have the authors conducted any experiments with subsampled random data to assess the sensitivity of the method to the amount of random data?\n- Have the authors conducted any experiments in image-based environments? Such settings may better reflect the nature of imitation from observation scenarios, where actions are not available and only visual state information is provided.\n\nI am willing to increase the score if most of the concerns in the weakness and questions sections are solved.\n\nLIMITATIONS:\nYes (in the appendix)", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 1, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}