SlowGuess/NeurIPS_2023_OCR huggingface dataset:
When Demonstrations Meet Generative World Models: A Maximum Likelihood Framework for Offline Inverse Reinforcement Learning
Siliang Zeng*
zeng0176@umn.edu
Alfredo Garcia
alfredo.garcia@tamu.edu
Chenliang Li*
chenliangli@tamu.edu
Offline inverse reinforcement learning (Offline IRL) aims to recover the structure of rewards and environment dynamics that underlie observed actions in a fixed, finite set of demonstrations from an expert agent. Accurate models of expertise in executing a task has applications in safety-sensitive applications such as clinical decision making and autonomous driving. However, the structure of an expert's preferences implicit in observed actions is closely linked to the expert's model of the environment dynamics (i.e. the "world" model). Thus, inaccurate models of the world obtained from finite data with limited coverage could compound inaccuracy in estimated rewards. To address this issue, we propose a bi-level optimization formulation of the estimation task wherein the upper level is likelihood maximization based upon a conservative model of the expert's policy (lower level). The policy model is conservative in that it maximizes reward subject to a penalty that is increasing in the uncertainty of the estimated model of the world. We propose a new algorithmic framework to solve the bi-level optimization problem formulation and provide statistical and computational guarantees of performance for the associated optimal reward estimator. Finally, we demonstrate that the proposed algorithm outperforms the state-of-the-art offline IRL and imitation learning benchmarks by a large margin, over the continuous control tasks in MuJoCo and different datasets in the D4RL benchmark<sup>2</sup>.
Reinforcement learning (RL) is a powerful and promising approach for solving large-scale sequential decision-making problems [1, 2, 3]. However, RL struggles to scale to the real-world applications due to two major limitations: 1) it heavily relies on the manually defined reward function [4], 2) it requires the online interactions with the environment [5]. In many application scenarios such as dialogue system [6] and robotics [7], it is difficult to manually design an appropriate reward for constructing the practical reinforcement learning system. Moreover, for some safety-sensitive applications like clinical decision making [8, 9] and autonomous driving [10, 11], online trials and errors are prohibited
2Our implementation is available at https://github.com/Cloud0723/Offline-MLIRL
Figure 1: Illustration of the modular structure in our algorithmic framework, Offline ML-IRL. In Offline ML-IRL, it first estimates a world model from the dataset of transition samples, and then implements an ML based offline IRL algorithm on the estimated world model to recover the ground-truth reward function from the collected expert trajectories.
due to the safety concern. Due to these limitations in the practical applications, a new paradigm – learning from demonstrations, which relies on historical datasets of demonstrations to model the agent for solving sequential decision-making problems – becomes increasingly popular. In such a paradigm, it is important to understand the demonstrator and imitate the demonstrator's behavior by only utilizing the collected demonstration dataset itself, without further interactions with either the demonstrator or the environment.
In this context, offline inverse reinforcement learning (offline IRL) has become a promising candidate to enable learning from demonstrations [12, 13, 14, 9, 15]. Different from the setting of standard IRL [16, 17, 18, 19, 20, 21] which recovers the reward function and imitates expert behavior at the expense of extensive interactions with the environment, offline IRL is designed to get rid of the requirement in online environment interactions by only leveraging a finite dataset of demonstrations. While offline IRL holds great promises in practical applications, its study is still in an early stage, and many key challenges and open research questions remain to be addressed. For example, one central challenge in offline IRL arises from the so-called distribution shift [22, 15] – that is, the situation where the recovered reward function and recovered policy cannot generalize well to new unseen states and actions in the real environment. Moreover, in the training process of the offline IRL, any inaccuracies in the sequential decision-making process induced by distribution shift will compound, leading to poor performance of the estimated reward function / policy in the real-world environment. This is due to the fact that offline IRL is trained upon fixed datasets, which only provide limited coverage to the dynamics model of the real environment. Although there are some recent progress in offline IRL [14, 9, 15], how to alleviate distribution shift in offline IRL is still rarely studied and not clearly understood. Witnessing recent advances in a closely related area, the offline reinforcement learning, which incorporates conservative policy training to avoid overestimation of values in unseen states induced by the distribution shift [23, 24, 25, 26, 27], in this work we aim to propose effective offline IRL method to alleviate distribution shift and recover high-quality reward function from collected demonstration datasets. Due to the space limitations, we refer readers to Appendix B for more related work.
Our Contributions. To alleviate distribution shift and recover high-quality reward function from fixed demonstration datasets, we propose to incorporate conservatism into a model-based setting and consider offline IRL as a maximum likelihood estimation (MLE) problem. Overall, the goal is to recover a reward that generates an optimal policy to maximize the likelihood over observed expert demonstrations. Towards this end, we propose a two-stage procedure (see Fig. 1 for an overview). In the first stage, we estimate the dynamics model (the world model) from collected transition samples; by leveraging uncertainty estimation techniques to quantify the model uncertainty, we are able to construct a conservative Markov decision process (conservative MDP) where the state-action pairs with high model uncertainty and low data coverage receive a high penalty value to avoid risky exploration in the unfamiliar region. In the second stage, we propose an IRL algorithm to recover the reward function, whose corresponding optimal policy under the conservative MDP constructed in the first stage maximizes the likelihood of observed expert demonstrations. To the best of our knowledge, it is the first time that ML-based formulation, as well as the associated statistical and computational guarantees for reward recovery, has been developed for offline IRL.
To summarize, our main contributions are listed as follows:
- We consider a formulation of offline IRL based on MLE over observed transition samples and expert trajectories. In the proposed formulation, we respectively model the transition dynamics and
the reward function as the maximum likelihood estimators to generate all observed transition samples and all collected expert demonstrations. We provide a statistical guarantee to ensure that the optimal reward function of the proposed formulation could be recovered as long as the collected dataset of transition samples has sufficient coverage on the expert-visited state-action space.
- We develop a computationally efficient algorithm to solve the proposed formulation of offline IRL. To avoid repeatedly solving the policy optimization problem under each reward estimate, we propose an algorithm which alternates between one reward update step and one conservative policy improvement step. Under nonlinear parameterization for the reward function, we provide the theoretical analysis to show that the proposed algorithm converges to an approximate stationary point in finite time. Moreover, when the reward is linearly parameterized and there is sufficient coverage on the expert-visited state-action space to construct the estimated world model, we further show that the proposed algorithm approximately finds the optimal reward estimator of the MLE formulation.
- We conduct extensive experiments by using robotic control tasks in MuJoCo and collected datasets in D4RL benchmark. We show that the proposed algorithm outperforms the state-of-the-art offline IRL such as [14, 15] and imitation learning methods such as [28], especially when the number of observed expert demonstrations is limited. Moreover, we transfer the recovered reward across different datasets to show that the proposed method can recover high-quality reward function from the expert demonstrations.
Markov decision process (MDP) is defined by the tuple  $(S, \mathcal{A}, P, \eta, r, \gamma)$ , which consists of the state space  $S$ , the action space  $\mathcal{A}$ , the transition dynamics  $P: S \times \mathcal{A} \times S \to [0,1]$ , the initial state distribution  $\eta(\cdot)$ , the reward function  $r: S \times \mathcal{A} \to \mathbb{R}$  and the discounted factor  $\gamma \in (0,1)$ . Under a transition dynamics model  $P$  and a policy  $\pi$ , we are able to further define the state-action visitation measure as  $d_P^\pi(s, a) := (1 - \gamma)\pi(a|s) \sum_{t=0}^{\infty} \gamma^t P^\pi(s_t = s | s_0 \sim \eta)$  for any state-action pair  $(s, a)$ .
Maximum entropy inverse reinforcement learning (MaxEnt-IRL) is a specific IRL formulation which aims to recover the ground-truth reward function and imitate the expert's policy from expert's demonstrations [18, 29, 20]. Let  $\tau^{\mathrm{E}}\coloneqq \{(s_t,a_t)\}_{t = 0}^{\infty}$  denotes the expert trajectory sampled from the expert policy  $\pi^{\mathrm{E}}$ ; let  $\tau^{\mathrm{A}}$  denote the trajectory generated by the RL agent with policy  $\pi$ . Then the MaxEnt-IRL is formulated as:
$$
\max  _ {r} \min  _ {\pi} \left\{\mathbb {E} _ {\tau^ {\mathrm {E}} \sim \pi^ {\mathrm {E}}} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \cdot r \left(s _ {t}, a _ {t}\right) \right] - \mathbb {E} _ {\tau^ {\mathrm {A}} \sim \pi} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \cdot r \left(s _ {t}, a _ {t}\right) \right] - H (\pi) \right\} \tag {1}
$$
where  $H(\pi) \coloneqq \mathbb{E}_{\tau \sim \pi} \left[ \sum_{t=0}^{\infty} -\gamma^{t} \log \pi(a_{t}|s_{t}) \right]$  denotes the causal entropy of the policy  $\pi$ . The MaxEnt-IRL formulation aims to recover the ground-truth reward function which assigns high rewards to the expert policy while assigning low rewards to any other policies. Although MaxEnt-IRL has been well-studied theoretically [29, 30, 21, 31] and has been applied to several practical applications [32, 33, 34], it needs to repeatedly solve policy optimization problems under each reward function and online interactions with the environment is inevitable. Such repeated policy optimization subroutine requires extensive online trials and errors in the environment and thus makes MaxEnt-IRL quite limited in practical applications.
Problem formulation Let us now consider an ML formulation of offline IRL. Given the transition dataset  $\mathcal{D} \coloneqq \{(s, a, s')\}$ , we train an estimated world model  $\widehat{P}(s'|s, a)$  for any  $s', s \in S$  and  $a \in \mathcal{A}$  (to be discussed in detail in Sec. 3). The constructed world model  $\widehat{P}$  will be utilized as an estimate of the ground-truth dynamics model  $P$ . Based on the estimated world model  $\widehat{P}$ , we propose a model-based offline approach for IRL from the ML perspective, given below:
$$
\max  _ {\theta} L (\theta) := \mathbb {E} _ {\tau^ {\mathrm {E}} \sim (\eta , \pi^ {\mathrm {E}}, P)} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \log \pi_ {\theta} \left(a _ {t} \mid s _ {t}\right) \right] \tag {2a}
$$
$$
s. t. \quad \pi_ {\theta} := \arg \max  _ {\pi} \mathbb {E} _ {\tau^ {\mathrm {A}} \sim (\eta , \pi , \widehat {P})} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \left(r \left(s _ {t}, a _ {t}; \theta\right) + U \left(s _ {t}, a _ {t}\right) + \mathcal {H} \left(\pi \left(\cdot \mid s _ {t}\right)\right)\right) \right], \tag {2b}
$$
where  $\mathcal{H}\big(\pi (\cdot |s)\big)\coloneqq \sum_{a\in \mathcal{A}} - \pi (a|s)\log \pi (a|s)$  denotes the entropy of the distribution  $\pi (\cdot |s);U(\cdot ,\cdot)$  is a penalty function to quantify the uncertainty of the estimated world model  $\hat{P} (\cdot |s,a)$  under any
state-action pair  $(s, a)$ . In practice, the penalty function is constructed based on uncertainty heuristics over an ensemble of estimated dynamics models [35, 25, 24, 36]. A comprehensive study of the choices of the penalty function can be found in [37]. Next, let us make a few remarks about the above formulation, which we name Offline ML-IRL.
First, the problem takes a bi-level optimization form, where the lower-level problem (2b) assumes that the parameterized reward function  $r(\cdot, \cdot; \theta)$  is fixed, and it describes the optimal policy  $\pi_{\theta}$  as a unique solution to solve the conservative MDP; On the other hand, the upper-level problem (2a) optimizes the reward function  $r(\cdot, \cdot; \theta)$  so that its corresponding optimal policy  $\pi_{\theta}$  maximizes the log-likelihood  $L(\theta)$  over observed expert trajectories.
Second, formulating the objective as a likelihood function is reasonable since it searches for an optimal reward function to explain the observed expert behavior within limited knowledge about the world (the world model  $\widehat{P}$  is constructed based on a finite and diverse dataset  $\mathcal{D} \coloneqq \{(s, a, s')\}$ ).
Third, the lower-level problem (2b) corresponds to a model-based offline RL problem under the current reward estimate. The policy obtained is conservative in that state-action pairs that are not well covered by the dataset are penalized with a measure of uncertainty in the estimated world model. The penalty function  $U(s,a)$  is used to quantify the model uncertainty and regularize the reward estimator. Therefore, the optimal policy  $\pi_{\theta}$  under the conservative MDP will not take risky exploration on those uncertain region of the state-action space where the transition dataset does not have sufficient coverage, and the constructed world model has high prediction uncertainty.
3 The world model and statistical guarantee
In this section, we construct the world model  $\widehat{P}$  from the transition dataset  $\mathcal{D} \coloneqq \{(s, a, s')\}$  and solve a certain approximated version of the formulation (2). Then we further show a high-quality reward estimator can be obtained with statistical guarantee.
Before proceeding, let us emphasize that one major challenge in solving (2) comes from the dynamics model mismatch between (2a) and (2b), which arises because the expert trajectory  $\tau^{\mathrm{E}}$  is generated from the ground-truth transition dynamics  $P$ , while the agent samples its trajectory  $\tau^{\mathrm{A}}$  through interacting with the estimated world model  $\hat{P}$ . To better understand the above challenge, next we will explicitly analyze the likelihood objective in (2) and understand the mismatch error. Towards this end, let us introduce below the notions of the soft  $Q$ -function and the soft value function of the conservative MDP in (2) (defined for any reward parameter  $\theta$  and the optimal policy  $\pi_{\theta}$ ):
$$
V _ {\theta} (s) := \mathbb {E} _ {\tau \sim (\eta , \pi_ {\theta}, \widehat {P})} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} \left(r \left(s _ {t}, a _ {t}; \theta\right) + U \left(s _ {t}, a _ {t}\right) + \mathcal {H} \left(\pi_ {\theta} (\cdot | s _ {t})\right)\right) \mid s _ {0} = s \right], \tag {3a}
$$
$$
Q _ {\theta} (s, a) := r (s, a; \theta) + U (s, a) + \gamma \mathbb {E} _ {s ^ {\prime} \sim \widehat {P} (\cdot | s, a)} \left[ V _ {\theta} \left(s ^ {\prime}\right) \right]. \tag {3b}
$$
According to [38, 39, 40], the optimal policy  $\pi_{\theta}$  and the optimal soft value function  $V_{\theta}$  have the following closed-form expressions under any state-action pair  $(s,a)$ :
$$
\pi_ {\theta} (a | s) = \frac {\exp Q _ {\theta} (s , a)}{\sum_ {\tilde {a} \in \mathcal {A}} \exp Q _ {\theta} (s , \tilde {a})}, \quad V _ {\theta} (s) = \log \left(\sum_ {a \in \mathcal {A}} \exp Q _ {\theta} (s, a)\right). \tag {4}
$$
Under the ground-truth dynamics model  $P$  and the initial distribution  $\eta(\cdot)$ , we further define the visitation measure  $d^{\mathrm{E}}(s, a)$  under the expert policy  $\pi^{\mathrm{E}}$  as below:
$$
d ^ {\mathrm {E}} (s, a) := (1 - \gamma) \pi^ {\mathrm {E}} (a | s) \sum_ {t = 0} ^ {\infty} \gamma^ {t} P ^ {\pi^ {\mathrm {E}}} \left(s _ {t} = s \mid s _ {0} \sim \eta\right). \tag {5}
$$
By plugging the closed-form solution of the optimal policy  $\pi_{\theta}$  into the objective function (2a), we can decompose the dynamics-model mismatch error from the likelihood function  $L(\theta)$  in (2).
Lemma 1. Under any reward parameter  $\theta$ , the objective  $L(\theta)$  in (2a) can be decomposed as below:
$$
L (\theta) = \widehat {L} (\theta) + \frac {\gamma}{1 - \gamma} \cdot \mathbb {E} _ {\left(s _ {t}, a _ {t}\right) \sim d ^ {\mathrm {E}} (\cdot , \cdot)} \left[ \sum_ {s ^ {\prime} \in \mathcal {S}} V _ {\theta} \left(s ^ {\prime}\right) \left(\widehat {P} \left(s ^ {\prime} \mid s, a\right) - P \left(s ^ {\prime} \mid s, a\right)\right) \right] \tag {6}
$$
where  $\widehat{L} (\theta)$  is a surrogate objective defined as:
$$
\widehat {L} (\theta) := \mathbb {E} _ {\tau^ {\mathrm {E}} \sim (\eta , \pi^ {\mathrm {E}}, P)} \left[ \sum_ {t = 0} ^ {\infty} \gamma^ {t} (r (s _ {t}, a _ {t}; \theta) + U (s _ {t}, a _ {t})) \right] - \mathbb {E} _ {s _ {0} \sim \eta (\cdot)} \left[ V _ {\theta} (s _ {0}) \right]. \tag {7}
$$
The detailed proof is included in Appendix D. In Lemma 1, we have shown that the likelihood function in (2) decomposes into two parts: a surrogate objective  $\widehat{L}(\theta)$  and an error term dependent on the dynamics model mismatch between  $\widehat{P}$  and  $P$ . As a remark, in the surrogate objective  $\widehat{L}(\cdot)$ , we separate the two dynamics models ( $\widehat{P}$  and  $P$ ) into two relatively independent components. Therefore, optimizing the surrogate objective is computationally tractable and we will propose an efficient algorithm to recover the reward parameter from it in the next section.
To further elaborate the connection between the likelihood objective  $L(\theta)$  and the surrogate objective  $\widehat{L}(\theta)$ , we first introduce the following assumption:
Assumption 1. For any reward parameter  $\theta$  and state-action pair  $(s,a)$ , following conditions hold:
$$
\left| r (s, a; \theta) \right| \leq C _ {r}, \quad \left| U (s, a) \right| \leq C _ {u} \tag {8}
$$
where  $C_r$  and  $C_u$  are positive constants.
As a remark, the assumption of the bounded reward is common in the literature of inverse reinforcement learning and imitation learning [41, 22, 42, 43]. Moreover, the assumption of the bounded penalty function holds true for common choices of the uncertainty heuristics [37], such as the max aleatoric penalty and the ensemble variance penalty. Then we can show the following results.
Lemma 2. Suppose Assumption 1 holds, then we obtain (where  $C_v$  is a positive constant):
$$
\left| L (\theta) - \widehat {L} (\theta) \right| \leq \frac {\gamma C _ {v}}{1 - \gamma} \cdot \mathbb {E} _ {(s, a) \sim d ^ {\mathrm {E}} (\cdot , \cdot)} \left[ \| P (\cdot | s, a) - \widehat {P} (\cdot | s, a) \| _ {1} \right]. \tag {9}
$$
Please see Appendix E for the detailed proof. The above lemma suggests that the gap between the likelihood function and its surrogate version is bounded by the model mismatch error  $\mathbb{E}_{(s,a)\sim d^{\mathrm{E}}(\cdot ,\cdot)}\left[\| P(\cdot |s,a) - \widehat{P} (\cdot |s,a)\| _1\right]$ . The fact that the objective approximation error  $|L(\theta) - \widehat{L} (\theta)|$  depends on the model mismatch error evaluated in the expert-visited state-action distribution  $d^{\mathrm{E}}(\cdot ,\cdot)$  is crucial to the construction of the world model  $\widehat{P}$ . Based on Lemma 2, we understand that full data coverage on the joint state-action space  $\mathcal{S}\times \mathcal{A}$  is not necessary. Instead, as long as the collected transition dataset  $\mathcal{D}:= \{(s,a,s')\}$  provides sufficient coverage on the expert-visited state-action space  $\Omega \coloneqq \{(s,a)|d^{\mathrm{E}}(s,a) > 0\}$ , then the surrogate objective  $\widehat{L} (\theta)$  will be an accurate approximation to the likelihood objective  $L(\theta)$ .
Intuitively, considering the goal is to recover a reward function to model expert behaviors which only lie in a quite limited region of the whole state-action space, data collection with full coverage can be redundant. This result is very useful in practice, since it serves to greatly reduce the efforts on data collection for constructing the world model. Moreover, it also matches recent theoretical understanding on offline reinforcement learning [44, 45, 46] and offline imitation learning [22], which show that it is enough to learn a good policy from offline data with partial coverage.
To analyze the sample complexity in the construction of the world model  $\widehat{P}$ , we quantitatively analyze the approximation error between  $L(\theta)$  and  $\widehat{L}(\theta)$ . In discrete MDPs, the cardinalities of both state space and action space are finite ( $|S| < \infty$  and  $|\mathcal{A}| < \infty$ ). Therefore, based on a collected transition dataset  $\mathcal{D} = \{(s, a, s')\}$ , we will use the empirical estimate to construct the world model  $\widehat{P}$ . Also recall that  $\Omega := \{(s, a) | d^{\mathrm{E}}(s, a) > 0\}$  denotes the set of expert-visited state-action pairs. Define  $\mathcal{S}^{\mathrm{E}} := \{s | \sum_{a \in \mathcal{A}} d^{\mathrm{E}}(s, a) > 0\} \subseteq \mathcal{S}$  as the set of expert-visited states. Using these definitions, we have the following result. The detailed proof is in Appendix H.
Proposition 1. For any  $\varepsilon \in (0,2)$ , suppose there are more than  $N$  data points on each state-action pair  $(s,a) \in \Omega$ , and the total number of the collected transition samples satisfies:
$$
\# \text {t r a n s i t i o n s a m p l e s} \geq | \Omega | \cdot N \geq \frac {c ^ {2} \cdot | \Omega | \cdot | \mathcal {S} ^ {\mathrm {E}} |}{\varepsilon^ {2}} \ln \left(\frac {| \Omega |}{\delta}\right)
$$
where  $c$  is a constant dependent on  $\delta$ . With probability greater than  $1 - \delta$ , the following results hold:
$$
\mathbb {E} _ {(s, a) \sim d ^ {\mathrm {E}} (\cdot , \cdot)} \left[ \| P (\cdot | s, a) - \widehat {P} (\cdot | s, a) \| _ {1} \right] \leq \varepsilon , \quad | L (\theta) - \widehat {L} (\theta) | \leq \frac {\gamma C _ {v}}{1 - \gamma} \varepsilon . \tag {10}
$$
The above result estimates the total number of samples needed to construct the estimated world model  $\widehat{P}$ , so that the surrogate objective  $\widehat{L}(\theta)$  can accurately approximate  $L(\theta)$ . A direct implication is that, the reward parameter obtained by solving  $\widehat{L}(\cdot)$  also guarantees strong performance. To be more specific, define the optimal reward parameters associated with  $L(\cdot)$  and  $\widehat{L}(\cdot)$  as below, respectively:
$$
\theta^ {*} \in \arg \max _ {\theta} L (\theta), \quad \hat {\theta} \in \arg \max _ {\theta} \widehat {L} (\theta).
$$
The next result characterizes the performance gap between the reward parameters  $\hat{\theta}$  and  $\theta^{*}$ . The detailed proof is included in Appendix I.
Theorem 1. For any  $\varepsilon \in (0, \frac{4\gamma C_v}{1 - \gamma})$ , suppose there are more than  $N$  data points on each state-action pair  $(s, a) \in \Omega$  and the number of transition dataset  $\mathcal{D}$  satisfies:
$$
\# \text {t r a n s i t i o n s a m p l e s} \geq | \Omega | \cdot N \geq \frac {4 \gamma^ {2} \cdot C _ {v} ^ {2} \cdot c ^ {2} \cdot | \Omega | \cdot | \mathcal {S} ^ {\mathrm {E}} |}{(1 - \gamma) ^ {2} \varepsilon^ {2}} \ln \left(\frac {| \Omega |}{\delta}\right)
$$
where  $c$  is a constant dependent on  $\delta$ . With probability greater than  $1 - \delta$ , the following result holds: