Bayesian Inverse Transition Learning for Offline Settings

Leo Benac 1 Sonali Parbhoo 2 Finale Doshi-Velez 1

Abstract

capturing important details of the environment as well as
counterfactual reasoning (Ha & Schmidhuber, 2018; Oh
et al., 2015; Buesing et al., 2018).

Offline Reinforcement learning is commonly used
for sequential decision-making in domains suchas healthcare and education, where the rewards
are known and the transition dynamics T mustbe estimated on the basis of batch data. A key
challenge for all tasks is how to learn a reliable estimate of the transition dynamics T that produce
near-optimal policies that are safe enough so thatthey never take actions that are far away from thebest action with respect to their value functions
and informative enough so that they communicate
the uncertainties they have. Using an expert’s
feedback, we propose a new constraint-based approach that captures our desiderata for reliably
learning a posterior distribution of the transition
dynamics T that is free from gradients. Our results demonstrate that by using our constraints,
we learn a high-performing policy, while considerably reducing the policy’s variance over different
datasets. We also explain how combining uncertainty estimation with these constraints can help
us infer a partial ranking of actions that produce
higher returns, and helps us infer safer and more
informative policies for planning.

In this paper, we are particularly interested in learning
the transition dynamics T in offline settings, that is fromalready-collected batch data. Such settings are common in
healthcare and education. However, learning the dynamics
T is a hard problem as the number of parameters to estimate
grows with the dimensions of state and action spaces. Thereis a tension between a model class for the dynamics that isexpressive enough to capture what is needed and one that issmall enough to learn (Zhu et al., 2023; Ayoub et al., 2020).
Moreover, in offline settings, we cannot interact with theenvironment to generate more data; we are limited to the
exploration (or lack thereof) to that the users who produced
the data (Herman et al., 2016).
Our goal in this work is to take advantage of the fact that,in many scenarios, the users who generated the trajectories
of batch data can be presumed to be near-optimal—or atthe very least, not highly sub-optimal. We consider a version of the learning using feedback from an expert throughdemonstrations that we call Inverse Transition Learning(ITL). Here, we presume that both the rewards R and anear-optimal policy πb are available to us; our goal is toestimate a posterior distribution of the dynamics T (either
for further optimization of the policy or for use in planningagainst other rewards R′ ). In later versions, we will relax
the requirement that the user policy πb is given.

1. Introduction
In standard planning scenarios, one is given the rewards
R and the transition dynamics of the environment T and
asked to compute the optimal set of actions or policy π.
However, in many real settings, the transition dynamics T is
not available. In such settings, model-based (Reinforcement
Learning) RL—that is, first learning the transition dynamics T and then planning—is useful because it can help us
be more data efficient, simulate data, encourage exploring,

Broadly, our approach consists of three parts: First, we
use feedback from the expert to create a set of constraints
that require that the value of actions taken by the expert
user is higher than the values of actions they never took.
Next, we create a gradient-free way to estimate the posterior
distribution over the dynamics T in a way that satisfies those
constraints. This process guarantees that policies associatedwith planning against any sample of the dynamics T will
be highly performant—potentially better than the original
expert—even with limited coverage of the state-action space.
We then demonstrate how planning against the maximum
likelihood estimate of the dynamics T M LE often results inpolicies that have high variance depending on the particularsample of trajectories in the offline batch. Then, we showthat having a posterior distribution over the dynamics T and

1

Department of Computer Science, Harvard University, Cambridge, MA, USA 2 Department of Engineering,
Imperial College London, London, UK. Correspondence
to:
Leo Benac <lbenac@g.harvard.edu>, Sonali Parbhoo <s.parbhoo@imperial.ac.uk>, Finale Doshi-Velez <finale@seas.harvard.edu>.
Interactive Learning with Implicit Human Feedback Workshop at
ICML 2023.

1

Inverse Transition Learning for Offline RL

such as T M LE fail under poor sample coverage as this is acommon way to estimate the transition dynamics T in suchoffline settings (Zhang et al., 2021). Other works (Reddy
et al., 2018; Golub et al., 2013) developed gradient-based
methods to learn the expert’s belief of the dynamics, wheresuch beliefs are suboptimal, whereas, in our setting, theexpert is uncertain what the best action is.

also a posterior over potential optimal policies results inhigher performing and more informative policies, as well as
lower variance across different batches of data.

2. Related Work
Learning from Demonstrations Learning behavior policies from demonstration trajectories produced by a nearoptimal expert through supervised learning is called Imitation Learning (IL). Different IL methods have been developed. (Ross et al., 2010) trains a policy at each time
step, (Ross et al., 2011) introduces an iterative algorithm
DAgger where at each iteration the agent needs to ask an
optimal expert a set of actions for some states that the agent
explored, then aggregate the dataset with this new information and retrain a policy on the aggregated dataset. (Kimet al., 2013) introduces Approximate Policy Iteration with
Demonstration (APID), where they combine expert data andRL signals and show that the policy that they learn is robust
to suboptimal demonstrations.

Constraint-based RL Several algorithms study IRL by imposing constraints on the rewards to induce, for instance,
safety. For instance, Fischer et al. (2021) presents Constrained Soft Reinforcement Learning (CSRL), an extensionof soft reinforcement learning to Constrained Markov Decision Processes (CMDPs) where the goal of the reward
maximization is regularized with an entropy objective to ensure safety. (Scobee & Sastry, 2019) introduce an approachbased on the Maximum Entropy IRL framework that allows
inferring which constraints can be added to the MDP to mostincrease the likelihood of observing these demonstrations.
Our work may be seen as a specific form of constraint-based
RL: we utilize observational data from an expert agent tocreate a set of constraints and perform inference on T within
a discrete MDP setting. In contrast to these approaches, weleverage (near) optimal data to estimate transitions for actions that lack data, ultimately resulting in a more accurate
T estimate than what traditional maximum likelihood estimates would produce.

Outside of learning the policy, learning the reward function
R may be of interest as it gives a succinct description of atask, and such reward function R can be used for planningand transfer learning. (Ng et al., 2000) introduce Inverse Reinforcement Learning (IRL) where the focus is on learning
the reward function R given T or access to a simulator to doroll-outs, as well as near-optimal demonstrations from anexpert and in some cases its policy. (Abbeel & Ng, 2004)
show that learning the reward function R through the principle of maximum margin can be used to learn the behavior
policy. (Ziebart et al., 2008) show how the principle ofMaximum Entropy can be used to learn the reward function
R and corresponding policies that are robust to suboptimal
demonstrations. (Ramachandran & Amir, 2007) solves the
non-identifiability of the reward function R by learning a
posterior distribution rather than a point estimate.

3. Preliminaries
Markov Decision Processes (MDPs) An MDP M can be
respresented as a tuple M = {S, A, T, γ, R} , where S is
the state space , A is the action space, T is the dynamics ofthe environment, γ is the discount factor and R is a bounded
reward function. In RL the goal is to find the best policy
π ∗ corresponding to a MDP M. In a discrete MDP there
exists at least one optimal deterministic policy. We measure
the quality of a policy π by looking at its corresponding
value functions. The value functions V π and Qπ are theexpected cumulative reward by taking actions with respect
to a policy π. V π (s) and Qπ (s, a) will both start from states but Qπ (s, a) will take action a before following π. They
can be computed through the Bellman Equations below,
where rt is the reward at time t:

ITL focuses on learning the transition dynamics T given
R, expert demonstrations, and its policy. Using Bayesianinference, we propose to learn a posterior distribution of thetransition dynamics T in an offline setting. Every sampleof this distribution is guaranteed to produce a near-optimal
policy downstream. The most closely related among theseworks is (Herman et al., 2016) where the authors present a
gradient-based IRL approach for simultaneously estimating
system dynamics and rewards via combined optimizationin tabular MDP’s. However, (Herman et al., 2016) does not
explicitly explore the relation between the estimated andtrue dynamics and learn the dynamics T by gradient-based
optimization. In contrast to these methods, we i) present
a constraint-based optimization that is free of gradients toguarantee learning a posterior distribution of T that yields
a performant and informative policy. We not only learn
the system dynamics but also characterize where methods

Bellman Equations The value of a policy π at state s isgiven by the expected sum of rewards,

V π (s)

=

E[

∞
X

γ t rt |s0 = s, π]

(1)

t=0

=

Ea [R(s, π(a|s))] + γEs′ [V π (s′ )]The value of a policy π at state s when performing action a2

Inverse Transition Learning for Offline RL

is given by,
Qπ (s, a)∞X= E[
γ t rt |s0 = s, a0 = a, π](ie. for the states where the expert user is certain of the best
action a∗ have the value of a∗ at least ϵ away from the otheractions and for the uncertain states have the actions selectedby the user expert ϵ close to each other, all with respect to
Tb).

(2)

t=0

= R(s, a) + γEs′ [V π (s′ )]π(3)Definition 4.3. (ϵ-optimality) A policy π is ϵ-optimal if it
only takes actions in the ϵ-ball for all states s with respect
to the true transition dynamics T , that is, only takes actionsin ϵ(s; T ).

π

Notation: Let V denote the vector of values V (s). We
use shorthand Ra , Qa , and Ta to represent the vectors R(·, a)and Q(·, a) and the matrix T (·|·, a). We also use use shorthand Rπ , Qπ , and Tπ to represent the vectors Ea∼π [Ra ],Ea∼π [Qa ] and the matrix Ea∼π [Ta ]. Q∗ represents the optimal Q functions in the True unknown environment T (ie.
assuming the agent always takes the best action possible a∗ ,where a∗ may change depending on the state s that we arein).

We assume that the expert policy πb , which is ϵ-optimal, isuniform in each of the actions in the ϵ ball ϵ(s; T ), where
T is the true unknown dynamics. This definition of suboptimality is similar to how clinicians would behave in real
healthcare settings, for states where there is a clear best treatment it makes sense for them to only pick that one treatment
but for other states where more than one treatment seems
appropriate it makes sense to assume complete uncertainty
among these treatments if no else prior information is given.

In the tabular setting the value functions can be calculated
directly through a closed-form solution of V π ,V π = Rπ + γTπ V π = (I − γTπ )−1 Rπ ,(4)

Definition 4.4. (deterministic/stochastic-policy state ) A
deterministic-policy state is a state for which the expert πbtakes only one action (ie. a state for which we know the best
action to take since we assume we are given the policy of theexpert πb ) and a stochastic-policy state is a state for which
the expert takes multiple actions (i.e. a state for which theexpert has uncertainty over a certain group of actions).

πand Q respectively,
Qπa = Ra + γTa V π = Ra + γTa (I − γTπ )−1 Rπ . (5)4. Methodology
In this section, we describe the setup of our methods, introduce two key definitions relating the sub-optimality of theexpert to T , and develop a set of constraints for both an optimal and sub-optimal expert. These constraints enable us toimpose the structure we would like to have when estimating
T . Based on these constraints, we demonstrate how one caninfer a reasonable estimate posterior on T .

Note that when ϵ = 0, the expert is fully optimal and hence
we would only get deterministic-policy states, whereas asϵ gets bigger we start having more stochastic-policy states.
A common method to estimate the dynamics T in offline
settings is to use MLE estimates. However since our batch
data D only contains expert demonstrations, there can bestates and action pairs (s, a) for which there is no data. A
common way around that is to assume we see every transition at least once before computing the MLE estimates,
which is also called smoothing (Zhang & Teng, 2021) . We
refer to this as T M LE . It is the equivalent of assuming auniform prior on T and setting T M LE to be the mean of thisposterior, assuming a Dirichlet-Multinomial Probabilistic
Model over T (ie. P (T |Data)), which we also refers to it
as the Unclipped Posterior:

Problem Setup We assume we know everything about the
MDP M except the true transition dynamics T (i.e. M \{T }). We assume we are also given an expert policy πband offline tabular observational batch data D, where D
is assumed to have been generated by rollouts of πb withrespect to the true unknown transition dynamics T . We
assume that the policy of the expert we are given, πb is
ϵ-optimal.
Definition 4.1. (ϵ-ball) For a state s, an action a belongs tothe ϵ-ball ϵ(s; T ′ ) if the action-value of that action Q(s, a)
is within ϵ of the optimal action-value maxa′ Q(s, a′ ) withrespect to the transition dynamics T ′ .

Probabilistic Model over T (Unclipped Posterior)
Prior :
Likelihood :Definition 4.2. (ϵ-ball property) We say that a point estimate Tb has the ϵ-ball property with respect to the truetransition dynamics T if they have the same ϵ-balls for every state s : ϵ(s; Tb) = ϵ(s; T ).

Posterior :Dir(1|s, a)M ultinomial(Ns,a |s, a)Dir(Ns,a + 1|s, a)

Ns,a and 1 are both vectors of dimensions |S|. Ns,a is thenumber of transitions in the batch data from state s andaction a. Using the assumptions we have made, we candevelop a set of constraints based on the expert policy πbHaving a transition dynamics estimate Tb that has the ϵ ballproperty is important as we want to make sure that it captures the same structure as the true transition dynamics T
3

Inverse Transition Learning for Offline RL

that highlights what properties we want our T to satisfy
by using the Closed Form Bellman Equations for tabular
MDPs.

Suboptimal Expert Policy Constraints:
where the expert is deterministic:

For (s, a∗ ) i.e.

1Ta∗ (I − γ Tbπb )−1 Rπb = ((I − γ Tbπb )−1 Rπb − Ra∗ ) (9)γ

4.1. Constraints on T given that the expert policy πb is
ϵ-optimal

For (s, a) such that a ∈/ ϵ(s) i.e. actions that the expert nevertakes:

We construct two different sets of constraints that T should
satisfy. The first set of constraints is applicable when theexpert is fully optimal, and the second when the expert issuboptimal. Tbπb in the constraints below could be sampledfrom the posterior of the data model or we could use Tbπb =TπMb LE . These are both reasonable choices since this is partof the state action space for which we have adequate data.

1((I − γ Tbπb )−1 Rπb − Ra − δs,a )γ(10)Finally, for (s, a) where a ∈ ϵ(s):Ta (I − γ Tbπb )−1 Rπb <Ta (I − γ Tbπb )−1 Rπb <Fully Optimal Expert Policy Constraints When πb =π∗ ,1((I − γ Tbπb )−1 Rπb − Ra + δs,a )
γ
(11)

and
1
Ta∗ (I − γ Tbπb )−1 Rπb = ((I − γ Tbπb )−1 Rπb − Ra∗ ) (6)γ1((I − γ Tbπb )−1 Rπb − Ra − δs,a ) < Ta (I − γ Tbπb )−1 Rπbγ(12)

and for a ̸= a∗
Ta (I − γ Tbπb )−1 Rπb <If πb is not fully optimal, meaning there are states for which
the expert takes more than 1 action (stochastic-policy states),
we can still develop similar constraints and guarantees. However in this case, if T satisfies the Suboptimal Expert Policy Constraints above, T will not necessarily have the ϵball property due to the nonlinearity of the actions in thestochastic-policy states. We can however still make sure that
the T ’s satisfying the constraints have the ϵ-ball property
by tuning the δs,a appropriately, where δs,a are constants.
Empirically, when δs,a = ϵ does not seem to output T ’s thathave the ϵ-ball property, it seems to be a good heuristic tomake the constraints tighter by increasing/decreasing δs,ain the inequality 10/ 11&12 respectively.

1((I − γ Tbπb )−1 Rπb − Ra − ϵ (7)γIn what follows, we show that if T satisfies the constraints
above, then T will capture the correct structure provided bythe ϵ-optimal expert.
Lemma 4.5. If T satisfies the above constraints then T will
recover the ϵ-ball property. That is, let a be any action suchthat a ̸= a∗ . Then,
Ta (I − γ Tbπb )−1 Rπb < γ1 ((I − γ Tbπb )−1 Rπb − Ra − ϵ)⇐⇒4.2. Estimating the Posterior over T given that theexpert is near-optimal

Ra + Ta γV πb + ϵ = Qπa b + ϵ < V πb = Qπa∗b(8)Using the batch data D, we can construct a posterior distribution over T , P (T |Data), that we will refer to as Unclipped.
T M LE is the mean of this posterior. However, the problem
with such estimates of T ’s is that they do not take advantageof the fact that the expert is acting near-optimally. The two
sets of constraints we developed impose T ′ s satisfying such
constraints to have the ϵ-ball property. We use those constraints to “Clip” the posterior distribution through rejection
sampling and model P (T |Data, expert is ϵ-optimal). The
idea is to sample from the posterior distribution constructedfrom the data (Unclipped) but to only accept the samples thatsatisfy our constraints. This leaves us with an empirical distribution of T ’s that recover the ϵ-ball property). We present
an algorithm to clip the Unclipped posterior in Algorithm 1.
We call this resulting posterior distribution Clipped.

Proof sketch: We used the fact that if T satisfy the constraints then Tπb = Tbπb (since πb = a∗ ). Hence we seethat having a Tb that satisfies these constraints will result inthe optimal action being at least ϵ better with respect to thevalue functions with respect to that Tb and hence recover the
ϵ-ball property.
Lemma 4.5 implies that if T satisfies the constraints then it
will always recover the best actions in deterministic-policy
states as well as never inducing actions outside of the ϵ-ballin stochastic-policy states. This is why we observe 100%
accuracy of our method in the Deterministic and a ∈ ϵ-balls
columns in Table 1.
4

Inverse Transition Learning for Offline RL

Algorithm 1 CLIP Posterior Rejection Sampling

big enough so that it is an environment interesting enough.
The True T is created such that transitioning from one state
to the next state is sometimes uniform and sometimes very
skewed towards one or a couple of states so that we havean environment that has a variety of behaviors. We use
γ = 0.95 and a reward function that is action dependentand such that the range of possible values depends on thenumber of states, |S| = 15.

Inputs: R, πb (ϵ expert policy), Posterior T (.|s, a) for
each (s,a), Number of samples N required
Output: Empirical CLIPPED Posterior of T (.|s, a) for
each (s,a)
While samples accepted < N
Sample Tbs,a ∼ P osterior(s′ |s, a) for each (s,a)
constrainta = Vπ − (Ra + Tba γVπ )bbfor each (s,a) where πb (a|s) = 0
While constrainta (s) < ϵ
Tbs,a ∼ P osterior(s′ |s, a)constrainta = Vπb − (Ra + Tba γVπb )for each (s,a) where πb (a|s) = 1
While constrainta (s) ̸= 0
Tbs,a ∼ P osterior(s′ |s, a)constrainta = Vπb − (Ra + Tba γVπb )for each (s,a) where s is a stochastic-policy stateand πb (a|s) > 0
While constrainta [s] < −δs,a
Tbs,a ∼ P osterior(s′ |s, a)constrainta = Vπb − (Ra + Tba γVπb )While constrainta [s] > δs,a
Tbs,a ∼ P osterior(s′ |s, a)constrainta = Vπb − (Ra + Tba γVπb )If Tb recover the ϵ ball property
Accept Tb
Else
Tune the different δs,a and start over.
Construct CLIPPED Posterior from all of the accepted
samples

Generating batch data D. For D, we simulate episodes
as follows: first, we pick a random state among the 15
possible states; next, we roll out the state using πb in thetrue environment T until we reach the terminal state or
until we reach 20 steps 1 This procedure is repeated for K
episodes. Based on D, we examine two different settings
namely, a low data setting (K = 15) and a high data setting(K = 300), as well as 3 different values of ϵ to get an idea of
how the task of estimating T varies across a different kindof dataset as well as different degree of optimality: ϵ = 0(corresponding to a fully optimal expert with 0 stochasticpolicy states); ϵ = 3 (3 stochastic-policy states); ϵ = 4 (6stochastic-policy states)
Baselines and Metrics. First, We compare the performance of our approach (Clipped Posterior) to the T M LE
and the original (Unclipped) Posterior distribution. We lookat the accuracy of deterministic-policy states, and the accuracy of stochastic-policy states, as well as how often. Next,
we look at how different methods perform at recovering thebest action over states, through planning on their respective
estimate of T by computing Q∗metric .
!
X
X
∗
∗
∗
∗
Qmetric (bπ) =Q (s, a ) −πb(a|s)Q (s, a)s∈S

5. Experimental Setupa∈A

This metric gives us a more quantitative idea of how good,the different methods are at estimating a T that captures thebest action when planning on such T , as well as quantifying
the importance of each mistake with respect to the Trueenvironment. The better an action is, the lower Q∗metric willbe, and the worse an action is the higher Q∗metric . We wantQ∗metric to be as close as possible to 0 (using π ∗ will achieve
such a result). In real life we would not be able to compute
such metrics, this is purely for analysis purposes. We also
report the result of the Q∗metric for the policy of the expert
πb . (Note that this policy is not obtained through planningon any T estimate, we just report it to get an idea of how
it performs compared to others. It is expected that this πb
performs very well since it is ϵ optimal by definition).

M LE

In our experiments, we look at how the T
performs
across different data and optimality settings, characterize
its mistakes as well as highlight certain issues that can arise
when learning under expert data. We show how combining
our constraints with uncertainty helps our method avoid andreduce such mistakes and pathologies and hence performs
better across different metrics and settings. We also showthat our constraints considerably reduce the variance ofthe policy we get over different batch data. Finally, wedemonstrate why combining uncertainty with constraints
can help us infer a ranking of actions in the ϵ-balls overthe stochastic-policy states which results in having more
informative, and higher-performing policies when doingplanning.

Training Details All of the results presented are averaged
over 1000 datasets. For the MLE results, we do planning

Setup For all the results we will show, we used a fixed
MDP of 15 states (+1 terminal state) and 6 actions so that it
is computationally feasible to run various experiments and1
Trajectories of longer than 20 steps are truncated to 20 steps
to reflect most realistic healthcare settings.

5

Inverse Transition Learning for Offline RL

on T M LE to get πb and compute the results whereas for theclipped and unclipped results, we do planning on 1000
different T ’s (sampled using the corresponding distribution)and get 1000 corresponding πb’s, then take the mean. For
each Batch Data D ∈ {D(i) }1000
i=1 we do the following:
MLE Method
Tb = T M LE−→−→

Value Iteration

posterior at inferring the best action, our method (Clipped)
has a lot less variance across different datasets thanks to theconstraints that enforce the T to recover the ϵ ball property
when planning.
Our approach enables quantifying the uncertainty andranking over actions for stochastic-policy states s thus
resulting in more informative and performant policies
for planning. The policy and Q values function we getwhen planning on our Clipped Posterior will quantify theuncertainty in a more informative way by creating a ranking
over the actions in the ϵ ball ϵ(s) for all the stochasticpolicy states s. This is particularly useful when we present
such policies to clinicians. The policy from the T M LE will
be deterministic even in states where we have uncertainty
which can be dangerous as T M LE is prone to making a lotof mistakes, while the policy of the expert has no idea of the
ranking of actions in the ϵ(s; T ) in stochastic-policy statess. The uncertainty modeled from the data coupled with ourconstraints used by the Clipped Posterior, is what helps us
infer that ranking while making sure that this uncertainty iscorrect with respect to the ϵ-ball definition of the expert. It
can also help us at inferring a better policy than the one weare given by the expert (See Table 2).

b(π M LE , QM LE ) = (bπ , Q)

Compute Results

Clipped or Unclipped Posterior Methodb (i) )}1000{Tb(i) }1000∼ Posterior−→{(b
π (i) , Q
i=1
i=1
Value Iteration−→

Empirical Meanb{(b
π , Q)}−→

Compute Results

6. Results
Our approach outperforms the baselines in terms ofaccuracy on both deterministic-policy and stochasticpolicy states. For deterministic-policy states, the moredata the more accurate we are across the settings and theless variability we observe. We also observe that in our
method (clipped), the constraints for deterministic-policy
states enforce 100% accuracy on such states while the MLE
and unclipped method, which does not have such constraints
cannot achieve such performance. These results are shownin Table 1. For stochastic-policy states, our introductionof constraints explicitly enforces that we will never pickactions that are ϵ away from the best action (bad mistakes).
In contrast, the MLE and unclipped methods do not havesuch a guarantee. What is also interesting to observe is thateven though our constraints do not say anything explicitly
about which action to take in the epsilon balls we see that our
method is still more accurate than the MLE and Unclipped
when inferring actions for stochastic-policy states. Thisshow how the constraints can help to infer the uncertainty
over stochastic-policy states. (See Table 1)
T M LE makes more bad mistakes, regardless of theamount of data used. Results from Table 1 show thathaving more data does not necessarily help in inferring better actions. Specifically, though T M LE uses more data, thisdoes not prevent it from making more bad mistakes (actions
not in the ϵ-balls), since certain states will still be hard toinfer even with more data. This is due to that the data only
covers a small part of the state action space and even inhigher data settings, a large part of the state action space
will remain unexplored. This is due to aleatoric uncertainty.

Figure 1. Result of the Q∗ metric across 1000 different
datasets/batch data, where the Q∗ metric measure quantitatively
how close the inferred policy πb is to the true unknown policy
π ∗ .Our method performs the best as well as exhibiting the least
amount of variance.

Our constraints produce policies that have considerably
less variance across all datasets in comparison to thebaselines. Results from Table 2 and Figure 1 show thatin addition to outperforming the MLE and the Unclipped
6Inverse Transition Learning for Offline RL

Table 2. Our method (Clipped) can recover a policy even better
than the policy of the expert πb which is already near optimal. It
recovers such a policy by having very little variance of various
batch data. This holds over different degrees of optimality.

Table 1. (Deterministic): accuracy in % of the deterministic-policy
states, (Stochastic): the accuracy of the stochastic-policy states(a ∈ ϵ-balls): the percentage of mistakes that are still ϵ-close to thebest action. (When ϵ = 0 there are no stochastic-policy states dueto the expert being fully optimal which is why there is no result
for the Stochastic and a ∈ ϵ-balls columns). On each metric, theClipped posterior achieve better results.

ϵ = 0: 0 stochastic-policy states
Low Data
MethodQ∗metric
Unclipped
142.17 ± 8.66
MLE
59.75 ± 51.99
Clipped
0 ±0Expert0 ±0High Data
MethodQ∗metric
Unclipped
99.0 ± 9.73
MLE
14.75 ± 6.98
Clipped
0 ±0Expert0 ±0

ϵ = 0: 0 stochastic-policy States
Low Data (Accuracy in %)MethodDeterministic Stochastic a ∈ ϵ-balls
Unclipped
48 ±9
N/A
N/A
MLE
67 ±1
N/A
N/A
Clipped100 ± 0
N/A
N/A
High Data (Accuracy in %)MethodDeterministic Stochastic a ∈ ϵ-balls
Unclipped
61 ±6
N/A
N/A
MLE
83 ±7
N/A
N/A
Clipped100 ± 0
N/A
N/A

MethodUnclipped
MLE
Clipped
MethodUnclipped
MLE
Clipped

MethodUnclipped
MLE
Clipped
MethodUnclipped
MLE
Clippedϵ = 3: 3 stochastic-policy States
Low Data
MethodQ∗metric
Unclipped
141.81 ± 8.92
MLE
62.71 ± 51.32
Clipped3.56 ±0.1
Expert3.60 ±0
High Data
MethodQ∗metric
Unclipped
100.83 ± 10.45
MLE
16.32 ± 9
Clipped3.56 ± 0.25
Expert3.60 ±0ϵ = 3: 3 Stochastic States
Low Data (Accuracy in %)Deterministic Stochastic a ∈ ϵ-balls
48 ±0937 ±2064 ±3765 ± 1044 ±2383 ±29100 ± 053 ±25100 ± 0High Data (Accuracy in %)Deterministic Stochastic a ∈ ϵ-balls
67 ±8
33 ±2530 ±3087 ±7
53 ±2870 ±38100 ± 055 ±29100 ± 0ϵ = 4: 6 stochastic-policy States
Low Data
MethodQ∗metric
Unclipped
143.52 ± 9.02
MLE
65.85 ± 50.83
Clipped9.45 ± 0.23
Expert9.4 ±0High Data
MethodQ∗metric
Unclipped
101.54 ± 10.81
MLE
20.11 ± 9.85
Clipped8.79 ± 0.6
Expert9.4 ±0ϵ = 4: 6 Stochastic States
Low Data (Accuracy in %)Deterministic Stochastic a ∈ ϵ-balls
53 ± 934 ±1581 ±1871 ±11
38 ±1687 ±17100 ± 046 ±18100 ± 0High Data (Accuracy in %)Deterministic Stochastic a ∈ ϵ-balls
81 ± 731 ±1738 ±2092 ± 652 ±2077 ±25100 ± 059 ±19100 ± 07Inverse Transition Learning for Offline RL

7. Discussion

Ayoub, A., Jia, Z., Szepesvari, C., Wang, M., and Yang, L.
Model-based reinforcement learning with value-targeted
regression. In International Conference on Machine
Learning, pp. 463–474. PMLR, 2020.

We focused on addressing the challenges associated withlearning the transition function, T , in a gradient-free manner,under offline, tabular, and inverse settings while capturing
our desired outcomes. First, using the expert’s feedback,
we introduced a novel constraint-based approach that explicitly incorporates our desiderata for learning T without
relying on gradients. By doing so, we mitigate the limitations and complexities that often arise when gradient-based
methods are employed. Second, we investigate the performance of T obtained through MLE, denoted as T M LE ,across various data and optimality settings. We comprehensively analyze the mistakes made by this T M LE whenlearning under an uneven coverage dataset due to the expert πb being ϵ-optimal. Third, we demonstrate that byintegrating our proposed constraints and incorporating uncertainty, our approach effectively avoids and reduces such
mistakes. Consequently, our method outperforms T M LE
across different evaluation metrics and in diverse settings.
Additionally, we showcase how our constraints significantly
decrease the variance of the policy generated from different
batch data, even in scenarios where these datasets exhibit
substantial variation. Furthermore, we highlights the benefits of combining uncertainty with constraints, particularlyin the context of planning. Our results indicate that thiscombination enables us to derive a ranking of the actionsin the ϵ-ball. Consequently, the policies derived from ourapproach are not only more informative by inferring a policythat is more discriminative in the stochastic-policy states but
also demonstrates superior performance in planning tasks.
In summary, our contributions include the development of aconstraint-based approach for learning T without gradients,a comprehensive analysis of the limitations of T M LE , andthe successful integration of constraints and uncertainty toenhance the performance and informativeness of policies.
Future work could extend this work assuming we are only
given the batch data and not the expert policy πb and extend
the method to partially observable domains or continuous
state and action spaces.

Buesing, L., Weber, T., Zwols, Y., Racaniere, S., Guez, A.,
Lespiau, J.-B., and Heess, N. Woulda, coulda, shoulda:
Counterfactually-guided policy search. arXiv preprint
arXiv:1811.06272, 2018.
Fischer, J., Eyberg, C., Werling, M., and Lauer, M.
Sampling-based inverse reinforcement learning algorithms with safety constraints. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems(IROS), pp. 791–798. IEEE, 2021.
Golub, M., Chase, S., and Yu, B. Learning an internal
dynamics model from control demonstration. In International Conference on Machine Learning, pp. 606–614.
PMLR, 2013.
Ha, D. and Schmidhuber, J. Recurrent world models facilitate policy evolution. Advances in neural information
processing systems, 31, 2018.
Herman, M., Gindele, T., Wagner, J., Schmitt, F., and Burgard, W. Inverse reinforcement learning with simultaneous estimation of rewards and dynamics. In Artificial
intelligence and statistics, pp. 102–110. PMLR, 2016.
Kim, B., Farahmand, A.-m., Pineau, J., and Precup, D.
Learning from limited demonstrations. Advances in Neural Information Processing Systems, 26, 2013.
Ng, A. Y., Russell, S., et al. Algorithms for inverse reinforcement learning. In Icml, volume 1, pp. 2, 2000.
Oh, J., Guo, X., Lee, H., Lewis, R. L., and Singh, S. Actionconditional video prediction using deep networks in atarigames. Advances in neural information processing systems, 28, 2015.
Ramachandran, D. and Amir, E. Bayesian inverse reinforcement learning. In IJCAI, volume 7, pp. 2586–2591,
2007.

Acknowledgements This material is based upon work
supported by the National Science Foundation under Grant
No. IIS-2007076. Any opinions, findings, and conclusions
or recommendations expressed in this material are those of
the author(s) and do not necessarily reflect the views of the
National Science Foundation.

Reddy, S., Dragan, A., and Levine, S. Where do you think
you’re going?: Inferring beliefs about dynamics frombehavior. Advances in Neural Information Processing
Systems, 31, 2018.
Ross, S., Gordon, G. J., and Bagnell, J. A. No-regret reductions for imitation learning and structured prediction.
Technical report, 2010.

References
Abbeel, P. and Ng, A. Y. Apprenticeship learning via inverse
reinforcement learning. In Proceedings of the twenty-firstinternational conference on Machine learning, pp. 1,
2004.

Ross, S., Gordon, G., and Bagnell, D. A reduction of imitation learning and structured prediction to no-regret online
learning. In Proceedings of the fourteenth international
8

Inverse Transition Learning for Offline RL

conference on artificial intelligence and statistics, pp.
627–635. JMLR Workshop and Conference Proceedings,
2011.
Scobee, D. R. and Sastry, S. S. Maximum likelihood constraint inference for inverse reinforcement learning. arXiv
preprint arXiv:1909.05477, 2019.
Zhang, K., Wang, Y., Du, J., Chu, B., Celi, L. A., Kindle, R.,
and Doshi-Velez, F. Identifying decision points for safe
and interpretable reinforcement learning in hypotension
treatment. arXiv preprint arXiv:2101.03309, 2021.
Zhang, Y. and Teng, Z. Natural language processing: amachine learning perspective. Cambridge University
Press, 2021.
Zhu, Y., Tang, Y.-H., and Kim, C. Learning stochastic dynamics with statistics-informed neural network. Journalof Computational Physics, 474:111819, 2023.
Ziebart, B. D., Maas, A. L., Bagnell, J. A., Dey, A. K., et al.
Maximum entropy inverse reinforcement learning. In
Aaai, volume 8, pp. 1433–1438. Chicago, IL, USA, 2008.

9Inverse Transition Learning for Offline RL

A. You can have an appendix here.
You can have as much text here as you want. The main body must be at most 8 pages long. For the final version, one more
page can be added. If you want, you can use an appendix like this one, even using the one-column format.

10