SlowGuess/NeurIPS_2023_OCR huggingface dataset:
If the behavior policy  $\pi$  is  $q_{\theta}(\boldsymbol{a} \mid s)$ , we can view the second term as a Monte Carlo estimation of the relative KL divergence between  $q$  and  $p$ . This estimator can suffer greatly from bias, as the KL should always be non-negative. To counteract this, we construct an unbiased, positively-constrained estimator following Schulman [97], using  $\mathbb{E}_{\boldsymbol{x} \sim q(\cdot)}[R(\boldsymbol{x})^{-1}] = \int p(\boldsymbol{x}) \, \mathrm{d}\boldsymbol{x} = 1$  and  $\log(x) \leq x - 1$ ,
$$
\mathbb {D} _ {\mathrm {K L}} [ q (\boldsymbol {x}) | | p (\boldsymbol {x}) ] = \mathbb {E} _ {\boldsymbol {x} \sim q (\cdot)} [ \log R (\boldsymbol {x}) ] = \mathbb {E} _ {\boldsymbol {x} \sim q (\cdot)} [ R (\boldsymbol {x}) ^ {- 1} - 1 + \log R (\boldsymbol {x}) ],
$$
This estimator results in replacing the second term of Equation 11 with
$$
\mathbb {E} _ {\boldsymbol {s}, \boldsymbol {a} \sim \rho_ {\pi}} \left[ r _ {\boldsymbol {\theta}} (\boldsymbol {s}, \boldsymbol {a}) \right]\rightarrow \mathbb {E} _ {\boldsymbol {s}, \boldsymbol {a} \sim \rho_ {\pi}} \left[ r _ {\boldsymbol {\theta}} (\boldsymbol {s}, \boldsymbol {a}) - \alpha + \alpha \exp (- r _ {\boldsymbol {\theta}} (\boldsymbol {s}, \boldsymbol {a}) / \alpha) \right].
$$
This objective is maximized concurrently with policy evaluation, using samples from the replay buffer. This reward refinement is similar to behavioral cloning via function-space variational inference (e.g., [98, 99]), enforcing the prior using the prior predictive distribution rather than the prior weight distribution.
In practice, the samples are taken from a replay buffer of online and offline data, so the action samples are taken from a different distribution to  $q_{\theta}$ . However, we still found the adjusted objective effective for reward regularization. Without it, the saddle-point refinement was less stable. Appendix G illustrates this refinement in a simple setting. Figure 45 ablates this component, where its effect is most beneficial when there are fewer demonstrations.
F Bounding the coherent reward
The coherent reward  $\alpha (\log q(\pmb {a}\mid s) - \log p(\pmb {a}\mid s))$  can be upper bounded if  $p(\pmb {a}\mid s) > 0$  whenever  $q(\pmb {a}\mid s) > 0$ . The bound is policy dependent. We use the bound in the continuous setting, where we use a tanh-transformed Gaussian policy. We add a small bias term  $\sigma_{\mathrm{min}}^2$  to the predictive variance in order to define the Gaussian likelihood upper bound. Inconveniently, the tanh change of variables term in the log-likelihood  $-\sum_{i = 1}^{d_a}\log (1 - \tanh^2 (u))$  [17] has no upper bound in theory, as the 'latent' action  $u\in [-\infty ,\infty ]$ , but in the Acme implementation this log-likelihood is bounded for numerical stability due to the inverse tanh operation. For action dimension  $d_{a}$  and uniform prior across action range  $[-1,1]^{d_a}$  and  $\alpha = d_a^{-1}$ , the upper bound is  $r(s,\pmb {a})\leq \frac{1}{d_a} (-0.5d_a\log 2\pi \sigma_{\mathrm{min}}^2 +c + d_a\log 2) = -0.5\log \pi \sigma_{\mathrm{min}}^2 /2 + \tilde{c},$  where  $c,\tilde{c}$  depends on the tanh clipping term.
G Visualizing reward refinement
To provide intuition on Theorem 2 and connect it to reward refinement (Appendix E), we build on the regression example from Figure 3 by viewing it as a continuous contextual bandit problem, i.e., a single-state MDP with no dynamics. Outside of the demonstration state distribution, we desire that the reward is uniformly zero, as we have no data to inform action preference. Figure 10 shows the result of applying CSIL and reward refinement to this MDP, where the state distribution is uniform. The 'desired' BC fit exhibits a coherent reward with the desired qualities, and the HETSTAT policy approximates this desired policy. The heteroscedastic MLP exhibits undesirable out-of-distribution (OOD) behavior, where arbitrary regions have strongly positive or negative rewards. Reward refinement improves the coherent reward of the heteroscedastic MLP to be more uniform OOD, which in turn makes the regression fit look more like a stationary policy. This result reinforces Theorem 2, which shows that KL-regularized BC is connected to a lower bound on an entropy-regularized game-theoretic IRL objective that uses the coherent reward.
Figure 10: We extend the regression problem from Figure 3 to highlight the effect of reward refinement detailed in Section E from the BC and IRL perspective. Refining the coherent reward with a minimax objective results in a BC policy that appears more stationary, as seen in the heteroscedatic MLP. The stationary heteroscedastic policy is also refined, but to a lesser degree, as it's approximately stationary by construction. The colourmap is shared between contour plots, and rewards are clipped at -10 to improve the visual colour range. Regression uncertainty is one standard deviation.
H Critic regularization
For the function approximation setting, CSIL requires an auxiliary loss term during learning to incorporate coherency. The full critic optimization problem is  $\min_{\phi}\mathcal{I}_{\mathcal{Q}}(\phi)$  for objective
$$
\mathcal {J} _ {\mathcal {Q}} (\phi) = \mathbb {E} _ {\boldsymbol {s}, \boldsymbol {a} \sim \mathcal {B}} \left[ \left(\mathcal {Q} _ {\phi} (\boldsymbol {s}, \boldsymbol {a}) - \mathcal {Q} ^ {*} (\boldsymbol {s}, \boldsymbol {a})\right) ^ {2} \right] + \mathbb {E} _ {\boldsymbol {s}, \boldsymbol {a} \sim \mathcal {D}} \left[ \left| \nabla_ {\boldsymbol {a}} \mathcal {Q} _ {\phi} (\boldsymbol {s}, \boldsymbol {a}) \right| ^ {2} \right], \tag {12}
$$
where  $\mathcal{Q}^*$  denotes the target  $Q$  values and  $\mathcal{B}$  denotes the replay buffer, that combines demonstrations  $\mathcal{D}$  and additional samples. The motivation is to shape the critic such that the demonstration actions are first-order optimal by minimizing the squared Frobenius norm of the action Jacobian.
IQLearn and PPIL adopt similar, but less explicit, regularization. Equation 10 of Garg et al. [42] is
$$
\max _ {\boldsymbol {\phi}} \mathbb {E} _ {\boldsymbol {s}, \boldsymbol {a} \sim \mathcal {B}} \left[ f \left(\mathcal {Q} _ {\boldsymbol {\phi}} (\boldsymbol {s}, \boldsymbol {a}) - \gamma \mathbb {E} _ {\boldsymbol {s ^ {\prime}} \sim \mathcal {P} (\cdot | \boldsymbol {s}, \boldsymbol {a})} [ \mathcal {V} _ {\boldsymbol {\phi}} ^ {\pi} (\boldsymbol {s} ^ {\prime}) ]\right) \right] - (1 - \gamma) \mathbb {E} _ {\boldsymbol {s} \sim \mu_ {0}} [ \mathcal {V} _ {\boldsymbol {\phi}} ^ {\pi} (\boldsymbol {s}) ],
$$
where  $\mathcal{V}_{\phi}^{\pi}(s) = \mathcal{Q}(s, a')$ ,  $a' \sim \pi(\cdot \mid s)$  and  $f$  is the concave function defining the implicit reward regularization. In the implementation [100], the 'v0' objective variant replaces the initial distribution  $\mu_0$  with the demonstration state distribution. This trick is deployed to improve regularization without affecting optimality (Section 5.3, Kostrikov et al. [41]). If the policy matches the expert actions with its mode, this term shapes a local maximum by minimizing the critic at action samples around the
Figure 11: Illustrating the two forms of critic regularization on a toy 1D problem for critic  $\mathcal{Q}(a)$  with regression target  $\mathsf{x}$ . Jacobian regularization minimizes the squared gradient and corresponds to CSIL. Sample regularization minimizes sampled nearby actions and corresponds to IQLearn and PPIL. Both approaches shape the critic to have zero gradient around the target action, denoted by the dotted lines. A standard two-layer MLP with 256 units and ELU actions is used for approximation.
Algorithm 2: Off-policy coherent soft imitation learning with function approximation
Data: Expert demonstrations  $\mathcal{D}$  , initial temperature  $\alpha$  , refinement temperature,  $\beta$    
parametric policy class  $q_{\theta}(\pmb {a}\mid \pmb {s})$  , prior policy  $p(\pmb {a}\mid \pmb {s})$  , regression regularizer  $\Psi$  , total steps  $T$    
Result:  $q_{\theta_N}(\pmb {a}\mid \pmb {s})$  , matching or improving the initial policy  $q_{\theta_1}(\pmb {a}\mid \pmb {s})$  // Pretrain coherent reward and critic   
Train initial policy from demonstrations,  $\pmb{\theta}_{1} = \arg \max_{\pmb{\theta}}\mathbb{E}_{\pmb{s},\pmb{a}\sim \mathcal{D}}[\log q_{\theta}(\pmb {a}\mid \pmb {s}) - \Psi (\pmb {\theta})]$  .   
Define fixed shaped coherent reward,  $\tilde{r}_{\theta_1}(s,a) = \alpha (\log q_{\theta_1}(a\mid s) - \log p(a\mid s))$  .   
Pretrain critic with SARSA on  $\mathcal{D}$  using  $\tilde{r}_{\theta_1}$    
for  $t = 1\rightarrow T$  do // finetune policy with reinforcement learning Interact with environment  $s_{t + 1} = \mathrm{Env}(s_t,a_t)$ $a_{t}\sim q_{\theta_{t}}(\cdot \mid s_{t})$  , store in replay buffer B; Optimize reward and critic using  $\mathcal{I}_{\mathcal{Q}}(\phi)$  and  $\mathcal{J}_r(\pmb {\theta})$  (Equations 12, 11) on minibatch from B; Update policy optimizing  $\mathcal{I}_{\pi}(\pmb {\theta})$  (Equation 13) on minibatch from B;   
end
expert. However, if the policy does not accurately match the expert actions, the effect of this term is harder to reason about. In Figure 11, we illustrate these two forms from critic regularization on a toy example, demonstrating that both can produce local maximums in the critic.
I Practical algorithm with function approximation
For deep imitation learning, CSIL adopts SAC for the SPI step in Algorithm 1, summarized in Algorithm 2. The critic objective for both pre-training and policy evaluation is the squared Bellman error. The policy objective  $\max_{\theta} \mathcal{I}_{\pi}(\theta)$  samples both the demonstrations and replay buffer,
$$
\mathcal {J} _ {\pi} (\boldsymbol {\theta}) = E _ {\boldsymbol {a} \sim q _ {\boldsymbol {\theta}} (\cdot | \boldsymbol {s}), \boldsymbol {s} \sim \mathcal {B} \cup \mathcal {D}} \left[ \mathcal {Q} (\boldsymbol {s}, \boldsymbol {a}) - \beta \left(\log q _ {\boldsymbol {\theta}} (\cdot | \boldsymbol {s}) - \log q _ {\boldsymbol {\theta} _ {1}} (\cdot | \boldsymbol {s})\right) \right]. \tag {13}
$$
The policy and demonstration data are combined equally. We did not experiment with tuning this ratio. Compared to SAC, we found we did not have to train multiple critics, but we did need target networks for the critics to stabilize learning. The critic and reward objectives were also evaluated on the policy and expert data each update. Moreover, like IQLearn and PPIL, we kept  $\beta$  constant and did not use an adaptive strategy.
J A divergence minimization perspective
Many imitation learning algorithms, such as GAIL and IQLearn, are derived from minimizing a divergence between the agent's state-action distribution and the expert's. While CSIL is derived through policy inversion, it also has a divergence minimization perspective, albeit with two minimization steps rather than one.
The first step involves recognising that maximum likelihood BC fit minimizes the forward KL divergence between the demonstration distribution  $p_{\mathcal{D}}(\boldsymbol{s}, \boldsymbol{a}) = p_{\mathcal{D}}(\boldsymbol{a} \mid \boldsymbol{s}) p_{\mathcal{D}}(\boldsymbol{s})$  and the policy,
$$
\boldsymbol {\theta} _ {1} = \arg \min  _ {\boldsymbol {\theta}} \mathbb {E} _ {\boldsymbol {s} \sim p _ {\mathcal {D}} (\cdot)} [ \mathbb {D} _ {\mathrm {K L}} [ p _ {\mathcal {D}} (\boldsymbol {a} \mid \boldsymbol {s}) | | q _ {\boldsymbol {\theta}} (\boldsymbol {a} \mid \boldsymbol {s}) ] ] = \arg \max  _ {\boldsymbol {\theta}} \mathbb {E} _ {\boldsymbol {s}, \boldsymbol {a} \sim \mathcal {D}} [ \log q _ {\boldsymbol {\theta}} (\boldsymbol {a} \mid \boldsymbol {s}) ].
$$
Using the forward KL means that the unknown log-likelihood of the data distribution is not required, and the policy can be trained using supervised learning and without environment interaction.
The second step involves rearranging the KL-regularized RL objective into two KL divergences,
$$
\begin{array}{l} \pmb {\theta} _ {*} = \arg \max _ {\pmb {\theta}} \mathbb {E} _ {\pmb {a} \sim q _ {\pmb {\theta}} (\cdot | \pmb {s}), \pmb {s} \sim \mu_ {\pi_ {\pmb {\theta}}} (\cdot)} \left[ \alpha \log \frac {q _ {\theta_ {1}} (\pmb {a} | \pmb {s})}{p (\pmb {a} | \pmb {s})} - \beta \log \frac {q _ {\theta} (\pmb {a} | \pmb {s})}{p (\pmb {a} | \pmb {s})} \right], \\ = \arg \min  _ {q _ {\theta}} \mathbb {E} _ {\boldsymbol {s} \sim \mu_ {q _ {\theta}} (\cdot)} [ \alpha \mathbb {D} _ {\mathrm {K L}} [ q _ {\theta} (\boldsymbol {a} \mid \boldsymbol {s}) | | q _ {\theta_ {1}} (\boldsymbol {a} \mid \boldsymbol {s}) ] - (1 - \beta / \alpha) \mathbb {D} _ {\mathrm {K L}} [ q _ {\theta} (\boldsymbol {a} \mid \boldsymbol {s}) | | p (\boldsymbol {a} \mid \boldsymbol {s}) ] ]. \\ \end{array}
$$
Note that since  $\beta \leq \alpha$ , then  $(1 - \beta / \alpha) \geq 0$ . The left-hand term minimizes the reverse KL between the agent and the expert, using the BC policy as a proxy. This is beneficial because the reverse conditional KL requires evaluating policy  $q_{\theta}$  on the environment. Moreover, the reverse KL requires log-likelihoods of the target distribution, which is why the BC policy is used as a proxy of the demonstration distribution. The right-hand term incorporates the coherent reward and encourages the policy to stay in states where the policy differs from the prior. As  $q_{\theta}$  should be close to the BC policy due to the left-hand term, the coherent reward shaping from Definition 1 should hold.
In this formulation of the objective, it is also clear to see that if  $\beta = \alpha$ , no RL finetuning will occur, and  $q_{\theta}$  will simply mimic the BC policy  $q_{\theta_1}$  due to the left-hand term.
K Theoretical results
This section provides derivations and proofs for the theoretical results in the main text.
K.1 Relative entropy regularization for reinforcement learning
This section re-derives KL-regularized RL and ME-IRL in a shared notation for the reader's reference.
Reinforcement learning. We consider the discounted, infinite-horizon setting with a hard KL constraint on the policy update with bound  $\epsilon$ ,
$$
\max  _ {q} \mathbb {E} _ {\boldsymbol {s} _ {t + 1} \sim \mathcal {P} (\cdot | \boldsymbol {s} _ {t}, \boldsymbol {a} _ {t}), \boldsymbol {a} _ {t} \sim q (\cdot | \boldsymbol {s} _ {t}), \boldsymbol {s} _ {0} \sim \mu_ {0} (\cdot)} [ \sum_ {t} \gamma^ {t} r (\boldsymbol {s} _ {t}, \boldsymbol {a} _ {t}) ] \quad \text {s . t .} \quad \mathbb {E} _ {\boldsymbol {s} \sim \nu_ {q} (\cdot)} [ \mathbb {D} _ {\mathrm {K L}} [ q (\boldsymbol {a} | \boldsymbol {s}) | | p (\boldsymbol {a} | \boldsymbol {s}) ] ] \leq \epsilon .
$$
This constrained objective is transcribed into the following Lagrangian objective following Van Hoof et al. [15], using the discounted stationary joint distribution  $d(\pmb{s},\pmb{a}) = q(\pmb{a} \mid \pmb{s})\nu(\pmb{s}) = (1 - \gamma)\rho(\pmb{s},\pmb{a}),$
$$
\begin{array}{l} \mathcal {L} (d, \lambda_ {1}, \lambda_ {2}, \lambda_ {3}) = (1 - \gamma) ^ {- 1} \int_ {\mathcal {S} \times \mathcal {A}} d (\boldsymbol {s}, \boldsymbol {a}) r (\boldsymbol {s}, \boldsymbol {a}) \mathrm {d} \boldsymbol {s} \mathrm {d} \boldsymbol {a} + \lambda_ {1} \left(1 - \int_ {\mathcal {S} \times \mathcal {A}} d (\boldsymbol {s}, \boldsymbol {a}) \mathrm {d} \boldsymbol {s} \mathrm {d} \boldsymbol {a}\right) \\ + \int_ {\mathcal {S}} \lambda_ {2} (\boldsymbol {s} ^ {\prime}) \left(\int_ {\mathcal {S} \times \mathcal {A}} \gamma \mathcal {P} (\boldsymbol {s} ^ {\prime} \mid \boldsymbol {s}, \boldsymbol {a}) d (\boldsymbol {s}, \boldsymbol {a}) d \boldsymbol {s} d \boldsymbol {a} + (1 - \gamma) \mu_ {0} (\boldsymbol {s} ^ {\prime}) - \int_ {\mathcal {A}} d (\boldsymbol {s} ^ {\prime}, \boldsymbol {a} ^ {\prime}) d \boldsymbol {a} ^ {\prime}\right) d \boldsymbol {s} ^ {\prime} \\ + \lambda_ {3} \left(\epsilon - \int_ {\mathcal {S}} \nu (\boldsymbol {s}) \int_ {\mathcal {A}} q (\boldsymbol {a} \mid \boldsymbol {s}) \log \frac {q (\boldsymbol {a} \mid \boldsymbol {s})}{p (\boldsymbol {a} \mid \boldsymbol {s})} \mathrm {d} \boldsymbol {a} \mathrm {d} \boldsymbol {s}\right), \\ \end{array}
$$
where  $\lambda_{2}$  is a function for an infinite-dimensional constraint. Solving for  $\partial \mathcal{L}(q,\lambda_1,\lambda_2,\lambda_3) / \partial d = 0$
$$
q (\pmb {a} \mid \pmb {s}) \propto \exp (\alpha^ {- 1} (\underbrace {r (\pmb {s} , \pmb {a}) + \gamma \mathbb {E} _ {\pmb {s} ^ {\prime} \sim \mathcal {P} (\cdot | \pmb {s} , \pmb {a})} [ \mathcal {V} (\pmb {s} ^ {\prime}) ]} _ {\mathcal {Q} (\pmb {s}, \pmb {a})} - \mathcal {V} (\pmb {s}))) p (\pmb {a} \mid \pmb {s}),
$$
with  $\alpha = (1 - \gamma)\lambda_3$  and  $\lambda_{2}(s) = (1 - \gamma)\mathcal{V}(s)$ , where  $\mathcal{V}$  can be interpreted as the soft value function,
$$
\mathcal {V} (\boldsymbol {s}) = \alpha \log \int_ {\mathcal {A}} \exp \left(\frac {1}{\alpha} \mathcal {Q} (\boldsymbol {s}, \boldsymbol {a})\right) p (\boldsymbol {a} \mid \boldsymbol {s}) d \boldsymbol {a} \quad \text {a s} \quad \int_ {\mathcal {A}} q (\boldsymbol {a} \mid \boldsymbol {s}) d \boldsymbol {a} = 1.
$$
To obtain the more convenient lower bound of the soft Bellman equation, we combine importance sampling and Jensen's inequality to the soft Bellman equation
$$
\begin{array}{l} \mathcal {V} (\boldsymbol {s}) = \alpha \log \int_ {\mathcal {A}} \exp \left(\frac {1}{\alpha} \mathcal {Q} (\boldsymbol {s}, \boldsymbol {a})\right) \frac {p (\boldsymbol {a} \mid \boldsymbol {s})}{q (\boldsymbol {a} \mid \boldsymbol {s})} q (\boldsymbol {a} \mid \boldsymbol {s}) d \boldsymbol {a}, \\ \geq \mathbb {E} _ {\pmb {a} \sim q (\cdot | \pmb {s})} \left[ \mathcal {Q} (\pmb {s}, \pmb {a}) - \alpha \log \frac {q (\pmb {a} | \pmb {s})}{p (\pmb {a} | \pmb {s})} \right]. \\ \end{array}
$$
This derivation is for the 'hard' KL constraint. In many implementations, including ours, the constraint is 'softened' where  $\lambda_3$  and, therefore,  $\alpha$  is a fixed hyperparameter.
Inverse reinforcement learning. As mentioned in Section 2 and Definition 2, ME-IRL is closely related to KL-regularize RL, but with the objective and a constraint reversed,
$$
\min _ {q} \mathbb {E} _ {\boldsymbol {s} \sim \nu_ {q} (\cdot)} [ \mathbb {D} _ {\mathrm {K L}} [ q (\boldsymbol {a} \mid \boldsymbol {s}) | | p (\boldsymbol {a} \mid \boldsymbol {s}) ] ] \quad \mathrm {s . t .} \quad \mathbb {E} _ {\boldsymbol {s}, \boldsymbol {a} \sim \rho_ {q}} [ \phi (\boldsymbol {s}, \boldsymbol {a}) ] = \mathbb {E} _ {\boldsymbol {s}, \boldsymbol {a} \sim \mathcal {D}} [ \phi (\boldsymbol {s}, \boldsymbol {a}) ].
$$
The constrained optimization is, therefore, transcribed into a similar Lagrangian,
$$
\begin{array}{l} \mathcal {L} (d, \lambda_ {1}, \lambda_ {2}, \boldsymbol {\lambda_ {3}}) = \int_ {\mathcal {S}} \nu (\boldsymbol {s}) \int q (\boldsymbol {a} \mid \boldsymbol {s}) \log \frac {q (\boldsymbol {a} \mid \boldsymbol {s})}{p (\boldsymbol {a} \mid \boldsymbol {s})} d \boldsymbol {a} d \boldsymbol {s} + \lambda_ {1} \left(1 - \int_ {\mathcal {S} \times \mathcal {A}} d (\boldsymbol {s}, \boldsymbol {a}) d \boldsymbol {s} d \boldsymbol {a}\right) \\ + \int_ {\mathcal {S}} \lambda_ {2} (\boldsymbol {s} ^ {\prime}) \left(\int_ {\mathcal {S} \times \mathcal {A}} \gamma \mathcal {P} (\boldsymbol {s} ^ {\prime} \mid \boldsymbol {s}, \boldsymbol {a}) d (\boldsymbol {s}, \boldsymbol {a}) d \boldsymbol {s} d \boldsymbol {a} + (1 - \gamma) \mu_ {0} (\boldsymbol {s} ^ {\prime}) - \int_ {\mathcal {A}} d (\boldsymbol {s} ^ {\prime}, \boldsymbol {a} ^ {\prime}) d \boldsymbol {a} ^ {\prime}\right) d \boldsymbol {s} ^ {\prime} \\ + \boldsymbol {\lambda} _ {3} ^ {\top} \left(\left(1 - \gamma\right) ^ {- 1} \int_ {\mathcal {S} \times \mathcal {A}} d (\boldsymbol {s}, \boldsymbol {a}) \phi (\boldsymbol {s}, \boldsymbol {a}) \mathrm {d} \boldsymbol {s} \mathrm {d} \boldsymbol {a} - \mathbb {E} _ {\boldsymbol {s}, \boldsymbol {a} \sim \mathcal {D}} [ \phi (\boldsymbol {s} _ {t}, \boldsymbol {a} _ {t}) ]\right). \\ \end{array}
$$
The consequence of this change is that the likelihood temperature is now implicit in the reward model  $r(\boldsymbol{s},\boldsymbol{a}) = (1 - \gamma)^{-1}\lambda_3^\top \phi (\boldsymbol{s},\boldsymbol{a})$  due to the Lagrange multiplier weights, this reward model is used in the soft Bellman equation instead of the true reward. The reward model is jointly optimized to minimize the apprenticeship error by matching the feature expectation.
K.2 Proof for Theorem 1
Proof. Equation 6 is derived by rearranging the posterior policy update in Equation 4. Equation 7 in derived by substituting the critic expression from Equation 6 into the KL-regularized Bellman equation (Equation 3),
$$
\begin{array}{l} \underbrace {\alpha \log \frac {q _ {\alpha} (\boldsymbol {a} \mid \boldsymbol {s})}{p (\boldsymbol {a} \mid \boldsymbol {s})} + \mathcal {V} _ {\alpha} (\boldsymbol {s})} _ {\mathcal {Q} (\boldsymbol {s}, \boldsymbol {a})} = \\ r(\boldsymbol {s},\boldsymbol {a}) + \gamma   \mathbb{E}_{\boldsymbol{s}^{\prime}\sim \mathcal{P}(\cdot |\boldsymbol {s},\boldsymbol {a}),  \boldsymbol{a}^{\prime}\sim q_{\alpha}(\cdot |\boldsymbol{s}^{\prime})}\left[\underbrace{\alpha\log\frac{q_{\alpha}(\boldsymbol{a}^{\prime}|\boldsymbol{s}^{\prime})}{p(\boldsymbol{a}^{\prime}|\boldsymbol{s}^{\prime})} + \mathcal{V}_{\alpha}(\boldsymbol{s}^{\prime})}_{\mathcal{Q}(\boldsymbol{s}^{\prime},\boldsymbol{a}^{\prime})}- \overline{\alpha\log\frac{q_{\alpha}(\boldsymbol{a}^{\prime}|\boldsymbol{s}^{\prime})}{p(\boldsymbol{a}^{\prime}|\boldsymbol{s}^{\prime})}}\right]. \\ \end{array}
$$
K.3 Proof for Theorem 2
Theorem 2 relies on the data processing inequality (Lemma 3) to connect the function-space KL seen in CSIL with the weight-space KL used in regularized BC.
Assumption 1. (Realizability) The optimal policy can be expressed by the parametric policy  $q_{\theta}(a|s)$ .
Lemma 2. (Coherent policy optimality). The optimal policy for a KL-regularized RL problem with coherent reward  $\tilde{r}(\boldsymbol{s},\boldsymbol{a}) = \alpha (\log q(\boldsymbol{a}\mid \boldsymbol{s}) - \log p(\boldsymbol{a}\mid \boldsymbol{s}))$  and temperature  $\alpha$  is  $q(\boldsymbol{a}\mid \boldsymbol{s})$ .
Proof. This result arises from the origin of the coherent reward from policy inversion and reward shaping (Theorem 1, Lemma 1). Moreover, considering the entropy-augmented reward  $r_{\alpha,\pi}(\pmb{s},\pmb{a}) = r(\pmb{s},\pmb{a}) - \alpha (\log \pi(\pmb{a} \mid \pmb{s}) - \log p(\pmb{a} \mid \pmb{s})$ , when  $r$  is the coherent reward  $\tilde{r}$  and  $\pi = q$ ,  $\tilde{r}_{\alpha,q}(\pmb{s},\pmb{a}) = 0 \forall \pmb{s} \in \mathcal{S}$ ,  $\pmb{a} \in \mathcal{A}$  so  $q(\pmb{a} \mid \pmb{s})$  is a solution to a soft policy iteration.
Lemma 3. (Data processing inequality, Wu [101], Theorem 4.1). For two stochastic processes parameterized with finite random variables  $\pmb{w} \sim p(\cdot)$ ,  $\pmb{w} \in \mathcal{W}$  and shared hypothesis spaces  $p(\pmb{y} \mid \pmb{x}, \pmb{w})$ , the conditional  $f$ -divergence in function space at points  $\pmb{X} \in \mathcal{X}^L$ ,  $L \in \mathbb{Z}_+$ , is upper bounded by the  $f$ -divergence in parameter space,
$$
\mathbb {D} _ {f} [ q (\boldsymbol {w}) | | p (\boldsymbol {w}) ] = \mathbb {D} _ {f} [ q (\boldsymbol {Y}, \boldsymbol {w} \mid \boldsymbol {X}) | | p (\boldsymbol {Y}, \boldsymbol {w} \mid \boldsymbol {X}) ] \geq \mathbb {D} _ {f} [ q (\boldsymbol {Y} \mid \boldsymbol {X}) | | p (\boldsymbol {Y} \mid \boldsymbol {X}) ].
$$
$$
\begin{array}{l} \mathbb {E} _ {\pmb {s}, \pmb {a} \sim \mathcal {D}} \left[ r _ {\pmb {\theta}} (\pmb {s}, \pmb {a}) \right] - \mathbb {E} _ {\pmb {a} \sim q _ {\pmb {\theta}} (\cdot | \pmb {s}), \pmb {s} \sim \mu_ {q _ {\pmb {\theta}}} (\cdot)} \left[ r _ {\pmb {\theta}} (\pmb {s}, \pmb {a}) - \beta (\log q _ {\pmb {\theta}} (\pmb {a} | \pmb {s}) - \log p (\pmb {a} | \pmb {s})) \right] \geq \\ \alpha \left(\mathbb {E} _ {\boldsymbol {s}, \boldsymbol {a} \sim \mathcal {D}} [ \log q _ {\boldsymbol {\theta}} (\boldsymbol {a} \mid \boldsymbol {s}) ] - \lambda \mathbb {D} _ {\mathrm {K L}} [ q _ {\boldsymbol {\theta}} (\boldsymbol {w}) | | p (\boldsymbol {w}) ] + \mathbb {E} _ {\boldsymbol {s}, \boldsymbol {a} \sim \mathcal {D}} [ \log p (\boldsymbol {a} \mid \boldsymbol {s}) ]\right). \\ \end{array}
$$