SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Joe Watson*†‡
London, United Kingdom {shhuang, heess} @google.com
Sandy H. Huang
†TU Darmstadt
Darmstadt, Germany
joe@robot-learning.de
Nicolas Heess
$^\ddagger$ Systems AI for Robot Learning
German Research Center for AI dfki.de
Imitation learning methods seek to learn from an expert either through behavioral cloning (BC) for the policy or inverse reinforcement learning (IRL) for the reward. Such methods enable agents to learn complex tasks from humans that are difficult to capture with hand-designed reward functions. Choosing between BC or IRL for imitation depends on the quality and state-action coverage of the demonstrations, as well as additional access to the Markov decision process. Hybrid strategies that combine BC and IRL are rare, as initial policy optimization against inaccurate rewards diminishes the benefit of pretraining the policy with BC. This work derives an imitation method that captures the strengths of both BC and IRL. In the entropy-regularized ('soft') reinforcement learning setting, we show that the behavioral-cloned policy can be used as both a shaped reward and a critic hypothesis space by inverting the regularized policy update. This coherency facilitates fine-tuning cloned policies using the reward estimate and additional interactions with the environment. Our approach conveniently achieves imitation learning through initial behavioral cloning and subsequent refinement via RL with online or offline data sources. The simplicity of the approach enables graceful scaling to high-dimensional and vision-based tasks, with stable learning and minimal hyperparameter tuning, in contrast to adversarial approaches. For the open-source implementation and simulation results, see joemwatson.github.io/csil.
Figure 1: By using regression rather than classification, our approach infers a shaped reward from expert data for this contextual bandit problem, whereas classifiers require additional non-expert data and may struggle to resolve the difference between expert  $(\times)$  and non-expert  $(\times)$  samples.
Imitation learning (IL) methods [1] provide algorithms for learning skills from expert demonstrations in domains such as robotics [2], either through direct mimicry (behavioral cloning, BC [3]) or inferring the latent reward (inverse reinforcement learning, IRL [4-6]) assuming a Markov decision process (MDP) [7, 8]. While BC is straightforward to implement, it is susceptible to covariate shift during evaluation [9]. IRL overcomes this problem but tackles the more complex task of jointly learning
*Corresponding author. Work done during an internship at Google DeepMind.
a reward and policy from interactions. So far, these two approaches to imitation have been largely separated [10], as optimizing the policy with an evolving reward estimate counteracts the benefit of pre-training the policy [11]. Combining the objectives would require careful weighting during learning and lacks convergence guarantees [12]. The benefit of combining BC and IRL is sample-efficient learning that enables any BC policy to be further improved using additional experience.
Contribution. Our IL method naturally combines BC and IRL by using the perspectives of entropy-regularized reinforcement learning (RL) [13-19] and gradient-based IRL [20, 21]. Our approach uses a BC policy to define an estimate of a shaped reward for which it is optimal, which can then be used to finetune the policy with additional knowledge, such as online interactions, offline data, or a dynamics model. Using the cloned policy to specify the reward avoids the need for careful regularization and hyperparameter tuning associated with adversarial imitation learning [11, 22], and we estimate a shaped variant of the true reward, rather than use a heuristic proxy, e.g., [23, 24]. In summary,
1. We use entropy-regularized RL to obtain a reward for which a behavioral-cloned policy is optimal, which we use to improve the policy and overcome the covariate shift problem.
2. We introduce approximate stationary stochastic process policies to implement this approach for continuous control by constructing specialized neural network architectures.
3. We show strong performance on online and offline imitation for high-dimensional and image-based continuous control tasks compared to current state-of-the-art methods.
2 Background & related work
A Markov decision process is a tuple  $\langle S, \mathcal{A}, \mathcal{P}, r, \gamma, \mu_0 \rangle$ , where  $\mathcal{S}$  is the state space,  $\mathcal{A}$  is the action space,  $\mathcal{P}: S \times \mathcal{A} \times S \to \mathbb{R}^+$  is the transition model,  $r: S \times \mathcal{A} \to \mathbb{R}$  is the reward function,  $\gamma$  is the discount factor, and  $\mu_0$  is the initial state distribution  $s_0 \sim \mu_0(\cdot)$ . When evaluating a policy  $\pi \in \Pi$ , we use the occupancy measure  $\rho_{\pi}(s, a) \triangleq \pi(a | s) \mu_{\pi}(s)$ ,  $\mu_{\pi}(s) \triangleq \sum_{t=0}^{\infty} \gamma^t \mu_t^\pi(s)$  where  $\mu_{t+1}^\pi(s') = \int \mathcal{P}(s' | s, a) \pi(a | s) \mu_t^\pi(s) ds da$ . This measure is used to compute infinite-horizon discounted expectations,  $\mathbb{E}_{s, a \sim \rho_\pi}[f(s, a)] = \mathbb{E}_{s_{t+1} \sim \mathcal{P}(s, a_t), a_t \sim \pi(\cdot | s_t)}[\sum_{t=0}^{\infty} \gamma^t f(s_t, a_t)]$ . Densities  $d_\pi(s, a) = (1 - \gamma) \rho_\pi(s, a)$  and  $\nu_\pi(s) = (1 - \gamma) \mu_\pi(s)$  are normalized measures. When the reward is unknown, imitation can be performed from a dataset  $\mathcal{D}$  of transitions ( $s, a, s'$ ) as demonstrations, obtained from the discounted occupancy measure of the expert policy  $\pi_* = \arg \max_\pi \mathbb{E}_{s, a \sim \rho_\pi}[r(s, a)]$ . The policy could be inferred directly or by inferring a reward and policy jointly, referred to as behavioral cloning and inverse reinforcement learning respectively.
Behavioral cloning. The simplest approach to imitation is to directly mimic the behavior using regression [3],  $\min_{\pi \in \Pi} l(\pi, \mathcal{D}) + \lambda \Psi(\pi)$ , with some loss function  $l$ , hypothesis space  $\Pi$  and regularizer  $\Psi$ . This method is effective with sufficient data coverage and an appropriate hypothesis space but suffers from compounding errors and cannot improve unless querying the expert [9]. The MIMIC-EXP algorithm of Rajaraman et al. [25] shows that the optimal no-interaction policy matches BC.
Inverse reinforcement learning. Rather than just inferring the policy, IRL infers the reward and policy jointly and relies on access to the underlying MDP [26, 5]. IRL iteratively finds a reward for which the expert policy is optimal compared to the set of possible policies while also finding the policy that maximizes this reward function. The learned policy seeks to match or even improve upon the expert using additional data beyond the demonstrations, such as environment interactions [6]. To avoid repeatedly solving the inner RL problem, one can consider the game-theoretic approach [27], where the optimization problem is a two-player zero-sum game where the policy and reward converge to the saddle point minimizing the 'apprenticeship error'  $\min_{\pi \in \Pi} \max_{r \in \mathcal{R}} \mathbb{E}_{\mathcal{D}}[r] - \mathbb{E}_{\rho_{\pi}}[r]$ . This approach is attractive as it learns the policy and reward concurrently [28]. However, in practice, saddle-point optimization can be challenging when scaling to high dimensions and can exhibit instabilities and hyperparameter sensitivity [11]. Compared to BC, IRL methods in theory scale to longer task horizons by overcoming compounding errors through environment interactions [29].
Maximum entropy inverse reinforcement learning & soft policy iteration. IRL is an underspecified problem [26], in particular in regions outside of the demonstration distribution. An effective way to address this is causal entropy regularization [30], by applying the principle of maximum entropy [31] to IRL (ME-IRL). Moreover, the entropy-regularized formulation has an elegant closed-form solution [30]. The approach can be expressed as an equivalent constrained minimum relative entropy problem for policy  $q(a \mid s)$  against a uniform prior  $p(a \mid s) = \mathcal{U}_A(a)$  using the Kullback
Leibler (KL) divergence and a constraint matching expected features  $\phi$ , with Lagrangian
$$
\min  _ {q, \boldsymbol {w}} \int_ {\mathcal {S}} \nu_ {q} (\boldsymbol {s}) \mathbb {D} _ {\mathrm {K L}} [ q (\boldsymbol {a} | \boldsymbol {s}) | | p (\boldsymbol {a} | \boldsymbol {s}) ] d s + \boldsymbol {w} ^ {\top} \left(\mathbb {E} _ {\boldsymbol {s}, \boldsymbol {a} \sim \mathcal {D}} [ \phi (\boldsymbol {s}, \boldsymbol {a}) ] - \mathbb {E} _ {\boldsymbol {s}, \boldsymbol {a} \sim \rho_ {q}} [ \phi (\boldsymbol {s}, \boldsymbol {a}) ]\right). \tag {1}
$$
The constraint term can be interpreted as the apprenticeship error, where  $\pmb{w}^{\top}\phi (\pmb {s},\pmb {a}) = r_{\pmb{w}}(\pmb {s},\pmb {a})$  Solving Equation 1 using dynamic programming yields the 'soft' Bellman equation [30],
$$
\mathcal {Q} (\boldsymbol {s}, \boldsymbol {a}) = r _ {\boldsymbol {w}} (\boldsymbol {s}, \boldsymbol {a}) + \gamma \mathbb {E} _ {\boldsymbol {s} ^ {\prime} \sim \mathcal {P} (\cdot | \boldsymbol {s}, \boldsymbol {a})} [ \mathcal {V} _ {\alpha} (\boldsymbol {s} ^ {\prime}) ], \mathcal {V} _ {\alpha} (\boldsymbol {s}) = \alpha \log \int_ {\mathcal {A}} \exp \left(\frac {1}{\alpha} \mathcal {Q} (\boldsymbol {s}, \boldsymbol {a})\right) p (\boldsymbol {a} | \boldsymbol {s}) d \boldsymbol {a}, \tag {2}
$$
for temperature  $\alpha = 1$ . Using Jensen's inequality and importance sampling, this target is typically replaced with a lower-bound [32], which has the same optimum and samples from the optimized policy rather than the initial policy like many practical deep RL algorithms [17]:
$$
\mathcal {Q} (\boldsymbol {s}, \boldsymbol {a}) = r _ {\boldsymbol {w}} (\boldsymbol {s}, \boldsymbol {a}) + \gamma \mathbb {E} _ {\boldsymbol {a} ^ {\prime} \sim q (\cdot | \boldsymbol {s} ^ {\prime}), \boldsymbol {s} ^ {\prime} \sim \mathcal {P} (\cdot | \boldsymbol {s}, \boldsymbol {a})} [ \mathcal {Q} (\boldsymbol {s} ^ {\prime}, \boldsymbol {a} ^ {\prime}) - \alpha (\log q (\boldsymbol {a} ^ {\prime} | \boldsymbol {s} ^ {\prime}) - \log p (\boldsymbol {a} ^ {\prime} | \boldsymbol {s} ^ {\prime})) ]. \tag {3}
$$
The policy update blends the exponentiated advantage function 'pseudo-likelihood' with the prior, as a form of regularized Boltzmann policy [17] that resembles a Bayes posterior [33],
$$
q _ {\alpha} (\boldsymbol {a} \mid \boldsymbol {s}) \propto \exp \left(\alpha^ {- 1} \left(\mathcal {Q} (\boldsymbol {s}, \boldsymbol {a}) - \mathcal {V} _ {\alpha} (\boldsymbol {s})\right) p (\boldsymbol {a} \mid \boldsymbol {s}\right). \tag {4}
$$
These regularized updates can also be used for RL, where it is the solution to a KL-regularized RL objective,  $\max_q\mathbb{E}_{\boldsymbol {s},\boldsymbol {a}\sim \rho_q}[\mathcal{Q}(\boldsymbol {s},\boldsymbol {a})] - \alpha \mathbb{D}_{\mathrm{KL}}[q(\boldsymbol {a}\mid s)\parallel p(\boldsymbol {a}\mid s)]$  , where the temperature  $\alpha$  now controls the strength of the regularization. This regularized policy update is known as soft- [14, 17] or posterior policy iteration [34, 35] (SPI, PPI), as it resembles a Bayesian update. In the function approximation setting, the update is performed in a variational fashion by minimizing the reverse KL divergence between the parametric policy  $q_{\theta}$  and the critic-derived update at sampled states  $s$  [17],
$$
\boldsymbol {\theta} _ {*} = \arg \min  _ {\boldsymbol {\theta}} \mathbb {E} _ {\boldsymbol {s} \sim \mathcal {B}} [ \mathbb {D} _ {\mathrm {K L}} [ q _ {\boldsymbol {\theta}} (\boldsymbol {a} \mid \boldsymbol {s}) | | q _ {\alpha} (\boldsymbol {a} \mid \boldsymbol {s}) ] ] = \arg \max  _ {\boldsymbol {\theta}} \mathcal {I} _ {\pi} (\boldsymbol {\theta}),
$$
$$
\mathcal {J} _ {\pi} (\boldsymbol {\theta}) = \mathbb {E} _ {\boldsymbol {a} \sim q _ {\boldsymbol {\theta}} (\cdot | \boldsymbol {s}), \boldsymbol {s} \sim \mathcal {B}} \left[ \mathcal {Q} (\boldsymbol {s}, \boldsymbol {a}) - \alpha \left(\log q _ {\boldsymbol {\theta}} (\boldsymbol {a} | \boldsymbol {s}) - \log p (\boldsymbol {a} | \boldsymbol {s})\right) \right]. \tag {5}
$$
The above objective  $\mathcal{I}_{\pi}$  can be maximized using reparameterized gradients and minibatches from replay buffer  $\mathcal{B}$  [17]. A complete derivation of SPI and ME-IRL is provided in Appendix K.1.
Gradient-based inverse reinforcement learning. An alternative IRL strategy is a gradient-based approach (GIRL) that avoids saddle-point optimization by learning a reward function such that the BC policy's policy gradient is zero [20], which satisfies first-order optimality. However, this approach does not remedy the ill-posed nature of IRL. Moreover, the Hessian is required for the sufficient condition of optimality [36], which is undesirable for policies with many parameters.
Related work. Prior state-of-the-art methods have combined the game-theoretic IRL objective [27] with entropy-regularized RL. These methods can be viewed as minimizing a divergence between the expert and policy, and include GAIL [37], AIRL [38],  $f$ -MAX [39], DAC [40], ValueDICE [41], IQLearn [42] and proximal point imitation learning (PPIL, [43]). These methods differ through their choice of on-policy policy gradient (GAIL) or off-policy actor-critic (DAC, IQLearn, PPI), and also how the minimax optimization is implemented, e.g., using a classifier (GAIL, DAC, AIRL), implicit reward functions (ValueDICE, IQLearn) or Lagrangian dual objective (PPIL). Alternative approaches to entropy-regularized IL use the Wasserstein metric (PWIL) [44], labelling of sparse proxy rewards (SQIL) [23], feature matching [30, 45-47], maximum likelihood [48] and matching state marginals [49] to specify the reward. Prior works at the intersection of IRL and BC include policy matching [50], policy-based GAIL classifiers [51], annealing between a BC and GAIL policy [12] and discriminator-weighted BC [52]. Our approach is inspired by gradient-based IRL [20, 36, 21], which avoids the game-theoretic objective and instead estimates the reward by analyzing the policy update steps with respect to the BC policy. An entropy-regularized GIRL setting was investigated in the context of learning from policy updates [53]. For further discussion on related work, see Appendix A.
3 Coherent imitation learning
For efficient and effective imitation learning, we would like to combine the simplicity of behavioral cloning with the structure of the MDP, as this would provide a means to initialize and refine the imitating policy through interactions with the environment. In this section, we propose such a method using a GIRL-inspired connection between BC and IRL in the entropy-regularized RL setting. By inverting the entropy-regularized policy update in Equation 4, we derive a reward for which the behavioral-cloning policy is optimal, a quality we refer to as coherence.
Inverting soft policy iteration and reward shaping. The  $\alpha$ -regularized closed-form policy based on Equation 4 can be inverted to provide expressions for the critic and reward (Theorem 1).
Theorem 1. (KL-regularized policy improvement inversion). Let  $p$  and  $q_{\alpha}$  be the prior and pseudoposterior policy given by posterior policy iteration (Equation 4). The critic can be expressed as
$$
\mathcal {Q} (\boldsymbol {s}, \boldsymbol {a}) = \alpha \log \frac {q _ {\alpha} (\boldsymbol {a} \mid \boldsymbol {s})}{p (\boldsymbol {a} \mid \boldsymbol {s})} + \mathcal {V} _ {\alpha} (\boldsymbol {s}), \quad \mathcal {V} _ {\alpha} (\boldsymbol {s}) = \alpha \log \int_ {\mathcal {A}} \exp \left(\frac {1}{\alpha} \mathcal {Q} (\boldsymbol {s}, \boldsymbol {a})\right) p (\boldsymbol {a} | \boldsymbol {s}) d \boldsymbol {a}. \tag {6}
$$
Substituting into the KL-regularized Bellman equation lower-bound from Equation 2,
$$
r (\boldsymbol {s}, \boldsymbol {a}) = \alpha \log \frac {q _ {\alpha} (\boldsymbol {a} \mid \boldsymbol {s})}{p (\boldsymbol {a} \mid \boldsymbol {s})} + \mathcal {V} _ {\alpha} (\boldsymbol {s}) - \gamma \mathbb {E} _ {\boldsymbol {s} ^ {\prime} \sim \mathcal {P} (\cdot | \boldsymbol {s}, \boldsymbol {a})} [ \mathcal {V} _ {\alpha} (\boldsymbol {s} ^ {\prime}) ]. \tag {7}
$$
The  $\mathcal{V}_{\alpha}(\boldsymbol{s})$  term is the 'soft' value function. We assume  $q_{\alpha}(\boldsymbol{a} \mid \boldsymbol{s}) = 0$  whenever  $p(\boldsymbol{a} \mid \boldsymbol{s}) = 0$ .
For the proof, see Appendix K.2. Jacq et al. [53] and Cao et al. [54] derived a similar inversion but for a maximum entropy policy iteration between two policies, rather than PPI between the prior and posterior. These expressions for the reward and critic are useful as they can be viewed as a valid form of shaping (Lemma 1) [54, 53], where  $\alpha (\log q(\pmb {a}\mid s) - \log p(\pmb {a}\mid s))$  is a shaped 'coherent' reward.
Lemma 1. (Reward shaping, Ng et al. [55]). For a reward function  $r: \mathcal{S} \times \mathcal{A} \to \mathbb{R}$  with optimal policy  $\pi \in \Pi$ , for a bounded state-based potential function  $\Psi: \mathcal{S} \to \mathbb{R}$ , a reward function  $\tilde{r}$  shaped by the potential function satisfying  $r(\boldsymbol{s}, \boldsymbol{a}) = \tilde{r}(\boldsymbol{s}, \boldsymbol{a}) + \Psi(\boldsymbol{s}) - \gamma \mathbb{E}_{\boldsymbol{s}' \sim \mathcal{P}(\cdot | \boldsymbol{s}, \boldsymbol{a})}[\Psi(\boldsymbol{s}')]$  has a shaped critic  $\tilde{\mathcal{Q}}$  satisfying  $\mathcal{Q}(\boldsymbol{s}, \boldsymbol{a}) = \tilde{\mathcal{Q}}(\boldsymbol{s}, \boldsymbol{a}) + \Psi(\boldsymbol{s})$  and the same optimal policy as the original reward.
Definition 1. The shaped 'coherent' reward and critic are derived from the log policy ratio by combining Lemma 1 and Theorem 1, with value function  $\mathcal{V}_{\alpha}(\boldsymbol{s})$  as the potential  $\Psi (\boldsymbol {s})$ . When policy  $q_{\alpha}(\boldsymbol {a}|\boldsymbol {s})$  models the data  $\mathcal{D}$  while matching its prior  $p$  otherwise, the density ratio should exhibit the following shaping
$$
\tilde {r} (\pmb {s}, \pmb {a}) = \alpha \log \frac {q _ {\alpha} (\pmb {a} \mid \pmb {s})}{p (\pmb {a} \mid \pmb {s})} \left\{ \begin{array}{l l} \ge 0 i f \pmb {s}, \pmb {a} \in \mathcal {D}, \\ <   0 i f \pmb {s} \in \mathcal {D}, \pmb {a} \notin \mathcal {D}, \\ = 0 i f \pmb {s} \notin \mathcal {D}, \forall \pmb {a} \in \mathcal {A}. \end{array} \right.
$$
In a continuous setting, the learned policy should capture this shaping approximately (Figure 2).
Figure 2: The coherent reward from Figure 1, made using a stationary Gaussian process policy, extended out-of-distribution. The reward approximates the shaping described in Definition 1.
Coherent soft imitation learning (CSIL, Algorithm 1) uses the BC policy to initialize the coherent reward and uses this reward to improve the policy further with additional interactions outside of  $\mathcal{D}$ . However, for the coherent reward to match Definition 1, the policy  $q(\pmb{a} \mid s)$  needs to match the policy prior  $p(\pmb{a} \mid s)$  outside of the data distribution. To achieve this, it is useful to recognize the policy  $q(\pmb{a} \mid s)$  as a 'pseudo-posterior' of prior  $p(\pmb{a} \mid s)$  and incorporate this Bayesian view into the BC step.
Imitation learning with pseudo-posteriors. The 'pseudo-posteriors' formalism [35, 56-59] is a generalized way of viewing the policy derived for ME-IRL and KL-regularized RL in Equation 4.
Definition 2. Pseudo-posteriors are solutions to KL-constrained or minimum divergence problems with an additional scalar objective or vector-valued constraint term and Lagrange multipliers  $\lambda$ ,
$$
\max  _ {q} \mathbb {E} _ {\boldsymbol {x} \sim q (\cdot)} [ f (\boldsymbol {x}) ] - \lambda (\mathbb {D} _ {\mathrm {K L}} [ q (\boldsymbol {x}) | | p (\boldsymbol {x}) ] - \epsilon) \quad \rightarrow q _ {\lambda} (\boldsymbol {x}) \propto \exp (\lambda^ {- 1} f (\boldsymbol {x})) p (\boldsymbol {x}).
$$
$$
\min  _ {q} \quad \mathbb {D} _ {\mathrm {K L}} [ q (\boldsymbol {x}) | | p (\boldsymbol {x}) ] - \boldsymbol {\lambda} ^ {\top} (\mathbb {E} _ {\boldsymbol {x} \sim q (\cdot)} [ \boldsymbol {f} (\boldsymbol {x}) ] - \boldsymbol {f} ^ {*}) \quad \rightarrow \quad q _ {\boldsymbol {\lambda}} (\boldsymbol {x}) \propto \exp (\boldsymbol {\lambda} ^ {\top} \boldsymbol {f} (\boldsymbol {x})) p (\boldsymbol {x}).
$$
The  $\exp (\pmb{\lambda}^{\top}\pmb {f}(\pmb {x}))$  term is an unnormalized pseudo-likelihood, as it facilities Bayesian inference to solve a regularized optimization problem specified by  $\pmb{f}$
Optimizing the policy distribution, the top objective captures KL-regularized RL, where  $f$  is the return, and the regularization is implemented as a constraint with bound  $\epsilon$  or soft penalty with constant  $\lambda$ . The bottom objective is the form seen in ME-IRL (Equation 1), where  $f$  is the feature space that defines the reward model. Contrasting these pseudo-posterior policies against BC with Gaussian processes, e.g., [60, 61], we can compare the Bayesian likelihood used for regression with the critic-based pseudo-likelihood obtained from imitation learning. The pseudo-likelihoods in entropy-regularized IL methods produce an effective imitating policy by incorporating the MDP and trajectory distribution because  $f$  captures the cumulative reward, compared to just the action prediction error typically captured by BC regression likelihoods. This point is expanded in Appendix B.
Algorithm 1: Coherent soft imitation learning (CSIL)
Data: Expert demonstrations  $\mathcal{D}$ , initial temperature  $\alpha$ , refinement temperature,  $\beta$ , parametric policy class  $q_{\theta}(a \mid s)$ , prior policy  $p(a \mid s)$ , regression regularizer  $\Psi$ , iterations  $N$
Result:  $q_{\theta_N}(a \mid s)$ , matching or improving the initial policy  $q_{\theta_1}(a \mid s)$
Train initial policy from demonstrations,  $\pmb{\theta}_{1} = \arg \max_{\pmb{\theta}}\mathbb{E}_{\pmb{s},\pmb{a}\sim \mathcal{D}}[\log q_{\pmb{\theta}}(\pmb {a}\mid \pmb {s}) - \Psi (\pmb {\theta})]$
Define fixed shaped coherent reward,  $\tilde{r}_{\theta_1}(\boldsymbol{s},\boldsymbol{a}) = \alpha (\log q_{\theta_1}(\boldsymbol{a}\mid \boldsymbol{s}) - \log p(\boldsymbol{a}\mid \boldsymbol{s}))$
Choose reference policy  $\pi (\pmb {a}\mid \pmb {s})$  to be  $p(\pmb {a}\mid \pmb {s})$  or  $q_{\theta_1}(\pmb {a}\mid \pmb {s})$  and initialize the shaped critic,
$$
\tilde {\mathcal {Q}} _ {1} (\boldsymbol {s}, \boldsymbol {a}) = \tilde {r} _ {\boldsymbol {\theta} _ {1}} (\boldsymbol {s}, \boldsymbol {a}) + \gamma \left( \right.\tilde {\mathcal {Q}} _ {1} \left(\boldsymbol {s} ^ {\prime}, \boldsymbol {a} ^ {\prime}\right) - \alpha \left(\log q _ {\boldsymbol {\theta} _ {1}} \left(\boldsymbol {a} ^ {\prime} \mid \boldsymbol {s} ^ {\prime}\right) - \log \pi \left(\boldsymbol {a} ^ {\prime} \mid \boldsymbol {s} ^ {\prime}\right)\right), \boldsymbol {s}, \boldsymbol {a}, \boldsymbol {s} ^ {\prime}, \boldsymbol {a} ^ {\prime} \sim \mathcal {D};
$$
for  $n = 2\to N$  do
// finetune policy with reinforcement learning
| Compute  $\tilde{\mathcal{Q}}_n$  and  $q_{\theta_n}$  using soft policy iteration (Section 2), e.g. SAC, with temperature  $\beta$ . end
Behavior cloning with pseudo-posteriors. Viewing the policy  $q(\boldsymbol{a} \mid s)$  as a pseudo-posterior has three main implications when performing behavior cloning and designing the policy and prior:
1. We perform conditional density estimation to maximize the posterior predictive likelihood, rather than using a data likelihood, which would require assuming additive noise [62, 33].
2. We require a hypothesis space  $p(\boldsymbol{a} \mid \boldsymbol{s}, \boldsymbol{w})$ , e.g., a tabular policy or function approximator, that is used to define both the prior and posterior through weights  $\boldsymbol{w}$ .
3. We need a fixed weight prior  $p(\boldsymbol{w})$  that results in a predictive distribution that matches the desired policy prior  $p(\boldsymbol{a} \mid \boldsymbol{s}) = \int p(\boldsymbol{a} \mid \boldsymbol{s}, \boldsymbol{w}) p(\boldsymbol{w}) \, \mathrm{d}\boldsymbol{w} \forall \boldsymbol{s} \in S$ .