SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Empirically, we find these two choices perform similarly, but the latter is easier to implement. So we will report results mainly based on the second realization. In our experiments, we consider two offline RL algorithms under this category, i.e., CRR, and IQL. They use two weighting schemes:  $f_{\mathrm{CRR}} = \exp \left[\left(Q_{\phi}(s,a) - \mathbb{E}_{a^{\prime}\sim \hat{\pi} (a|s)}Q(s,a^{\prime})\right) / \tau_{\mathrm{CRR}}\right]$  and  $f_{\mathrm{IQL}} = \exp \left[(Q_{\phi}(s,a) - V_{\psi}(s)) / \tau_{\mathrm{IQL}}\right]$ , where  $\tau$  refers to the temperature parameter and  $V_{\psi}(s)$  is an additional value network parameterized by  $\psi$ . We defer the details of these two algorithms to Appendix A.
4.4 Comparison to Diffusion-QL
Now we are ready to compare our method and Diffusion-QL comprehensively. Though our EDP shares the same policy parametrization as Diffusion-QL, it differs from Diffusion-QL significantly in the training algorithm. As a result, the computational efficiency and generality of diffusion policies have been improved substantially.
Efficiency The diffusion policy affects both policy evaluation (Eqn. (6)) and policy improvement (Eqn. (7)). First, calculating  $L_{\mathrm{TD}}(\phi)$  in policy evaluation requires drawing the next action from it. Diffusion-QL uses DDPM sampling while EDP employs a DPM-Solver, which can reduce the sampling steps from 1000 to 15, thus accelerating the training. Second, in policy improvement, Diffusion-QL again applies DDPM for sampling. Then, it calculates the loss function based on sampled actions and backpropagates through the sampling process for network update. This means it needs to forward and backward a neural network for  $K$  times each training iteration. As a result, Diffusion-QL can only work with small  $K$ , e.g.,  $5 \sim 100$ . In comparison, our training scheme only passes through the network once an iteration, no matter how big  $K$  is. This enables EDP to use a larger  $K$  (1000 in our experiments) to train diffusion policy on the more fine-grained scale. The results in Tab. 1 also show a larger  $K$  can give better performance.
Generality Diffusion-QL can only work with direct policy optimization, which contains only a small portion of algorithms. Moreover, thanks to their flexibility and high performance, the likelihood-based algorithms are preferred for some tasks (e.g., Antmaze). Our method successfully makes diffusion trainable with any RL algorithm.
4.5 Controlled Sampling from Diffusion Policies
Traditionally, a continuous policy is represented with a state-conditional Gaussian distribution. During evaluation time, a policy executes deterministically to reduce variance by outputting the distribution mean as an action. However, with diffusion policies, we can only randomly draw a sample from the underlying distribution without access to its statistics. As a result, the sampling process is noisy, and the evaluation is of high variance. We consider the following method to reduce variance.
Energy-based Action Selection (EAS) Recall that the goal of (offline) RL is to learn a policy that can maximize the cumulative return or values. Though the policy  $\pi_{\theta}$  is stochastic, the learned  $Q_{\phi}$  provides a deterministic critic for action evaluation. We can sample a few actions randomly, then use  $Q_{\phi}$  for selection among them to eliminate randomness. EAS first samples  $N$  actions from  $\pi_{\theta}$  by using any samplers (i.e., DPM-Solver), then sample one of them with weights proportional to  $e^{Q(\boldsymbol{s},\boldsymbol{a})}$ . This procedure can be understood as sampling from an improved policy  $p(\boldsymbol{a}|\boldsymbol{s}) \propto e^{Q(\boldsymbol{s},\boldsymbol{a})}\pi_{\theta}(\boldsymbol{a}|\boldsymbol{s})$ . All results will be reported based on EAS. See Appendix. C.4 for the other two methods.
We conduct extensive experiments on the D4RL benchmark [2] to verify the following assumptions: 1) Our diffusion policy is much more efficient than the previous one regarding training and evaluation costs. 2) Our diffusion policy is a generic policy class that can be learned through direct and likelihood-based policy learning methods. We also provide various ablation studies on the critical components for better understanding.
Baselines We evaluate our method on four domains in D4RL, including Gym-locomotion, AntMaze, Adroit, and Kitchen. For each domain, we consider extensive baselines to provide a thorough evaluation. The simplest method is the classic behavior cloning (BC) baseline and  $10\%$  BC that performs behavior cloning on the best  $10\%$  data. TD3+BC [6] combines off-policy reinforcement
learning algorithms with BC. OneStepRL [1] first conducts policy evaluation to obtain the Q-value of the behavior policy from the offline dataset, then use it for policy improvement. AWAC [25], AWR [28], and CRR [38] improve policy improvement by adding advantage-based weights to policy loss functions. CQL [17] and IQL [15] constrain the policy evaluation process by making conservative Q updates or replacing the max operator with expectile regression. We also consider the Decision Transformer (DT) [4] baseline that maps offline RL as a sequence-to-sequence translation problem.
Experimental Setup We keep the backbone network architecture the same for all tasks and algorithms, which is a 3-layer MLP (hidden size 256) with Mish [23] activation function following Diffusion-QL [37]. For the noise prediction network  $\epsilon_{\theta}(\pmb{a}^k, k; \pmb{s})$  in diffusion policy, we first encode timestep  $k$  with sinusoidal embedding [36], then concatenate it with the noisy action  $\pmb{a}^k$  and the conditional state  $\pmb{s}$ . We use the Adam [13] to optimize both diffusion policy and the Q networks. The models are trained for 2000 epochs on Gym-locomotion and 1000 epochs on the other three domains. Each epoch consists of 1000 iterations of policy updates with batch size 256. For DPM-Solver [20], we use the third-order version and set the model call steps to 15. We reimplement DQL strictly following the official PyTorch code [27] for fair comparison and we refer to DQL (JAX) for all sample efficiency comparisons. We defer the complete list of all hyperparameters to the appendix due to space limits. Throughout this paper, the results are reported by averaging 5 random seeds.
Evaluation Protocol We consider two evaluation metrics in this paper. First, online model selection (OMS), proposed by Diffusion-QL [37], selects the best-performing model throughout the whole training process. However, though OMS can reflect an algorithm's capacity, it is cheating, especially when the training procedure is volatile on some of the tasks. Therefore, we propose another metric to focus on the training stability and quality, which is running average at training (RAT). RAT calculates the running average of evaluation performance for ten consecutive checkpoints during training and reports the last score as the final performance.
5.1 Efficiency and Reproducibility
In this section, we focus on the training and evaluation efficiency of our efficient diffusion policy. We choose the OMS evaluation metric to make a fair comparison with the baseline method Diffusion-QL [37]. We consider four variants of EDP to understand how each component contributes to the high efficiency of our method. 1) EDP is the complete version of our method. It uses the action approximation technique in Eqn. (9) for policy training and uses DPM-Solver for sampling. 2)  $EDP w/o$  DPM modifies EDP by replacing DPM-Solver with the original DDPM sampling method in Eqn. 4. 3)  $EDP w/o$  AP removes the action approxima
tion technique. 4) DQL (JAX) is our Jax implementation of Diffusion-QL.
Figure 2: Training and evaluation speed comparison. The training IPS are: 4.66, 22.30, 50.94, 38.4, and 116.21. The sampling SPS are: 18.67, 123.70, 123.06, 411.0, 411.79.
We first benchmark and compare the training/evaluation speed of the above three variants and Diffusion-QL. We choose walker2d-medium-expert-v2 as the testbed. For training speed, we run each algorithm for 10,000 iterations of policy updates and calculate the corresponding iterations-per-second (IPS). Similarly, we sample 10,000 transitions by interacting with the environment and calculate the corresponding steps-per-second (SPS) for evaluation speed. Based on the visualization in Fig. 2, by taking DQL (JAX) as the baseline, we are able to attribute the performance boost to specific techniques proposed. Specifically, we can observe that action approximation makes  $2.3\mathrm{x}$  training and  $3.3\mathrm{x}$  sampling faster, while using the DPM-Solver adds an additional  $2.3\mathrm{x}$  training speedup. We can observe that DQL (JAX) is  $5\times$  faster than Diffusion-QL, which means our Jax implementation is more computationally efficient than Diffusion-QL's PyTorch code. This demonstrates that both the action approximation technique and DPM-Solver play a critical role in making the training of diffusion policy efficient. However, this technique does not affect the sampling procedure; thus,  $EDP w/o$  DPM and DQL (JAX) are on par with each other regarding sampling speed.
Table 2: Average normalized score on the D4RL benchmark. FF denotes the original policies parameterized with feed-forward neural networks, while EDP is our efficient diffusion policy. Results of baselines are taken directly from [15]. For the missing results of  $10\%$ , AWAC, and OneStep RL, we re-implement the baselines and report the results. Our results are reported with the RAT metric.
Dataset	BC	10%BC	DT	AWAC	OneStep	CQL	TD3+BC	CRR	IQL
FF	EDP	FF	EDP	FF	EDP
halfcheetah-medium-v2	42.6	42.5	42.6	43.5	48.4	44.0	48.3	52.1	44.0	49.2	47.4	48.1
hopper-medium-v2	52.9	56.9	67.6	57.0	59.6	58.5	59.3	81.9	58.5	78.7	66.3	63.1
walker2d-medium-v2	75.3	75.0	74.0	72.4	81.8	72.5	83.7	86.9	72.5	82.5	78.3	85.4
halfcheetah-medium-replay-v2	36.6	40.6	36.6	40.5	38.1	45.5	44.6	49.4	45.5	43.5	44.2	43.8
hopper-medium-replay-v2	18.1	75.9	82.7	37.2	97.5	95.0	60.9	101.0	95.0	99.0	94.7	99.1
walker2d-medium-replay-v2	26.0	62.5	66.6	27.0	49.5	77.2	81.8	94.9	77.2	63.3	73.9	84.0
halfcheetah-medium-expert-v2	55.2	92.9	86.8	42.8	93.4	91.6	90.7	95.5	91.6	85.6	86.7	86.7
hopper-medium-expert-v2	52.5	110.9	107.6	55.8	103.3	105.4	98.0	97.4	105.4	92.9	91.5	99.6
walker2d-medium-expert-v2	107.5	109.0	108.1	74.5	113.0	108.8	110.1	110.2	108.8	110.1	109.6	109.0
average	51.9	74.0	74.7	50.1	76.1	77.6	75.3	85.5	77.6	78.3	77.0	79.9
kitchen-complete-v0	65.0	7.2	-	39.3	57.0	43.8	2.2	61.5	43.8	73.9	62.5	75.5
kitchen-partial-v0	38.0	66.8	-	36.6	53.1	49.8	0.7	52.8	49.8	40.0	46.3	46.3
kitchen-mixed-v0	51.5	50.9	-	22.0	47.6	51.0	0.0	60.8	51.0	46.1	51.0	56.5
average	51.5	41.6	-	32.6	52.6	48.2	1.0	58.4	48.2	53.3	53.3	59.4
pen-human-v0	63.9	-2.0	-	15.6	71.8	37.5	5.9	48.2	37.5	70.2	71.5	72.7
pen-cloned-v0	37.0	0.0	-	24.7	60.0	39.2	17.2	15.9	39.2	54.0	37.3	70.0
average	50.5	-1.0	-	20.2	65.9	38.4	11.6	32.1	38.4	62.1	54.4	71.3
antmaze-umaze-v0	54.6	62.8	59.2	56.7	64.3	74.0	40.2	96.6	0.0	95.9	87.5	94.2
antmaze-umaze-diverse-v0	45.6	50.2	53.0	49.3	60.7	84.0	58.0	69.5	41.9	15.9	62.2	79.0
antmaze-medium-play-v0	0.0	5.4	0.0	0.0	0.3	61.2	0.2	0.0	0.0	33.5	71.2	81.8
antmaze-medium-diverse-v0	0.0	9.8	0.0	0.7	0.0	53.7	0.0	6.4	0.0	32.7	70.0	82.3
antmaze-large-play-v0	0.0	0.0	0.0	0.0	0.0	15.8	0.0	1.6	0.0	26.0	39.6	42.3
antmaze-large-diverse-v0	0.0	6.0	0.0	1.0	0.0	14.9	0.0	4.4	0.0	58.5	47.5	60.6
average	16.7	22.4	18.7	18.0	20.9	50.6	16.4	29.8	7.0	43.8	63.0	73.4
To show that EDP does not hurt performance, we compare the normalized scores of all tasks in Tab. 1. The results for Diffusion-QL are directly copied from the paper, where each task is carefully tuned with its own hyperparameters. Instead, DQL (JAX) and EDP use the same hyperparameters for tasks belonging to the same domain. Moreover, since EDP is more computationally efficient, we can train a better score model with large  $K = 1000$ , which is larger than the one (5~100) used in Diffusion-QL. Note that training diffusion policies with large K is impossible
in Diffusion-QL, as it needs to forward and backward the neural network K times. Finally, we can observe that EDP and DQL (JAX) are comparable with Diffusion-QL on gym-locomotion tasks but much better on the other three domains. Hence, efficient diffusion policy can boost sample efficiency and improve performance by enabling diffusion training in fine-grained noise prediction.
Table 1: The performance of Diffusion-QL with efficient diffusion policy. The results for Diffusion-QL are directly quoted from [37]. All results are reported based on the OMS metric.
Dataset	Diffusion-QL	DQL (JAX)	EDP
locomotion	89.4	89.4	90.3
antmaze	75.4	78.6	77.9
adroit	68.3	66.5	89.1
kitchen	71.6	83.0	80.5
5.2 Generality and Overall Results
This section aims to evaluate whether EDP is a general policy class. To this end, we train it with direct policy optimization (TD3) and likelihood-based policy optimization (CRR, IQL) methods. Then, we compare them with their feed-forward counterparts and other baseline methods in Table 2. All scores for diffusion policies are reported using the RAT metric, while other scores are directly quoted from their paper. It shows that EDP can beat the standard Gaussian policy parameterized with an MLP on all domains and for all the three algorithms considered. On the gym-locomotion domain,  $\mathrm{EDP} + \mathrm{TD3}$  gives the best performance (average score 85.5), while likelihood-based policy learning methods are slightly worse. However, on the other three domains (Kitchen, Adroit, and Antmaze),  $\mathrm{EDP} + \mathrm{IQL}$  beats all the other methods by a large margin (more than 10 average scores). Therefore, we conclude that EDP can serve as a plug-in policy class for different RL methods.
Evaluation Metrics To reveal that evaluation metric is important, we train EDP with TD3 algorithms on three selected environments: walker2d-medium-expert-v2, hopper-medium-expert-v2, and antmaze-medium-diverse-v0. We then compare the scores for OMS (best) and RAT (average) by plotting the training curves in Fig. 3. On
walker2d, the training is stable; thus, both OMS and RAT scores steadily grow and result in close final scores. A similar trend can be observed on the hopper but with a more significant gap between these two metrics. However, these two metrics diverge significantly when the training succeeds and then crashes on antmaze. Therefore, OMS is misleading and can not give a reliable evaluation of algorithms, which explains the necessity of using RAT in Sec. 5.2.
Figure 3: Training curves for EDP +TD3 on three representative environments. Average represents RAT, Best represents OMS.
Energy-Based Action Selection We notice that energy-based action selection (EAS) is a general method and can also be used for arbitrary policies. We apply EAS to normal TD3+BC and find no improvement, which shows EAS is only necessary for diffusion sampling. The results are deferred to the Appendix. Moreover, set the number of actions used in
EAS from 1 to 200, and report the performance on gym-locomotions tasks in Fig. 4. It shows the normalized score monotonically grows as the number of actions increases on 8 out of 9 tasks. In our main experiments, we set the number of actions to 10 by trading off the performance and computation efficiency.
Figure 4: Performance of different number of actions used in EAS. The experiments are conducted on the nine locomotion tasks.
Number of Samples
DPM-Solver We are using DPM-Solver to speed up the sampling process of diffusion policies. The number of models in DPM-Solver is an important hyper-parameter that affects sampling efficiency and quality. We vary this number from 3 to 30 and compare the performance on gym-locomotion tasks in Fig. 5. We can observe that the performance increases as more
steps of model calls are used. The performance gradually plateaus after 15 model calls. Therefore, we use 15 in our main experiments.
Figure 5: Performance of DPM-Solver with varying steps. The experiments are conducted on the nine locomotion tasks.
Diffusion policy has emerged as an expressive policy class for offline reinforcement learning. Despite its effectiveness, diffusion policy is limited by two drawbacks, hindering it from wider applications. First, training a diffusion policy requires to forward and backward through a long parameterized Markov chain, which is computationally expensive. Second, the diffusion policy is a restricted policy class that can not work with likelihood-based RL algorithms, which are preferred in many scenarios. We propose efficient diffusion policy (EDP) to address these limitations and make diffusion policies faster, better, and more general. EDP relies on an action approximation to construct actions from corrupted ones, thus avoiding running the Markov chain for action sampling at training. Our benchmarking shows that EDP achieves  $25 \times$  speedup over Diffusion-QL at training time on the gym-locomotion tasks in D4RL. We conducted extensive experiments by training EDP with various offline RL algorithms, including TD3, CRR, and IQL, the results clearly justify the superiority of diffusion policies over Gaussian policies. As a result, EDP set new state-of-the-art on all four domains in D4RL.
[39] Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361, 2019.
[40] Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. Advances in neural information processing systems, 34:28954-28967, 2021.
[41] Yang Yue, Bingyi Kang, Xiao Ma, Gao Huang, Shiji Song, and Shuicheng Yan. Offline prioritized experience replay. arXiv preprint arXiv:2306.05412, 2023.
[42] Yang Yue, Bingyi Kang, Xiao Ma, Zhongwen Xu, Gao Huang, and Shuicheng Yan. Boosting offline reinforcement learning via data rebalancing. arXiv preprint arXiv:2210.09241, 2022.
In this paper, we propose an efficient yet powerful policy class for offline reinforcement learning. We show that this method is superior to most existing methods on simulated robotic tasks. However, some war robots or weapon robots might employ our EDP to learn strategic agents considering the generalization ability of EDP. Depending on the specific application scenarios, it might be harmful to domestic privacy and safety.
A Reinforcement Learning Algorithms
In this section, we introduce the details of the RL algorithms experimented.
A.1 TD3+BC
TD3 [8] is a popular off-policy RL algorithm for continuous control. TD3 improves DDPG [19] by addressing the value overestimation issue. Specifically, TD3 adopts a double Q-learning paradigm that computes the TD-target  $\hat{Q}(s, a)$  as
$$
\hat {Q} (\boldsymbol {s}, \boldsymbol {a}) = r (\boldsymbol {s}, \boldsymbol {a}) + \gamma \min  \left(Q _ {1} \left(\boldsymbol {s} ^ {\prime}, \boldsymbol {a} ^ {\prime}\right), Q _ {2} \left(\boldsymbol {s} ^ {\prime}, \boldsymbol {a} ^ {\prime}\right)\right), \quad \boldsymbol {a} ^ {\prime} = \pi \left(\boldsymbol {s} ^ {\prime}\right), \tag {14}
$$
where  $\pi(s')$  is a deterministic policy,  $Q_{1}$  and  $Q_{2}$  are two independent value networks. Specifically, TD3 takes only  $Q_{1}$  for policy improvement
$$
\pi^ {*} = \underset {\pi} {\arg \max } \mathbb {E} [ Q _ {1} (\boldsymbol {s}, \hat {\boldsymbol {a}}) ], \quad \hat {a} = \pi (\boldsymbol {s}). \tag {15}
$$
Built on top of TD3, TD3+BC simply adds an additional behavior cloning term for its policy improvement
$$
\pi^ {*} = \arg \max  _ {\pi} \mathbb {E} _ {s, a \sim \mathcal {D}} \left[ Q (\boldsymbol {s}, \hat {\boldsymbol {a}}) - \alpha (\hat {a} - a) ^ {2} \right], \quad \hat {\boldsymbol {a}} = \pi (\boldsymbol {s}), \tag {16}
$$
where  $\alpha$  is a hyper-parameter that balances these two terms. However, as the scale of Q-values are different from the behavior cloning loss, TD3+BC normalizes it with  $\frac{1}{N}\sum_{i=1}^{N}|Q(\boldsymbol{s}_i,\hat{\boldsymbol{a}}_i)|$  for numerical stability.
In the context of EDP, we observe that 1) the behavior cloning term can be naturally achieved as a diffusion loss as defined in Eqn. 5; 2) the sampled action  $\hat{a}$  is replaced by action approximation as in Eqn. 9 for efficient policy improvement. Different from the original TD3+BC that uses only  $Q_{1}$  for policy improvement, we sample from  $Q_{1}$  and  $Q_{2}$  with equal probability for each policy improvement step.
A.2 Critic Regularized Regression
CRR follows Advantage Weighted Regression (AWR) [29], which is a simple yet effective off-policy RL method, for policy improvement. Specifically, AWR considers a constrained policy improvement step
$$
\arg \max  _ {\pi} \int_ {s} d _ {\mu} (s) \int_ {a} \pi (\boldsymbol {a} \mid s) A (\boldsymbol {s}, \boldsymbol {a}) d \boldsymbol {a} d s, \quad \text {s . t .} \quad \int_ {s} d _ {\mu} (s) D _ {\mathrm {K L}} [ \pi (\cdot \mid s) \| \mu (\cdot \mid s) ] \leq \epsilon , \tag {17}
$$
where  $A(s, a)$  is the advantage function,  $\mu$  is a behavior policy that is used to generate trajectories during off-policy RL,  $d_{\mu}(s)$  is the state distribution induced by  $\mu$ ,  $D_{\mathrm{KL}}$  is the KL divergence, and  $\epsilon$  is a threshold parameter. This constrained optimization problem can be solved in closed form, which gives an optimal policy of
$$
\pi^ {*} (\boldsymbol {a} \mid \boldsymbol {s}) = \frac {1}{Z (\boldsymbol {s})} \mu (\boldsymbol {a} \mid \boldsymbol {s}) \exp \left(\frac {1}{\beta} A (\boldsymbol {s}, \boldsymbol {a})\right), \tag {18}
$$
with  $Z(s)$  being the partition function and  $\beta$  is a hyper-parameter. For policy improvement, AWR simply distills the one-step improved optimal policy to the learning policy  $\pi$  by minimizing the
KL-divergence
$$
\begin{array}{l} \arg \min _ {\pi} \mathbb {E} _ {\boldsymbol {s} \sim d _ {\mu} (\boldsymbol {s})} \left[ D _ {\mathrm {K L}} \left[ \pi^ {*} (\cdot | \boldsymbol {s}) \| \pi (\cdot | \boldsymbol {s}) \right] \right] \\ = \arg \max  _ {\pi} \mathbb {E} _ {\boldsymbol {s} \sim d _ {\mu} (\boldsymbol {s}), \boldsymbol {a} \sim \mu (\cdot | \boldsymbol {s})} \left[ \log \pi (\boldsymbol {a} \mid \boldsymbol {s}) \exp \left(\frac {1}{\beta} A (\boldsymbol {s}, \boldsymbol {a})\right) \right] \tag {19} \\ \end{array}
$$
CRR performs policy improvement in the same way as AWR, which simply replaces the sampling distribution with a fixed dataset
$$
\arg \max  _ {\pi} \mathbb {E} _ {\boldsymbol {s}, \boldsymbol {a} \sim \mathcal {D}} \left[ \log \pi (\boldsymbol {a} \mid \boldsymbol {s}) \exp \left(\frac {1}{\beta} A (\boldsymbol {s}, \boldsymbol {a})\right) \right]. \tag {20}
$$
As a result, this naturally imposes an implicit constraint on its policy improvement step.
However, as computing the log-likelihood  $\log \pi (a\mid s)$  is intractable in diffusion models, in practical implementations, we use Eqn. 13 instead. In addition, we compute the advantage by
$$
A (\boldsymbol {s}, \boldsymbol {a}) = \min  \left(Q _ {1} (\boldsymbol {s}, \boldsymbol {a}) - Q _ {2} (\boldsymbol {s}, \boldsymbol {a})\right) - \frac {1}{N} \sum_ {i = 1} ^ {N} \min  \left(Q _ {1} (\boldsymbol {s}, \hat {\boldsymbol {a}} _ {i}), Q _ {2} (\boldsymbol {s}, \hat {\boldsymbol {a}} _ {i})\right), \tag {21}
$$
where  $\hat{a}_i\sim \mathcal{N}(\hat{\pmb{a}}^0,\pmb {\sigma})$  is a sampled action with the mean of approximated action and an additional standard deviation. In our experiment, we found using a fixed standard deviation, an identity matrix of  $\pmb {\sigma} = \mathbb{I}$ $\beta = 1$  , and sample size  $N = 10$  generally produce a good performance.
A.3 Implicit Q Learning