SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this section, we present the TD7 algorithm (TD3+4 additions). We begin by introducing the use of checkpoints in RL to improve the stability of RL agents. We then combine SALE with checkpoints and various previous algorithmic modifications to TD3 [Fujimoto et al., 2018] to create a single RL algorithm for both the online and offline setting.
5.1 Policy Checkpoints
Deep RL algorithms are notoriously unstable [Henderson et al., 2017]. The unreliable nature of deep RL algorithms suggest a need for stabilizing techniques. While we can often directly address the source of instability, some amount of instability is inherent to the combination of function approximation and RL. In this section, we propose the use of checkpoints, to preserve evaluation performance, irrespective of the quality of the current learned policy.
A checkpoint is a snapshot of the parameters of a model, captured at a specific time during training. In supervised learning, checkpoints are often used to recall a previous set of high-performing parameters based on validation error, and maintain a consistent performance across evaluations [Vaswani et al., 2017, Kenton and Toutanova, 2019]. Yet this technique is surprisingly absent from the deep RL toolkit for stabilizing policy performance.
In RL, using the checkpoint of a policy that obtained a high reward during training, instead of the current policy, could improve the stability of the performance at test time.
For off-policy deep RL algorithms, the standard training paradigm is to train after each time step (typically at a one-to-one ratio: one gradient step for one data point). However, this means that the policy changes throughout each episode, making it hard to evaluate the performance. Similar to many on-policy algorithms [Williams, 1992, Schulman et al., 2017], we propose to keep the policy fixed for several assessment episodes, then batch the training that would have occurred.
- Standard off-policy RL: Collect a data point  $\rightarrow$  train once.
- Proposed: Collect  $N$  data points over several assessment episodes  $\rightarrow$  train  $N$  times.
In a similar manner to evolutionary approaches [Salimans et al., 2017], we can use these assessment episodes to judge if the current policy outperforms the previous best policy and checkpoint accordingly. At evaluation time, the checkpoint policy is used, rather than the current policy.
We make two additional modifications to this basic strategy.
Minimum over mean. Setting aside practical considerations, the optimal approach would be to evaluate the average performance of each policy using as many trials as possible. However, to preserve learning speed and sample efficiency, it is only sensible to use a handful of trials. As such, to penalize unstable policies using a finite number of assessment episodes, we use the minimum performance, rather than the mean performance. This approach also means that extra assessment episodes do not need to be wasted on poorly performing policies, since training can resume early if the performance of any episode falls below the checkpoint performance.
Variable assessment length. In Appendix F, we examine the caliber of policies trained with a varied number of assessment episodes and observe that a surprisingly high number of episodes  $(20+)$  can be used without compromising the performance of the final policy. However, the use of many assessment episodes negatively impacts the early performance of the agent. Freezing training for many episodes means that the environment is explored by a stale policy, reducing data diversity, and delaying feedback from policy updates. To counteract this effect, we restrict the number of assessment episodes used during the initial phase of training before increasing it.
Additional details of our approach to policy checkpoints can be found in Appendix F.
5.2 TD7
TD7 is based on TD3 [Fujimoto et al., 2018] with LAP [Fujimoto et al., 2020], a behavior cloning term for offline RL [Fujimoto and Gu, 2021], SALE (Section 4.1), and policy checkpoints (Section 5.1).
LAP. Gathered experience is stored in a replay buffer [Lin, 1992] and sampled according to LAP [Fujimoto et al., 2020], a prioritized replay buffer  $D$  [Schaul et al., 2016] where a transi
Algorithm 1 Online TD7
1: Initialize: ▷ Before training
· Policy πt+1, value function Qt+1, encoders (ft+1, gt+1).
· Target policy πt, target value function Qt, fixed encoders (ft, gt), target fixed encoders (ft-1, gt-1).
· Checkpoint policy πc, checkpoint encoder fc.  
2: for episode = 1 to final Episode do ▷ Data collection  
3: Using current policy πt+1, collect transitions and store in the LAP replay buffer.  
4: if checkpoint_condition then ▷ Checkpointing  
5: if actor πt+1 outperforms checkpoint policy πc then  
6: Update checkpoint networks πc ← πt+1, fc ← ft.  
7: for i = 1 to timesteps_since_training do ▷ Training  
8: Sample transitions from LAP replay buffer (Equation 10).  
9: Train encoder (Equation 2), value function (Equations 6 & 9), and policy (Equation 11).  
10: if target_update频率 steps have passed then  
11: Update target networks (Equation 8).
$\triangleright$  Detailed hyperparameter explanations found in the Appendix.
tion tuple  $i\coloneqq (s,a,r,s^{\prime})$  is sampled with probability
$$
p (i) = \frac {\operatorname* {m a x} \left(| \delta (i) | ^ {\alpha} , 1\right)}{\sum_ {j \in D} \operatorname* {m a x} \left(| \delta (j) | ^ {\alpha} , 1\right)}, \quad \text {w h e r e} \delta (i) := Q (s, a) - y, \tag {10}
$$
where  $y$  is the learning target. The amount of prioritization used is controlled by a hyperparameter  $\alpha$ . Furthermore, the value function loss uses the Huber loss [Huber et al., 1964], rather than the MSE.
Offline RL. To make TD7 amenable to the offline RL setting, we add a behavior cloning loss to the policy update [Silver et al., 2014], inspired by TD3+BC [Fujimoto and Gu, 2021]:
$$
\pi \approx \underset {\pi} {\operatorname {a r g m a x}} \mathbb {E} _ {(s, a) \sim D} \left[ Q (s, \pi (s)) - \lambda | \mathbb {E} _ {s \sim D} [ Q (s, \pi (s)) ] | _ {\times} (\pi (s) - a) ^ {2} \right]. \tag {11}
$$
The same loss function is used for both offline and online RL, where  $\lambda = 0$  for the online setting.  $|\cdot |_{\times}$  denotes the stop-gradient operation. Unlike TD3+BC, we do not normalize the state vectors. Checkpoints are not used in the offline setting, as there is no interaction with the environment.
Both the value function and policy use the SALE embeddings as input, which we omit from the equations above for simplicity. Pseudocode for TD7 is described in Algorithm 1.
In this section, we evaluate the performance of TD7 in both the online and offline regimes. A detailed description of the experimental setup, baselines, and hyperparameters can be found in the Appendix, along with additional learning curves and ablation studies.
Online. Using OpenAI gym [Brockman et al., 2016], we benchmark TD7 against TD3 [Fujimoto et al., 2018], SAC [Haarnoja et al., 2018], TQC [Kuznetsov et al., 2020], and  $\mathrm{TD3 + OFE}$  [Ota et al., 2020] on the MuJoCo environments [Todorov et al., 2012]. SAC and  $\mathrm{TD3 + OFE}$  results are from re-implementations based on author descriptions [Haarnoja et al., 2018, Ota et al., 2020]. TD3 and TQC results use author-provided code [Fujimoto et al., 2018, Kuznetsov et al., 2020], with a consistent evaluation protocol for all methods. Learning curves are displayed in Figure 4 and final and intermediate results are listed in Table 1.
Although TQC and TD3+OFE use per-environment hyperparameters along with larger and more computationally expensive architectures, TD7 outperforms these baselines significantly in terms of both early (300k time steps) and final performance (5M time steps). At 300k time steps, TD7 often surpasses the performance of TD3 at 5M time steps, highlighting the considerable performance gains.
Offline. We benchmark TD7 against CQL [Kumar et al., 2020b], TD3+BC [Fujimoto and Gu, 2021], IQL [Kostrikov et al., 2021] and  $\mathcal{X}$ -QL [Garg et al., 2023] using the MuJoCo datasets in D4RL [Todorov et al., 2012, Fu et al., 2021]. While there are methods that use per-dataset
Figure 4: Learning curves on the MuJoCo benchmark. Results are averaged over 10 seeds. The shaded area captures a  $95\%$  confidence interval around the average performance.
Table 1: Average performance on the MuJoCo benchmark at  $300\mathrm{k}$ , 1M, and 5M time steps, over 10 trials, where  $\pm$  captures a  $95\%$  confidence interval. The highest performance is highlighted. Any performance which is not statistically significantly worse than the highest performance (according to a Welch's  $t$ -test with significance level 0.05) is highlighted.
Environment	Time step	TD3	SAC	TQC	TD3+OFE	TD7
HalfCheetah	300k	7715 ± 633	8052 ± 515	7006 ± 891	11294 ± 247	15031 ± 401
1M	10574 ± 897	10484 ± 659	12349 ± 878	13758 ± 544	17434 ± 155
5M	14337 ± 1491	15526 ± 697	17459 ± 258	16596 ± 164	18165 ± 255
Hopper	300k	1289 ± 768	2370 ± 626	3251 ± 461	1581 ± 682	2948 ± 464
1M	3226 ± 315	2785 ± 634	3526 ± 244	3121 ± 506	3512 ± 315
5M	3682 ± 83	3167 ± 485	3462 ± 818	3423 ± 584	4075 ± 225
Walker2d	300k	1101 ± 386	1989 ± 500	2812 ± 838	4018 ± 570	5379 ± 328
1M	3946 ± 292	4314 ± 256	5321 ± 322	5195 ± 512	6097 ± 570
5M	5078 ± 343	5681 ± 329	6137 ± 1194	6379 ± 332	7397 ± 454
Ant	300k	1704 ± 655	1478 ± 354	1830 ± 572	6348 ± 441	6171 ± 831
1M	3942 ± 1030	3681 ± 506	3582 ± 1093	7398 ± 118	8509 ± 422
5M	5589 ± 758	4615 ± 2022	6329 ± 1510	8547 ± 84	10133 ± 966
Humanoid	300k	1344 ± 365	1997 ± 483	3117 ± 910	3181 ± 771	5332 ± 714
1M	5165 ± 145	4909 ± 364	6029 ± 531	6032 ± 334	7429 ± 153
5M	5433 ± 245	6555 ± 279	8361 ± 1364	8951 ± 246	10281 ± 588
hyperparameters to attain higher total results, we omit these methods because it makes it difficult to directly compare. Baseline results are obtained by re-running author-provided code with a single set of hyperparameters and a consistent evaluation protocol. Final performance is reported in Table 2.
TD7 outperforms all baselines. Since TD7 and  $\mathrm{TD3 + BC}$  employ the same approach to offline RL, the significant performance gap highlights the effectiveness of SALE in the offline setting.
Ablation study. In Figure 5 we report the results of an ablation study over the components of TD7 (SALE, checkpoints, LAP). The interaction between components is explored further in Appendix G.
Run time. To understand the computational cost of using SALE and the TD7 algorithm, we benchmark the run time of each of the online baselines with identical computational resources and deep learning framework. The results are reported in Figure 6.
Table 2: Average final performance on the D4RL benchmark after training for 1M time steps. over 10 trials, where  $\pm$  captures a  $95\%$  confidence interval. The highest performance is highlighted. Any performance which is not statistically significantly worse than the highest performance (according to a Welch's  $t$  -test with significance level 0.05) is highlighted.
Environment	Dataset	CQL	TD3+BC	IQL	X-QL	TD7
HalfCheetah	Medium	46.7 ± 0.3	48.1 ± 0.1	47.4 ± 0.2	47.4 ± 0.1	58.0 ± 0.4
Medium-Replay	45.5 ± 0.3	44.6 ± 0.4	43.9 ± 1.3	44.2 ± 0.7	53.8 ± 0.8
Medium-Expert	76.8 ± 7.4	93.7 ± 0.9	89.6 ± 3.5	90.2 ± 2.7	104.6 ± 1.6
Hopper	Medium	59.3 ± 3.3	59.1 ± 3.0	63.9 ± 4.9	67.7 ± 3.6	76.1 ± 5.1
Medium-Replay	78.8 ± 10.9	52.0 ± 10.6	93.4 ± 7.8	82.0 ± 14.9	91.1 ± 8.0
Medium-Expert	79.9 ± 19.8	98.1 ± 10.7	64.2 ± 32.0	92.0 ± 10.0	108.2 ± 4.8
Walker2d	Medium	81.4 ± 1.7	84.3 ± 0.8	84.2 ± 1.6	79.2 ± 4.0	91.1 ± 7.8
Medium-Replay	79.9 ± 3.6	81.0 ± 3.4	71.2 ± 8.3	61.8 ± 7.7	89.7 ± 4.7
Medium-Expert	108.5 ± 1.2	110.5 ± 0.4	108.9 ± 1.4	110.3 ± 0.2	111.8 ± 0.6
Total	656.7 ± 24.3	671.3 ± 15.7	666.7 ± 34.6	674.9 ± 20.4	784.4 ± 14.1
Representation learning has been typically reserved for image-based tasks, where the observations are large and unstructured. However, by learning embeddings which consider the interaction between state and action, we make representation learning more broadly applicable to low-level states. We introduce SALE, a method for learning state-action embeddings by considering a latent space dynamics model. Through an extensive empirical evaluation, we investigate various design choices in SALE.
We highlight the risk of extrapolation error [Fujimoto et al., 2019] due to the increase in input dimensions from using state-action embeddings, but show this instability can be corrected by clipping the target with seen values. We further introduce stability by including policy checkpoints.
While both SALE and policy checkpoints are general-purpose techniques that can be included with most RL methods, we combine them with TD3 and several other recent improvements [Fujimoto et al., 2020, Fujimoto and Gu, 2021] to introduce the TD7 algorithm. We find our TD7 algorithm is able to match the performance of expensive offline algorithms and significantly outperform the state-of-the-art continuous control methods in both final performance and early learning.
This research was enabled in part by support provided by Calcul Quebec and the Digital Research Alliance of Canada.
Figure 5: Ablation study over the components of TD7. The y-axis corresponds to the average performance over all five MuJoCo tasks, normalized with respect to the performance of TD7 at 5M time steps. The shaded area captures a  $95\%$  confidence interval. The impact can be ranked (1) SALE, (2) LAP, (3) policy checkpoints.
Figure 6: Run time of each method for 1M time steps on the HalfCheetah environment, using the same hardware and deep learning framework (PyTorch [Paszke et al., 2019]).
Herke Van Hoof, Nutan Chen, Maximilian Karl, Patrick van der Smagt, and Jan Peters. Stable reinforcement learning with autoencoders for tactile and visual data. In 2016 IEEE/RSJ international conference on intelligent robots and systems (IROS), pages 3928-3934. IEEE, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Manuel Watter, Jost Springenberg, Joschka Boedecker, and Martin Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. Advances in neural information processing systems, 28, 2015.
William Whitney, Rajat Agarwal, Kyunghyun Cho, and Abhinav Gupta. Dynamics-aware embeddings. In International Conference on Learning Representations, 2020.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229-256, 1992.
Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Improving sample efficiency in model-free reinforcement learning from images. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 10674-10681, 2021.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. In International Conference on Learning Representations, 2022.
Amy Zhang, Harsh Satija, and Joelle Pineau. Decoupling dynamics and reward for transfer learning. arXiv preprint arXiv:1804.10689, 2018.
Amy Zhang, Rowan Thomas McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. In International Conference on Learning Representations, 2020.
A TD7 Additional Details 17
A.1 Algorithm 17
A.2 Hyperparameters 19
B Experimental Details 21
CBaselines 22
C.1 TD3 & TD3+OFE Hyperparameters 22
C.2 SAC & TQC Hyperparameters 25
C.3 Offline RL Baselines 28
D Design Study 29
D.1 Learning Target 30
D.2 Network Input 32
D.3 Normalization 34
D.4 End-to-end 37
E Extrapolation Error Study 38
E.1 Extrapolation Error 38
E.2 Empirical Analysis 38
F Policy Checkpoints 41
F.1 Motivation 41
F.2 Method 41
F.3 Empirical Analysis 43
G Ablation Study 46
G.1 Main Components 46
G.2 Subcomponents 47
H Offline RL Learning Curves 50
I Run time 51
J Limitations 52
A TD7 Additional Details
A.1 Algorithm
TD7 (TD3+4 additions) has several networks and sub-components:
- Two value functions  $(Q_{t + 1,1}, Q_{t + 1,2})$ .
- Two target value functions  $(Q_{t,1}, Q_{t,2})$ .
A policy network  $\pi_{t + 1}$
- A target policy network  $\pi_t$ .
- An encoder, with sub-components  $(f_{t + 1},g_{t + 1})$
- A fixed encoder, with sub-components  $(f_t, g_t)$ .
- A target fixed encoder with sub-components  $(f_{t - 1},g_{t - 1})$
- A checkpoint policy  $\pi_c$  and checkpoint encoder  $f_c$  ( $g$  is not needed).
Encoder: The encoder is composed of two sub-networks  $(f_{t + 1}(s),g_{t + 1})(z^s,a)$  , where each network outputs an embedding:
$$
z ^ {s} := f (s), \quad z ^ {s a} := g \left(z ^ {s}, a\right). \tag {12}
$$
At each training step, the encoder is updated with the following loss:
$$
\begin{array}{l} \mathcal {L} \left(f _ {t + 1}, g _ {t + 1}\right) := \left(g _ {t + 1} \left(f _ {t + 1} (s), a\right) - \left| f _ {t + 1} \left(s ^ {\prime}\right) \right| _ {\times}\right) ^ {2} (13) \\ = \left(z _ {t + 1} ^ {s a} - \left| z _ {t + 1} ^ {s ^ {\prime}} \right| _ {\times}\right) ^ {2}, (14) \\ \end{array}
$$
where  $|\cdot|_{\times}$  is the stop-gradient operation.
Value function: TD7 uses a pair of value functions (as motivated by TD3 [Fujimoto et al., 2018])  $(Q_{t + 1,1}, Q_{t + 1,2})$ , each taking input  $[z_{t - 1}^{s' a'}, z_{t - 1}^{s'}, s', a']$ . At each training step, both value functions are updated with the following loss:
$$
\mathcal {L} \left(Q _ {t + 1}\right) := \operatorname {H u b e r} \left(\operatorname {t a r g e t} - Q _ {t + 1} \left(z _ {t} ^ {s a}, z _ {t} ^ {s}, s, a\right)\right), \tag {15}
$$
$$
\text {t a r g e t} := r + \gamma \operatorname {c l i p} \left(\min  \left(Q _ {t, 1} (x), Q _ {t, 2} (x)\right), Q _ {\min }, Q _ {\max }\right), \tag {16}
$$
$$
x := \left[ z _ {t - 1} ^ {s ^ {\prime} a ^ {\prime}}, z _ {t - 1} ^ {s ^ {\prime}}, s ^ {\prime}, a ^ {\prime} \right], \tag {17}
$$
$$
a ^ {\prime} := \pi_ {t} \left(z _ {t - 1} ^ {s ^ {\prime}}, s ^ {\prime}\right) + \epsilon , \tag {18}
$$
$$
\epsilon \sim \operatorname {c l i p} \left(\mathcal {N} \left(0, \sigma^ {2}\right), - c, c\right). \tag {19}
$$
Taking the minimum of the value functions is from TD3's Clipped Double Q-learning (CDQ) [Fujimoto et al., 2018]). The use of Huber loss [Huber et al., 1964] is in accordance to TD3 with the Loss-Adjusted Prioritized (LAP) experience replay [Fujimoto et al., 2020].  $a'$  is sampled and clipped in the same manner as TD3 [Fujimoto et al., 2018]. The same embeddings  $(z_t^{sa}, z_t^s)$  are used for each value function.  $Q_{\mathrm{min}}$  and  $Q_{\mathrm{max}}$  are updated at each time step:
$$
Q _ {\min } \leftarrow \min  \left(Q _ {\min }, \text {t a r g e t}\right), \tag {20}
$$
$$
Q _ {\max } \leftarrow \max  \left(Q _ {\max }, \text {t a r g e t}\right), \tag {21}
$$
where target is defined by Equation 16.
Policy: TD7 uses a single policy network which takes input  $[z^s, s]$ . On every second training step (according to TD3's delayed policy updates [Fujimoto et al., 2018]) the policy  $\pi_{t+1}$  is updated with the following loss:
$$
\mathcal {L} \left(\pi_ {t + 1}\right) := - Q + \lambda \left| \mathbb {E} _ {s \sim D} [ Q ] \right| _ {\times} \left(a _ {\pi} - a\right) ^ {2}, \tag {22}
$$
$$
Q := 0. 5 \left(Q _ {t + 1, 1} (x) + Q _ {t + 1, 2} (x)\right) \tag {23}
$$
$$
x := \left[ z _ {t} ^ {s a _ {\pi}}, z _ {t} ^ {s}, s, a _ {\pi} \right], \tag {24}
$$
$$
a _ {\pi} := \pi_ {t + 1} \left(z _ {t} ^ {s}, s\right). \tag {25}
$$
The policy loss is the deterministic policy gradient (DPG) [Silver et al., 2014] with a behavior cloning term to regularize [Fujimoto and Gu, 2021].  $\lambda$  is set to 0 for online RL.
After every target_update-frequency (250) training steps, the iteration is updated and each target (and fixed) network copies the network of the higher iteration:
$$
\left(Q _ {t, 1}, Q _ {t, 2}\right) \leftarrow \left(Q _ {t + 1, 1}, Q _ {t + 1, 2}\right), \tag {26}
$$
$$
\pi_ {t} \leftarrow \pi_ {t + 1}, \tag {27}
$$
$$
\left(f _ {t - 1}, g _ {t - 1}\right) \leftarrow \left(f _ {t}, g _ {t}\right), \tag {28}
$$
$$
\left(f _ {t}, g _ {t}\right) \leftarrow \left(f _ {t + 1}, g _ {t + 1}\right). \tag {29}
$$
The checkpoint policy and checkpoint encoder are only used at test time (see Appendix F).
LAP: Gathered experience is stored in a replay buffer [Lin, 1992] and sampled according to LAP [Fujimoto et al., 2020], a prioritized replay buffer  $D$  [Schaul et al., 2016] where a transition tuple  $i \coloneqq (s, a, r, s')$  is sampled with probability
$$
p (i) = \frac {\operatorname* {m a x} \left(| \delta (i) | ^ {\alpha} , 1\right)}{\sum_ {j \in D} \operatorname* {m a x} \left(| \delta (j) | ^ {\alpha} , 1\right)}, \tag {30}
$$
$$
| \delta (i) | := \max  \left(| Q _ {t + 1, 1} \left(z _ {t} ^ {s a}, z _ {t} ^ {s}, s, a\right) - \text {t a r g e t} |, | Q _ {t + 1, 2} \left(z _ {t} ^ {s a}, z _ {t} ^ {s}, s, a\right) - \text {t a r g e t} |\right), \tag {31}
$$
where target is defined by Equation 16. As suggested by Fujimoto et al. [2020],  $|\delta(i)|$  is defined by the maximum absolute error of both value functions. The amount of prioritization used is controlled by a hyperparameter  $\alpha$ . New transitions are assigned the maximum priority of any sample in the replay buffer.
We outline the train function of TD7 in Algorithm 2. There is no difference between the train function of online TD7 and offline TD7 other than the value of  $\lambda$ , which is 0 for online and 0.1 for offline.
Algorithm 2 TD7 Train Function
1: Sample transition from LAP replay buffer with probability (Equation 30).  
2: Train encoder (Equation 13).  
3: Train value function (Equation 15).  
4: Update  $(Q_{\mathrm{min}}, Q_{\mathrm{max}})$  (Equations 20 & 21).  
5: if  $i$  mod policy_update频率 = 0 then  
6: Train policy (Equation 22).  
7: if  $i$  mod target_update频率 = 0 then  
8: Update target networks (Equation 26).
A.2 Hyperparameters
The action space is assumed to be in the range  $[-1, 1]$  (and is normalized if otherwise). Besides a few exceptions mentioned below, hyperparameters (and architecture) are taken directly from TD3 https://github.com/sfujim/TD3.
Table 3: TD7 Hyperparameters.
Hyperparameter	Value
TD3[Fujimoto et al., 2018]	Target policy noise σ	N(0,0.22)
Target policy noise clipping c	(-0.5,0.5)
Policy update frequency	2
LAP[Fujimoto et al., 2020]	Probability smoothing α	0.4
Minimum priority	1
TD3+BC[Fujimoto and Gu, 2021]	Behavior cloning weight λ (Online)	0.0
Behavior cloning weight λ (Offline)	0.1
Policy Checkpoints(Appendix F)	Checkpoint criteria	minimum
Early assessment episodes	1
Late assessment episodes	20
Early time steps	750k
Criteria reset weight	0.9
Exploration	Initial random exploration time steps	25k
Exploration noise	N(0,0.12)
Common	Discount factor γ	0.99
Replay buffer capacity	1M
Mini-batch size	256
Target update frequency	250
Optimizer	(Shared) Optimizer	Adam [Kingma and Ba, 2014]
(Shared) Learning rate	3e-4
Besides algorithmic difference from TD3, there are three implementation-level changes:
1. Rather than only using  $Q_{1}$ , both value functions are used when updating the policy (Equation 22).
2. The value function uses ELU activation functions [Clevert et al., 2015] rather than ReLU activation functions.
3. The target network is updated periodically (every 250 time steps) rather than using an exponential moving average at every time step. This change is necessary due to the use of fixed encoders.
We evaluate the importance of these changes in our ablation study (Appendix G). Network architecture details are described in Pseudocode 1.