SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\hat {V} ^ {\pi} (s) \leq V ^ {\pi} (s) + \left(I - \gamma P ^ {\pi}\right) ^ {- 1} E _ {a \sim \pi (a | s)} \frac {C _ {r , t , \delta} R _ {\max}}{(1 - \gamma) \sqrt {| D (s , a) |}} - \alpha \left(I - \gamma P ^ {\pi}\right) ^ {- 1} \left[ \frac {d (s)}{d _ {u} (s)} - 1 \right] \tag {20}
$$
, where  $P^{\pi}$  is the transition matrix coupled with the policy  $\pi$  and  $P^{\pi}V(s) = E_{a^{\prime}\sim \pi (a^{\prime}|s^{\prime})s^{\prime}\sim P(s^{\prime}|s,a^{\prime})}[V(s^{\prime})]$ .
Then the expectation of  $V^{\pi}(s)$  under distribution  $d(s)$  satisfies:
$$
\begin{array}{l} E _ {s \sim d (s)} \hat {V} ^ {\pi} (s) \leq E _ {s \sim d (s)} (V ^ {\pi} (s)) + E _ {s \sim d (s)} (I - \gamma P ^ {\pi}) ^ {- 1} E _ {a \sim \pi (a | s)} \frac {C _ {r , t , \delta} R _ {m a x}}{(1 - \gamma) \sqrt {| D (s , a) |}} \\ - \alpha \underbrace {E _ {s \sim d (s)} \left(I - \gamma P ^ {\pi}\right) ^ {- 1} \left[ \frac {d (s)}{d _ {u} (s)} - 1 \right]} _ {> 0} \tag {21} \\ \end{array}
$$
$$
\mathrm {W h e n} \alpha \geq \frac {E _ {s \sim d (s)} E _ {a \sim \pi (a | s)} \frac {C _ {r , t , \delta R _ {m a x}}}{(1 - \gamma) \sqrt {| D (s , a) |}}}{E _ {s \sim d (s)} [ \frac {d (s)}{d _ {u} (s)} - 1 ])}, E _ {s \sim d (s)} \hat {V} ^ {\pi} (s) \leq E _ {s \sim d (s)} (V ^ {\pi} (s)).
$$
Proof of Theorem 3.3: The expectation of  $V^{\pi}(s)$  under distribution  $d(s)$  satisfies:
$$
\begin{array}{l} E _ {s \sim d _ {u} (s)} \hat {V} ^ {\pi} (s) \leq E _ {s \sim d _ {u} (s)} \left(V ^ {\pi} (s)\right) + E _ {s \sim d _ {u} (s)} \left(I - \gamma P ^ {\pi}\right) ^ {- 1} E _ {a \sim \pi (a | s)} \frac {C _ {r , t , \delta} R _ {\max}}{(1 - \gamma) \sqrt {| D (s , a) |}} \tag {22} \\ - \alpha E _ {s \sim d _ {u} (s)} (I - \gamma P ^ {\pi}) ^ {- 1} \left[ \frac {d (s)}{d _ {u} (s)} - 1 \right]) \\ \end{array}
$$
Noticed that the last term:
$$
\sum_ {s \sim d _ {u} (s)} \left(\frac {d _ {f} (s)}{d _ {u} (s)} - 1\right) = \sum_ {s} d _ {u} (s) \left(\frac {d _ {f} (s)}{d _ {u} (s)} - 1\right) = \sum_ {s} d _ {f} (s) - \sum_ {s} d _ {u} (s) = 0 \tag {23}
$$
We obtain that:
$$
E _ {s \sim d _ {u} (s)} \hat {V} ^ {\pi} (s) \leq E _ {s \sim d _ {u} (s)} \left(V ^ {\pi} (s)\right) + E _ {s \sim d _ {u} (s)} \left(I - \gamma P ^ {\pi}\right) ^ {- 1} E _ {a \sim \pi (a | s)} \frac {C _ {r , t , \delta} R _ {m a x}}{(1 - \gamma) \sqrt {| D (s , a) |}} \tag {24}
$$
Proof of Theorem 3.4: Recall that the expression of the V-function iterate is given by:
$$
\hat {V} ^ {k + 1} (s) = \mathcal {B} ^ {\pi^ {k}} \hat {V} ^ {k} (s) - \alpha \left[ \frac {d (s)}{d _ {u} (s)} - 1 \right] \forall s, k \tag {25}
$$
Now the expectation of  $V^{\pi}(s)$  under distribution  $d_{u}(s)$  is given by:
$$
E _ {s \sim d _ {u} (s)} \hat {V} ^ {k + 1} (s) = E _ {s \sim d _ {u} (s)} \left[ \mathcal {B} ^ {\pi^ {k}} \hat {V} ^ {k} (s) - \alpha \left[ \frac {d (s)}{d _ {u} (s)} - 1 \right] \right] = E _ {s \sim d _ {u} (s)} \mathcal {B} ^ {\pi^ {k}} \hat {V} ^ {k} (s) \tag {26}
$$
The expectation of  $V^{\pi}(s)$  under distribution  $d(s)$  is given by:
$$
E _ {s \sim d (s)} \hat {V} ^ {k + 1} (s) = E _ {s \sim d (s)} \mathcal {B} ^ {\pi^ {k}} \hat {V} ^ {k} (s) - \alpha \left[ \frac {d (s)}{d _ {u} (s)} - 1 \right] = E _ {s \sim d (s)} \mathcal {B} ^ {\pi^ {k}} \hat {V} ^ {k} (s) - \alpha E _ {s \sim d (s)} \left[ \frac {d (s)}{d _ {u} (s)} - 1 \right] \tag {27}
$$
Thus we can show that:
$$
\begin{array}{l} E _ {s \sim d _ {u} (s)} \hat {V} ^ {k + 1} (s) - E _ {s \sim d (s)} \hat {V} ^ {k + 1} (s) = E _ {s \sim d _ {u} (s)} \mathcal {B} ^ {\pi^ {k}} \hat {V} ^ {k} (s) - E _ {s \sim d (s)} \mathcal {B} ^ {\pi^ {k}} \hat {V} ^ {k} (s) + \alpha E _ {s \sim d (s)} [ \frac {d (s)}{d _ {u} (s)} - 1 ] \\ = E _ {s \sim d _ {u} (s)} V ^ {k + 1} (s) - E _ {s \sim d (s)} V ^ {k + 1} (s) - E _ {s \sim d (s)} \left[ \mathcal {B} ^ {\pi^ {k}} \left(\hat {V} ^ {k} - V ^ {k}\right) \right] \\ + E _ {s \sim d _ {u} (s)} \left[ \mathcal {B} ^ {\pi^ {k}} \left(\hat {V} ^ {k} - V ^ {k}\right) \right] + \alpha E _ {s \sim d (s)} \left[ \frac {d (s)}{d _ {u} (s)} - 1 \right] \tag {28} \\ \end{array}
$$
By choosing  $\alpha$ :
$$
\alpha > \frac {E _ {s \sim d (s)} \left[ \mathcal {B} ^ {\pi^ {k}} \left(\hat {V} ^ {k} - V ^ {k}\right) \right] - E _ {s \sim d _ {u} (s)} \left[ \mathcal {B} ^ {\pi^ {k}} \left(\hat {V} ^ {k} - V ^ {k}\right) \right]}{E _ {s \sim d (s)} \left[ \frac {d (s)}{d _ {u} (s)} - 1 \right]} \tag {29}
$$
We have  $E_{s \sim d_u(s)} \hat{V}^{k+1}(s) - E_{s \sim d(s)} \hat{V}^{k+1}(s) > E_{s \sim d_u(s)} V^{k+1}(s) - E_{s \sim d(s)} V^{k+1}(s)$  hold.
Proof of Theorem 3.5:  $\hat{V}$  is obtained by solving the recursive Bellman fixed point equation in the empirical MDP, with an altered reward,  $r(s,a) - \alpha \left[\frac{d(s)}{d_u(s)} -1\right]$ , hence the optimal policy  $\pi^{*}(a|s)$  obtained by optimizing the value under Eq. 3.5.
Proof of Theorem 3.6: The proof of this statement is divided into two parts. We first relates the return of  $\pi^{*}$  in the empirical MDP  $\hat{M}$  with the return of  $\pi_{\beta}$ , we can get:
$$
J \left(\pi^ {*}, \hat {M}\right) - \alpha \frac {1}{1 - \gamma} \mathbb {E} _ {s \sim d _ {\hat {M}} ^ {*} (s)} [ \frac {d (s)}{d _ {u} (s)} - 1 ] \geq J \left(\pi_ {\beta}, \hat {M}\right) - 0 = J \left(\pi_ {\beta}, \hat {M}\right) \tag {30}
$$
Algorithm 1 CSVE based Offline RL Algorithm
Input: data  $D = \{(s,a,r,s')\}$    
Parametered Models:  $Q_{\theta},V_{\psi},\pi_{\phi},Q_{\overline{\theta}},M_{\nu}$    
Hyperparameters:  $\alpha ,\lambda$  , learning rates  $\eta_{\theta},\eta_{\psi},\eta_{\phi},\omega$    
Train the transition model with the static dataset  $D$ $M_{\nu}\gets \mathrm{train}(D)$    
Train the conservative value estimation and policy functions. Initialize function parameters  $\theta_0,\psi_0,\phi_0,\bar{\theta}_0 = \theta_0$    
for step  $k = 1$  to  $N$  do   
 $\psi_k\gets \psi_{k - 1} - \eta_\psi \nabla_\psi L_V^\pi (V_\psi ;\overline{\hat{Q}_{\theta_k}})$ $\theta_{k}\gets \theta_{k - 1} - \eta_{\theta}\nabla_{\theta}L_{Q}^{\pi}(Q_{\theta};\hat{V}_{\psi_{k}})$ $\phi_k\gets \phi_{k - 1} - \eta_\phi \nabla_\phi L_\pi^+ (\pi_\phi)$ $\bar{\theta}_k\gets \omega \bar{\theta}_{k - 1} + (1 - \omega)\theta_k$    
end for
The next step is to bound the difference between  $J(\pi_{\beta}, \hat{M})$  and  $J(\pi_{\beta}, M)$  and the difference between  $J(\pi^{*}, \hat{M})$  and  $J(\pi^{*}, M)$ . We quote a useful lemma from [4] (Lemma D.4.1):
Lemma A.3. For any MDP  $M$ , an empirical MDP  $\hat{M}$  generated by sampling actions according to the behavior policy  $\pi_{\beta}$  and a given policy  $\pi$ ,
$$
| J (\pi , \hat {M}) - J (\pi , M) | \leq \left(\frac {C _ {r , \delta}}{1 - \gamma} + \frac {\gamma R _ {\max} C _ {T , \delta}}{(1 - \gamma) ^ {2}}\right) \mathbb {E} _ {s \sim d _ {\hat {M}} ^ {\pi^ {*}} (s)} \left[ \frac {\sqrt {| \mathcal {A} |}}{\sqrt {| \mathcal {D} (s) |}} \sqrt {E _ {a \sim \pi (a | s)} \left(\frac {\pi (a | s)}{\pi_ {\beta} (a | s)}\right)} \right] \tag {31}
$$
Setting  $\pi$  in the above lemma as  $\pi_{\beta}$ , we get:
$$
| J (\pi_ {\beta}, \hat {M}) - J (\pi_ {\beta}, M) | \leq \left(\frac {C _ {r , \delta}}{1 - \gamma} + \frac {\gamma R _ {\max} C _ {T , \delta}}{(1 - \gamma) ^ {2}}\right) \mathbb {E} _ {s \sim d _ {\hat {M}} ^ {\pi^ {*}} (s)} [ \frac {\sqrt {| \mathcal {A} |}}{\sqrt {| \mathcal {D} (s) |}} \sqrt {E _ {a \sim \pi^ {*} (a | s)} \left(\frac {\pi^ {*} (a | s)}{\pi_ {\beta} (a | s)}\right)} ] \tag {32}
$$
given that  $\sqrt{E_{a\sim\pi^{*}(a|s)}[\frac{\pi^{*}(a|s)}{\pi_{\beta}(a|s)}}$  is a pointwise upper bound of  $\sqrt{E_{a\sim\pi_{\beta}(a|s)}[\frac{\pi_{\beta}(a|s)}{\pi_{\beta}(a|s)}}$  ([4]). Thus we get,
$$
\begin{array}{l} J (\pi^ {*}, \hat {M}) \geq J (\pi_ {\beta}, \hat {M}) - 2 (\frac {C _ {r , \delta}}{1 - \gamma} + \frac {\gamma R _ {m a x} C _ {T , \delta}}{(1 - \gamma) ^ {2}}) \mathbb {E} _ {s \sim d _ {\hat {M}} ^ {\pi^ {*}} (s)} [ \frac {\sqrt {| \mathcal {A} |}}{\sqrt {| \mathcal {D} (s) |}} \sqrt {E _ {a \sim \pi^ {*} (a | s)} (\frac {\pi^ {*} (a | s)}{\pi_ {\beta} (a | s)})} ] \\ + \alpha \frac {1}{1 - \gamma} \mathbb {E} _ {s \sim d _ {\hat {M}} ^ {\pi} (s)} [ \frac {d (s)}{d _ {u} (s)} - 1 ] \tag {33} \\ \end{array}
$$
Here, the second term represents the sampling error, which arises due to the discrepancy between  $\hat{M}$  and  $M$ . The third term signifies the enhancement in policy performance attributed to our algorithm in  $\hat{M}$ . It's worth noting that when the first term is minimized, smaller values of  $\alpha$  can achieve improvements over the behavior policy.
B CSVE Algorithm and Implementation Details
In Section 4, we d e t e i l e d the c o m p l e x f o r m a l d e s c r i p t i o n s o f t h e C S V E p r a c t i c a l o f f l n e RL algorithm. Here we consolidate those details and present the full deep offline reinforcement learning algorithm as illustrated in Alg. 1. In particular, the dynamic model, value functions, and policy are parameterized with deep neural networks and optimized via stochastic gradient descent methods.
We implement our method based on an offline deep reinforcement learning library d3rlpy [34]. The code is available at: https://github.com/2023AnonymousAuthor/csve.
Table 3: Hyper-parameters of CSVE evaluation
Hyper-parameters	Value and description
B	5, number of ensembles in dynamics model
α	10, to control the penalty of OOD states
τ	10, budget parameter in Eq. 8
β	In Gym domain, 3 for random and medium tasks, 0.1 for the other tasks; In Adroit domain, 30 for human and cloned tasks, 0.01 for expert tasks
γ	0.99, discount factor.
H	1 million for Mujoco while 0.1 million for Adroit tasks.
w	0.005, target network smoothing coefficient.
lr of actor	3e-4, policy learning rate
lr of critic	1e-4, critic learning rate
C Additional Ablation Study
Effect of model errors. Compared to traditional model-based offline RL algorithms, CSVE exhibits greater resilience to model biases. To access this resilience quantitatively, we measured the performance impact of model biases using the average L2 error in transition prediction as an indicator. As shown in Fig. 3, the influence of model bias on RL performance is CSVE is marginal. Specifically, in the halfcheetah task, there is no observable impact of model errors on scores, model errors show no discernible impact on scores. For the hopper and walker2d tasks, only a minor decline in scores is observed as the errors escalate.
Figure 3: Effect of the model biases to performance scores. The correlation coefficient is  $-0.32$ ,  $-0.34$ , and  $-0.29$  respectively.
D Experimental Details and Complementary Results
D.1 Hyper-parameters of CSVE evaluation in experiments
Table 3 provides a detailed breakdown of the hyper-parameters used for evaluating CSVE in our experiments.
D.2 Details of Baseline CQL-AWR
To facilitate a direct comparison between the effects of conservative state value estimation and Q-value estimation, we formulated a baseline method named CQL-AWR as detailed below:
$$
\begin{array}{l} \hat {Q} ^ {k + 1} \gets \arg \min _ {Q} \alpha \left(E _ {s \sim D, a \sim \pi (a | s)} [ Q (s, a) ] - E _ {s \sim D, a \sim \hat {\pi} _ {\beta} (a | s)} [ Q (s, a) ]\right) + \frac {1}{2} E _ {s, a, s ^ {\prime} \sim D} [ (Q (s, a) - \hat {\beta} _ {\pi} \hat {Q} ^ {k} (s, a)) ^ {2} ] \\ \pi \gets \arg \min _ {\pi^ {\prime}} L _ {\pi} (\pi^ {\prime}) = - E _ {s, a \sim D} \left[ \log \pi^ {\prime} (a | s) \exp \left(\beta \hat {A} ^ {k + 1} (s, a)\right) \right] - \lambda E _ {s \sim D, a \sim \pi^ {\prime} (s)} \left[ \hat {Q} ^ {k + 1} (s, a) \right] \\ \mathrm {w h e r e} \hat {A} ^ {k + 1} (s, a) = \hat {Q} ^ {k + 1} (s, a) - \mathbb {E} _ {a \sim \pi} [ \hat {Q} ^ {k + 1} (s, a) ]. \\ \end{array}
$$
Figure 4: Effect of  $\lambda$  to Score (upper figures) and  $L_{\pi}$  loss in Eq. 9 (bottom figures)
In CQL-AWR, the critic adopts a standard CQL equation, while the policy improvement part uses an AWR extension combined with novel action exploration as denoted by the conservative Q function. When juxtaposed with our CSVE implementation, the policy segment of CQL-AWR mirrors ours, with the primary distinction being that its exploration is rooted in a Q-based and model-free approach.
D.3 Reproduction of COMBO
In Table 1 of our main paper, we used the results of COMBO as presented in the literature [23]. Here we detail additional attempts to reproduce the results and compare the performance of CSVE with COMBO.
Official Code. We initially aimed to rerun the experiment using the official COMBO code provided by the authors. The code is implemented in Tensoflow 1.x and relies on software versions from 2018. Despite our best efforts to recreate the computational environment, we encountered challenges in reproducing the reported results. For instance, Fig. 5 illustrates the asymptotic performance on medium datasets up to 1000 epochs, where the scores have been normalized based on SAC performance metrics. Notably, for both the hopper and walker2d tasks, the performance scores exhibited significant variability. The average scores over the last 10 epochs for halfcheetah, hopper, and walker2d were 71.7, 65.3, and -0.26, respectively. Furthermore, we observed that even when using the D4RL v0 dataset, COMBO demonstrated similar performance patterns when recommended hyper-parameters were applied.
Figure 5: Return of official COMBO implementation on D4RL mujoco v2 tasks, fixing seed=0.
JAX-based optimized implementation Code [35]. We also tested one recent re-implementation available in RIQL. This version is regarded as the most highly-tuned implementation to date. The results of our tests can be found in Fig.6. For the random and expert datasets, we applied the same hyper-parameters as those used for the medium and medium-expert datasets, respectively. For all other datasets, we adhered to the default hyper-parameters provided by the authors [35]. Despite these efforts, when we compared our outcomes with the original authors' results (as shown in Table 10 and Fig.7 of [35]), our reproduced results consistently exhibited both lower performance scores and greater variability.
Figure 6: Return of an optimized COMBO implementation[35] on D4RL mujoco v2 tasks. The data are obtained by running with 5 seeds for each task, and the dynamics model has 7 ensembles.
D.4 Effect of Exploration on Near Dataset Distributions
As discussed in Section 3.1 and 4.2, the appropriate selection of exploration on the distribution  $(d)$  beyond data  $(d_u)$  should help policy improvement. The factor  $\lambda$  in Eq. 10 controls the trade-off on such 'bonus' exploration and complying with the data-implied behavior policy.
In section 5.2, we examined the effect of  $\lambda$  on the medium datasets of mujoco tasks. Now let us further take the medium-replay type of datasets for more analysis of its effect. In the experiments, with fixed  $\beta = 0.1$ , we investigate  $\lambda$  values of  $\{0.0, 0.5, 1.0, 3.0\}$ . As shown in the upper figures in Fig. 4,  $\lambda$  shows an obvious effect on policy performance and variances during training. In general, there is a value under which increasing  $\lambda$  leads to performance improvement, while above which further increasing  $\lambda$  hurts performance. For example, with  $\lambda = 3.0$  in hopper-medium-replay task and walker2d-medium-replay task, the performance gets worse than with smaller  $\lambda$  values. The value of  $\lambda$  is task-specific, and we find that its effect is highly related to the loss in Eq. 9 which can be observed by comparing the bottom and upper figures in Fig. 4. Thus, in practice, we can choose proper  $\lambda$  according to the above loss without online interaction.
D.5 Conservative State Value Estimation by Perturbing Data State with Noise
In this section, we investigate a model-free method for sampling OOD states and compare its results with the model-based method adopted in section 4.
The model-free method samples OOD states by randomly adding Gaussian noise to the sampled states from the data. Specifically, we replace the Eq.5 with the following Eq. 34, and other parts are consistent with the previous technology.
$$
\begin{array}{l} \hat {V} ^ {k + 1} \leftarrow \arg \min  _ {V} L _ {V} ^ {\pi} (V; \overline {{\hat {Q} ^ {k}}}) = \alpha \left(E _ {s \sim D, s ^ {\prime} = s + N \left(0, \sigma^ {2}\right)} [ V (s ^ {\prime}) ] - E _ {s \sim D} [ V (s) ]\right) \tag {34} \\ + E _ {s \sim D} \left[ (E _ {a \sim \pi (\cdot | s)} [ \hat {Q} ^ {k} (s, a) ] - V (s)) ^ {2} \right]. \\ \end{array}
$$
The experimental results on the Mujoco control tasks are summarized in Table 4. As shown, with different noise levels  $(\sigma^2)$ , the model-free CSVE performs worse than our original model-based CSVE implementation; and for some problems, the model-free method shows very large variances across seeds. Intuitively, if the noise level covers the reasonable state distribution around data, its performance is good; otherwise, it misbehaves. Unfortunately, it is hard to find a noise level that is consistent for different tasks or even the same tasks with different seeds.
Table 4: Performance comparison on Gym control tasks. The results of different noise levels are over three seeds.
CQL	CSVE	σ2=0.05	σ2=0.1	σ2=0.15
Random	HalfCheetah	17.5 ± 1.5	26.7 ± 2.0	20.8 ± 0.4	20.4 ± 1.3	18.6 ± 1.1
Hopper	7.9 ± 0.4	27.0 ± 8.5	4.5 ± 3.1	14.2 ± 15.3	6.7 ± 5.4
Walker2D	5.1 ± 1.3	6.1 ± 0.8	3.9 ± 3.8	7.5 ± 6.9	1.7 ± 3.5
Medium	HalfCheetah	47.0 ± 0.5	48.6 ± 0.0	48.2 ± 0.2	47.5 ± 0.0	46.0 ± 0.9
Hopper	53.0 ± 28.5	99.4 ± 5.3	36.9 ± 32.6	46.1 ± 2.1	18.4 ± 30.6
Walker2D	73.3 ± 17.7	82.5 ± 1.5	81.5 ± 1.0	75.5 ± 1.9	78.6 ± 2, 9
Medium Replay	HalfCheetah	45.5 ± 0.7	54.8 ± 0.8	44.8 ± 0.4	44.1 ± 0.5	43.8 ± 0.4
Hopper	88.7 ± 12.9	91.7 ± 0.3	85.5 ± 3.0	78.3 ± 4.3	70.2 ± 12.0
Walker2D	81.8 ± 2.7	78.5 ± 1.8	78.7 ± 3.3	76.8 ± 1.3	66.8 ± 4.0
Medium Expert	HalfCheetah	75.6 ± 25.7	93.1 ± 0.3	87.5 ± 6.0	89.7 ± 6.6	93.8 ± 1.6
Hopper	105.6 ± 12.9	95.2 ± 3.8	63.2 ± 54.4	99.0 ± 11.0	37.6 ± 63.9
Walker2D	107.9 ± 1.6	109.0 ± 0.1	108.4 ± 1.9	109.5 ± 1.3	110.4 ± 0.6
Expert	HalfCheetah	96.3 ± 1.3	93.8 ± 0.1	59.0 ± 28.6	67.5 ± 21.9	75.3 ± 27.3
Hopper	96.5 ± 28.0	111.2 ± 0.6	67.3 ± 57.7	109.2 ± 2.4	109.4 ± 2.1
Walker2D	108.5 ± 0.5	108.5 ± 0.0	109.7 ± 1.1	108.9 ± 1.6	108.6 ± 0.3