SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} f ^ {\pi} (s, a) = \mathcal {T} _ {\mathrm {S V R}} ^ {\pi} f ^ {\pi} (s, a) \\ = r (s, a) + \gamma \mathbb {E} _ {s ^ {\prime}} \mathbb {E} _ {a ^ {\prime} \sim \pi (\cdot | s ^ {\prime})} [ f ^ {\pi} (s ^ {\prime}, a ^ {\prime}) ] \\ = r (s, a) + \gamma \mathbb {E} _ {s ^ {\prime}} \mathbb {E} _ {a ^ {\prime} \sim \pi (\cdot | s ^ {\prime})} \left[ \mathcal {T} _ {\mathrm {S V R}} ^ {\pi} f ^ {\pi} \left(s ^ {\prime}, a ^ {\prime}\right) \right] \\ \leq r (s, a) + \gamma \mathbb {E} _ {s ^ {\prime}} \mathbb {E} _ {a ^ {\prime} \sim \pi (\cdot | s ^ {\prime})} [ \mathcal {T} ^ {\pi} f ^ {\pi} (s ^ {\prime}, a ^ {\prime}) ] \\ = r (s, a) + \gamma \mathbb {E} _ {s ^ {\prime}} \mathbb {E} _ {a ^ {\prime} \sim \pi (\cdot | s ^ {\prime})} [ r (s ^ {\prime}, a ^ {\prime}) + \gamma \mathbb {E} _ {s ^ {\prime \prime}} \mathbb {E} _ {a ^ {\prime \prime} \sim \pi (\cdot | s ^ {\prime \prime})} [ f ^ {\pi} (s ^ {\prime \prime}, a ^ {\prime \prime}) ] ] \\ = r (s, a) + \gamma \mathbb {E} _ {s ^ {\prime}} \mathbb {E} _ {a ^ {\prime} \sim \pi (\cdot | s ^ {\prime})} [ r (s ^ {\prime}, a ^ {\prime}) + \gamma \mathbb {E} _ {s ^ {\prime \prime}} \mathbb {E} _ {a ^ {\prime \prime} \sim \pi (\cdot | s ^ {\prime \prime})} [ \mathcal {T} _ {\mathrm {S V R}} ^ {\pi} (s ^ {\prime \prime}, a ^ {\prime \prime}) ] ] \\ \leq r (s, a) + \gamma \mathbb {E} _ {s ^ {\prime}} \mathbb {E} _ {a ^ {\prime} \sim \pi (\cdot | s ^ {\prime})} [ r (s ^ {\prime}, a ^ {\prime}) + \gamma \mathbb {E} _ {s ^ {\prime \prime}} \mathbb {E} _ {a ^ {\prime \prime} \sim \pi (\cdot | s ^ {\prime \prime})} [ T ^ {\pi} f ^ {\pi} (s ^ {\prime \prime}, a ^ {\prime \prime}) ] ] \\ \end{array}
$$
$$
\begin{array}{l} \leq \mathbb {E} _ {\pi} \left[ \sum_ {j = 0} ^ {\infty} \gamma^ {j} r \left(s _ {t + j}, a _ {t + j}\right) | s _ {t} = s, a _ {t} = a \right] \\ = Q ^ {\pi} (s, a) \\ \end{array}
$$
In conclusion, for any  $\pi$ , the fixed point of  $\mathcal{T}_{\mathrm{SVR}}^{\pi}$  satisfies
$$
\left\{ \begin{array}{l l} Q _ {\min } \leq f ^ {\pi} (s, a) \leq Q ^ {\pi} (s, a), & \beta (a | s) > 0, \\ f ^ {\pi} (s, a) = Q _ {\min }, & \beta (a | s) = 0. \end{array} \right. \tag {27}
$$
Theorem 3 (Fixed point, Theorem 1). SVR yields support-constrained  $\pi_i : \pi_i \in \Pi$ . The fixed point of  $\mathcal{T}_{\mathrm{SVR}}^{\pi_i}$  is
$$
f ^ {\pi_ {i}} (s, a) = \left\{ \begin{array}{l l} Q ^ {\pi_ {i}} (s, a), & \beta (a | s) > 0, \\ Q _ {\min }, & \beta (a | s) = 0. \end{array} \right. \tag {28}
$$
Proof. By Proposition 4,  $\mathcal{T}_{\mathrm{SVR}}^{\pi}$  is a contraction and has a unique fixed point. For any  $\pi_i$ , the fixed point of policy evaluation satisfies  $f^{\pi_i} = \mathcal{T}_{\mathrm{SVR}}^{\pi} f^{\pi_i}$ .
According to Proposition 5, the policy improvement at this iteration satisfies
$$
\pi_ {i + 1} (a | s) = \mathbb {I} \left[ a = \underset {a ^ {\prime}} {\operatorname {a r g m a x}} f ^ {\pi_ {i}} \left(s, a ^ {\prime}\right) \right] = \mathbb {I} \left[ a = \underset {a ^ {\prime} \in \operatorname {s u p p} (\beta \cdot | s))} {\operatorname {a r g m a x}} f ^ {\pi_ {i}} \left(s, a ^ {\prime}\right) \right] \tag {29}
$$
Therefore, for arbitrary initial policy  $\pi_0$ ,  $\pi_i$  ( $i > 0$ ) in SVR is support-constrained:  $\pi_i \in \Pi, i > 0$ .
Now we show the fixed point of  $\mathcal{T}_{\mathrm{SVR}}^{\pi_i}$  ( $i > 0$ ). Assume the fixed point is  $f^{\pi_i}$ :
$$
f ^ {\pi_ {i}} (s, a) = \mathcal {T} _ {\mathrm {S V R}} ^ {\pi_ {i}} f ^ {\pi_ {i}} (s, a) = \left\{ \begin{array}{l l} \mathcal {T} ^ {\pi_ {i}} f ^ {\pi_ {i}} (s, a), & \beta (a | s) > 0, \\ Q _ {\min }, & \text {e l s e .} \end{array} \right. \tag {30}
$$
For  $\beta (a|s) > 0$
$$
\begin{array}{l} f ^ {\pi_ {i}} (s, a) = \mathcal {T} _ {\text {S V R}} ^ {\pi_ {i}} f ^ {\pi_ {i}} (s, a) \\ = r (s, a) + \gamma \mathbb {E} _ {s ^ {\prime}} \mathbb {E} _ {a ^ {\prime} \sim \pi_ {i} (\cdot | s ^ {\prime})} \left[ f ^ {\pi_ {i}} \left(s ^ {\prime}, a ^ {\prime}\right) \right] \\ = r (s, a) + \gamma \mathbb {E} _ {s ^ {\prime}} \mathbb {E} _ {a ^ {\prime} \sim \pi_ {i} (\cdot | s ^ {\prime})} [ \mathcal {T} _ {\mathrm {S V R}} ^ {\pi_ {i}} f ^ {\pi_ {i}} (s ^ {\prime}, a ^ {\prime}) ] \\ = r (s, a) + \gamma \mathbb {E} _ {s ^ {\prime}} \mathbb {E} _ {a ^ {\prime} \sim \pi_ {i} (\cdot | s ^ {\prime})} [ r (s ^ {\prime}, a ^ {\prime}) + \gamma \mathbb {E} _ {s ^ {\prime \prime}} \mathbb {E} _ {a ^ {\prime \prime} \sim \pi_ {i} (\cdot | s ^ {\prime \prime})} [ f ^ {\pi_ {i}} (s ^ {\prime \prime}, a ^ {\prime \prime}) ] ] \\ = r (s, a) + \gamma \mathbb {E} _ {s ^ {\prime}} \mathbb {E} _ {a ^ {\prime} \sim \pi_ {i} (\cdot | s ^ {\prime})} [ r (s ^ {\prime}, a ^ {\prime}) + \gamma \mathbb {E} _ {s ^ {\prime \prime}} \mathbb {E} _ {a ^ {\prime \prime} \sim \pi_ {i} (\cdot | s ^ {\prime \prime})} [ \mathcal {T} _ {\mathrm {S V R}} ^ {\pi_ {i}} f ^ {\pi_ {i}} (s ^ {\prime \prime}, a ^ {\prime \prime}) ] ] \\ \end{array}
$$
$$
\begin{array}{l} = \mathbb {E} _ {\pi_ {i}} \left[ \sum_ {j = 0} ^ {\infty} \gamma^ {j} r \left(s _ {t + j}, a _ {t + j}\right) | s _ {t} = s, a _ {t} = a \right] \\ = Q ^ {\pi_ {i}} (s, a) \\ \end{array}
$$
The fourth equality holds because  $\pi_i$  is support-constrained and thus the expectation  $\mathbb{E}_{a' \sim \pi_i(\cdot | s')}$  has non-zero mass only on support-constrained  $a'$ , i.e.,  $\beta(a'|s') > 0$ .
Therefore, for the policies  $\pi_i$  ( $i > 0$ ) in SVR, the fixed point of  $\mathcal{T}_{\mathrm{SVR}}^{\pi_i}$  is
$$
f ^ {\pi_ {i}} (s, a) = \left\{ \begin{array}{l l} Q ^ {\pi_ {i}} (s, a), & \beta (a | s) > 0, \\ Q _ {\min }, & \beta (a | s) = 0. \end{array} \right. \tag {31}
$$
This fixed point provides unbiased  $Q$ -values for all ID actions and underestimated  $Q$ -values for all OOD actions.
Proposition 6 (Proposition 3). Only in the  $ID$  region  $(\beta(a|s) > 0)$  and when the evaluated policy is support-constrained  $(\pi \in \Pi)$ ,  $\mathcal{T}_{\mathrm{CQL}}^{\pi}$  is a contraction operator. Its fixed point satisfies
$$
f ^ {\pi} (s, a) = Q ^ {\pi} (s, a) - \frac {\alpha}{1 - \gamma} \left(\rho_ {s a} ^ {\pi}\right) ^ {T} \left(\frac {\pi}{\beta} - 1\right), \beta (a | s) > 0. \tag {32}
$$
$$
\mathcal {T} _ {\mathrm {C Q L}} ^ {\pi} Q (s, a) = \left\{ \begin{array}{l l} \mathcal {T} ^ {\pi} Q (s, a) - \alpha \left(\frac {\pi (a | s)}{\beta (a | s)} - 1\right), & \beta (a | s) > 0, \\ - \infty , & \beta (a | s) = 0, \pi (a | s) > 0, \\ Q (s, a), & \beta (a | s) = 0, \pi (a | s) = 0. \end{array} \right. \tag {33}
$$
We first consider the OOD region  $(\beta (a|s) = 0)$ . We separate it into  $\pi (a|s) = 0$  and  $\pi (a|s) > 0$ .
For all  $(s,a)$  s.t.  $\beta (a|s) = 0$  and  $\pi (a|s) = 0$
$$
\left| \mathcal {T} _ {\mathrm {C Q L}} ^ {\pi} f _ {1} (s, a) - \mathcal {T} _ {\mathrm {C Q L}} ^ {\pi} f _ {2} (s, a) \right| = \left| f _ {1} (s, a) - f _ {2} (s, a) \right| \tag {34}
$$
For all  $(s,a)$  s.t.  $\beta (a|s) = 0$  and  $\pi (a|s) > 0$
$$
\left| \mathcal {T} _ {\mathrm {C Q L}} ^ {\pi} f _ {1} (s, a) - \mathcal {T} _ {\mathrm {C Q L}} ^ {\pi} f _ {2} (s, a) \right| = | \infty - \infty | \tag {35}
$$
In both cases,  $\mathcal{T}_{\mathrm{CQL}}^{\pi}$  is not a contraction. Therefore, the contraction region of  $\mathcal{T}_{\mathrm{CQL}}^{\pi}$  (if exists) cannot contain the OOD region  $(\beta(a|s) = 0)$ .
Now we consider the ID region  $(\beta (a|s) > 0)$ .
If  $\pi$  is not support-constrained, for all  $(s,a)$  s.t.  $\beta (a|s) > 0$  and  $\forall \pi \notin \Pi$ , we have
$$
\begin{array}{l} \left| \mathcal {T} _ {\mathrm {C Q L}} ^ {\pi} f _ {1} (s, a) - \mathcal {T} _ {\mathrm {C Q L}} ^ {\pi} f _ {2} (s, a) \right| = \left| \mathcal {T} ^ {\pi} f _ {1} (s, a) - \mathcal {T} ^ {\pi} f _ {2} (s, a) \right| \\ = \left| \gamma \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a), a ^ {\prime} \sim \pi (\cdot | s ^ {\prime})} \left[ f _ {1} \left(s ^ {\prime}, a ^ {\prime}\right) - f _ {2} \left(s ^ {\prime}, a ^ {\prime}\right) \right] \right| \\ \end{array}
$$
Since  $\pi \notin \Pi$ , the expectation  $\mathbb{E}_{a' \sim \pi(\cdot | s')}$  can have non-zero mass on some  $(s', a')$  in the OOD region. Thus  $\mathcal{T}_{\mathrm{CQL}}^{\pi}$  cannot be a contraction.
If  $\pi$  is support-constrained, for all  $(s,a)$  s.t.  $\beta (a|s) > 0$  and  $\forall \pi \in \Pi$ , we have
$$
\begin{array}{l} \left| \mathcal {T} _ {\mathrm {C Q L}} ^ {\pi} f _ {1} (s, a) - \mathcal {T} _ {\mathrm {C Q L}} ^ {\pi} f _ {2} (s, a) \right| = \left| \mathcal {T} ^ {\pi} f _ {1} (s, a) - \mathcal {T} ^ {\pi} f _ {2} (s, a) \right| \\ = \left| \gamma \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a), a ^ {\prime} \sim \pi (\cdot | s ^ {\prime})} \left[ f _ {1} \left(s ^ {\prime}, a ^ {\prime}\right) - f _ {2} \left(s ^ {\prime}, a ^ {\prime}\right) \right] \right| \\ \leq \gamma \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a), a ^ {\prime} \sim \pi (\cdot | s ^ {\prime})} [ \left| f _ {1} \left(s ^ {\prime}, a ^ {\prime}\right) - f _ {2} \left(s ^ {\prime}, a ^ {\prime}\right) \right| ] \tag {37} \\ \leq \gamma \max  _ {(s, a): \beta (a | s) > 0} | f _ {1} (s, a) - f _ {2} (s, a) | \\ \end{array}
$$
The last inequality holds because  $\pi \in \Pi$  and thus the expectation  $\mathbb{E}_{a' \sim \pi(\cdot|s')}$  has non-zero mass only on support-constrained  $a'$ , i.e.,  $\beta(a'|s') > 0$ .
Therefore, in the ID region  $(\beta (a|s) > 0)$  and when  $\forall \pi \in \Pi$ ,  $\mathcal{T}_{\mathrm{CQL}}^{\pi}$  is a  $\gamma$ -contraction operator. Assume the fixed point is  $f^{\pi}$ . We have
$$
f ^ {\pi} (s, a) = \mathcal {T} _ {\mathrm {C Q L}} ^ {\pi} f ^ {\pi} (s, a) = \mathcal {T} ^ {\pi} f ^ {\pi} (s, a) - \alpha \left(\frac {\pi (a | s)}{\beta (a | s)} - 1\right), \beta (a | s) > 0. \tag {38}
$$
We define  $R$  as the vector of reward function, and  $P^{\pi}$  as the transition matrix on state-action pairs induced by policy  $\pi$ :  $P_{(s,a),(s',a')}^{\pi} \coloneqq P(s'|s,a)\pi(a'|s')$ . Now Eq. (38) can be written in vector form:
$$
\begin{array}{l} f ^ {\pi} (s, a) = \left[ R + \gamma P ^ {\pi} f ^ {\pi} \right] (s, a) - \alpha \left(\frac {\pi (a | s)}{\beta (a | s)} - 1\right), \beta (a | s) > 0 \\ \Rightarrow f ^ {\pi} (s, a) = \left(I - \gamma P ^ {\pi}\right) ^ {- 1} \left[ R - \alpha \left(\frac {\pi}{\beta} - 1\right) \right] (s, a), \beta (a | s) > 0 \\ \Rightarrow f ^ {\pi} (s, a) = Q ^ {\pi} (s, a) - \alpha \left[ (I - \gamma P ^ {\pi}) ^ {- 1} \left(\frac {\pi}{\beta} - 1\right) \right] (s, a), \beta (a | s) > 0 \\ \end{array}
$$
The row  $(s,a)$  of the matrix  $(I - \gamma P^{\pi})^{-1}$  is  $\rho_{sa}^{\pi} / (1 - \gamma)$ , i.e., the unnormalized discounted state-action occupancy induced by policy  $\pi$  with initial state-action pair  $(s,a)$  (Lemma 1.6 in [1]). Therefore, in the ID region  $(\beta (a|s) > 0)$  and when  $\pi \in \Pi$ , the fixed point is:
$$
f ^ {\pi} (s, a) = Q ^ {\pi} (s, a) - \frac {\alpha}{1 - \gamma} \left(\rho_ {s a} ^ {\pi}\right) ^ {T} \left(\frac {\pi}{\beta} - 1\right), \beta (a | s) > 0.
$$
Theorem 4 (Strict policy improvement to support-constrained optimal, Theorem 2). SVR yields support-constrained  $\pi_{i}$  and guarantees monotonic performance improvement:
$$
V ^ {\pi_ {i + 1}} (s) \geq V ^ {\pi_ {i}} (s) \quad \forall s, \tag {39}
$$
Proof. In Theorem 3, we have proved that SVR yields support-constrained  $\pi_{i}$  and the fixed point of policy evaluation is:
$$
f ^ {\pi_ {i}} (s, a) = \left\{ \begin{array}{l l} Q ^ {\pi_ {i}} (s, a), & \beta (a | s) > 0, \\ Q _ {\min }, & \beta (a | s) = 0. \end{array} \right. \tag {40}
$$
Now we prove the monotonic improvement results. We start out with the performance difference lemma [16]. Given two policies  $\pi', \pi$ ,
$$
V ^ {\pi^ {\prime}} (s) - V ^ {\pi} (s) = \frac {1}{1 - \gamma} \mathbb {E} _ {s ^ {\prime} \sim d ^ {\pi^ {\prime}, s}} \left[ A ^ {\pi} \left(s ^ {\prime}, \pi^ {\prime} \left(s ^ {\prime}\right)\right) \right] \tag {41}
$$
where  $d^{\pi', s}$  is the normalized discounted state occupancy induced by policy  $\pi'$  from starting state  $s$ . Thus for  $i > 0$ ,
$$
\begin{array}{l} V ^ {\pi_ {i + 1}} (s) - V ^ {\pi_ {i}} (s) = \frac {1}{1 - \gamma} \mathbb {E} _ {s ^ {\prime} \sim d ^ {\pi_ {i + 1}, s}} \left[ A ^ {\pi_ {i}} \left(s ^ {\prime}, \pi_ {i + 1} \left(s ^ {\prime}\right)\right) \right] (42) \\ = \frac {1}{1 - \gamma} \mathbb {E} _ {s ^ {\prime} \sim d ^ {\pi_ {i + 1}, s}} \left[ \mathbb {E} _ {a \sim \pi_ {i + 1}} Q ^ {\pi_ {i}} \left(s ^ {\prime}, a\right) - \mathbb {E} _ {a \sim \pi_ {i}} Q ^ {\pi_ {i}} \left(s ^ {\prime}, a\right) \right] (43) \\ = \frac {1}{1 - \gamma} \mathbb {E} _ {s ^ {\prime} \sim d ^ {\pi_ {i + 1}, s}} \left[ \mathbb {E} _ {a \sim \pi_ {i + 1}} f ^ {\pi_ {i}} \left(s ^ {\prime}, a\right) - \mathbb {E} _ {a \sim \pi_ {i}} f ^ {\pi_ {i}} \left(s ^ {\prime}, a\right) \right] (44) \\ \geq 0 (45) \\ \end{array}
$$
The third equality holds because  $\pi_i$  ( $i > 0$ ) is support-constrained and  $f^{\pi_i}(s, a) = Q^{\pi_i}(s, a)$  for  $\beta(a|s) > 0$  by Eq. (40). The last inequality holds because  $\pi_{i+1}$  is the greedy policy with respect to  $f^{\pi_i}(s, a)$  and it holds that  $\mathbb{E}_{a \sim \pi_{i+1}} f^{\pi_i}(s', a) \geq \mathbb{E}_{a \sim \pi_i} f^{\pi_i}(s', a)$  at every  $s'$ .
When the improvement  $V^{\pi_{i+1}}(s) - V^{\pi_i}(s)$  is 0 at every state  $s$ , it implies  $\mathbb{E}_{a \sim \pi_{i+1}} f^{\pi_i}(s, a) = \mathbb{E}_{a \sim \pi_i} f^{\pi_i}(s, a)$  at every  $s$ , because the point mass on  $s$  is "contained" in  $d^{\pi_{i+1},s}$ :  $d^{\pi_{i+1},s}(s) > 0$ .
$$
f ^ {\pi_ {i}} (s, a) = \mathcal {T} _ {\text {S V R}} ^ {\pi_ {i}} f ^ {\pi_ {i}} (s, a) = \left\{ \begin{array}{l l} r (s, a) + \gamma \mathbb {E} _ {s ^ {\prime}} \mathbb {E} _ {a ^ {\prime} \sim \pi_ {i} (\cdot | s ^ {\prime})} [ f ^ {\pi_ {i}} \left(s ^ {\prime}, a ^ {\prime}\right) ], & \beta (a | s) > 0, \\ Q _ {\min }, & \beta (a | s) = 0. \end{array} \right. \tag {46}
$$
$$
\begin{array}{l} f ^ {\pi_ {i}} (s, a) = r (s, a) + \gamma \mathbb {E} _ {s ^ {\prime}} \mathbb {E} _ {a ^ {\prime} \sim \pi_ {i} (\cdot | s ^ {\prime})} [ f ^ {\pi_ {i}} \left(s ^ {\prime}, a ^ {\prime}\right) ] (47) \\ = r (s, a) + \gamma \mathbb {E} _ {s ^ {\prime}} \mathbb {E} _ {a ^ {\prime} \sim \pi_ {i + 1} (\cdot | s ^ {\prime})} \left[ f ^ {\pi_ {i}} \left(s ^ {\prime}, a ^ {\prime}\right) \right] (48) \\ = r (s, a) + \gamma \mathbb {E} _ {s ^ {\prime}} \left[ \max  _ {a ^ {\prime} \in \operatorname {s u p p} (\beta (\cdot | s ^ {\prime}))} f ^ {\pi_ {i}} \left(s ^ {\prime}, a ^ {\prime}\right) \right] (49) \\ \end{array}
$$
The last equality holds as  $\pi_{i + 1}$  is support-constrained and is greedy with respect to  $f^{\pi_i}$ . It can be seen that Eq. (49) is the support-constrained Bellman optimality equation in Eq. (19):
$$
f ^ {\pi_ {i}} (s, a) = \left\{ \begin{array}{l l} Q _ {\Pi} ^ {*}, & \beta (a | s) > 0, \\ Q _ {\min }, & \beta (a | s) = 0. \end{array} \right. \tag {50}
$$
$$
\begin{array}{l} \pi_ {i + 1} (a | s) = \mathbb {I} \left[ a = \underset {a ^ {\prime}} {\operatorname {a r g m a x}} f ^ {\pi_ {i}} \left(s, a ^ {\prime}\right) \right] (51) \\ = \mathbb {I} \left[ a = \underset {a ^ {\prime} \in \operatorname {s u p p} (\beta (\cdot | s))} {\operatorname {a r g m a x}} f ^ {\pi_ {i}} (s, a ^ {\prime}) \right] (52) \\ = \mathbb {I} \left[ a = \underset {a ^ {\prime} \in \operatorname {s u p p} (\beta (\cdot | s))} {\operatorname {a r g m a x}} Q _ {\Pi} ^ {*} (s, a ^ {\prime}) \right] (53) \\ = \pi_ {\Pi} ^ {*} (54) \\ \end{array}
$$
where the last equality is the definition of the support-constrained optimal policy  $\pi_{\Pi}^{*}$ .
To conclude, the policy iteration with  $\mathcal{T}_{\mathrm{SVR}}^{\pi}$  guarantees monotonic performance improvement, and the improvement is strict until the optimal support-constrained policy  $\pi_{\Pi}^{*}$  is found.
B.1 Experimental details on D4RL experiments
Table 2: Hyperparameters in SVR.
Hyperparameter	Value
SVR	Optimizer	Adam [17]
Critic learning rate	3 × 10-4
Actor learning rate	3 × 10-4with cosine schedule
Batch size	256
Discount factor	0.99
Number of iterations	106
Target update rate τ	0.005
Policy update frequency	2
Number of Critics	4
Penalty coefficient α	{0.001, 0.02} for Gym-MuJoCo {10} for Adroit
Standard deviation of u	0.2
Architecture	Actor	input-256-256-output
Critic	input-256-256-1
All hyperparameters of SVR are included in Table 2. Note that the only hyperparameter we tuned is the penalty coefficient  $\alpha$ . We use  $\alpha = 10$  for Adroit tasks and  $\alpha = \{0.001, 0.02\}$  for Gym-MuJoCo tasks ( $\lambda = 0.02$  for expert and medium-expert datasets,  $\lambda = 0.001$  for medium, medium-replay, random datasets). The discrepancy of  $\alpha$  in this two domains is due to their characteristics. Adroit
Figure 6: Parameter study on the penalty coefficient  $\alpha$  of SVR. The curves are averaged over 4 random seeds, with the shaded area representing the standard deviation across seeds.
datasets are mostly collected by human behavior and the data coverage is very narrow, so strong value regularization is needed to keep value functions from overestimation. We set the standard deviation of the sampling distribution in SVR as 0.2, which our experiments have demonstrated to be an insensitive parameter. And following TD3+BC [8], we normalize the states in all datasets.
For evaluation, we average returns over 10 evaluation trajectories and 5 random seeds on all tasks. The reported results are the normalized scores, which are offered by the D4RL benchmark [7] to measure how the learned policy compared with random and expert policy:
$$
\mathrm {D 4 R L} \text {s c o r e} = 1 0 0 \times \frac {\text {l e a n e d p o l i c y r e t u r n - r a n d o m p o l i c y r e t u r n}}{\text {e x p e r t p o l i c y r e t u r n - r a n d o m p o l i c y r e t u r n}}
$$
B.2 Experimental details on tabular maze experiments
As depicted in Fig. 1(a) in the paper, the task is to navigate from bottom-left to top-right in as few steps as possible, with a wall in the middle. The agent receives a reward of 0 for reaching the goal and  $-1$  for all other transitions. Episodes are terminated after 100 steps and  $\gamma$  is set to 0.9. We first collect 10,000 transitions using a random policy. Then we remove all the transitions containing rightward actions in the bottom-left  $4 \times 4$  region to introduce OOD actions. It makes the optimal support-constrained policy (see Fig. 1(b) in the paper) differ from the actual optimal policy in the environment. To magnify the impact of bootstrapping from OOD actions, we used optimistic initialization for each algorithm (i.e., initialized all  $Q$ -values with  $Q_{\max}$ ). This ensures overestimation occurs in some states and we can observe how well the algorithms mitigate it.
Here we are verifying the support-constrained optimality of the SVR operator in the tabular MDP, and the actual updates of CQL and SVR are Eq. (4) and Eq. (7) in the paper respectively. We run experiments over 5 random seeds that affect dataset collection and policy/Q initialization.
B.3 Experimental details on noisy dataset experiments
In noisy dataset experiments, we construct a "noisy" dataset by combining the random and expert datasets in D4RL with different expert ratios. The size of the combined dataset is set to  $1 \times 10^{6}$ . In some environments, the size of the random or expert dataset in D4RL is slightly smaller than  $1 \times 10^{6}$ , so we directly utilize the corresponding D4RL dataset when expert ratio is 0 or 1.
The hyperparameters of SVR follow Table 2:  $\alpha = 0.001$  for expert ratio of 0 and  $\alpha = 0.02$  for all other expert ratios. For CQL, we tune its regularization coefficient in  $\{5,10,20,30\}$  (performs relatively well in this range) and present the best results obtained for each dataset.
C.1 Ablation on the penalty coefficient
We also conduct an ablation study on the penalty coefficient  $\alpha$  of SVR in Fig. 6. Experimental results indicate that different datasets have different requirements for  $\alpha$ , but choosing  $\alpha \in [1e - 3, 1e - 2]$  generally induces relatively satisfying performance.
Table 3: Comparisons with additional baselines on the D4RL benchmark.
Dataset	APAC	GAN-Joint	SPOT	MCQ	IAC	SVR (Ours)
halfcheetah-med	58.4±2.6	44.0±0.2	58.4±1.0	64.3±0.2	51.6±0.3	60.5±1.2
hopper-med	93.8±5.4	86.4±10.9	86.0±8.7	78.4±4.3	74.6±11.5	103.5±0.4
walker2d-med	58.5±7.4	69.9±6.4	86.4±2.7	91.0±0.4	85.2±0.4	92.4±1.2
halfcheetah-med-rep	57.3±2.8	33.4±2.4	52.2±1.2	56.8±0.6	47.2±0.3	52.5±3.0
hopper-med-rep	51.9±5.6	30.9±3.2	100.2±1.9	101.6±0.8	103.2±1.0	103.7±1.3
walker2d-med-rep	16.3±5.3	6.7±2.2	91.6±2.8	91.3±5.7	93.2±1.8	95.6±2.5
halfcheetah-med-exp	-	72.6±11.1	86.9±4.3	87.5±1.3	92.9±0.7	94.2±2.2
hopper-med-exp	-	71.1±10.7	99.3±7.1	111.2±0.1	109.3±4.0	111.2±0.9
walker2d-med-exp	-	79.6±1.9	112.0±0.5	114.2±0.7	110.1±0.1	109.3±0.2
gym total	-	495.2	796.3	773.0	767.3	822.9
pen-expert	-	134.5±10.8	116.3±22.3	-	103.3±12.2	138.9±9.2
pen-human	-	71.0±23.2	91.4±13.4	68.5±6.5	45.9±23.0	73.1±12.1
pen-cloned	-	27.6±7.1	47.9±20.5	49.4±4.3	36.4±9.2	70.2±17.4
adroit total	-	233.1	255.6	-	185.6	282.2
Figure 7: (Left) Performance of SVR under different behavioral model checkpoints, which are obtained at different steps in the behavioral model training process. (Right) Training loss of the behavioral model  $\beta_{\omega}$  at different training steps. The curves are averaged over 4 random seeds, with the shaded area representing the standard deviation across seeds.
C.2 Comparisons with additional baselines
To make a more comprehensive comparison, we also compare SVR with APAC [52], GAN-Joint [49], SPOT [43], MCQ [27], and IAC [51] on the D4RL benchmark. The results are shown in Table 3. The results indicate that our methods have better performance than these baselines.
C.3 Empirical study on the behavior model error of SVR
To empirically investigate SVR under different behavior model errors, we run SVR using different checkpoints of the behavior model, which are obtained at different steps in the behavior model training process (Eq. (15) in the paper). The model error is controlled by the number of steps taken to train the behavioral model. The results are shown in Fig. 7. We observe that the performance of SVR increases with the number of training steps of the behavioral model. Notably, the performance of SVR stabilizes at a high level after only  $10^{2}$  steps of behavioral model training, where the model has not been adequately trained. Theoretically, in Eq. (16) in the paper, the error of  $\beta_{\omega}$  only affects the weights of rewarding ID actions. Thus, an imperfect model can make the maximization and minimization of ID  $Q$ -values not cancel out well, but have little effect on OOD ones (still penalizing all OOD Q-values). We hypothesize that this is the reason why SVR can still achieve good performance with an imperfect behavior model.
C.4 Computational cost
We test the runtime of SVR on halfcheetah-medium-replay-v2 on a GeForce RTX 3090. The results of SVR and other baselines are shown in Table 4. It takes  $2\mathrm{h}40\mathrm{min}$  for SVR to finish the task, which is comparable to other baselines. Note that it only takes two minutes for the pre-training part. SVR is computationally more efficient than CQL, because the practical CQL algorithm needs to sample 10
Table 4: Runtime of TD3BC, IQL, CQL, SVR for halfcheetah-medium-replay-v2 on a GeForce RTX 3090.
Algorithm	TD3BC	IQL	CQL	SVR	pre-training in SVR
Runtime	1h	1h50min	4h10min	2h40min	2min
Figure 8: Learning Curves of SVR on Gym-MuJoCo Tasks.
actions at each state to compute log-sum-exp [23], while SVR only needs to sample a single action at each state. SVR can achieve better performance with lower computational costs.
C.5 Learning curves of SVR
Learning curves on Gym-MuJoCo tasks and Adroit tasks are presented in Fig. 8 and Fig. 9, respectively. The curves are averaged over 5 random seeds, with the shaded area representing the standard deviation across seeds.
Societal. Offline reinforcement learning (RL) holds significant promise in facilitating and expanding practical applications of RL, including domains such as robotics, recommendation systems, healthcare,
Figure 9: Learning Curves of SVR on Adroit Tasks.