SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} L _ {Q} (\theta) = \mathbb {E} _ {(s, a, s ^ {\prime}) \sim \mathcal {D}} \left[ \left(Q _ {\theta} (s, a) - R (s, a) - \gamma \mathbb {E} _ {a ^ {\prime} \sim \pi_ {\phi} (\cdot | s ^ {\prime})} Q _ {\theta^ {\prime}} \left(s ^ {\prime}, a ^ {\prime}\right)\right) ^ {2} \right] \\ + \alpha \left(\mathbb {E} _ {s \sim \mathcal {D}, a \sim u} \left[ \left(Q _ {\theta} (s, a) - Q _ {\min }\right) ^ {2} \right] - \mathbb {E} _ {s, a \sim \mathcal {D}} \left[ \frac {u (a | s)}{\beta_ {\omega} (a | s)} \left(Q _ {\theta} (s, a) - Q _ {\min }\right) ^ {2} \right]\right) \tag {16} \\ \end{array}
$$
where  $\alpha$  is a hyperparameter and  $Q_{\mathrm{min}} \coloneqq R_{\mathrm{min}} / (1 - \gamma)$ . For the environment where  $R_{\mathrm{min}}$  is unknown, we choose the smallest reward in the dataset as  $R_{\mathrm{min}}$ .
Overall Algorithm. Putting everything together, we summarize our final algorithm in Algorithm 1. Our algorithm first trains the estimated behavior policy to obtain the behavior density. Then it turns to the actor-critic framework for policy training.
In this section, we conduct several experiments to justify the validity of our method. We aim to answer five questions: (1) Does SVR actually converge to the optimal support-constrained policy? (2) Does SVR perform better than previous methods on standard offline RL benchmarks? (3) When does SVR empirically benefit the most compared to the density-based regularization? (4) How should we select the sampling distribution of SVR in practice? (5) How does the implementation of each component affect SVR? More experimental details and results are provided in Appendix B and C.
4.1 Support-constrained optimality in the tabular setting
We use a simple maze environment to verify the support-constrained optimality of SVR. As depicted in Fig. 1(a), the task is to navigate from bottom-left to top-right, with a wall in the middle. The agent receives a reward of 0 for reaching the goal and  $-1$  for all other transitions. Episodes are terminated after 100 steps and  $\gamma$  is set to 0.9. We first collect 10,000 transitions using a random policy. Then we remove all the transitions containing rightward actions in the bottom-left  $4\times 4$  region to introduce OOD actions. It makes the optimal support-constrained policy (see Fig. 1(b)) differ from the actual optimal policy in the environment.
We test three algorithms: vanilla policy iteration [41], CQL [23], and SVR. Fig. 1(c, d, e) depict their learned value functions and policies respectively. To show convergence results, we also present
the learning curves in Fig. 2, along with the performance of the behavior policy and the optimal support-constrained policy. The results indicate that vanilla policy iteration has a severe overestimation of value functions, leading to poor performance. On the other hand, although CQL does not overestimate  $V$  functions, it fails to converge and performs poorly when the dataset is highly suboptimal. In contrast, SVR converges to the optimal support-constrained policy and the learned value function closely matches the true support-constrained optimal value function, verifying Theorem 2.
Figure 2: Learning curves of SVR, CQL and vanilla policy iteration on the toy maze.
Table 1: Averaged normalized scores on the D4RL benchmarks over five random seeds. Note that m = medium, m-r = medium-replay, m-e = medium-expert, e = expert, r = random.
Dataset	BC	OneStep	TD3BC	AWAC	BCQ	BEAR	UWAC	CQL	IQL	SVR (Ours)
halfcheetah-m	42.0	50.4	48.3	47.9	46.6	43.0	42.2	47.0	47.4	60.5±1.2
hopper-m	56.2	87.5	59.3	59.8	59.4	51.8	50.9	53.0	66.2	103.5±0.4
walker2d-m	71.0	84.8	83.7	83.1	71.8	-0.2	75.4	73.3	78.3	92.4±1.2
halfcheetah-m-r	36.4	42.7	44.6	44.8	42.2	36.3	35.9	45.5	44.2	52.5±3.0
hopper-m-r	21.8	98.5	60.9	69.8	60.9	52.2	25.3	88.7	94.7	103.7±1.3
walker2d-m-r	24.9	61.7	81.8	78.1	57.0	7.0	23.6	81.8	73.8	95.6±2.5
halfcheetah-m-e	59.6	75.1	90.7	64.9	95.4	46.0	42.7	75.6	86.7	94.2±2.2
hopper-m-e	51.7	108.6	98.0	100.1	106.9	50.6	44.9	105.6	91.5	111.2±0.9
walker2d-m-e	101.2	111.3	110.1	110.0	107.7	22.1	96.5	107.9	109.6	109.3±0.2
halfcheetah-e	92.9	88.2	96.7	81.7	89.9	92.7	92.9	96.3	95.0	96.1±0.7
hopper-e	110.9	106.9	107.8	109.5	109.0	54.6	110.5	96.5	109.4	111.1±0.4
walker2d-e	107.7	110.7	110.2	110.1	106.3	106.6	108.4	108.5	109.9	110.0±0.2
halfcheetah-r	2.6	2.3	11.0	6.1	2.2	2.3	2.3	17.5	13.1	27.2±1.2
hopper-r	4.1	5.6	8.5	9.2	7.8	3.9	2.7	7.9	7.9	31.0±0.3
walker2d-r	1.2	6.9	1.6	0.2	4.9	12.8	2.0	5.1	5.4	2.2±1.5
gym-v2 total	784.2	1041.2	1013.2	975.6	968.0	581.7	756.2	1010.2	1033.1	1200.5
pen-expert	85.1	61.6	111.0	115.2	114.9	105.9	111.9	107.0	110.2	138.9±9.2
pen-human	34.4	73.7	54.9	25.5	68.9	-1.0	21.7	37.5	71.5	73.1±12.1
pen-cloned	56.9	31.8	63.8	10.4	44.0	26.5	33.1	39.2	37.3	70.2±17.4
adroit-v0 total	176.4	167.1	229.7	151.1	227.8	131.4	166.7	183.7	219.0	282.2
4.2 Comparisons on D4RL benchmarks
Then we evaluate our approach on the D4RL benchmarks [7]. We compare SVR with prior state-of-the-art offline RL methods, including BC [37], BCQ [10], BEAR [22], OneStep RL [4], TD3+BC [8], AWAC [33], UWAC [45], CQL [23], and IQL [21].
The results are reported in Table 1. In both Gym-MuJoCo and Adroit domains, SVR achieves state-of-the-art performance and outperforms prior value regularization methods (CQL, UWAC) by a large margin. We also observe that SVR has strong advantages on the sub-optimal datasets (medium, medium-replay, random). This is because SVR exploits the optimal support-constrained policy in the dataset in a theoretically sound way and is less affected by the average quality of the dataset. In addition, while BCQ and BEAR are the policy constraint methods that are also designed to search for the optimal policy within the behavior support, they have much poorer performance. As stated in previous works [11, 44], they are limited respectively by the errors of the generative model and the unsuitability of using Maximum Mean Discrepancy (MMD) to characterize the support constraint. In contrast, without querying out-of-dataset behavior density, SVR takes the form of value regularization to search for the optimal support-constrained policy and achieves superior performance. For learning curves and more experimental details, please see Section B in the supplementary material.
4.3 Comparisons on noisy datasets.
In this section, we aim to validate that, compared with the existing density-based value regularization, the supported value regularization of SVR will benefit when the dataset contains a large portion of bad actions. To this end, we construct a "noisy" dataset by combining the random dataset and the expert dataset, and then evaluate SVR and CQL under different expert ratios.
The results are shown in Fig. 3. In all environments, SVR outperforms CQL over nearly all expert ratios. Moreover, as the expert ratio gets lower, the improvement is more significant. The performance of CQL is susceptible to the expert ratio and exhibits a sharp decrease at the expert ratio  $20\%$ , while SVR can still retain good or even expert performance (most spark in Hopper and Walker2d).
4.4 Empirical study on the sampling distribution of SVR
In this section, we investigate how to practically select the sampling distribution  $u$  in SVR, which controls the relative strength of penalizing different OOD  $Q$ -values. In our implementation,  $u$  is the
Figure 3: Evaluations of SVR (supported value regularization) vs CQL (density value regularization) on noisy datasets, which are made by mixing random and expert datasets with varying expert ratios.
Figure 4: Learning curves of SVR with different sampling distribution  $u$ .  $\sigma$  is the standard deviation of Gaussian. SVR is able to converge to good performance under various sampling distributions.
Gaussian with the same mean as the current policy. Here we vary its standard deviation  $\sigma$  and present the corresponding results in Fig. 4. We do not show the results for  $\sigma > 0.7$  because such a large  $\sigma$  would cause most sampled actions to lie at the boundary of the action space. Instead, we use Uniform distribution to represent  $\sigma = \infty$ .
As shown in Fig. 4, SVR is able to converge to good performance over a very wide range of  $\sigma$ . However, if  $\sigma$  is too small, SVR may not be able to adequately penalize all OOD  $Q$ -values and the IS ratio will have a large variance, thereby disrupting the learning process (see  $\sigma = 0.1$  in hopper-m-r). Conversely, if  $\sigma$  is too large or if the uniform distribution is used, sampling from  $u$  may fail to emphasize the key areas where overestimation is most likely to occur (as indicated by the current policy), leading to insufficient mitigation of overestimation and inferior performance (see Uniform in hopper-m, hopper-m-r, walker2d-m-r and  $\sigma = 0.7$  in hopper-m).
4.5 Effects of components in SVR
Density estimator. We speculate that a more precise behavior density estimator may further improve SVR. So following previous works [10, 43], we consider to replace the Gaussian density estimator  $\beta_{\omega}$  with the conditional variational autoencoder [18, 40]. We refer to this variant as SVR-VAE. The results of SVR-VAE are shown in Fig. 5 (left). Overall, SVR-VAE has only marginal performance gains over SVR.
Figure 5: Percentage performance difference of the SVR variant compared to the original algorithm. (Left) SVR with VAE density estimator. (Right) SVR with self-normalized importance sampling. HC = HalfCheetah, H = Hopper, W = Walker2d.
IS techniques. Self-Normalized Importance Sampling (SNIS) [35] is sometimes adopted to reduce the high variance
of IS [38]. The SNIS estimator has lower variance but is biased [35]. Here we test an SVR variant SVR-SNIS, which normalizes the IS ratio across the batch. As shown in Fig. 5 (right), the perfor
mance of SVR-SNIS is worse in hopper-m and hopper-m-e, and slightly better in other tasks. This discrepancy may be due to the varying requirements for variance and bias across different tasks.
Offline RL. In offline RL, extrapolation error and overestimation caused by OOD actions pose significant challenges [10]. Among various solutions, value regularization methods aim to introduce conservatism in value estimation [23, 20, 29, 6, 3, 27], while policy constraint approaches enforce proximity between the trained policy and the behavior policy, either explicitly via divergence penalties [44, 22, 13, 8], implicitly by weighted behavior cloning [5, 36, 33, 42, 31], or directly through specific parameterization of the policy [10, 11, 53]. Another line of the methods, instead, opt for in-sample learning, which involves formulating the Bellman target with only the actions in the dataset [4, 28, 21, 51, 47]. However, the performance of most existing methods is largely confined by the average quality of the dataset [25]. In contrast, SVR guarantees the convergence to the optimal support-constrained policy and achieve superior performance in sub-optimal datasets.
Value regularization. We divide the value regularization methods in offline RL into two categories: direct  $Q$  penalization and Bellman target reduction. The former involves adding penalties to the critic learning loss, trying to penalize OOD  $Q$ -values [23, 50, 46, 6, 27]. Due to the difficulty of determining the OOD region, they typically penalize  $Q$ -values under the current policy distribution [23, 3, 27, 48] and optionally incorporate a maximization term under the data distribution for milder pessimism [23, 50, 6]. Thus, existing works of this category belong to policy density based value regularization, rather than ID/OOD based. On the other hand, the Bellman target reduction methods either subtract an uncertainty quantifier from the Bellman target [14, 3] or directly use the minimum of  $Q$  ensembles [34] to compute the target [2, 9]. However, if not incorporating additional penalties, they do not provide learning signals on OOD  $Q$ -values [2]. It empirically works because after the minimization over ensemble (or subtracting the uncertainty quantifier), the random generalized OOD  $Q$ -values will be smaller than ID ones with high probability, reducing the impact of OOD  $Q$ -values on learning. However, it requires a large number of  $Q$  networks and is much more computationally expensive [2].
Support-constrained optimality. In absence of additional information beyond the dataset, the optimal support-constrained policy represents the best policy that we can hope to obtain. Recent works have considered various policy constraints to learn it, but their performance still leaves considerable room for improvement. For example, BEAR [22] attempts to keep the learned policy within the support of the behavior policy by minimizing MMD between them. However, this choice lacks a theoretical guarantee, and Wu et al. [44] empirically found that MMD has no performance gain over KL. Other works adopt a specific parameterization of the policy to constrain it to the behavior support [10, 11]. They first pre-train a generative model (e.g., VAE) for the behavior policy. Then during learning, they sample several actions at each state and choose the one with the highest  $Q$ -value as the output of the policy, aiming to obtain the maximum within the behavior support. However, these methods are vulnerable to model errors [11] and have a high computational cost for generating sufficient actions for each state. Instead of working on policy constraints, SVR is the first value regularization algorithm to achieve support-constrained optimality. Moreover, SVR only needs to query the behavior density of in-dataset  $(s,a)$  pairs, making it less susceptible to model errors.
In this work, we propose a novel value regularization method, SVR, which penalizes the  $Q$ -values for all OOD actions while maintaining standard Bellman updates for ID ones. We show that the policy evaluation operator of SVR is a contraction, whose fixed point outputs unbiased  $Q$ -values in ID region and underestimated  $Q$ -values in OOD region. Furthermore, SVR guarantees strict policy improvement until convergence to the optimal support-constrained policy. Empirical results validate the theoretical properties of SVR and demonstrate its SoTA performance on the D4RL benchmarks.
One limitation of SVR lies in the need of pre-training the behavior model. An exciting direction for future work would be to achieve supported value regularization without explicit behavior policy estimation.
This work was supported by the National Key R&D Program of China under Grant 2018AAA0102801, National Natural Science Foundation of China under Grant 61827804.
[51] Hongchang Zhang, Yixiu Mao, Boyuan Wang, Shuncheng He, Yi Xu, and Xiangyang Ji. In-sample actor critic for offline reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=dfDv0WU853R.
[52] Jing Zhang, Chi Zhang, Wenjia Wang, and Bing-Yi Jing. Apac: Authorized probability-controlled actor-critic for offline reinforcement learning. arXiv preprint arXiv:2301.12130, 2023.
[53] Wenxuan Zhou, Sujay Bajracharya, and David Held. Plas: Latent action space for offline reinforcement learning. In Conference on Robot Learning, pages 1719-1735. PMLR, 2021.
In this section, we present the proofs for all the theories in the paper.
We first restate the two definitions in the paper.
Definition 3 (Support-constrained policy, Definition 1). The support-constrained policy class  $\Pi$  is defined as
$$
\Pi = \{\pi \mid \pi (a | s) = 0 \text {w h e n e v e r} \beta (a | s) = 0 \} \tag {17}
$$
Definition 4 (Optimal support-constrained policy, Definition 2). The optimal support-constrained policy  $\pi_{\Pi}^{*}$  is:
$$
\pi_ {\Pi} ^ {*} (a | s) := \mathbb {I} \left[ a = \underset {a ^ {\prime} \in \operatorname {s u p p} (\beta (\cdot | s))} {\operatorname {a r g m a x}} Q _ {\Pi} ^ {*} (s, a ^ {\prime}) \right] \tag {18}
$$
$$
Q _ {\Pi} ^ {*} (s, a) = R (s, a) + \gamma \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a)} \left[ \max  _ {a ^ {\prime} \in \operatorname {s u p p} \left(\beta \left(\cdot | s ^ {\prime}\right)\right)} Q _ {\Pi} ^ {*} \left(s ^ {\prime}, a ^ {\prime}\right) \right]. \tag {19}
$$
Proposition 4 (Contraction, Proposition 1). In the whole  $S \times \mathcal{A}$  space and for any  $\pi$ ,  $\mathcal{T}_{\mathrm{SVR}}^{\pi}$  is a  $\gamma$ -contraction operator in the  $\mathcal{L}_{\infty}$  norm.
$$
\mathcal {T} _ {\mathrm {S V R}} ^ {\pi} Q (s, a) = \left\{ \begin{array}{l l} \mathcal {T} ^ {\pi} Q (s, a), & \beta (a | s) > 0, \\ Q _ {\min }, & \text {e l s e .} \end{array} \right. \tag {20}
$$
Let  $f_{1}$  and  $f_{2}$  be two arbitrary functions.
For all  $(s,a)$  s.t.  $\beta (a|s) = 0$  , we have
$$
\left| \mathcal {T} _ {\mathrm {S V R}} ^ {\pi} f _ {1} (s, a) - \mathcal {T} _ {\mathrm {S V R}} ^ {\pi} f _ {2} (s, a) \right| = Q _ {\min } - Q _ {\min } \leq \gamma \| f _ {1} - f _ {2} \| _ {\infty} \tag {21}
$$
For all  $(s,a)$  s.t.  $\beta (a|s) > 0$  , we have
$$
\begin{array}{l} \left. \left| \mathcal {T} _ {\mathrm {S V R}} ^ {\pi} f _ {1} (s, a) - \mathcal {T} _ {\mathrm {S V R}} ^ {\pi} f _ {2} (s, a) \right| = \left| \gamma \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a), a ^ {\prime} \sim \pi (\cdot | s)} \left[ f _ {1} \left(s ^ {\prime}, a ^ {\prime}\right) - f _ {2} \left(s ^ {\prime}, a ^ {\prime}\right) \right] \right| \right. \\ \leq \gamma \mathbb {E} _ {s ^ {\prime} \sim P (\cdot | s, a), a ^ {\prime} \sim \pi (\cdot | s)} [ | f _ {1} (s ^ {\prime}, a ^ {\prime}) - f _ {2} (s ^ {\prime}, a ^ {\prime}) | ] \tag {22} \\ \leq \gamma \| f _ {1} - f _ {2} \| _ {\infty} \\ \end{array}
$$
Therefore, we have  $\| \mathcal{T}_{\mathrm{SVR}}^{\pi}f_{1} - \mathcal{T}_{\mathrm{SVR}}^{\pi}f_{2}\|_{\infty}\leq \gamma \| f_{1} - f_{2}\|_{\infty}$
Proposition 5 (Proposition 2). For any  $\pi$ , the fixed point of  $\mathcal{T}_{\mathrm{SVR}}^{\pi}$  satisfies
$$
\left\{ \begin{array}{l l} Q _ {\min } \leq f ^ {\pi} (s, a) \leq Q ^ {\pi} (s, a), & \beta (a | s) > 0, \\ f ^ {\pi} (s, a) = Q _ {\min }, & \beta (a | s) = 0. \end{array} \right. \tag {23}
$$
Proof. By Proposition 4,  $\mathcal{T}_{\mathrm{SVR}}^{\pi}$  is a contraction and has a unique fixed point. Assume the fixed point is  $f^{\pi}$ . Thus,
$$
f ^ {\pi} (s, a) = \mathcal {T} _ {\mathrm {S V R}} ^ {\pi} f ^ {\pi} (s, a) = \left\{ \begin{array}{l l} \mathcal {T} ^ {\pi} f ^ {\pi} (s, a), & \beta (a | s) > 0, \\ Q _ {\min }, & \text {e l s e .} \end{array} \right. \tag {24}
$$
Define  $(\hat{s},\hat{a})\coloneqq \mathrm{argmin}_{s\in S,a\in \mathrm{supp}(\beta (\cdot |s))}f^{\pi}(s,a)$ , and  $p^\pi (s,a)\coloneqq \mathbb{E}_{s'\sim P(\cdot |s,a),a'\sim \pi (\cdot |s')}\pi (a'|s')$
As  $\beta (\hat{a} |\hat{s}) > 0$
$$
\begin{array}{l} f ^ {\pi} (\hat {s}, \hat {a}) = \mathcal {T} _ {\mathrm {S V R}} ^ {\pi} f ^ {\pi} (\hat {s}, \hat {a}) \\ = r (\hat {s}, \hat {a}) + \gamma \mathbb {E} _ {\hat {s} ^ {\prime}} \mathbb {E} _ {\hat {a} ^ {\prime} \sim \pi (\cdot | \hat {s} ^ {\prime})} [ f ^ {\pi} (\hat {s} ^ {\prime}, \hat {a} ^ {\prime}) ] \\ = r (\hat {s}, \hat {a}) + \gamma \mathbb {E} _ {\hat {s} ^ {\prime}} \mathbb {E} _ {\hat {a} ^ {\prime} \sim \pi (\cdot | \hat {s} ^ {\prime})} [ \mathcal {T} _ {\mathrm {S V R}} ^ {\pi} f ^ {\pi} (\hat {s} ^ {\prime}, \hat {a} ^ {\prime}) ] \\ = r (\hat {s}, \hat {a}) + \gamma \mathbb {E} _ {\hat {s} ^ {\prime}} \left[ \sum_ {\hat {a} ^ {\prime} \in \operatorname {s u p p} \left(\beta \left(\cdot | \hat {s} ^ {\prime}\right)\right)} \pi \left(\hat {a} ^ {\prime} | \hat {s} ^ {\prime}\right) \mathcal {T} ^ {\pi} f ^ {\pi} \left(\hat {s} ^ {\prime}, \hat {a} ^ {\prime}\right) + \sum_ {\hat {a} ^ {\prime} \notin \operatorname {s u p p} \left(\beta \left(\cdot | \hat {s} ^ {\prime}\right)\right)} \pi \left(\hat {a} ^ {\prime} | \hat {s} ^ {\prime}\right) Q _ {\min } \right] \\ = r (\hat {s}, \hat {a}) + \gamma \mathbb {E} _ {\hat {s} ^ {\prime}} \left[ \sum_ {\hat {a} ^ {\prime} \in \operatorname {s u p p} (\beta (\cdot | \hat {s} ^ {\prime}))} \pi (\hat {a} ^ {\prime} | \hat {s} ^ {\prime}) f ^ {\pi} (\hat {s} ^ {\prime}, \hat {a} ^ {\prime}) + \sum_ {\hat {a} ^ {\prime} \notin \operatorname {s u p p} (\beta (\cdot | \hat {s} ^ {\prime}))} \pi (\hat {a} ^ {\prime} | \hat {s} ^ {\prime}) Q _ {\min } \right] \\ \geq r _ {\min } + \gamma \mathbb {E} _ {\hat {s} ^ {\prime}} \left[ \sum_ {\hat {a} ^ {\prime} \in \operatorname {s u p p} (\beta (\cdot | \hat {s} ^ {\prime}))} \pi (\hat {a} ^ {\prime} | \hat {s} ^ {\prime}) f ^ {\pi} (\hat {s}, \hat {a}) + \sum_ {\hat {a} ^ {\prime} \notin \operatorname {s u p p} (\beta (\cdot | \hat {s} ^ {\prime}))} \pi (\hat {a} ^ {\prime} | \hat {s} ^ {\prime}) Q _ {\min } \right] \\ \end{array}
$$
The last inequality holds because  $f^{\pi}(\hat{s},\hat{a})\coloneqq \min_{s\in \mathcal{S},a\in \operatorname {supp}(\beta (\cdot |s))}f^{\pi}(s,a)$ .
Now we define a shorthand
$$
\lambda := \mathbb {E} _ {\hat {s} ^ {\prime}} \left[ \sum_ {\hat {a} ^ {\prime} \in \operatorname {s u p p} (\beta (\cdot | \hat {s} ^ {\prime}))} \pi (\hat {a} ^ {\prime} | \hat {s} ^ {\prime}) \right]
$$
$$
f ^ {\pi} (\hat {s}, \hat {a}) \geq r _ {\min } + \gamma \lambda f ^ {\pi} (\hat {s}, \hat {a}) + \gamma (1 - \lambda) Q _ {\min }
$$
Because  $\lambda \in [0,1]$ , we have
$$
\begin{array}{l} f ^ {\pi} (\hat {s}, \hat {a}) \geq \frac {r _ {\operatorname* {m i n}} + \gamma (1 - \lambda) Q _ {\operatorname* {m i n}}}{1 - \gamma \lambda} \\ = \frac {(1 - \gamma) Q _ {\operatorname* {m i n}} + \gamma (1 - \lambda) Q _ {\operatorname* {m i n}}}{1 - \gamma \lambda} \\ = Q _ {\min } \\ \end{array}
$$
Therefore, for all  $(s,a)$  s.t.  $\beta (a|s) > 0$ , it holds that  $f^{\pi}(s,a)\geq f^{\pi}(\hat{s},\hat{a})\geq Q_{\min}$ . Besides, For  $(s,a)$  s.t.  $\beta (a|s) = 0$ , it holds that  $f^{\pi}(s,a) = Q_{\min}$ . Thus  $f^{\pi}(s,a)\geq Q_{\min}$ ,  $\forall s,a$ .
Before we prove  $f^{\pi}(s,a) \leq Q^{\pi}(s,a)$  when  $\beta (a|s) > 0$ , we first prove  $T_{\mathrm{SVR}}^{\pi}f^{\pi}(s,a) \leq T^{\pi}f^{\pi}(s,a), \forall s,a$
For any  $(s,a)$ , We have
$$
\begin{array}{l} \mathcal {T} ^ {\pi} f ^ {\pi} (s, a) = r (s, a) + \gamma \mathbb {E} _ {s ^ {\prime}} \mathbb {E} _ {a ^ {\prime} \sim \pi (\cdot | s ^ {\prime})} [ f ^ {\pi} (s ^ {\prime}, a ^ {\prime}) ] \\ \geq r _ {\min } + \gamma \mathbb {E} _ {s ^ {\prime}} \mathbb {E} _ {a ^ {\prime} \sim \pi (\cdot | s ^ {\prime})} [ Q _ {\min } ] \\ = Q _ {\min } \tag {25} \\ \end{array}
$$
$$
\mathcal {T} _ {\mathrm {S V R}} ^ {\pi} f ^ {\pi} (s, a) = \left\{ \begin{array}{l l} \mathcal {T} ^ {\pi} f ^ {\pi} (s, a) \leq \mathcal {T} ^ {\pi} f ^ {\pi} (s, a), & \beta (a | s) > 0, \\ Q _ {\min } \leq \mathcal {T} ^ {\pi} f ^ {\pi} (s, a), & \text {e l s e .} \end{array} \right. \tag {26}
$$
Thus, it holds that  $\mathcal{T}_{\mathrm{SVR}}^{\pi}f^{\pi}(s,a)\leq \mathcal{T}^{\pi}f^{\pi}(s,a),\forall s,a.$
Now we prove  $f^{\pi}(s, a) \leq Q^{\pi}(s, a)$  when  $\beta(a|s) > 0$ . For  $\beta(a|s) > 0$ ,