Quasi-optimal Reinforcement Learning with Continuous Actions
Authors: Yuhan Li
Wenzhuo Zhou
Ruoqing Zhu
Venue: ICLR 2023 poster (2023)
Keywords: Continuous Treatments
Markov Decision Process
Safe Action Allocation

Abstract: Many real-world applications of reinforcement learning (RL) require making decisions in continuous action environments. In particular, determining the optimal dose level plays a vital role in developing medical treatment regimes. One challenge in adapting existing RL algorithms to medical applications, however, is that the popular infinite support stochastic policies, e.g., Gaussian policy, may assign riskily high dosages and harm patients seriously. Hence, it is important to induce a policy class whose support only contains near-optimal actions, and shrink the action-searching area for effectiveness and reliability. To achieve this, we develop a novel quasi-optimal learning algorithm, which can be easily optimized in off-policy settings with guaranteed convergence under general function approximations. Theoretically, we analyze the consistency, sample complexity, adaptability, and convergence of the proposed algorithm. We evaluate our algorithm with comprehensive simulated experiments and a dose suggestion real application to Ohio Type 1 diabetes dataset.

Published as a conference paper at ICLR 2023
QUASI-OPTIMAL REINFORCEMENT LEARNING WITH CONTINUOUS ACTIONS
Yuhan Li
Wenzhuo Zhou
Ruoqing Zhu
"University of Illinois Urbana Champaign University of California Irvine "Equal contribution
ABSTRACT
Many real-world applications of reinforcement learning (RL) require making de- cisions in continuous action environments. In particular, determining the optimal dose level plays vital role in developing medical treatment regimes. One chal- lenge adapting existing RL algorithms to medical applications, however, is that the popular infinite support stochastic policies, e.g., Gaussian policy, may assign riskily high dosages and harm patients seriously. Hence, it is important induce a policy class whose support only contains near-optimal actions, and shrink the action-searching area for effectiveness and reliability. To achieve this, we develop novel quasi-optimal learning algorithm. which can be casily opti- mized in off-policy settings with guaranteed convergence under general function approximations. Theoretically, we analyze the consistency, sample complexity, adaptability, and convergence of the proposed algorithm. We evaluate our algorithm with comprehensive simulated experiments and a dose suggestion real application to Ohio Type diabetes dataset.
INTRODUCTION
Learning good strategies in a continuous action space is important for many real-world problems (Lillicrap al.) 2015), including precision medicine, autonomous driving, etc. In particular, when developing new dynamic regime to guide the use of medical treatments, it is often necessary to decide the optimal dose level (Murphy 2003 Laber et al. 2014 Chen et al. [0066] [Zhou et al.) 202T). In infinite horizon sequential
cettsion-making settings (Luckett et al. 2019 [Sh et al. 2001 learning such a dynamic treatment regime falls into a reinforcement learning (RL) framework. Many RL
algorithms (Miih et al. 2013 Silver et al., 2017 Nachum et al. 2017 Chow et al., 2018b Hessel jet al. 2018) have achieved considerable success when the action space is finite. A straightforward approach to adapting these methods to continuous domains is to discretize the continuous action space. However, this strategy either causes a large bias in coarse discretization (Lee et al.) 2018a Cai
al. 2021a b) or suffers from the the curse of dimensionality (Chou et al. 2017) for fine-grid.
There has been recent progress on model-free reinforcement learning in continuous action spaces without utilizing discretization. In pasecy-based methods (Williams) 1992 Sutton et al. [999 Silver
et al. 2014: Duan et al.) 2016). a Gaussian distribution is used frequently for policy distribution representation, while its mean and variance are parameterized using function approximation and updated via policy gradient descent. In addition, many actor-critic based approaches, e.g., soft actor-critic (Haarnoja et al. (018b), ensemble critic (Fujimoto et al.) 20188) and Smoothie (Nachum al. 2a). have been developed to improve the performance in continuous action spaces. These works target to model a Gaussian policy for action allocations as well.
However, there are two less-investigated issues in the aforrmentioned RI. approaches, especially for their applications in the healthcare (Fatemi et al. [021] Yu et al., (02I). First, existing methods that use an infinite support Gaussian policy as the treatment policy may assign arbitrarily high dose levels, which may potentially harm the patient Yanase et al., 2020). Hence, these approaches are not reliable in practice due safety and ethical concerns. It would be more desirable to develop policy class to identify the near-optn optimal (Tang et al.] 2020), or at least safe, action regions, and reduce
Correspondence Wenzhuo Zhou wwenzhuz3 @ ucci.edu>> and Runqing Zhu <rqzhu iddu>

---

Published as a conference paper at ICLR 2023
the optimal action search area for reliability and effectiveness. Those actions out of the identified region are discriminated as non- non-optimal, and would be screened out with zero densities in the policy distribution. Second, for many real-world applications, the action spaces are bounded due practical constraints. Examples include autonomous driving with a limited steering angle and dose assignment with a budget or safety constraint. In these scenarios, modeling an optimal policy by infinite support probability distribution, e.g., Gaussian policy, would inevitably introduce a non non-negligible off-support bias shown in Figure 2 In consequence, the off-support bias damages the performance of policy learning and results in a biased decision- -aaiing procedure. Instead, constructing a policy class with finite but adjustable support might be one of the demanding solutions.
In this work, we take substantial step towards solving the aforementioned issues by developing a novel quasi-optimal learning algorithm. Our development hinges upon a novel quasi-optimal Bellman operator and stationarity equation, which is solved via minimizing an unbiased kernel embedding loss. Quasi-optimal learning estimates an implicit stochastic policy distribution whose support region only contains near-optimal actions. In addition, our algorithm overcomes the difficulties of the non- smoothness learning issue and the double sampling issue (Baird 1995), and can be easily optimized using sampled transitions in off off-policy scenarios without training instability and divergence. The main contribution of this paper can be summarized as follows:
We construct a novel Bellman operator and develop a reliable stochastic policy class, which is able identify quasi-optimal action regions in scenarios with a bounded or unbounded action space. This address the shortcomings of existing approaches relying on modeling an optimal policy with infinite support distributions.
We formalize an unbiased learning framework for estimating the designed quasi-optimal policy. Our framework avoids the double sampling issue and can be optimized using sampled transitions, which is beneficial in offline policy optimization tasks.
We thoroughly investigate the theoretical properties of the quasi-optimal learning algorithm, including the adaptability of the quasi-optimal policy class, the loss consistency, the finite- sample bound for performance error, and the convergence analysis of the algorithm. Empirical analyses are conducted with comprehensive numerical experiments and real- world case study, to evaluate the model performance in practice.
2 PRELIMINARIES
Notations We first give an introduction to our notations. For two strictly positive sequences (m)))m22 and {T (m))m21 the notation (m) S ({(m)) |m>1 means that there exists sufficiently small constant 2 0 such that (n)) S cT()). - and lloc denote the L norm and supremum- norm, respectively. We define the set indicator function 1l
1 x E set 0 otherwise. The notation P,, denotes the empirical measure i.e., P, 1 > -i=1- For two sets No and N1, the notation No N, indicates that the set No excluding the elements in the set N₁. We write [Nol as the cardinality of the set No. For any Borel set N2, we denote (N2) as the Borel measure of N2. We denote a probability simplex over a space F by (FF), and in particular, Acowex (F) indicates the convex probability simplex over F. We denote [-] as the floor function, and use O as the convention.
Background A Markov decision process (MDP) detined as tuple < S, S,A, >, where S is the state space, A is the action space. P: S x A + A(S) is the unknown transitional keree,, R x S x A + R is a bounded reward function, and ? E [0. 1) is the discounted factor. In this paper, we focus on the scenario of continuous action space. We assume the offline data consists of n i.i.d. trajectories, i.e., D1.,. (S), A/, R S?....,.... S. AT R,. +f -== Ji=1, where the length ..... of trajectory T is assumed to be non-random for simplicity. A policy π map from the state space to the action space π: S + A. The learning goal is to search optimal policy x² which maximizes the expected discounted sum of rewards. V.T(s) = Ex |>e- k-1 - == =] is the value function under policy x, where E,, is taken by assuming that the system follows policy æ, and the Q-function is defined as QT (s. a) = Ex [Ex=> 2 R = s, A' A' =]] In a hime-homogenous Markov process (Puterman 2014 V." (s) and Q (s, a) do not depend on 1. The optimal value function V* is the unique fixed point of the Bellman operator B, (V)s) == max, (g++:=((..)) R +V = s. A' -===0p a). Then (s) = V*(() for any E S. An optimal policy π can be obtained by taking the greedy action of Q" a), that is n (s) = arg max, (' a). For the rest of the paper, we use the short notation Es/s, for the conditional expectation E. and Es A s544 is short for As St) ((St) ,A)), where v is a some fixed distribution and T6 is some behavior policy.
2

---

Published as a conference paper at ICLR 2023
3 METHODOLOGY
To start with, we first revisit the Bellman optimality equation via a policy explicit view,
BV BV (s) = max (.-x)(1)). |tt [R(S*+) ,, a) + (*)).
(I)
To obtain the optimal policy 'T" and value function V an optimization idea is to minimize the discrepancy between the two sides of the equation under a 7.2 loss. Unfortunately, there are several major challenges when it comes to optimization: (1) Non-smoothness: the Bellman operator involves a non-smoothed hard-max operator, which leads training instability; (2) Policy class: As discussed in Section [ it is necessary to induce optimal policy class whose support consists of quasi-optimal sub-regions for reliability, and avoids off-support bias Figure (3) Double sampling: the unknown conditional expectation [s, is required to be double sampled for obtaining an unbiased sample approximation for Es.+1|s.a. However, this is usually infeasible in real-world environments; (4) Off-policy data: directly minimizing the Bellman error is not easy incorporate off-policy data. To address these issues, we propose a quasi-optimal counterpart of the Bellman equation (T).
3.1 QUASI-OPTIMAL BELLMAN OPERATOR
In this subsection, we aim to tackle the first two challenges. We propose a quasi-optimal counterpart for the Bellman operator B that simultaneously circumvents the non-smoothness obstacles, and induce a novel policy class which can identify quasi-optimal sub-regions continuous action spaces.
We leverage the Legendre- Fennheel transform HhriarttUrruty & Lemaréchal 2012) on the Bellman operator B. For a convex probability simplex Aguaves (A) and a strongly convex and continuous proximity function prox(1)) Aeaves (A) - R. the Fenchel transform counterpart of B is defined as
()) = max
(()(ssa)T((as) + (prox(T(a(s))) da,
(2)
#E (A) aEA
where (7, a) = (a 8, + + and (s) is the unique fixed point of the quasi-optimal Bellman operator Bμ. Note that, besides the smoothing purpose, are also
interested in constructing a stochastic optimal policy class that can screen out the non-optimal and sub-optimal actions. Therefore, we further define special prox function class motivated by
the rationale of q-logarithm as prox(x) logg = =(1->--1) where Sac prox ( (als))da
q-1
q-1 (1- x (aaas)d essentially generalize the Shannon's 's entropy (Martins al. 2020). In this paper, we focus on the setting that q =
Assumption 3.1. For any policy distribution constant, i.e., (-1s)) s for all E S.
(A)mer(A), its density is bounded above by
This ussumption avoids some extreme cases where a stochastic policy distribution degenerates to be deterministie. the following, we show several nice properties of the proposed Bellman operator. Proximal Approximation The operator B,, is a proximal approximation to B. This delivers two messages: firstly, the approximation bias is upper bounded; secondly, the operator B,, is smoothed substitute for B3 In particular, Theorem 3.1 demonstrates that the approximation bias can vanish to zero for small enough μ. In addition, the operator B,, has differentiable and analytical form ( which justifies that B,, is a smoothed counterpart of B, see Corollary (S. in Appendix for details.
Theorem 3.1 (Proximal bias). Under Assumption 3.1 for any s S and value function V, B2 Vs)) (s) E [p(1 C),
1 Uweww, (2 241)
/ Jrew. 013(s,a)do,
(2 = 41
(3)
o(W.)
where W, denotes the the support of .ř in for a given state s.
Quasi-optimal Support Region In addition to the proximal approximation property, another unique and important property B,, inducing a policy *" whose support region contains all the actions with action-value higher than certain threshold. The induced policy nj is bridged from the oracle Q-function:
(als) = (7,
Snew, (1, (W,,
1 1)
2μ
(W.))
3

---

Published as a conference paper at ICLR 2023
(s(sa))
(s
-2
-2
0
2
-1
0
2
Figure An illustrating example of the quasi-optimal sub-regions. In the left panel, the lowest admissible action-value corresponds to the horizontal red dashed line, and the integral difference is the shadowed pink area, which equals 2p. As shown in the right panel, when μ decreases, the pink area shrinks, and the quasi-optimal sub-regions become narrower.
where the support of TH i.e., W. =- UnEA al screeniing set (a) with
screening set set == : aEA: J
JEEMiN
Vj(s,a')da' (MM))))))() > 20j. (5)
M.(a) = U (1110)(aa))2)))0)(())) d'EA
(6)
This mechanism allows us to identify multiple sub-regions in the entire action space which only contains near-optimal actions, and weed out the sub-optimal and non-optimal support regions. Note that, the identified sub-region might not be joint in general. which is beneficial to the situation that the true Q-function has multiple modes. The screening set in (5) indicates that the threshold parameter μ not only controls the degree of smoothness, but also determines how the quasi-optimal region behaves and controls the screening intensity, as shown in Figure
3.2
q GAUSSSINN POLICY DISTRIBUTION
In this section, we bridge the induced policy distri- bution = to an explainable g-Gaussian distribution. The g-Gaussian distribution is less favored for heavy tails, which makes it widely used in practice to model the effect of external stochasticity (d'Onoriio 2013).
Geussin Policy
0.75
Gouneat Palcy Ford
Palcyy 1/wa
Than Aaran
In continuous actions problems, e.g., medical dose suggestion, the q- Gaussian distribution a more suit- able choice than the Gaussian distribution for policy modeling, since it can filter out non-optimal and risky dose levels, i.e., too high too low dosage.
Denaly
0.50
of bix
0.25
Motivated by the fact that the induced policy .. is feasible to identify quasi-optimal support sub- regions, and g-Gaussian policy distribution can re- alize bounded support, conjectured that the q- Gaussian policy distribution might be recovered from the induced policy Th Fortunately, the g-Gaussian policy distribution is indeed special case of the induced policy if Q7, (s, a) is concavely quadratic function with respect to the action a. We illustrate this phenomenon Theorem 3.2
0.00
3
Action
Figure 2: An illustrating example of bounded action space and g-Gaussian policy distribu- tion. The Gaussian policy assigns non-zero probabilities density to all actions, even for those actions outside of the true action space support boundary. This causes the off off-support bias. In contrast, the q g-Gaussian policy re- lieves such off- off-support bias blessed by the boundedness of the quasi-optimal region.
Theorem 3.2. Suppose (s, is a concavely quadratic function over a E A, i.e. (s, = -o1 (s)aa + az(s)a + as(s) = QN (s. a) where on (s), az (s). ass(s) are functions over s S and (1(s) 0 for all s, then the induced policy distribution Tú (-|s) would follow q- Gaussian distribution with a density function
(01(() (a
a2(s)
(a1(s)
(als)
= TH (als),
2p
20 (s)
2
12
4

---

Published as a conference paper at ICLR 2023
and closed-form quasi-optinal support region
W (22(s) (12a{(5))) a2(s) + (12a)()) (01
W
(8)
201 (s)
The policy distribution distribution with mean
TH .a(s)) where the maximum action-value attains, i.e., Q (2,,-1110 221
T/ (-1s) behaves as a affine transformation of the standard q-Gaussian
(s)
arg a O (s, a). Note that the width of the quasi-optimal region is (12a](+)u) determined by (1
the threshold parameter μ.. The actions within the region IR' \ W.Y. are discriminated as the non-optimal and would be assigned with zero probability densities. For small μr, i.e., strong screening intensity, narrow region would be identified as the quasi-optimal, which yields a relatively conservative action recommendation. In contrast, with large Jr, more actions are included in the support. In an extreme case, W degenerates to IR as + co. In Theorem 1f 4. of μl affects the induced policy distribution formally.
Section we investigate how the intensity
So far, we have obtained the closed-form representations for the general policy (-1s) and q-Gaussian policy " However, how make policy estimation remains unknown. Indicated by the challenges in Section 3 we need to address the double sampling issue and utilize off-policy data in optimization. Both challenges cannot be easily solved by minimizing the Bellman error. Fortunately, the kernel embedding helps us to bypass the difficulties.
3.3 KERNEL EMBEDDING ON QUASI-OPTIMAL ERROR
In this subsection, we introduce the quasi-optimal learning framework for solving the induced policy TH. First, we establish stationary equation in Theorem 3.3 This helps incorporate off-policy
data. Then we leverage the idea of the kernel embedding Gretton et al. 2012) to obtain an unbiased empirical loss without the double sampling issue.
Theorem 3.3 (Stationarity equation). Let Vi be a fixed point of the quasi-optimal Bellman operator
By and t is the induced policy in (4). For any E S,a A, and μ (0, oo), the pair (Vi, mx) satisfies the following equation:
Ege+1 1s,0 [R(s++1 .),a) + (V puprox
n(s) + - Vu(s). (9) ( (Tp
Here prox (x) = 2r 1, nys) 8 I 0) and (s, a): SxA R are Lagrange multipliers that (s, Ny (a|s) O. The discrepancy between the two sides of p is "quasi-optimal error".
The equation ( connects quasi-optimal value function y and policy function T along with any arbitrary state-action pair. This provides an easy way to incorporate off-policy data, i.e., the state-action pairs which are sampled from state-action visitation under the behavior policy, without adjusting the distribution mismatch.
Min-max Optimization One way to solve the equation p is minimizing the quasi-optimal error under a 2 loss function. Unfortunately, the double sampling issue would still appear if replacing the unknown Ege-ijs.a [R( s, a) + (V*(+++) in the quasi-optimal error by its mample bootstrapping counterpart R + (V*+(t++) Alternatively, inspired by the average Bellman error (Jiang et al. (0117), we propose to minimize a weighted average quasi-optimal error, and the unwanted conditional variance of the bootstrapping counterpart under I22 loss could vanish. We define the loss L(V,, pr, TT. 1), x, u)
Est, [u (St, A²) (Gv,..,, AS s :(((n((((())))
+ ( (0) - VS*)),
where Sv (s, a, 8') == A(s', s, a) + (V')s') μprox x (IT,
(als)) and u(-): S x A - R is
bounded function in 2 space (C,) := {u E L ||a|| < C.). Essentially, the weight function is to fit the discrepancy of ( and promotes the sample points with large quasi-optimal errors.
As L(V+ The 7. u) = 0 holds for any function, this leads to minimax optimization:
min max (V na u.) V Va oe (Coo)
(10)
5

---

Published as a conference paper at ICLR 2023
Algorithm 1 Quasi-optimal Learning in Continuous Action Spaces
1: Input observed transition pairs data A R, s4++1 ): :t TT
2: Initialize the parameters of interests (6, {) = (0" the mini- size no. the learning rate
Oro. the prox parameter μ., the kernel bandwidth bwg. and the stopping criterion E.
3: For iterations j = 1 to k
4: Randomly sample a mini-batch {(S) 51 A. R 5(++) = -
Th
5: Decay the learning rate aj = (()
6: Compute stochastic gradients with respect to and E: Vo =P VoLy and Vg =Pro -ProVeli
VeLo. 7: Update the parameters of interest as A t- G aj VoLu, , e + ----1 - ag VeV.Lu.
8: Stop if (0), .)) (00-1, -1 S.
9: Return -- 05, E g)
Kernel Representation Solving the minimax optimization problem (10) is unstable, and it is also intractable due to the difficulty for the representation of u in L space. Fortunately, we identify continuity invariance between the reward function and the optimal weight function u² (see Theorem S.2 in Appendix). The optimal u" (-) is continuous as long as the reward function is continuous, which is widely satisfied in real-world applications.
for positive definite kernel K, bounded
reproducing kernel Hilbert space (RKHS) HRKHS (Co) == {u E HRKHS ||u] K < Co has diminishing approximation error to any continuous function class as Co - ac (Bach) 2017). This together with continuity invariance provides us basis for representing the weight function in bounded RKHS. This kernel representation further leads to a closed-fore of the inner optimization maximizer (Gretton et al. 2012). The detailed derivation is provided in Theorem S.3 in Appendix. Upon this, the minimax optimization is reduced to only minimizing the loss
LV Egg
(Av, n (S*, A S K((" A A')AA, u. (5,, S0, (11)
where Ay,., (s, s') == Ov..ma (s, a.a 8 - (]s) + x(s.a) Vu((s and A", is an independent copy of transition pair (S*, A', St
It observes that the loss Ly is symmetric and kernel represented. This motivates U-statistic estimator to obtain the sample loss. Given the observed data, Dia with n trajectories of length T. thus the total loss Lu can be aggregated as the empirical mean of n i.i.d. within trajectory loss:
unbiased
use trajectory- baased U-statistic estimator capture the within-trajectoon loss,
min Eo = Pn 77
[Av a (S) AC, ( A A4)Ay, (5,,A
s
ISisKST
s.t. w(als) 2 0. T (als) (a)s) 0 and ry(s) E (-wC, 0] for all s SE A.
(12)
The sample loss Eu is unbiased and consistent with the population loss Lu. The consistency is justified in Theorem 4.2 via examining the tail behavior of Lv. In essence, solving the equation (12) a computationally intensive non-linear programming problem. Alternatively, we convert the constrained problem to unconstrained problem by restricting the Lagrange multipliers. Thus, it can be solved by an unconstrained true gradient algorithm, i.e., Algorithm [under function approximation (Vp Tee TJ, w) = (V mu 1f, w)); see Appendix for details.
THEORY
In this section, we study the theoretical properties of the proposed method. First, study some general properties of the proposed quasi-optimal Bellman operator, given in Proposition S.T and S.2 of Appendix. In Theorem 4.1 we disclose the effect of the intensity of prox parameter f on the induced optimal policy distribution. Moreover, a non-asymptotic concentration bound is established in Theorem 4.2 showing the consistency and measuring the rate of convergence of Lú to Lv. Further, the overall performance error of the algorithm is given in Theorem 4.3, where the performance error is decomposed as the four sources. Finally, we show that the proposed quasi-optimal learning is convergent algorithm. Before we present the theoretical results, we introduce some assumptions on the boundedness condition of the MDP and the sample trajectory properties, respectively.

---

Published as a conference paper at ICLR 2023
Assumption 4.1. The reward function R(s', s. a.) is uniformly bounded, iee, |R(-))loo I &Rmax-
Assumption 4.2. Suppose {5"
(121 A is a strictly stationary and exponentially B-mixing sequence with a mixing coefficient (mm) I exp(-6mm) for m 2 1. We further assume that the behavior policy Th, which is used to collect the offline data D satisfies that minae To(aas) > O.
Theorem 4.1 (Policy Adaptability). Under Assumption 4.1 for all S, the quasi-optimal policy distribution T (-1)) degenerates to a uniform distribution over ((A) as + 0c, and T (-1s)
concentrates in a point mass as μ - 0 and C 00.
Theorem 4.1 formally investigates the effect of μr on (-)s). In extreme case that μ 0, C oc, only the action maximizing Q7 a) would be included in the quasi- region. In the following, we establish a non-asymptotic concentration inequality for the empirical loss in the non-i.i.d. case. we have e-divergence of
Theorem 4.2. For any μ E (0, oo) and O, under Assumptions ICu - Lul bounded in probability, i.e.,
4.1 4.2
P(ICo - Lul > x) 5 C1 exep (- max +(
eT -C VT max
+ C3 exp
T
C2MMiss
log T log (T) /T
M1
max
where Cr and C3 are some constants depending on ō1 respectively, and M Theorem 4.2 implies that Ly is a consistent estimator to Lu, and thus avoiding the double sampling issue. Note that the concentration bound is sharper than the bound established in Chakrabortty & Kuchibhotla (2018) since we utilize a novel temporal correlatedness structure to decompose the U-statistic. We now analyze the performance error between the finite sample learner and true solution, which can be decomposed into four source errors.
Rmax + C.C.
Theorem 4.3, Under Assumption 4.1 4.2 let V be the optimizer from Algorithm and V is the optimal value function and Kmin be the smallest eigenvalue corresponding of (2 x A) space. With probability 1 - 5, the performance error is upper bounded by
orthonormal basis
1 V* < Kmin (1 77)
C1
(V
C5 CrsDraim log
(80,
( V A Co [T/2
n
generalization CrIO
C7 1-C++1-Clv1y v1)
+cs 1 Do
+ Eapprosimation errors
(1-
proximal bias
optimization evror
where Â Dram log, T/2] + log(5) + log* ( (CoC Cocpeain P-dim(E)) P-dim (2)), and CA, .... Cs are some constants. Here P-dim(-) denotes the pseudo- dimension operator (Györf 2010). and θ1, O2, and 32 are are function spaces for Vu:Tup w and n, respectively. The Capproximation evror is from parametrization (V/+,,
2 Dy-dim P-dim((1)) + P-dim(
"r,,,...,.... on (Vus T w, n)). The above sample complexity bound gives an insight into the performance error of the proposed algorithm. The generalization error Egen (LLVTT) if is as the same order of T, the proximal bias E prox Ou²²) and the optimization error Eoptim = O(1/k) for k iterations. Although the prox function introduces proximal bias in the quasi-optimal Bellman operator B,, it leads to smoothed approximation for B. There exists a trade-off between the proximal bias and approximation error. As the increase of μ. it enlarges the proximal bias but decreases the approximation error since true function space becomes more smoothed and easy for function approximation. On the other hand, a small μ leads to small proximal bias but relatively large approximation error.
Theorem 4.4. Suppose Ey in Algorithm 1 differentiable, but not necessarily convex, and its gradient (O, is Mi-Lipcchitz and Var(Ve + Ve) S og And suppose that the learning rate A for some A Z and e is sufficient small. Let ke with ME. ToV3
)a,) are set to aj- = min 2
(,(2) PCk = -) (ay1 Mea,)) for 1....,,k ... Then, if (0,E) is the optimization solution and
(0',g') is the first step solution, we have
Ko McAAko Mc + (. s 22Mc (C (Ou(0))))) min (0) 0.5
ou
AooMc Vio
7

---

Published as a conference paper at ICLR 2023
Theorem 4.4 implies that the quasi-optimal learning algorithm is converges to stationary point with a sub-linear rate (1///k)) even if the empirical loss is non-convex. The property serves as a basis for applying non-linear function approximation with convergent guarantees. Theorem 44.4is adapted from Corollary 2.2 in Ghadimi & Lan (2013) under a decay learning rate and a Fuclidean stopping criterion. The convergence of Algorithm| blessed by our unbiased stochastic gradient estimator.
RELATED WORKS
In this work, we propose a provably convergent and sample efficient off-policy optimization algorithm. Our learning algorithm trained in a fully offline fashion, without any future online interaction with the environment. This connects our work to offline RL algorithms (Lange et al., 2012: Levine et al. 2020 Due to space limitations, we defer the discussion on offline RL in Appendix.
Algorithmically, our work is related to the entropy-regularize reinforcement learning algorithms (Rawlik et al. [012] Haarnoja et al. (017)) but these works are fundamentally different from ours. Our formulation is motivated by constructing proximal counterpart of the Bellman operator, which serves as a basis for the latter quasi-oracle learning algorithm. Besides, the major drawback of the existing algorithms (Lee et al. 2018b Chow et al. 2018b Vieillard et al. 2020) is the lack of theoretical guarantees when accompanied by function approximation. It is not clear whether the algorithm is convergent, generalizable, and consistent. In contrast, our algorithm is thoroughly examined on both theoretical and empirical fronts. Nachum et al. (2017); (hoow et al. (2018b) exploit an analogous stationarity condition as in Theorem (3. and minimize the upper bound of the error, which is biased and encounters double sampling issue. In contrast, our work leverages the kernel embedding to bypass the double sampling issue, and is provably consistent. Unlike our algorithm, the algorithms in continuous control problems, e.g.. (Haarnoja et al. 2018b Nachum et al., [20 8b) Lee et al. 2019) do not check the policy optimality, but separately model pre-specified policy class. This may introduce an additional bias if the pre-specified policy class is misspecified.
Our approach exemplifies more recent efforts that aim learn optimal policy with continuous actions (Lillicrap et al., 2015). One of our key innovations is to develop policy class that can identify quasi-optimal sub-regions and the induced policy has a closed-form regarding value function. This distinguishes us from the approaches, e.g., (Silver et al. 2014; Mnih et al. 2016: Kumar et al. 22019) 2020p. These methods typically require prior know ledge to determine pre-specified policy class and commonly use Gaussian family distribution, but unfortunately facing the risk from off-support bias.
Our work is also relevant to reward, e.g., the quantile of return, it draws connections to our algorithm. Given potential application scenarios, quasi-optimal learning is also related to RL in healthcare domain and the trade-offs between safety and optimality. [Tang al. (2020) constructs set- valuve policies of near-optimal actions allowing the interaction between the clinician and the decision support system. However, their method is not applicable in a fully offline setting. Fatemi et al. (2021) assesses regions of risk and identifies treatments to avoid in a safety-critical environment. Nevertheless, near-optimal regret guarantee vacuous in their framework. Due to page limits, we provide a detailed discussion on safe and healthcare RL in Appendix.
RL. When the risk measure is defined based on the
6 EXPERIMENTS
In this section, we evaluate our proposed method on synthetic and real environments. We compare our method to the state-of-the-art baselines including DDPG (Lillicrap et al. (2015). SAC (Haarnoja) al., 201 Sa), BEAR (Kumar et al., 2019). Greedy-GQ (Ertefaie Strawderman (018)) V-Learning (Luckett et al.] 2019). We also compete with two safe RI. algorithms CQL (Kumar et al. 2020) and IQN (Dabney et al. 2018a) for a comprchensive comparison from the safety RL point of view.
INN
COL
S4C REAR
Duady 60
Sangle
Sampae 22t
Sample Size
Sansae
Figure 3: The boxplot of the discounted return over 50 repeated experiments.
8

---

Published as a conference paper at ICLR 2023
Synthetic Data The four enviromments are simulated to mimic the real environments for continuous treatment applications. In Environment and II, we consider a bounded action space to evaluate the potential of quasi-optimal learning for addressing off-support bias. The design of Environment III is to mimic safety-critical environment by incorporating the notion of safety into the reward function (Tia et al. 2020), i.e., the optimal dosage is unique, and high dosage leads excessive toxicity while a lower dosage is ineffective (Zagg et al. [014)). This is helpful for examining safety performance. In Environment IV, all the methods are implemented and compared in more complex environment. The detailed discussion on the experiment designs and settings is deferred to Section D in Appendix.
Figure 3 shows that our proposed method outperforms competing methods with a relatively small variance. This mainly benefits from identifying the quasi-optimal region, which guarantees the suggested action near-optimal, hence improving the performance. In comparison, SAC and BEAR use Gaussian policy and assign non-negligible positive densities to all actions, even for the non- optimal ones, which damages the model performance. Meanwhile, even though safe RI. methods (i.e., CQL and IQN) show better performance and smaller variance compared with non-safe methods, their performance is still negatively affected by assigning non-zero densities to non-optimal actions. In addition, in Environment I and II with bounded action support, the competing methods are affected by an off-sutport bias which lowers their discounted return. In Environment III and IV, the performance gains of the proposed method are mainly from the well-recover of the quasi-optimal regions. Also, note that our algorithm achieves stable performance in small sample size settings, which is blessed by the smoothness and optimization-friinnn -rrenddyy of our algorithm. This is promising as limited data is common in medical applications. Additional experiment results including safety criterion, i.c., distribution of the discounted sum of rewards, sensitivity analysis of μ are provided in Appendix.
Real Data: A Ohio Type Diabetes Case Study Ohio type 1 diabetes (OhioTIDM) dataset (Marling & Bunescu 2020). which contains cohorts of patients with Type-1 diabetes, each patient with weeks of life-event data including health status measurements and insulin injection dosage. Clinicians are interested in adjusting insulin injection dose levels (Marling & Bunescu 2020 Bao et al. 2011) based on patient's health status to maintain the glucose level in certain range for safe dose suggestions. As each individual has dramatically distinctive glucose dynamics, We follow Zhu ct al. (2020) to regard each patient data as independent dataset, and the data from each day as a trajectory. The state variables are health status measurements, and the action space is a bounded insulin dose range. The glycemic index is regarded as reward function to measure the goodness of dose suggestion. See more details of the experiment setup in Appendix. As shown in Table the proposed method achieves the best performance among almost all patients. The proposed method mitigates the off-support bias in this bounded dosage space and outperforms the competing methods. This finding is consistent with the results in the synthetic data and demonstrates the potential of our method in continuous action spaces. Besides model performance, we illustrate the safety guarantee of the quasi-optimal learning with additional experiments results and analyses in Appendix.
Table 1: The discounted return for the policy improvement based on 50 repeated experiments.
Patient
Proposed
DDPG
SAC
BEAR
Greedy-GQ
GQ
VL
CQL
IQN
540
18.6 + 0.6 14.1 2.3
14.2+1.2
13.7 =09
15.5 2.4
14.1 +
2.4
17.0 0.9 18.2 + 0.9
544
11.0 1.5
75
5.9
2.9
8.1
9.3
1.0
552
6.4
4.8
5.7
3.6
4.1
5.2
6.7 6.1
567
29.9
3..0
27.3
2..6
24.8
29. 10880000111111
002-283155111
584
32.1 27.0
23.3
26.9
1.2
596
5.5
4.1
0.5
2.7
1.
3.7
4.6 1.7
559
24
1.21 201111
19.6
097
17.3
200.6..22..222.1 20.6
22,1
1.2
563
11.6 0.6 8.4
9.3
8.4
9.2
8.8
9.4
0.8
570
25.0 0.8
26.1
25.8
0.8
22.8
22.6
0.9
0.8
575
15.5 = 1.0
10.4
14
10.2
1.0
5.7
8.5
02.
1.2
588
18.6 1 0.7 14.2
13
13.5
12.0
- 0.9
10.0
8.6
1.8
1.3
591
15.4 = 10 12.3
11.9
0.6
12.8
0.7
10.7
10.5
0.6
15.2
7
CONCLUSIONS
We introduce novel quasi-oracle learning algorithm for continuous action allocations, which is particularly useful in determining the dose level when developing medical treatment regimes. The quasi-optimal learning algorithm is provably convergent in off aff-policy cases, and PAC bound is provided to analyze its sample complexity. The promising results arise some interesting directions for future works, including extending the framework to online settings interacting with environments.
9

---

Published as a conference paper at ICLR 2023
8 ACKNOWLEDGMENTS
The authors are grateful to the anonymous reviewers and area chair for valuable comments and suggestions. Ruoqing Zhu's 's research partially supported by grant from the National Science Foundation DMS-2210657.
REPRODUCIBILITY STATEMENT
We include the reproducible code for all the experiments and the guideline for access to the Ohio Type I Diabetes dataset in GitHub link [nttps://github. www./ooiaaann255 Quasi-opt: i imal Lea -Larrning The experiment details are provided Appendix for reproducible purpose. All proofs of main theorems and addition theorems are included in Appendix.
10 ETHICS STATEMENT
The proposed method aims at finding optimal policy in continuous action space, with special focus on medical applications. We believe the proposed work has potential applications in sequential treatment dose suggestion, e.g., managing diabetes through insulin injection. We admit that the proposed method needs additional validation experiments in controlled settings for practical use in medical applications to avoid abundant risks. The proposed algorithm should not be used as stand-alone tool nor replacement of human experts.
REFERENCES
András Antos, Csaba Szepesvári, and Rémi Munos. Value-iteration based fitted policy iteration: learning with single trajectory. In 2007 IEEE international symposium approximate dynamic programming and reinforcement learning, pp. 330-337. IEEE, 2007.
András Antos, Csaba Szepesvári, and Rémi Munos. Learning near-optimal policies with bellman- residual minimization based fitted policy iteration and single sample path. Machine Learning, 71 (1):89-129, 2008.
Miguel Angel Arcones and Bin Yu. Central limit theorems for empirical andu-processes of stationary mixing sequences. Journal of Theoretical Probability, 7(1):47-71, 1994.
Francis Bach. Breaking the curse of dimensionality with convex neural networks. The Journal of Machine Learning Research, 18(1 18(1):629-681, 2017.
Leemon Baird. Residual algorithms: Reinforcement learning with function approximation. In Machine Learning Proceedings 1995, pp. 30-37. Elsevier, 1995.
Jiansong Bao, Heather R Gilbertson, Robyn Gray, Diane Munns, Gabrielle Howard, Peter Petocz, Stephen Colagiuri, and Jennie C Brand-Miller. Improving the estimation of mealtime insulin dose adults with type diabetes: the normal insulin demand for dose adjustment (nidda) study. Diabetes Care, 34(10):214--2151 2011.
Dimitri P Bertsekas. Nonlinear programming. Journal of the Operational Research Society, 48(3): 334-334, 1997.
Hengrui Cai, Chengchun Shi, Rui Song, and Wenbin Lu. Deep jump learning for off-policy evaluation in continuous treatment settings. Advances in Neural Information Processing Systems, 34:15285- 15300, 2021a.
Hengrui Cai, Chengchun Shi, Rui Song, and Wenbin Lu. Jump interval-learning for individualized decision making. arXiv preprint arXiv:2111.08883 2021b.
Abhishek Chakrabortty and Arun Kumar Kuchibhotla. Tail bounds for canonical u-statistics and u-processes with unbounded kernlss. 2018.
10

---

Published as a conference paper at ICLR 2023
Guanhua Chen, Donglin Zeng, and Michael R Kosorok. Personalized dose finding using outcome weighted learning. Journal of the American Statistical Association, 11 1596 2016.
Jinglin Chen and Nan Jiang. Information-theeoriicc considerations in batch reinforcement learning. In Internatioal Conference on Machine Learning, pp. 1042-1051, PMLR, 2019.
Po-Wei Chou, Daniel Maturana, and Sebastian Scherer. Improving stochastic policy gradients in continuous control with deep reinforcement learning using the beta distribution. In International conference on machine learning, pp. 834 843. PMLR, 2017.
Yinlam Chow, Ofir Nachum, Edgar Duzmaz, and Mohammad Ghavamzadeh. A Iyapunov- based approach to safe reinforcement learning, Advances in neural information processing systems, 31, 2018a.
Yinlam Chow, Ofir Nachum, and Mohammad Ghavamzadeh. Path consistency learning in tsallis entropy regularized mdps. In International Conferencee Machine Learning, pp. 979-988, 201 8b.
Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. In International conference machine learning, pp. 1096- 1105. PMLR, 2018a.
Will Dabney, Mark Rowland, Mare Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018b.
Bo Dai, Albert Shaw, Lihong Li, Lin Xiao, Niao He, Zhen Liu, Jianshu Chen, and Le Song. Sbeed: Convergent reinforcement learning with nonlinear function approximation. In International Conference on Machine Leurning, pp. 1134-1134. PMLR, 2018
Alberto d' d'Onofrio. Bounded noises in physics, biology, and engineering. Springer, 2013.
Yoel Drori and Ohad Shamir. The complexity of finding stationary points with stochastic gradient descent. In International Conference on Machine Learning, pp. 2558-2667. PMLR, 2020.
Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In International conference on machine learning, PMLR, 2016.
pp.
Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6, 2005.
Ashkan Ertefaic and Robert L Strawderman. Constructing dynamic treatment regimes over indefinite time horizons. Biometrika, (564)):9639977,000
Amir-massoud Farahmand, Mohammad Ghavamzadeh, Csaba Szepesvári, and Shie Mannor. Reg- ularized policy iteration with nonparametric function spaces. The Journal of Machine Learning Research, 171444809-4874, 2016.
Mchdi Fatemi, Taylor W Killian, Jayakumar Subramanian, and Marzych Ghassemi. Medical dead- ends and learning identify high-risk states and treatments. Advances in Neural Information Processing Systems, 34:4856-48777 4870, 2021.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference on machine learning, pp. 1587-1596. PMLR, 2018.
Javier Garcia and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437-1480, 2015.
Saced Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization, 23(4):2341-236 2013.
Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf. and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773, 2012.

---

Published as a conference paper at ICLR 2023
Shangding Gu, Long Yang, Yali Du, Guang Chen, Florian Walter, Jun Wang, Yaodong Yang, and Alois Knoll. A review of safe reinforcement learning: Methods, theory and applications. arXiv preprint arXiv:2205.10330, 2022.
László Györfi. A distribution-Fii theory of nonparametric regression, 2010.
Tuomas Haarnoja, Haoran Tang. Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International Conference on Machine Learning, pp. 1352-1361. PMLR, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International Conference on Machine Learning, pp. 1861-1870. PMLR, 2018a.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv: 18/2.05905 2018b.
Fang Han. An exponential inequality for u-statistics under mixing conditions. Journal of Theoretical Probability, 31(1):556-573 2018.
Katharine E Henry, David Hager, Peter J Pronovost, and Suchi Saria. A targeted real-time early warning score (trewscore) for septic shock. Science translational medicine, 7(299):299ra122 122- 299ra122, 2015.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Thirty-second AAAI conference on artificial intelligence, 2018.
Jean-Baptiste Hiriart-Urruty and Claude Lemaréchal. Fundamentals of Convex Analysis. Springer Science & Business Media, 2012.
Wassily Hoeffding. Probability inequalities for sums of bounded random variables. The Collected Works of Wassily Hoeffding, pp-
Springer, 1994.
Matthew W Hoffman, Alessandro Lazaric, Mohammad Ghavamzadeh, and Rémi Munos. Regularized least squares temporal difference learning with nested 2 and 1 penalization. In European Workshop Reinforcement Learning. pp. 102-1144 14. Springer, 2011.
Yan Jia, John Burden, Tom Lawton, and Ibrahim Habli. Safe reinforcement learning for sepsis treatment. In 2020 IEEE International Conference on Healthcare Informatics (ICHI), pp. 1-7. IEEE, 2020.
Nan Jiang, Akshay Krishnamurthy, Alekh Agarwal, John Langford, and Robert E Schapire Contex- tual decision processes with low bellman rank are pac-learnable. In International Conference on Machine Learning, pp. 1704-1713. PMLR, 2017.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 191.6994,,.6988, 2014.
Matthieu Komorowski, Leo A Celi, Omar Badawi, Anthony C Gordon, and A Aldo Faisal. The artificial intelligence clinician learns optimal treatment strategies for sepsis in intensive care. Nature medicine,
2018.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1179-1199 2020.
Eric B Laber, Daniel J Lizotte, Min Qian, William E Pelham, and Susan A Murphy. Dynamic treatment regimes: Technical challenges and applications. Electronic journal of statistics, 8(1): 1225, 2014.
12

---

Published as a conference paper at ICLR 2023
Michail G Lagoudakis and Ronald Parr. Least-squares policy iteration. The Journal of Machine Learning Research, 4:1107-1149, 2003.
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforcement learning, pp. 45-73. Springer, 2012.
Kyowoon Lee, Sol-A Kim, Jaesik Choi, and Seong- Whan Lee. Deep reinforcement learning in continuous action spaces: a case study in the game of simulated curling. In International conference on machine learning, pp. 2937-2946. PMLR, 2018a.
Kyungjae Lee, Sungjoon Choi, and Songhwai Oh. Sparse markov decision processes with causal sparse tsallis entropy regularization for reinforcement learning. IEEE Robotics and Automation Letters, 34)):1466-147, 2018b.
Kyungjae Lee, Sungyub Kim, Sungbin Lim, Sungjoon Choi, and Songhwai Oh. Tsallis reinforcement learning: A unified framework for maximum entropy reinforcement learning. arXiv preprint arXiv: 09009992200003 2019.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 2020.
Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Frez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.0291) 2015.
Daniel Luckett, Eric B Laber, Anna Kahkoska, David M Maahs, Elizabeth Mayer-Davis, and Michael R Kosorok. Estimating dynamic treatment regimes in mobile health using v-learming. Journal of the American Statistical Association, 2019.
Daniel J Luckett, Eric B Laber, Anna R Kahkoska, David M Maahs, Elizabeth Mayer-Davis and Michael Kosorok. Estimating dynamic treatment regimes in mobile health using v-learning. Journal of the American Statistical Association, 1 115(530):692-7066 2020.
Xiaoteng Ma, Li Xia, Zhengyuan Zhou, Jun Yang, and Qianchuan Zhao. Dsac: distributional soft actor critic for risk-sensitive reinforcement learning. arXiv preprint arXiv:2004. 14547, 2020.
Hamid Maei, Csaba Szepesvári, Shalabh Bhatnagar, and Richard S Sutton. Toward off-policy learning control with function approximation. In Proceedings of the 27th International Conference on Machine Learning (ICML-10), pp. 719-726, 2010.
Cindy Marling and Razvan Bunescu. The ohiot dm dataset for blood glucose level prediction: Update 2020. KHID@ IJCAI, 2020.
André Martins, António Farinhas, Marcos Treviso, Vlad Niculae, Pedro Aguiar, and Mario Figueiredo. Sparse and continuous attention mechanisms. Advances in Neural Information Processing Systems, 33:20989-210011, 2020.
Borislav Mavrin, Hengshuai Yao, Linglong Kong, Kaiwen Wu, and Yaoliang Yu. Distributional reinforcement learning for efficient exploration. In International conference machine learning, pp. 4424-4434. PMLR, 2019.
Florence Merlevède, Magda Peligrad, Emmanuel Rio, et al. Bernstein inequality and moderate deviations under strong mixing conditions. In High dimensional probability V: the Luminy volume, pp. 273-292. Institute of Mathematical Statistics, 2009.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 1rX1v:1312.5602, 2013.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1938-1937. PMLR, 2016.
13

---

Published as a conference paper at ICLR 2023
Tetsuro Morimura, Masashi Sugiyama, Hisashi Kashima, Hirotaka Hachiya, and Toshiyuki Tanaka. Nonparametric return distribution approximation for reinforcement learning. In JCML, 2010.
Rémi Munos and Csaba Szepesvári. Finite-time bounds for fitted value iteration. Journal of Machine Learning Rexearch, 9(5), 2008.
Susan A Murphy. Optimal dynamic treatment regimes. Journal of the Royal Statistical Society: Series (Statistical Methodology), 65(2):331-355, 2003.
Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Bridging the gap between value and policy based reinforcement learning. In Advances in Neural Information Processing Systems, pp. 2775-2785, 2017.
Ofir Nachum, Mohammad Norouzi, George Tucker, and Dale Schuurmans. Smoothed action value functions for learning gaussian policies. In International Conference on Machine Learning, pp. 3692-3700. PMLR, 2a.
Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Trust-pel: An off-policy trust region method for continuous control. In International Conference Learning Representations, 2018b.
Tu- Hoa Pham, Giovanni De Magistris, and Ryuki Tachibana. Optlaee--paccical constrained opti- mization for deep reinforcement learning in the real world. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 6236-6243. IEEE, 2018.
David Pollard. Convergence of stochastic processes. Springer Science & Business Media, 2012.
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
Aniruddh Raghu, Matthieu Komorowski, Imran Ahmed, Leo Celi, Peter Szolovits, and Marzych Ghassemi. Deep reinforcement learning for sepsis treatment. arXiv preprint drXiv:1711.09600) 2017.
Konrad Rawlik, Marc Toussaint, and Sethu Vijayakumar. On stochastic optimal control and rein- forcement learning by approximate inference. Proceedings of Robotics: Science and Systems VIII, 2012.
Martin Riedmiller. Neural fitted q iteration- first experiences with a data efficient neural reinforcement learning method. In European conference on machine learning, pp. 317-328. Springer, 2005.
Christian P Robert, George Casella, and George Casella. Monte Carlo statistical methods, volume 2. Springer, 1999.
David Rodbard. Interpretation of continuous glucose monitoring data: Glycemic variability and quality of glycemic control. Diabetes Technology Therapeutics, 11(S1):S-55, 2009.
Bruno Scherrer, Victor Gabillon, Mohammad Ghavamzadeh, and Matthieu Geist. Approximate modified policy itcration. arXiv preprint 100.3,30 2012.
Takuma Seno and Michita Imai. d3rlpy: An offline deep reinforcement learning library. arXiv preprint arXiv:2 112111100000 /.03888, 2021.
Chengehun Shi, Alin Fan, Rui Song, and Wenbin Lu. High- nigh-dimensional a-learning for optimal dynamic treatment regimes. Annals of Statistics, 46(3):925 967, 2018.
Chengchun Shi, Shengxing Zhang, Wenbin Lu, and Rui Song. Statistical inference of the value function for reinforcement learning in infinite-horizon settings. Journal of the Royal Statistical Society. Series B: Staristical Methodology, 2021.
Chengchun Shi, Shikai Luo, Yuan Le, Hongtu Zhu, and Rui Song. Statistically efficient advantage learning for offline reinforcement learning in infinite horizons. Journal of the American Statistical Association, pp. 1-14, 2022.
14

---

Published as a conference paper at ICLR 2023
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International conference on machine learning, pP- 387-395. PMLR, 2014.
David Silver, Julian Schritwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676)355-3333 2017.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
Csaba Szepesvári. Algorithms for reinforcement learning. Synthesis lectures on artificial intelligence and machine learning, 4(1):1-103, 2010.
Aviv Tamar, Yonatan Glassner, and Shie Mannor. Optimizing the evar via sampling. In Twenty-Ninth AAAI Conference Artificial Intelligence, 2015.
Shengpu Tang, Aditya Modi, Michael Sjoding, and Jenna Wiens. Clinician -n -oop decision making: Reinforcement learning with near oetrmml set-valued policies. In International Conference Machine Learning. pp. 9387-9396. PMLR, 2020.
Nino Vieillard, Olivier Pietquin, and Matthieu Geist. Munchausen reinforcement learning. Advances in Neural Information Processing Systems, 33:4235-4246 2020.
Robert Vincent. Reinforcement learning in models of adaptive medical treatment strategies. McGill University (Canada), 2014.
Ronald J Williams.
Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3):299-22 256, 1992.
Tengyang Xie and Nan Jiang. Q approximation schemes for batch reinforcement learning: A theoretical comparison. In Conference on Uncertainty in Artificial Intelligence, pp. 550-559, PMLR, 2020.
Fumitaka Yanase, Tomoko Fujii, Thummaporn Naorungroj. Alessandro Belletti, Nora Luethi, Ani- tra C Carr, Paul J Young, and Rinaldo Bellomo. Harm of iv high-dose vitamin therapy in adult patients: a scoping review. Critical care medicine, 48(7)xe620-e628, 2020.
Ken-ichi Yoshihara. Limiting behavior u-statistics for stationary, absolutely regular processes. Zeitschrift für Walnsscheemiichkettstheerrr und verwandte Gebiete, 35(3):237-252 1976.
Bin Yu. Rates of convergence for empirical processes of stationary mixing sequences. The Annals of Probability, pp. 116, 1994.
Chao Yu, Jiming Liu, Shamim Nemati, and Guosheng Yin. Reinforcement learning in healthcare: A survey, ACM Computing Surveys (CSUR), 55(1):1-36, 2021.
Yong Zang, Jack Lee, and Ying Yuan. Adaptive designs for identifying optimal biological dose for molecularly targeted agents. Clinical Trials, 13))3319-327, 2014.
Wenzhuo Zhou, Ruoqing Zhu, and Donglin Zeng. A parsimonious personalized dose-finding model via dimension reduction. Biometrika, 108(3):643-63) 659, 2021.
Wenzhuo Zhou, Ruoqing Zhu, and Annie Qu. Estimating optimal infinite horizon dynamic treatment regimes via pt-learning. Journal of the American Statistical Association, pp. 1-14, 2022.
Liangyu Zhu, Wenbin Lu, and Rui Song. Causal effect estimation and optimal dose suggestions in mobile health. In International Conference Machine Learning, pp. 11588-11598 PMLR, 2020.
15

---

Published as a conference paper at ICLR 2023
"Supplementary Materials for Quasi-optimal Reinforcement Learning with Continuous Actions"
Appendices
Additional Related Works
18
Technical Proofs
19
B.1
Proots Constructing QpasiaOptima Bellman Operator
19
B.I.1
Proof of Theorem S.I
19
B.1.2
Proof of Corollary
20
B.1.3
Proof of Theorem 3.1
20
Proof of Theorem 3.2
20
B.2
Proofs on Quasi-Optimal Staionarity Equation
................................
21
B.2.1
Proof of Theorem 3.3
21
B.3
Proofs Kernel Representation
22
B.3.1 Proof of Theorem
22
B.3.2
Proof of Theorem S.3
23
B.4
Proots on Generic Properties of Quasi-optimal Bellman Operator
24
B.4.
Proof of Proposition S.1
24
B.4.2
Proof of Proposition S.2
25
B.5
5
Proof of Theorem
25
B,6 6
Proof of Lemma S.I
27
Proof of Theorem
28
B.
Proof of Theorem
32
9 Proof of Theorem
Practical Implementation)
40
Experiment Details and Additional Results
41
D.I Details of Simulation Settings and Real Data Analysis
41
D.2 Additional Experiment Details
42
D.3 Additional Experiment Results
44
D.3.I Sensitivity Analyses
44
D.3.2 Distribution Evaluation Criterion
44
16

---

Published as a conference paper at ICLR 2023
D.3.3
Model Performance on Large Dataset
45
D.3.4 Safe Transitions and Learned Policy Distribution
45
17

---

Published as a conference paper at ICLR 2023
A ADDITIONAL RELATED WORKS
We discuss additional related works in this section.
Safe RL Safe Reinforcement Learning (safe-RL) aims at finding optimal policy while ensuring safety (Garcia & Fernández] 2015). In the safe-RL framework, the definition of safety and its guarantee varies based on the specific purpose of learning tasks. In our view, there three mainstream works for safe RL.
ensuring safe action allocations in the exploration process by incorporating prior knowledge, which often exists in online RL settings (Pham et al., 2018).
Safe Exploration:
Safety Constraints: finding an optimal policy that satisfies external user-specified safe constraints (Chow al., 2018a Gu et al. 2022).
Risk-sensitivity and Conservatism: finding policy maximizing the infinite-horizon cumula- tive discounted reward while incorporating the notion of risk (Morimura et al. 2010; Mavrin et al., 2019), e.g., value at risk (quantile), percentile performance, chance, the variance of return.
In medical applications, specifying explicit constraints is typically hard to realize in practice (Vincent 2014). Alternatively, the notion of safety usually incorporated in the design of reward functions, where high-risk actions lead to significantly low reward (Raghu et al. 2017 Jia al. 2020).
Based on these, our quasi-optimal learning is closely related to the risk-sensitive RL framework, which aims to control value at risk to ensure safety. For example, maintaining the discounted return above a certain threshold (Tamar et al., 2015), reducing the variability of performance by avoiding extremely low performance (Ma al. 2020), target to maximize the robust performance criterion, e.g., quantile of the discounted return (Dabney et al. (2018b). Commonly used algorithms in risk-sensitive RL include conservative Q-learning (CQL:; Kumar al. (2020) and implicit quantile network (IQN; (Dabney et al.] (018a)))) CQL learns a conservative Q-function such that the expected value of a policy under this Q-function lower-bounds its true value and thus avoids selecting high-risk actions with over-estimation action value. IQN models the full quantile function for the state-action return distribution and yields risk-sensitive policies. For a more comprehensive empirical study, we compare the proposed algorithm with the aforementioned two safe RL baselines, conduct additional numerical experiments and analyze the results from the safety point of view.
RL in healthcare Reinforcement learning has a wide variety of applications in healthcare (Yu et al. 202T). Some of the recent works aim to solve safety issues when applying RI. healthcare domains. Tang et al. (2020) considers identifying set-valued policies with near-optimal actions, which allows incorporating expert knowledge from clinicians to assist in decision making. As the same rationale in our proposed quasi-optimal region, Tang et al. (200)) also utilizes the value function to threshold a near-optimal action set. However, this method is only developed on discrete action space, and it is still not directly applicable in fully offline settings. |Fatemi et al. (202T) considers identifying high-risk states in data-constrained offline settings by training two separate Q functions that model the probability of negative outcomes and positive outcomes respectively. They target to identify treatments proportional to their chance of leading to dead-ends, and attain safcty by excluding these treatments from consideration. However, as they aim to identify possible "dead-ends" of state space and treatments, there exists a trade-off between safety and optimality. In particular, it still has a gap for optimal treatment allocations.
Other interesting works in RL for healthcare including Henry et al. (2015): Komorowski et al. (2018) adopt RL algorithms for sepsis treatment recommendations, Jia et al. (2020) redefine the state variables and reward function to reflect practical safety concerns in sepsis treatments. We refer
readers to (Yu al. T02T) for a more comprehensive review.
Offline RL.
The domain approaches of offline RL include fitted Q-iteration (FQI; Ernst et al. (2005) :
Riedmiller 2005);|Munos Munos & Szepesvári (2008); Szepesvári (210)) ), fitted policy iteration (Antos
et 2007 Lagoudakis & Parr] 2003] Scherrer et al. 20. Bellman Residual Minimization (BRM;
Antos et al. (2008); Hoffman et al. (201 1): Farahmand et al. (2016): Dai et al. (2018) ; Chen & Jiag
(2019); Xie & Jiang (2020), gradient O-learning (Maei et al. 2010 Ertefaie & Strawderman 2018
and Advantage learning (Murphy) 2003, Shi et al. 2018: 2022). We refer the reader to Levine et al.
(2020) for more comprehensive discussions on the topics of the offline RL.
18

---

Published as a conference paper at ICLR 2023
In the aforementioned mainstreams of works, ours is closely related to the Bellman Residual Mini- mization. They learn the value function by solving a nested optimization problem, where the function space used for the inner and outer optimization must be the same. From the perspective of the couple optimization, their inner optimization plays a similar role as the inner maximization of the min-max framework. In addition to the fundamental difference in derivation, reduced to a single minimization problem aided by the kernel representation, while they have to solve an unstable minimax optimization problem. Most importantly, our quasi-optimal learning framework provides practical way to learn a reliable policy in continuous action space via quasi-optimal region identifications. to the best of our knowledge, no existing RL algorithms can achieve this.
min-max optimization
be
B TECHNICAL PROOFS
B.1 PROOFS ON CONSTRUCTING QUASI-OPTIMAL BELLMAN OPERATOR
B.1.1 PROOF OF THEERRM S.T
Theorem S.1. Assume the induced policy has density function T (a|s) < for all a, where C is
given constant. Then the proximal Bellman operator B,, in equation (2)
has closed form equivalent:
- JOEW..
Jaew.. Q7(0, a))a
1
2
(8, a)
BA (+)
da
2Wo((Ws,1)
o(W.11)
24
Co(We.))oww. Qj(s, Co(W,,2) Saew,, a7
WW ((W,,))
2o(
o(V.,1)
(13)
where Ws,1 refers the set ta A: (als) > 0}, Wa,2 refers the set {a A: A: (als)) C}.
Proof: The proof is mainly to check the KKT conditions of the maximization. The Lagrangian function of the RHS of can be expressed as follows:
L 0,, 22) E (s) [Q a)) (prox(((ass) ni(s) ( a(als)da-)
+ (s, - (2 a=)(sa)((((ss) CC.
The following KKT conditions are necessary for the maximizer Tu the equation:
Primal: SaeA (T]
0.- = 0. (alss) sc. C. Duality: क (1 a) 2 O. (2( 0.
Complementary slackness: (1 as) (als) = 0. =2(x () C) 0. Stationarity: (s,a) a) + p(1 - (als) - (iss) + (s) #2(8, a) = 0.
We can obtain the equation for T (a|s) from the stationary condition such that
(als) = 2 2μ [π(s)) (s, - (1 + w2 a)].
Combined with complementary slackness condition,
If (als) 0, then कn (8, a) 2 0, #2(8, a) = 0, thus (s, a) S 7(s)
IfC > 0, then कn (s) =2(s,a) = 0. thus i(s) - + CnC > 07
ii(s) - -M.
If (als) C, then क (1 a) 0. =2 a) s 0. thus (s, 2 i(s) p CμC.
Therefore, (8, (s)ef a) can be expressed as:
(s) if - 2nC > (s, a) > i(s)) - (14)
S i(s)) -
(als) =
(ī(s)) - (2(5
2 2p
C
(s 2 riss) - # 24C
19

---

Published as a conference paper at ICLR 2023
Mcanwhile, notice that SaEA (2(5,a) a)
show that (i(s) has closed form:
î(s) ++ Jaew., Qi(s, a)da 2μ + 24Coo(Ws2)
o(W.11)
where Wa,1 refers the set {a A: C' (a(as) C), and (WW.,1) ( refers the interval length of the corresponding set. We take 1j(s) back to (14). we then have
0}, W. refers the set fa A (als)
as S n(s)) -M
(als)
(z(s,)
Sacwa. Qi(s,a)da (WCo( a(W..))
if #+ 2nC > (p(s, a) > fi(8)
2po(Waa)
c
if (5, 2 ni(s) a + 2pC
(15)
We finally plug in the closed form of xp(als) to by some algebra, we have
/ 1- JeEW...
Joew.. (7
2
(1) = - u11 Co(Wo,1) SaEw., (i(s, Co(W,,2) Jaew., (2
da
o(W..))
2f
uC*o(WWs))loWWs..) (WW,))
20(Wo.1)
((W.,1)
B.1.2 PROOF OF COROLLARY S.I
Corollary S.1. When o(W,2) O, we denote W1 as W, the closed in 133 can be simplified
as
(Wwww. (7, a''aa' 241)²
B₂ (s) -- 4u
02'
oWW)
JeE
Proof: We plug in (W..2) to 33. then could obtain the result.
B.1.3 PROOF OF THEOREM 3.1
Proof of Theorem 3.1: For any generic value function (s) and the corresponding generic Q-function Q(s, a), we first build the lower bound:
B₁ (s)
max A (A)
Bann 4(1 - [aass))
2
max
(enn<) - μC
me Seuve (A)
BV (s) + p(1
For the upper bound:
(s)
B (((s,))) #11 [(ass))
max
n E (A)(A)
max H mede (A)(A)
(a +
Bs) + n.
Therefore, have (s) BV)ss E [u(1 C), 14.
B. PROOF OF THEBREM
Proof of Theorem 3.2 Suppose (s, = -a) + + as(s) with (s) > 0. We
assume the density won't reach its boundary value C for this theorem, and we proceed by simplifying (s(s) as ag for i = 1, 2, 3. By Equation we have
(5,
Snew, (s,
T (als)
2μ
(Wo
o(W.)
20

---

Published as a conference paper at ICLR 2023
We first try find the support set of (a)ss)) Since Qj takes the maximum value at y 02 by the symmetric property of quadratic function, the support set should be of the form W, 201 0). Additionally, the boundary point of the support set should be the solution of -lv-1,++](
(s)s,a) Saew, (, a)da
0,
(W,) with respect to a. Thus, we can find the boundary point of the support set by solving the equation with respect to I:
2/
(VW)
1 aa 112 + az2y +1)++a++++ 003 21 9-
cut (-aia² aza ai
7
[1203
It turns out that /
Thus, the support set has the closed-form
201
02 (12afu)
02 + (12a7
W.
aaaa
201
201
Therefore o(V.) = (12afu)3 and
Jaew, Qpls, (120īu)² (WV,
-30
03
+
24pri 24
We plug in the result to the closed form of (als), and obtain the probability density function
(als)
02
2p
201
12
is clear that the resulting distribution of (a|s) of the exact form of G-Gusssian distribution with 0, B = on and centered at 201
B.2 PROOFS ON QUASI-OPTIMAL STAIONARITY EQUATION
B.2.1 PROOF OF THEOREM 3.3
Proof of Theorem 3.3 By the stationary condition from Theorem S.I we have
(s, a) + m(1 (a]s)) 7(s)) + (1, a2) =0, therefore, by the definition of (s, we have
Esr+' |s.a [R(++11
0.
(16)
Notice that Egst (R( 8, a)] = r(s, a), and we take expectation with respect to a following the policy distribution = (als) from both sides of (T6).
Eawny (a|a) [r(s, a) + 7 L (S* + p(1 (als)) i7(s) + wi (s, a) w2(s,a)],
0 (a(s) [r(s,a) a) 7 [V, (S*+ + p(1 - (als)) - i(s) + (s - ())),a)) da.
JaEA
According to the proximal Bellman optimality equation B,, (s) Vi where (s) is the fixed point of By,. With the explicit definition of V we observe that
0 / (als) [r(s, a) + AEA
(S++1 + p(1 - - (a]s)) da S.cA pTj (als)da
I T((a)))T())da +
I (0
(als)s) (8, adda
REA
aEA
DEA
(3))s) /
JOEA / (als)da I [
[h]a|s) (s (s, a)
JOEA
JAEA
- -f (22)))) a)da
21

---

Published as a conference paper at ICLR 2023
Meanwhile SaeA (s(a)s)((s)dda ri(s) Saca Tz(als)da Saed mj(als)<i (s, a)da and SaEA (p(()))=2(s,,)dde slackness, we further have
fi(s) by the property of density, C SaeA we(5, a) by complete
i())) -C cf ueA
V 11 TH (als)da - (u())
W2(8, 0.
aEAA
Since 0 S (a]s) - C, thus y IaEA Tp ( (als)da= - pB (als) E [0, CJ. Therefore,
n(s) = ri(s) ( cf a)(s,a)da, -C ff (2(s,a)da)
uEA
aEd
The stationary condition can be reformulated
E/ [R(S++1, ,,, ( -uprox (T (s) (200) (s) 0.
(17)
Obviously, (T/ Vit) is a solution for the above equation for some (ss), ( a), and 22 a). such that
(1, a) 2 (,,2((8,) 2 0, (1) [a(als) = 0, (2) (C (a(als)) = 0 and ()s)ss E - - W.C C L w2(8, a)da, a)da -c -cf 32(5
aad
When (Wa.2) = 0, we have W2(8, a) =0. Plugging in to equation (17 and denote We,1 as Wa we have the exact form of (
B.3 PROOFS ON KERNEL REPRESENTATION
B.3.1 PROOF OF THEOREM S.2
Theorem S.2. We define the optimal weight function as u² = arg max (g C3 uu 7, u). Let CCS x A) be all continuous functions S x A. For any (s, a) E S x A and s' E the optimal weight function functionn u' (St, At) E (C)) n CSS x A) and unique if the reward function R(s', s,a) and the transition kernel (s
(s are continuous over as)
Proof: Denote ū = Iv,.,, (S", A S(+1) n1S5) + (SS - Vy((). It follows from the definition of CV Tp, w, u), we have that
min max 1e
( (') - ( min max (Est, [9V,,,, (5 - -
(
7 a
min max ((9v,,,, A', A', - A')) V)(())), uuS", u(S", A'))² Vi a
(S" - (1() + ( Va(s)), VCou (OV,,,, A s(t1)) 11(8") + (S" A\) (s())) A -n(s))+ - |
min Vi
min
A
Cjū u
(SS",A4) - Vv((S)))
| | 12
min
((Gv,.,, (St, Ar, 1 - (5() + (55 A²) - Vi((S)); (Gv...,, A st,tt)) (((5))+
(SSt, (A)) - v)((")))
1 VCo Gv... . (S* S - n(S²) + *($", - -V)))v(((s)))) (2
min V
where the third equality is obtained by maximization condition of the inner product between and Sy (St, A ((t++)) - ms()) + ((St,A') - VS())) is that the two terms should have the same direction; the fourth equality is obtained by the equality condition of the Sehchy- inequality.
22

---

Published as a conference paper at ICLR 2023
Such finding indicates that there exists closed form solution of the the optimal weight function ut, such that
u asss a) =y (s, n7s)) + ass (s),
which is equal to ū when (Vu, Rx) (Vmmm))
Notice that for given μ4, We is fully determined by (7, a), thus by Equation 3 we have that T2 (als), yj (s) is continuous over (8, Additionally, by the complete slackness and stationary
condition in Theorem [S.T] we have
u -7(()) + ((, a) - V/),, if (s, + 0;
(7(s) = - -1 - μ + 2pm + (7)),, if (ss O.
Since y TH can be represented by functions of Q (s, a), the Lagrange multipliers (n(s) + (s, can also be represented by a function of Q (s, a), and is also continuous over (7, a).
As TH (a]s), (+),, (s)) + (8, a) are all continuous over (7, a), we only need to prove that (1, a) is continuous over (s, a). By the stationarity equation in Theorem 3.3 [ 8, a)] = g(Q) (s, a)). Since the reward function R(s', s, s. and the transition kernel P(s'ls, a) are continuous
over (8, a) by assumption, Q7 (s,.0 a) is continuous for any (s, a) as Eg (R(sss 8, a)] is continuous for any (s, a). Therefore, the optimal weight function u² (s, a) is continuous over any arbitrary state-action pair (s,a).
B.3.2 PROOF OF THEOREM
Theorem S.3. Suppose n' E H is reproduced by a universal kernel K(-,)), then the minimax
optimizer I0 can be decoupled to a single-stage minimization problem as
CU = Egt,3 A,,, $,11,111,,111 [( , - ( 8*) - V2 (5*))
min 7p.
3 - n(5)) + ((A' 55) - vv((") C.K A" ) (gv.... (3* A A
where (8", " 51+1) an independent copy of the transition pair (5t , 1
Proof: Let Es+ A [( A', St+1 '7(S*) (A*)S*) Vy,(s)) k(, (S*, AA))]]
and define the inner product (:, Havus in H It follows from the definition of L(V,, Tp. w, and kernel reproducing property we have,
min max L² (Vus Tux 17, u.
min max (Es [(Vv,.T, (S', s(, - (S*) x SS V2 (S*) (S*, A9)))
min max Es [( (St, A S4++1 (5 + + (A' 59) V/ (5
K (:::5t, A'), u (5", ,'))) MaKKs
min
max
Esy, [(Qv..s ('t,1 - n (S') + x (4'1 (A'15') - V/ (s'))
K (-;-+tAA))) .. A²)
HRKHS
min 20
ESi.A [( (st, A st+,) - (5") + ~ - -Vp (5¹))
2
Vouu
K (.:S,,A))
7 HAKHS
Maxus
23

---

Published as a conference paper at ICLR 2023
where the last equality holds because of the maximization of inner product between i and Est A [G,,,,, (5", A', A', st --0-n((())))))) + ((415") - (=())) K AA)] should have the same direction. Then we have,
(Est,nc [(9v (5" ( # S' V/ (S*) - K (-::5tA4))
min 2a
VCoul ||0|| ) 2
HAKKS
(Est,a (8yy... A 17 (5 + ~ (A'1 15) - Vi (S') K((:5',A'))
min 70
E [OVV (5 (', (5*) + x (1 11 - V/ (5*)) K( A')]
Cou
[ MuKKs
[ HnKns
HRKHS
min
(Es",A" [(9v
A (" 1) + x (A' - Vi (5")) K :::5,A4]]
CoEzs, Ae (Q (3,.T1, (
+ (A' 5タ) vz(( K (.:3", [T)]]]] HnKnn
where the first equality is by the equality condition of Cauchy-Schward inequality, i.e. // ///l Husis is linear dependent of EEst, .A" [(9v.., (8*, A St+1) myS²) + (A*15*) (x() K- St A
Then, by the reproducing property of K(S", A Å'), we have
min T
max NEH
CV T.. w.
E 3 (GV... A', ((())))((())+((())(() -V2(8))
min
Co (o A'; K(3", (("","":))) Haxms (9V..,, A' A (t)
((5¹) + -
min Vi
St E E 1 /t Co (gV, (5t, g1+1) - ()() + 5 11)) - Vr((s))
K(s", A45 T') (9V, (5', 1', ( - 1/
min
g 3t A At , Co (āV (S" A - a(5 -')) Vu((9))
3
A": A') @V... ( , (S+) + - vv
Thus, we finish the proof.
B.4 PROOFS ON GENERIC PROPERTIES OF QUASI-OPTIMAL BELLMAN OPERATOR
B.4.1 PROOF OF PROPOSITION
Proposition S.1. The quasi-optimal Bellman operator By is "-contractive with respect to the supreme norm over S. That is [B, - B,,V' S > - VVlloo. for any generic value functions (V.V" S S R).
Proposition S.I justifies that there exists a unique fixed point of B,, i.c., Vr, indicating that the quasi-optimal value function VY and the induced policy म) are well defined and unique.
Proof: By the definition of B,,, the explicit form corresponding to V is as follows:
B2 (s) = max Eg Eawz(1s) Es s.a [R(S++1, s) + +V( ) + uprrx (T(a))))
24

---

Published as a conference paper at ICLR 2023
For any two arbitrary value functions V and V we have
(s)) B₁ (s)lloo
154 8,,)) + +V((t+)) + (xprox
max (e
max
(.
Eg [ .,,a) +V*
uprox +
52
$ max Bawr(1ss) B |s [R(S .,,)) + (V + pprox puprox (T(al))))
Bawa(-1a) [Bstt1 1 .8,a) + + pxpro (T(aaa))
(((stt)--V(sstt -
= max 7 1
S >>IV((s - (s)
B.4.2 PROOF OF PROPOSITIONSS2 S.2
Proposition S.2. For any E S, the performance error between (s) and (s) satisfies
p. max CI.11
I - V-Il
1 1-
where C is the upper bound for induced policy Ty.
Proof of Proposition S.2
1 = ||8 VI BV 1100
< V/ - V + ||, BV
Notice that ||B V B
7 VlV Vo by Theorem S. and ||B BV S
max Cl. 1} by Proposition
3.1 Therefore,
(1-7) - Vo S.. mmx - C).1].
We finish the proof.
B.5
PROOF OF THEOREM 4.1
Proof of Theorem We first prove that when μ + 00, = would degenerate to uniform distribution over A. By (4, we only need to prove that for arbitrary small 0
Q,( a)
Saew, Q2(s,a)da 2Po(WW)
o(A) <E
2μ
o(W))
Lower bound:
(s,a)
See (s,
as)
o(W.) max (s,
(18)
2p
2uo (W.)
a(We)
2p
(W,)
((W.)
Q (8)
maxa
(),)aa)
a')
1
(19)
2p
2/1
((A)
Thus, we aim to prove that
(z a) max as))
-
2μ
Let 5* be the unique fixed point of ( and Hmax max H(T), where
((T) Ba (a]s))]
25

---

Published as a conference paper at ICLR 2023
Let (etr(s, == 15 [R(S+), s,a]], by the definition of Qu have
(7(8,a)
g
10.0
[V (5t+1)]
r(s, a)
2/1
2/
2/
(s,aa)
/ 15.0 [Vi (5t+1) V (s++1)
B
[V*
r(s, a)
2p
241
2/1
2/1
Therefore,
07 a)
10yHmax
r(s,a)
7/Baraa (V* (8')
+
2/
2(1 71)
2/
2p
(20)
27 (s, a)
Joy JyHHmax
Rmax
1
2px
2(1 (1-7)
2(1 771
Meanwhile, from another perspective, the proximal Bellman operator (2) can be treated
MDP with the immediate reward r(s, a) + p1d (T((ss) for given Combine with the fact that
HuHmaa -max E
Σ 1=
7" 1-1
um (141))))) = = = a
Let TH argmax, ((Tiaa))((((()))).....)))))))))) then
(5,a))
H max
(s,
2
max E 7
(1 μoدt (A' (S') 1S = =
2μ
2(1 - 7)
2/
1=2
T (s)
20
Em
1-1 (μ - μ (A')5)) S = 8, = a
2/1
00
r (*)
Ex
5² = 5, =
2/1
1=1
Rmax
2(1 7)p
(21)
Based 20 and 21. have
(s,a))
max Qi(s, a²)
(5,
7 Hmax
+ HHuax
max.' Q7 (s,
2p
2p
2/
2(1 -11-r)
2(1 (11
24
(22)
Rmax
(1 7)μ
Similarly, we also have
27 (s. a)
max (s,
Rmax
(23)
2p
2p
(1 7)
Therefore, we have the lower bound approaching a(A)-
For the upper bound, we have SAEA (als)dd thus
f aEA
(s(aa)
J Qi
o(W.) da
21
2.o(W.)
mina (x()aa))
Swew. (i
o(W.) da
JOEA
24
(W,,
min.""
(i(88a")
Swew. (o,
(AA
2p
2p
o(W))
26

---

Published as a conference paper at ICLR 2023
By 23
we then have
()))
Jaew, Q, a)da
(s,a))
max." Qi(s, a")
(24)
2/
(W)(W,)
(WW)
2p
2p
maxa" (3 a")
Sarww Qz d')
1
24
24
oWV))
Rmax
(25)
((A)
(1-7)p
Therefore, by the lower bound and upper bound, we conclude that T. (als) will decay to the uniform distribution as μ oc.
For the case when + O, we prove that TH would converge to the uniform distribution with the length of the support set equal to G. Therefore, when C - oc, it will converge to the point mass. According (T)). we only need prove (WW.1) 0 as μ Meanwhile by Theorem (S.I). E W.,1. if
(( - ( ai(s,d')da - 2p + 20Co(W E (0, 2/Co.
As μ 0, (W.)))) O. Thus, by squeeze theorem, have (s (JarEw., (', -2 21Coo(Ws2) - 0 - 0, which equivalent to o(W., (Q
Je'E.... (s 0 for all E Ws,1-
Therefore, Wa,1 could only include with the same value of (s, a), which should only be series of points rather than an interval. Thus, (WW,,) = 0, and T (als) would converge to uniform distribution with interval length +.
B.6 PROOF OF LEMMA S.1
Before we prove the main result, we first provide helper lemma for studying the boundedness of the symmetric kernel in the U-statistic.
Lemma S.1. Under Assumption for any E S, a E A and μ E (0, oo), we have that (s,,)) (7s) + wss a) - vu(s) s Mmaxs
sup sES.nEA
where Mmax I- Rmax + MCC
Proof of Lemma
S.I.
Dy,ot, (8, a, 8') 1)(5) + ((5 Vi(()) RRs', + VV('') - ((s) (s, a) Vs(s)
<Rmax + + + FC + (Vy(s') VV -2pT + (s,
By checking the KKT conditions, we can further simplify the term (a). Specifically,
1. If म.. 0, then w 2 0. By the stationarity equation (9) have
(a) a(s,a)
-T(s) - Qu a) + Vs(s)
Rmnax + Riax HH
Rmax + HH
(H Egsar (a(ss)
11
1 1-7
Rmax 1 + WH
1-
Rmax-
1- ?
27

---

Published as a conference paper at ICLR 2023
2. If nH (0, CJ, then
(a) = -pt < O.
Therefore,
g5, (8, 0') - 116)) + 2 (8, - (V((s)
SBmax + μ + PC - (V'('") - Vu((s) + 1-
Rmax
< Rmax μ + P. + 7 Rmax + HH
-Rmax +uH 2
Rmax
1 7
7
1 1-7
S 4 Rmax + μC + μ - pH
< 4 Rmax + P.C.
Thus, we gain the upper bound. For the lower bound, the same technique is applied, and we can also gain that
4
Ov,,n, s') ((s) + as Vv(ss= 2 1-7 Rmax H.C.
Therefore, this completes the proof.
B.7 PROOF OF THEOREM
Proof of Theorem 4.2 We first define an operator P from 9V vv A* , sktl) to Gy (Sk, At 1 (S*) + (S* Ah) to simplify the expression, such that
PGv,,,, (S* A , == Gv (S* A ,*,sktt - n(S*) + (AA sh:
We further define several other notations
T
-=
Σ
K(S), A4s A " (8 A , Va(('))
2
ISJAKS
(PGv (St AK (St,,,,, Aht) - Vn((St)}
K (s A" S4,1, 3041
: K 5t 1 A³) (PDV (S (', Vi (5 PGy,... (5 (3,, 3441) V (3t)
Let the expectation with respect to stationary trajectory and i.dd training set as Ey and IE respectively. For any finite threshold parameter μ < co and any O, have
P (( Lu >e) x) =P (1o6 E (UT) + E (UT) Lu >c)
P Eo E (Ur)
P [E (UT)
Lul E) 2
(#)
For (1), since the Gaussian kernel satisfy that [K(:) [1 S 1, then by Lemma
S.I
we have
K (s.a, 8 a, 8)) < M2 max:
for any 8, 8, a, a. By Hoeffding's 's inequality, we have
ne²
(i) $ 2xxp
(26)
2M4 max
For the term (it), the expectation of Up as (U)))) can be calculated as follows:
(T (1)
Er K A Sh (PG ( VV,(s)))
ISIRKET
(PGv, V,,,,, (S* A - (=(
28

---

Published as a conference paper at ICLR 2023
If with-in trajectory samples are independent, then it is obvious that
Er(UT) Er K
A', (
N' (tt =
However, for weakly dependent data, dependency may introduce an additional bias term (UT)) U³ thus we further decompose the term (ii)
S
( P(E (UT) E E (UT) + E E -U
(8i4)
(6v)
For the term (iii), follow similar idea to novel decomposition of the variance term of U-statistic from Han (2018). The idea is to break down the summation of U-statistic into numerous parts, where the current time is affected by randomness, and the historical time will be canceled out after conditioning on the future.
As [- )] is bounded by M under the mixing condition of Assumption 4.2 the exponential inequality from Merlevède et al. (2009) can be applied to bound each decomposition part.
Then we follow the Theorem 3.1 from Han (2018) that for any 60.
M
M Maax log log(4T) log T -1 }. (27)
P( (Ur) TE Er (Ur)] > 60) S 2exp
max Tejoy
Too
where C is some constant.
Then, we proceed to bound the term (iv). By Hoeffding decomposition of kernel function K (s", 11 S8 At, Stt). there exist kernel functions K2 K1 A (1,5t+1) and K2 (St A' stt
s S Stt such that
K1 (s) = ErK 3, - -U,
K2 (,,0,8';3,0,8) = ((,,,,,';;,;ū K (8,0,8') - Ki (3, U* and K, A S(+1) = 0, Rr (5,,4 ,5, position of U7, we have
A S1,)) Then by Hoeffding decom-
2 T n
UT = U
Σ
Ki A', + URS'
Taking the expectation from both sides:
T
Er(UT]= = U +
Er A", (", + ET(URR) Rg.
= U + ET(URR] Rg.
Therefore, by Lyapunov inequality. we can bound the bias term
Er [Ur] U+
Er
U
S Ey
U
U2 Kg ET
Σ
Ey
K2 (Sti,
Ali Shh +1, S4, A4, S41+))
(28)
1shn Shi
4 T2(T 1)2*
Kg Ahe, Shat1; A'2, Al, Slat])
We proceed by the discussing the relationship between hy, h2 11,11 1z.
29

---

Published as a conference paper at ICLR 2023
Case 1.1: If S h1 S h2 S 1 S S T and L2 - 11-12 h2-- Under the mixing condition assumption, and by Generalized Correlation inequality in Lemma 2 of, we have
ET K2 (S AA s , , s Kg (Sto A shatt ,/,, ,
(M 1/1 B1/3 (h22 h1);
where 1/r + 1/8 1, -1.
Case 1.2: If S h1 S h2 S 1 S la2 S and h1 h2 S 12- h.
Similar as Case 1.1, we have
Er K2 "shh,,A, , shut1 sh , s4tt) K2 (she, A shatt, sl+t]]
<4 (M 1/r 8//s (12 11).
Combine Case 1.1 and Case 1.2, we apply the bounded inequalities (2.17-2.21) from Yoshihara (9766), and have the following result
Σ
Er [2 (Shr A Shath , A', s"++))
1sh ShashSls5
R₂ (Shr shats
S' A
s/+t
S
Σ 1--
ET
K2 (S A shith, , gh+t)
K2 Ahe shatt: She , +
Σ ha-dish-d Shonhho
ET K2 (5 A shit , ,41
R₂ (Shr Ahe, shets Al, s/ttt
S M max
T
(j (j+1))8//()) = 0 ( M2² max TD--).
3=1
where
2 2 #+1 I-51
(29)
6 1 st1
Case 2: 1f < h1 <4 < h2 < l2 < T.
Using similar technique Case 1.1 and 1.2, have
Σ
Ey K2 (hh: t , ; sh, sh++t
Ighish STo
K2 (She, Ar,,shatt,
Al2, s't
Σ
Ey K2 (shr A s , ,
1a- Ihh
.2 A shatl, S/3 A'2,S'2+11 +
Σ 11-disho-h h2 1sh SlishisusT
Er Kg (hh: shutl ,
K₂ (Sho, Shatts S/2 A'', st++)]
= =0 OM2 max
(M
30

---

Published as a conference paper at ICLR 2023
Case 3: 1f h S 1 T and 1 I h2 = 12 12 T. Following the same technique, we have
Σ Σ
A Ey K2 " shutl, ,' s(itt)
K2 " , A sk,
I Σ
Σ
By K2 (5 , , s(1-1)
ST
R2 (Sh2, Ale
, Ala
s12t1) ,
+
Σ
Σ
Er K2 (S A , ,
K2 (Stt , ghatt S/z Al ,/,,
S U2 max T2 + M T
3//3 (j) O (M2 [Maxxx
max
1=1
Case 4: 1 S h1 =1 T and 1 S h2 S 12 S.
Using the same technique, we can obtain the same rate as follows:
Σ Ighi SchimiiTION SIShSSITT
E Er Kg (5*1 , ; , ,
K2 (St2, , , 1/2 slttt)
=0 (M T
Combine Case 1-4 with the equation 28 we conclude that
BBT U < CoMe max T 7 a.8.
We further use the continuous mapping theorem to conclude that
E [[T(( BU < M max T '
0.8.,
(30)
where r is defined in 29 and C/ is a constant.
As > O, we have T 2 T-2. 2. Combine 27 and 30 for sufficiently large T, we have
E ET (Ur) (11) ((E
ET (UT) EU*)
TC₁ (c/2 C.M T T22+))/22
< 2exp
max
Mixxx M2 max (e/2 Cg Ma T (1+r)/2) log T log log 47
T/4 - To1cCy max
T (1+T)/2 TC₁ C/
(1+r)
(31)
exp
M4 Maax M2 Maax (c/2 C/ max T (1++)//2 log T log log 47
(2+t)//2,ee(////// M2 To1e//4 /4- (1+T))+)/ C1
max + C1 c/2 M T
= expp
M4 M2 (E/2 M2 T (1(++)//2) log T log log 47
Then by the monotonicity of exp(.),
(2- max - T- C M1 2M max 2/1 M4 MAx + log Tog log 4T M2 c/2
(1+T)/210g log log M
e/4 T ( max + TT= C' M M
(32)
M4 max
+ log T log log 47 M2 max e/2
1/2 log T log log 4T
cC C/C' cM2 VT
M max
6/2 M2 M2 max /VT) log log log 47 + M1 max
31

---

Published as a conference paper at ICLR 2023
where 04 is constant. Combine 26) and 32, simplify the terms and then
P(IEU Cul e) S C1 exp
+ ( C2Mmas log T log log(T) ) + C's exp Ma max
e² C2e VT
nnc²
M max
vT
where C1 C2. C3 are some constants depending on ò respectively, and Mmax I-7 Rmax + P.C.
B.8 PROOF OF THEOREM
Proof of Theorem 4.3 To bound the performance error, we first decompose as
V V 1;
I v !/ V* + Eapproximation emnr
where the first term is the optimization error and the last term is the approximation error. Then we proceed to bound
10 - 12 <
|| M
11 + V
V 1|
where ViT" satisfying the stationarity equation (9) and V is the unique fixed point of B. First, we move to bound A1. Follow similar kernel reproducing property and a eigen decomposition spirit in
Bertsekas| (997) Sutton Barto (2018); Zhou al. 2022), we have
(Cu (V P
2h of x Cv [))++ +
Kmin
2/ (uprox ( (A') puprox ( (T (A (1115)) (775 (S*) - ((S))) + (Sf4 ,A' w(S 1
2 2l (Est+ijss [ (5t+1) - Es ist (S(+ - (V22 (5") - -5"
Then by
| (44)5t) - uprrxx ( (441 11 Sp ( (A S Cμ².
and the auxiliary functions (5)) E CH, 0] for any s E S, then
(S*) - s15tt) S (Cp C₂)) = (C₁))
[ (S*) m7 (S)) s
2
(Lu ( , 0ft,005 - (V , 17, x))
Kmin
Then we conclude that ( -
Cs(Lu
De 0gg q $ LL (V -.pp 17, x0))
Cofe
(S*)
+
Kmin (1 - -2
(1 7)
Cs(Lu (V,,, M 2 ,pa,, x Cij
Cofu
+ (1 - -2
Kimin (1 - 7)
where C3 and Co are some constants, and
C inf (V
Lu CV (Wn
Now, we have the remainder term Ag to bound.
A2 S 1
11 11 - V'
We first bound 4/. For any E S, then we have that
B₁ v (s) = max Bann( $3.2 R( + V (s - [
R(S 3, + T (5t+1) + (prox(f
Eanx, St41/sa
R(S s. + マ, (5t+1 1) + 141 - (a((ss
=BawH,((e), St
(.o)).()))) set1|s,a
R(S a) + ソ/ (5t+1) + # (aass
(a)s(
32

---

Published as a conference paper at ICLR 2023
As (V#*, (T.)) is the solution of the stationarity equation,
(- suttle.o R(S 88,0) VV (5tH))
" + 10 [ (s)
and since E (+)))) [ S ur, then we have
B V (8) S (
For the lower bound, Ba-H,((le), |s, [R(s*+1 ,,,)) a) + 1V ( + ppppl((ass) | 5-5==|
= 2 -Cp
so similarly, we conclude that
C₁ +
(z).
If follows the definition of the proximal Bellman operator Bμ and due to the monotonicity of the Bellman operator that B,, (is)) 2 B, (s) for generic value functions Vi(s) 2 V2(s), and the (s)(s) 2 " (s) for any generic value function V, where B is the proximal Bellman evaluation operator, i.e.,
T V(s)= == Ea-sa(le), st)))))))) R(S ,s,st),,,,a + +V((*+)) + /prox (F)((s)))
Note that, V " is unique fixed point of the Bellman operator B thus lim; (B )V, (s) (s) " (s), where i E Z+ And for any initial value function. e.g., V lim (B,)) (8) Vi)()) holds. Therefore the following inequality holds that
(s) lim B V (s) s lim (B y (VÆ-- + +c) (s) s lim (B₂)) (v²= +Cu) (s)
1000
/ P (8) Σ Cun S Vi()) + Cμ (1-
VT lim (B,)) V
(33)
We repeatedly apply a similar procedure, without loss of generality. We first show one step that
(B, VT (s)) & (V² (s) + Cp)) = (VZ + Cuy S V (s) + Cμ + Cuy.
Then apply infinite many time B,,, then have that
Cμ Cjyy-1 = = (s) +
V (s) = lim (B,)'V < (8)
(34)
(1 7)
Combine with the inequalities
we immediately have that
C₁ (1-7)
V
122 5
Next, by Proposition S.2 we have
1 max - CI.1]
V Vo
1-7
Now, we need bound the excess risk. The excess risk can be decomposed into approximation error and estimation error, i.e.
LV f6 n7,, ,,))) Ct = (
inf
Cu Cu o 17=1 ,002)) Lt:
1
+ Lu (v% (D9,, 200 7/51, )66)
inf (V ma f x
LV 1 , 17851 ,006))
33

---

Published as a conference paper at ICLR 2023
where appprox is the approximation error and Aea is the estimation error. The approximation error is assumed to be zero in our proof for simplicity, At first, we consider to bound the estimation error.
Lu
( 1
mia 1781 x
inf
Lv 1s mh ,nft, xxf2)
(V/1 f 2f11, x
= Lv (v0) T 17 )
Lo Ti. Ta wwf)
S Cu (V #6= 1/5t 2
LV F.. xfa) + Eo (V=" ,1 _58))= - -Cy (D a62 Thi 17 (2,
/
Lu
Vol #0 175 @ H
ad1 . @ (Lo u x Eo 1 The 1781 soft) Ev [0). H
(V 1 , - nf wft) - LV ah ,p,, m
2
sup da xxx (V/ an
where offt, whs are Lagrange multipliers satisfying minimal Bayes risk associ- ated with Vi TT for the rest of this proof. Observe that the randomness of (y =² if ( xxxxxxxxx/ee( xx, sffa, Eo x² sff4, s0f2 can be decom- posed into two parts, one is from the n number of i.i.d. trajectories and another one is from the dependent transition within each trajectory. For each single trajectory, we define the quantity
" (V Th² ,7/11, $1)) =Ay," .x (S), A1, ( A/: St At)A,, . (S), V("),, where Er is defined as taking expectation to single stationary trajectory and E is defined as taking expectation to i.i.d. trajectory random variable D2, respectively. Without loss of generality, we assume Co- = 1. The U-statistic approximation for EU() is as follows:
UT(V 1 , T ",,nn 1ff2)
2
Σ
[A 22 AS, (S, ..) (Av mn (S A/, v
T(T
Then the uniform process is bounded by
(V m np x02) - Ev /
T m 17 xf2)
sup 02 (
(11)
$
sup
(V 1, ;
02 17 x 02 175 x22 - -p(D Ey (U (Vi.
P(Pan) ET (U* (V m ff و Ppl P
Pe.. Ur (v a 1nft m
sup (11)) de =0) x
P2 02 1 52) - p(D D (U T m ا Lv # =h
sup
n
ET (V
) - Ur(V u m. _ w()]
sup 1 K
where Pl (P is the empirical measure with respect to Dam = {D.) and we simply denotes it as P. in the following proof. The last term is the bound for uniform process w.r.t sum of trajectories. In this sense, it is necessary to bound
A2
sup
BT (V - , f x0 Ur n off = +1)]
(V 3151
since the trajectories [D.} are i.dd Now, we process to bound A As can be re-expressed as the empirical process of {D, in measure F, such that
1a w.r.t the probability space (IN, Fx, P)) equipped with empirical
A
sup
E[ET (U"(" m of x - EU (UV (V . . [
(V mm (f
0z EG(VA a : - " -,; - GVV , ,T - ; D.)l
sup (V w of (1
34

---

Published as a conference paper at ICLR 2023
where G(V = ,f , is the random function associated with random variable Dj- To bound A 1. is needed calculate the covering number N' Fo, {D)) by by Pollard's tail inequal-
ity (Pollard 2012). where the function space is the composite space Fele G(O, x 22 x E x E2). 2 , : 41) A11) Hy , (S) Ai )
Specifically, G(V Tpf 1781, D,) Er (V a
M(V ff2.7f (f D:)], where M(V x nft 50; D,)
Next, we proceed to bound the distance in composite space Fo.e- In particular, let (V T , n/f,, s (Vo'', Tfft" o78a'' a E 01 x O2 3 x Eg are two arbitrary functions, then the empiri- cal norm distance w.n.t (D.)%= for the two function can be upper bounded by
and M(V mbf 1f2, : D.) = Ave m m (S), AV. 11
Pn
n w D:) G(V FR 0= G(V
ThH oz .n , :D))
Pn
BT [MV) T n w s (I) .A'K(SS (5 (5, (') TH my ;,;; D:)]
Ey [M(2'' . 0o ofl se (S) 5 A. Th 0z' ,fa1" _ [Di]]
=F P ET K(S A (M T .f s (.). T . wa D:)
M(V
of sa D.M T ./all , :D))] Th Lo
-P,, ET K(S", S (M th if se D. MV T. 2 D,)-
M(V . o , D. MV t of stt D.))
<Pn, m ET K(S", S x사) ET (M 2 of = D:) M MVV11' / n20 .rfaa" _oaaaa :
ET (M())) P. o nnf1, ,, D.) M (V 1 क mp مoم |.i.}
< <Mmax.
PnErIInS mf5a1 | + uP in Er prox (m prox (T ')
+ P,rEr((VV 1 - -V (+1V vol '1 + + Br =
02' = 1maxx1 1 (1 - -ffal + ulP 1T 02 mp + (1 + 1),,, V/ - v00 + Pr 1100 x p
Mmaxx, (Pall7 - -fsi' 11x + HPP, 1|,1 2/ 1l + (F 1 - / 11 + P - 1loo), / where Mmax,1 = 2.Maux. Therefore, as the proximal parameter 0 S 1 S /max < 0o, for any > the metric entropy N ((Mmax + 4) Fe.e, (Dij can be bound with respect to
-v,
separate metric entropy of (0,, O2, E1, Denote min(2((max + Mmaxx 1) as C, then V((mmax + 1e, Fo.s: (Di)
SN 01, (D.)).)) N Oz O2. (C..,,)) N (2) (C..
To bound these factors, we first introduce a idea of pseudo-dimension. that is, for any set X, any points ..:.N E N any class F of functions on X' taking values in [0, CJ with pseudo-dimension Dy < 6o and any 0, we have
(2cC) Dr
N (Cc, < e < +1) E
Therefore, we have
N (2((Maxx + 4), Fo.g, (D:)
(2eMmax
Da, + DDz + +D, D22
Sea (De, +))(De, + (D= (D=1 + (Dz, + +1)
Ce
35

---

Published as a conference paper at ICLR 2023
which implies
N N 2 Fo,s (Di)
Sea (De, + +1) +1) (Dz, +1) (8(umax + 4) max
Do, DD2 D
Ce
Dro.s
:0
where Ci = et (De, + (Do, + +1) +1) 8 8(Amax -4) M max""
De, + Dog - D21 D3g
and DFo.s De, + Des + Dz, Dz, i.e., the "effective" psuedo dimension.
Then we apply Pollard tail inequality, for any 2 32/c², we have P(
BG(V T rf : D,) P.G((V. . . 2 ?
sup (Vr")
(
Dio.cc
nc2
<SC₁
exp
512M2 max
Then obtain
EG(V T T ,,p,,,,2; P. G(V n of w0: D.i1
E
sup [V up ff (e)
BG(V 1
# " o ; P.
2 22 ff w2; D.)l 1)dl
sup
P(
sup
RG(V 0, TH
s 5 - P. Gi
ग 82 S ; Di)l > l)di
(Va) N 3781 x0,
+
P((
sup
BG(V H T 17 D.) P,,.(V o tf w D.)l t)dt
(V, :
02 , (f) xx
<u (
8C1 (1)
D.o.
nt²
exp
dt
512M2 max
64C1 (2) Dro.e
TuU
=u +
exp
n
512M2 max
With probability 1 8, minimizing the RHS with respect u, and plug the minimizer in, have
E
sup
BG(VE Th (2; PV.G(V m. f/,,111 1 22 D.)
L
f2)
64D
(8C (1))
where C2 = 8C1. Therefore, we conclude that, with probability 1 - 8, we have
64D log 8C1
OsDras log g
A S
Next, we proceed to bound Ag. To simply the notation, we denote the U-statistic kernel as
R(st, A St A') = Avgi ft (S), At,
(St. A&; 3t A1) d2 (S) St, S(+1),
By Hoeffding's decomposition of kernel function K(St S A²), there exists kernel functions A') = 0. The
K (St, A') and K2(S", At;5t, A') that (5", At) 0 and Ez,2(s, 5 U-statistic Up can be decomposed into
2 T T
Ux E.
Σ El
Kg (S (') + UR. and E, Ur| E, [U ET
36

---

Published as a conference paper at ICLR 2023
where UR,- := (V "x :n"., w) is defined similarly in the proof of Theorem 4.2 The details of the decomposition can be seen in the proof of Theorem 4.2 The term Ag can be immediately decomposed as follows
A2
sup (V Tp x
UT 1 TH
of [ - ET BTIUUTUUVV (UT((U))) u mp ,/1,, w(ر
BT Th , ET (Ur(UU(V Tp 178 mلf
sup
(V 2 , , 1 x x
82 1/ x0 B BU Th n7 mo UV 1 nM
sup (1 m16 5 x0,x3
A
sup
ET Rs
(V
, n x #
Note that the second term is not exactly zero since the samples are weakly dependent. But next, we will show that A3 converges to zero. First, we check the conditions of Lemma 3.1 in Arcones & Yu 1994). Observe that K(,-)) S and according to Lemma S.1. then
K (st A", )
S M K (S",A'3",A7 A')
M
sup (V ff (0
Therefore, the kernel K is uniformly bounded function. Under Assumption 4.2 that (mm S m 4 for 8 > 1. Therefore, (mm)m²) - 0. By using similar technique of calculating the metric entropy. for any c > O, we have the covering number that
N Fo, ll I1aa) S
< 00
Then the conditions of Lemma 3.1 in Arcones & Yu 1994) are satisfied, we have
|VTUR, (Ve) fB T -h E (,,,,, = (1))
sup (V 82 ni =2)) x x
sup (1 mn ff E0 x0,xE,
BB UR, H s TH , =
= (1).
(35)
02 Since UR,( (V : Tx 17f1, x is uniformly bounded, then
[UR, (V0, 00 1781, x < <0
sup (V .2 (55))
As xr T + 0o, then
E
sup
UR RR (VD,
P , T ,/ w
sup (1741 o ,
UR, (V , oh (,,, > x} + 0,
[VV 22 .Ff)
which means sup (v d2 then as T + x0
(2) URR (V8, T .18i, w()) is uniformly integrable. Combine with
the weak convergence in 35
A sup
EUUR, (V8,
B2 2
s w
)1 S E
sup
(V H , f {2)] + 0.
(v (2)
(V
(22
Then we move to bound J The U-statistic Up is not degenerate, so we adopt Hoeffding's represen- tation (Hoeffding 1994) such that it reduces the problem "first-order" analysis. Specifically, let a (I)) is the collection of all permutations of {1, 2,,2.... ..., T}, the U-statistic Ur can be re-expressed as
To
UT (V-), The 17,,, 62)
1 T! (T)
K (x((o)))(oott)
To
37

---

Published as a conference paper at ICLR 2023
where To [T/2]. By the trick, we have the following inequality
2
Σ
ET.UU((V -
K (X',))) - ET
sup
T(T 1)
27 (V 18 2 20f2)
ISiJiST
1
K (x0(0)x(((oot)) - -Er EU(UT(V / nf, xf
sup 22 (Ve) 1 of .xf2)
T ((T)
To
sup
1 -Σ To i=
Σ
K
[x xx
1
Σ a(T)
ET (V ,
[
T ((T)
T!
(V my fofa) (off)
sup
1 Σ
To Σ
K (xo()) x
(UT(UV
T
82 1 S
71
(V 1 1551 (062)
a(T)
1 To
Σ ET a(T)
Σ
K (xa(t),xa(tttt)) (x(Tott)) ET(UT(VV x 17 ](01
sup
To 1=1
(V a (002)
1 To
R (x x((00)))
Err Err nh 6 الو
sup
82 (V my ofرر ,xfz)
To t=1
1 To
1 To
sup 82 (V/" 1a (,511
Σ
K (x',
(x1xX(7Ott)
Ex
K x (((x))))))))
To t=1
To
t=1
1 To
To
sup (V 17 - (44)
Σ 1=1
G
xx)
- ET
To
To
where X (x', (T++()) which itself is two-dimensional stationary sequences under mix- ing condition. Note that the last term is the expectation of the suprema of the empirical process 1/To Σt= C(X') ET/T/ To G(X*) on the space Go.g. The distance in Go.e can be bounded by the following,
N( min (2 + 4maxx 1|e, Go.e, (X*)
SN <N(CE, 01, (D.iL))) N (Ce, Oz. (Di) N (Ce, N (DiHi=))
M2eMmax
Do, Do,+Doz -D,, +Dz
=e' (De, + (Dz, + (Dz, +1)
Õe
which implies
( L Go.5. (8*)//] ]t=1
16
(6Mmax+332) +32)
maxe Da, DD,+Doz+ +Do, Da, + Dr,
Sed (Do, + (D), +1) (D), (D=, +1)
Ce
=C's (1) Døs.c
where D6.. = De, + De, + Dz, + Dz,- First, without loss of generality, let To = 22mr, kr, for appropriate positive integers mi, KT, as in Yu 1994). Follow Lemma in Antos et al. (2008), we obtain that
To
To
P(
sup
1
G
x
Er
G
{xt)
2 2
(V1
B2 S
(2
To
t=1
To
1=1
Doa.c
Cs
exp
(-4Camr,, 2m ((Ko)))
38

---

Published as a conference paper at ICLR 2023
where C4
1
If Dē 2 2, and let B(m) 2 exp(-61m),T N 1, MT
2
8A4max
(CaToe2/8)) (
moTo = To/ (2KTo), where Doo. 2, C3. Ca, on, we apply Lemma 14 Antos
ct al. (2008), then
2m BozD + C1 (
Doo.c
exp (-4C2m1, 8
and we have, with probability 6.
2A(A/51
2
CaTo
2A/2/81 V1)
C1/T/2]
where
(c,C,
C220 Doo.e
A (Dg.. 2) log To + log(e/6) log*
A Dg g 22 log (T/) + log(e/6) + log"
C2Cs C2
Doo.
Now, we conclude that
C1
log 1811
2A V1)
1
1=
Kmin (1 71)
n
C/
C2 11 C|
71- +CB 100
+ Epprooximation error
CCCC Doole /2)
where A (Do../ 22 log (TT/2]) + log(e/8) + log
Dão.e P-dim(O1)
P-dim 02) P-dim(E,) P-dim(E, and Cı,..., C5 some constants. Adapt the notations for the constants number from Theorem 4.2 By some algebra, we conclude that
1, - V= 11
C1
Cs dim log
8Cs
2 2 V VI)
+
Kmin (1 - 7/2
Co
T/2
generalization
C| + Cs V. DeL.
O
Eppproximation error
(1-7)²
proximal bias
optimization
where A Desan log( (222)
log( log (Coo
Cecoraan 2
Dr P-dim(O1) + P-dim(O2)
P-dim E1
P-dim (2)) and Cg., ..., Cs are some constants.
B.9 PROOF OF THEOREM
We note that SGD converges has global convergence stationary point with sublinear rate in the case of convexity. However, the resulting dose not typically holds for the non-convex analysis. The intuition behind the proof is that quasi-optimal algorithm be regarded as special case of the randomized stochastic descent (RSD) algorithm for solving the non-convex minimization problem.
The convergence analysis of for randomized stochastic descent algorithm has been established in Corollary 2.2 of (Ghadimi Lan 2013). That is, RSD is provably convergent to a stationary point. Follow Theorem 3 in (Drori & Shamir 2020), unbiased SGD algorithm, i.e., the quasi-optimal
39

---

Published as a conference paper at ICLR 2023
algorithm with diminishing learning rate and evaluated Euclidean distance. Therefore, it suffices to show that the gradient of the loss is unbiased.
Now show that the gradient is unbiased, follows
Ve, Cu = = Vo, (7V - v ( ,'),,,,,,,,,, A') ,
+ AV (5", A', S K((" S Vo, (1V (5t-1) - -V2 (59)
Va,Cu = E 2p Vo (A'1') A SA)AA,, 4 T ($ A", (3t1))
- AV A', St K((S , (,)5',A'(22((( TH (A' (A'15*))
V., Cu = E V (S A)K(SS S A.). (5" S
+ AV (S", S K(SS, ",, M. ($) (²)
Cu E VEn((S))((S A (',AA),,, ($", S
AV A', A A V.)
We conclude that the gradient estimator is unbiased. Follow Theorem 3 in (Drori & Shamir 2020). under the conditions stated in Theorem 4.4 we adapt Corollary 2.2 to our quasi-optimal algorithm, completes the proof.
C PRACTICAL IMPLEMENTATION
In practice, {V Th. w needs to be parameterized for practical implementation. However, noticing that V and "i are both associated with Q with closed-form expressions (4) Thus, we propose to represent (Vi,, x by modeling Q Additionally, by modeling Q7 as a quadratic function, the induced policy would follow q-Gaussian distribution. Therefore, we model the coefficients associated with the quadratic form as a linear combination of basis function (p(s) such that (,, a; f) exp {)){{())))a) the m-diimensional basis function, and 0 = [01, 02, 0z]" the 3m-dimensional parameters we need to estimate. The advantage of such parametrization lies in that the parameter space could be reduced.
0 ((s)a + 6J ((s), where p(s) = (s), (s), Pm (s)]"
To solve the constrained optimization problem, we propose a computationally efficient algorithm by transforming the original constrained optimization problem into an unconstrained minimiza- tion problem. Specifically, we impose restrictions on the representation of Lagrangian multipliers (n)(), (s, a)) so that they satisfy their constraints automatically. Although such re-parametrization may sacrifice model ficxibility, it gains great computational advantage as the unconstrained optimiza- tion problem would be much simpler. To be specific, we parametrize w
wss a; max (0, (x(s,;;))
+ Saewi(s) (1, (W.)
(36)
2/1
o(W))
Therefore, (SS 2 0 and T (A*(S*) ((*t A') 0 are automatically satisfied. Also, by specifying the expression of Lagrangian multipliers, (1, a) share the same set of parameters 0 as (Vi T.)) We also define
CC
((s { 1 exp(exo bo))
(37)
where bn is the sigmoid's ' midpoint and ko is the logistic growth rate. By flipping the sigmoid function to parametrize v7 E), the constraint 17(s) E
[- C-C is also automatically satisfied.
40

---

Published as a conference paper at ICLR 2023
D EXPERIMENT DETAILS AND ADDITIONAL RESULTS
For the reproducing purpose, we include our code for all the experiments and the guideline for access the Ohio Type Diabetes dataset an anonymous GitHub link https: btttps://noonyoouuu 4open science / Quasi opt imal Learning witt Cont inuous Treatments 9BB88
D.1 DETAILS OF SIMULATION SETTINGS AND REAL DATA ANALYSIS
The details of the data generative model of each environment in Section 6 are stated below:
Environment I: We consider bounded action space where A = [0, 11, and 2-dimensional state
space. A ~ Unif(O, 1), the state transition function is defined as 1-exp( 45)
S.. = Teoxpi- 45) 57,1 + 0.255 + S St+ --epp(-00 T4exp( (:) S/ + 0.255 St + c1,2
where 1 1/22 sid N.0,0.5%) and the reward function defined as
R 3 (- e S/,2 (AA)) + (S + S + 0.5) S + +122)
Environment II: We consider bounded action space where A [0, 1]. and 2-dimensional state space. A ~ Uni 1), the state transition function is defined as
0. S/ 0.74 - 1) + 0.255 9 where c i.4 (0, 0.52), and R = ( - +0:25(244- Environment II: We consider an unbounded action space where A = (-o0, oo), and 8-dimensional state space. We sampled action uniformly from bounded space, A Unif(-100 100, 100), while it is allowed to select actions on R for the learned policy. The state transition function is defined as, S N(u,tt E), where Σ is a pre-specified covariance matrix, and p4 [p].1.... H(s)..
c1,p-
[+1 exp(A A5/1100 + (4,) exp( exp(- + m(,1)
P exp(A//100) + m4,) exp(= + m(,))
for j= 1,2,344
exp( 1--A//11000 + (xp())))((((AA//1100 +(,))) m4,)))
Pij
exp( - 11-1A//100 + v(,) + eex((=(-A4//1000
1100 + x(,)))
for j j= 5,6,,,7,8
R - exp( ( /2+ (100 + + + 00.50
Environment IV: This environment shares the same transition kemel as Environment III, the only difference is the reward function here
R² = /2)3 + | /2)² + S + S + + (S + + +
For all four environments, we consider different sample sizes where the number of trajectories (25, 50}, and the length of each trajectory T {24, 363. The discount factor 7 is 0.9.
Motivation of Synthetic experiment design: We aim to test the performance of our proposed method on the settings of bounded and unbounded continuous action space with unimodal and multimodal reward functions. The motivation for testing the proposed method in bounded action space is to test the proposed method could potentially handle the off-support bias, as illustrated in Figure 2. The reason for considering multimodal synthetic environment is to evaluate the quasi-optimal policy class (q- (g-Gaussian policy class) works in relatively complex situation. Especially for the q Gaussian policy distribution which is unimodal, it is necessary to test if the q Gauussia policy still works and is robust to the scenario where the optimal policy might multimodally behaving.
We make summary of the synthetic experiments as follows:
Environment I:
Setting: Bounded action space and unimodal reward function
41

---

Published as a conference paper at ICLR 2023
Purpose:
To evaluate if the quasi-optimal learning works in the scenario where it might suffer the off- off-sut bias issue as the continuous action space is bounded.
Environment
Setting: Bounded action space and multimodal reward function
In addition to the purpose in Environment I, we aim to implement quasi- learning in a more challenging environment. Also, this for evaluating the robustness of the unimodal q- g-Gaussian policy under the scenario that the true optimal policy follows multimodal probability distribution.
Purpose:
Environment III:
Setting: High- High-dimension state space and well-separated reward function. The design of the well-separated reward function causes the effect that the selection of non-optimal or sub-optimal actions greatly damages the rewards and increases the risk.
Purpose: To evaluate the reliability/safety of quasi-optimal learning. We aim to examine if quasi-optimal learning could perform well in this scenario. As we expect quasi-optimal learning is able to identify the quasi-optimal sub-regions and avoids choosing those non- optimal/sub-optimg actions which greatly damage the performance.
Environment IV:
Setting: High-dimension state space and complex well-separated reward function.
Purpose: In addition to the purpose in Environment III, we target to evaluate the quasi- optimal learning in more complex environment, imposing great challenges recovering the quasi quasi-optimal regions for the proposed method. Indeed, imposing more complex struc- tures reward function indicates imposing difficulties on value function learning and thus imposes great challenges on identifying quasi-optimal regions.
Ohio Type Diabetes Dataset: For individuals the first cohort, we treat glucose level, carbon- hydrate intake, and acceleration level as state variables, i.e., S S and S For individuals in the second cohort, heart rate is used instead of acceleration level as S7 The reward function is defined as
R 1( > 14
+ +15) (0) - 80)²
30
D.2 ADDITIONAL EXPERIMENT DETAILS
In our implementation, since the objective function, Ly may not be convex with respect to (B, We determine the initial point by randomly generating 200 initial values for all parameters and selecting the with the smallest objective function value.
For the discretization- based methods, i.e., Greedy-GQ and V-learning, we discretize the original action space into 20 bins for implementation in synthetic experiments and 14 bins for real data analysis. The number of bins is chosen by analyzing the distribution of action and the scale of rewards, where too few bins could not lead an accurate approximation of the whole dynamic, and too many bins may damage the performance of these methods. We use radial basis to approximate value functions for these two methods based on the recommendation of the original implementation (Ertefaae & Strawderman 2018: Luckett et al. 2019
For the DeepRL-based continuous control methods, i.e DDPG, SAC, BEAR, CQL and IQN, we implement them mainly based on well-known offline deep reinforcement learning library (Seno & Imai 2021). For the general optimization and function approximation settings, we use multi-layer perceptron (MLP) with 2 hidden layers, each with 32 nodes for function approximation. We set the batch size to be 64, and use ReLU function as the activation function. In addition to the summary provided below, the initial learning rate is chosen from the set {3 x 10-4 1 10-4 4,3 x 10 10-53 53. We use Adam (Kingma & Ba 2014) as the optimizer for learning the neural network parameters. We set the discounted factor be 7 0.9 for all experiments.
42

---

Published as a conference paper at ICLR 2023
To evaluate the policy obtained from the proposed method in synthetic experiments, generate 100 independent trajectories. each with a length of 100 based on the learned policy. We use rejection sampling (Robert et al., 1999] randomly sample each action by the induced density Ty (a(s) and calculate the discounted sum of reward for each trajectory. We compare the discounted return each method. The boxplot of synthetic experiments results based on 50 runs is presented in Figure
For real data analysis, since the data-generating process is unknown, we follow Luckett et al. (2020) to utilize the Monte Carlo approximation of the estimated V-function of the initial state of each trajectory to evaluate the performance of each method. To better evaluate the stability and performance of each method, we randomly select 10 or 20 trajectories from each individual based on available trajectories 50 times and apply all methods to the selected data. The baseline refers to the observed discounted return. The mean and standard deviation of the improvements on the Monto Carlo discounted returns are presented in [
We report all hyperparameters used in training and additional experiment results in this section. The value of is selected from the set {0.01, 0.05, 0.1, 0.2, 0.3, 0.5). We select μ by cross-validation for each experiment, specifically we select μ with the largest fitted V-function value on the initial states of each trajectory, i... Pr, ( -1 (1-2)- >) u, where we mitigate the effect of the threshold parameter μ.. In implementation, we set C = 5 for all synthetic experiments and real data analysis, and check that the induced policy Ty never reaches the boundary value.
We set the learning rate aj for the jthh iteration is be 1+dyJ' where ag is the learning rate of the initial iteration, and d is the decay rate of the learning rate. When n 25, we set the batch size to be 5, and when n = 50, we set the batch size to be 7. We use the L2 distance of iterative parameters as the stopping criterion for the SGD algorithm. The μ selected for each experiment, along with the learning rates and their descent rates, are shown in Table 3 and 4
Table 2: Hyperparameters for each synthetic environment
Hyperparameters
Environment
Environment
Environment
Environment IV
j
0.1
0.05
0.05
0.05
Learning Rate
0.002
0.0005
10-5
10-5
Descent Rate
10 10-4
10-4
10"4
10
Table 3: Hyperparameters for Ohio Type I Diabetes Analysis (Cohort T)
Patient ID
540
544
552
567
584
596
1
0.1
0.1
0.1
0.05
0.05
0.1
Learning Rate
0.001
0.001
0.001
0.0005
0.001
0.02
Descent Rate
10-4
10-4
10
10-4
10-4
2 x 10-4
Table 4: Hyperparameters for Ohio Type Diabetes Analysis (Cohort II)
Patient ID
559
563
570
575
588
591
P
0.2
0.1
0.2
0.05
0.05
0.05
Learning Rate
0.005
0.0001
0.005
0.0001
0.0001
0.0001
Descent Rate
10
10-4
10
10-4
10-4
10-4
Table 5: The mean running time in seconds of each method over 50 experiment runs in Environment I. The synthetic experiments are conducted single 2. GHz Dual-Core Intel Core i5 CPU
n
T
Proposed
SAC
DDPG
BEAR Greedy-GQ
25
24
23.11
22.17
14.31
35.42
11.39
36
28.91
28.12
18.35
42.16
14.47
50
24
28.88
29.91
19.42
46.73
15.62
36
45.23
44.46
36.82
63.54
24.81
43

---

Published as a conference paper at ICLR 2023
Table 6: The mean running time seconds of each method over 50 experiment runs in Environment
II. The synthetic experiments conducted single 2.3 GHz Dual-Core Intel Core
15 CPU
n
T
Proposed
SAC
DDPG
BEAR Greedy-GQ
25
24
30.93
27.29
20.12
44.01
14.56
36
39.34
36.91
26.43
52.86
19.43
50
24
41,12
42.25
28.42
55.17
21.56
36
60.16
56.47
45.71
72.12
32.14
D.3 ADDITIONAL EXPERIMENT RESULTS
D.3.1 SENSITIVITY ANALYSES
To validate the -rold-aali procedure practice and analyze the effect of μa on model perfor- mance, conduct sensitivity analyses for the change of μ.. Results summerized in Figure 4. This confirms that the vross-validation procedure indeed selects a proper which maximizes the discounted return.
1.0
331
005
900
05
35
100
005
33
03
115
815
32
03
03
000
815
03
03
Figure 4: The sensitivity analyses of μ over 50 repeated experiments
D.3.2 DISTRIBUTION EVALUATION CRITERION
To measure the performance on safety, we aim to evaluate the distribution Monte- Carlo discounted sum of rewards for each roll-out trajectory Dabney al. (2088)), instead of its empirical mean, i.e., discounted return.
In partieular, generate 100 trajectories under the learned policy and record the discounted sum of rewards of each single trajectory. Then we draw the density plots in Figure for all four environments. As shown in Figure the distribution of the quasi-optimal learning shows thinner tail the left. This is aligned to two safe RL algorithms IQN and CQL. The phenomenon indicates that there is less chance enter a low reward trajectory which is damaged by allocating highly-rish actions. However, the non-safe RL approach SAC is more evenly distributed on both extremes; Hence, SAC may enter a low reward trajectory with higher probability (heavier left tail) compared to the quasi-optimal learning and two safe RI baselines. This validates that quasi-optimal learning can avoid risky actions as the other two safe RL baselines.
Enwoeeenc
EnwcerDEE
AAG:
CD.
X.T
Nehoo Pooeset
10'
500
Figure 5: The distribution of Monte-Carlo discounted of rewards over 50 repeated experiments.
44

---

Published as a conference paper at ICLR 2023
D.3.3 MODEL PERFORMANCE ON LARGE DATASET
We evaluate the model performance large sample size scenarios (10,000 transition pairs (n 100, T = 100) for all four environments. The results presented in Figure 6 Deep RL baseline methods have some improvement in the model performance and variance reduction with increased training samples. Meanwhile, the quasi-optimal learning still outperforms all competing methods as shown in Figure g
Methods
Proposed
CQL
DEAR
Environment
Envrroneent.V
Envirommnts
Figure 6: The boxplot of discounted return over 30 repeated experiments with sample size N = 100, 100.
D.3.4 SAFE TRANSITIONS AND LEARNED POLICY DISTRIBUTION
We illustrate the safety of the proposed method via evaluating the proportion of safe transition, i.e., from fixed current state to safe transition state. The goal of the OhioTIM case study to maintain the glucose level in a safe range. The safe state in this study is defined as the state where the glucose level is within the range of 80-140 mg/dL. The reward function, i.c., the index of glycemic control (Rodbard) 2009).
Safe Transition:
1( 140) 1 +(S 80)(S) 80)²
R
30
where ST, the glucose level of patient i decision stage. This reward setting tends favor the safe range and penalize the risky scenario where the glucose level is out of the range of 80-140 mg/dL.
The details of the evaluation procedure are summarized in the following. In offline OhioTIM dataset, we pick up the observed states which transited to risky states, i.e., the states out of the safe range of glucose level. On the picked-up states, calculate the proportion of safe transition, in which the corresponding transition states are sampled from the transition kernel under the learned policy. The transition kemel is estimated by maximum likelihood estimation from the offline dataset.
We summarize the results of the safe proportions on 1000 transition samplings Figure As shown. the quasi-optimal learning achieves 82.2% safe proportions, which outperforms 67.3% in safe RL baseline IQN and 44.6% non-safe RL baseline SAC. By the results, we may conclude that quasi-optimal learning enjoys better safety guarantee when applied to the medical domain.
45

---

Published as a conference paper at ICLR 2023
0.8
Proportion
0.4
0.2
0.0
Proposed
IQN
SÁC
Methods
Figure 7: Proportions of Safe Transition from each Method
The Learned Policy Distribution: In this dimension, we illustrate the validity of the quasi-optimal policy distribution on a fixed state. In ThioT IM dataset, we select patient state with glucose level of 217 mg/dL, which is moderate hyperglycemia. On this state, we draw density plot in Figure for the policy distribution learned by the quasi-optimal learning. IQN, and SAC. Figure 8 shows that the quasi-optimal learning identified support regions [3.15,6.7 6.19). As the patient is under moderate hyperglycemia, the moderate insulin dosage, i.e., [3.15, 6.1], works well to decrease the glucose level into a safe range. Meanwhile, avoids overly dropping the patient's 's glucose level and causes hypoglycemia. In comparison, SAC risky it has non-negligible probability of assigning too low and too high insulin dosage to the patient. The policy learned by the safe RL algorithm IQN tends avoid assigning extreme dosage, but it has wider support than the learned by quasi-optimal learning. Regarding efficiency or safety, the quasi-optimal has certain advantages compared with IQN in this case.
0.5
Proposed
0.4
IQN
SAC
Density
0.3
0.2
0.1
0.0
0.0
2.5
5.0
7.5
Dose Level (Bolus)
Figure 8: The Learned Policy Distribution of each Method for the Same Given State
46


Reviews:
[{"review_id": "oOskRmLWGK", "reviewer": "Reviewer_xW9e", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper addresses learning in continuous action-spaces where circumspection of \"optimal\" behavior is necessary, specifically focused on healthcare. The paper proposes a method by which to constrain the policy search space to only those areas that are \"quasi-optimal\". Detailed theoretical justification and analysis of the proposed quasi-optimal Bellman operator is provided as well as preliminary application of the approach to a diabetes management dataset. ", "summary_of_the_review": "I found this paper to be very well motivated and throughly justified from a theoretical perspective. It's clear that the quasi-optimal learning strategy has some bearing for appropriately constraining policies in continuous action settings. I am less convinced about it's overall use in important safety-critical environments such as healthcare. Conceptually I am aligned with the proposed direction but I cannot currently advocate for the paper's publication at this stage. I would need to see more thorough experimental analysis of how the proposed method actually avoids high-risk or dangerous action decisions in comparison to prior methods to demonstrate that the proposed method is working as intended. The cumulative \"score\" in Table 1 does not communicate that to me. It's nice to see that the proposed method learns \"better\" policies but I'm not sure that it actually is fulfilling the design of the approach. ", "strength_and_weaknesses": "I'll first list the **strengths** of this paper:\n - The problem setting (constraining continuous action spaces to remain safe) is very important for the potential use of contemporary RL algorithms in real-world scenarios. \n- The authors have appropriately motivated the design of the approach from a theoretical foundation. \n- I found the theoretical justification and design of the quasi-optimal Bellman operator (aided by using q-Gaussian distributions) to be really interesting.\n- For the most part the derivation of the methodology is clear and easy to follow (specific examples of where this is not clear will be listed below in the weakness section). The authors have taken great care to list out the failings or limitations of each component and then how the next concepts address this limitation. This made following the line of reasoning and development much easier.\n\nNow for the **weaknesses**:\n- The use of $\\mu$ is not entirely clear. It's not clearly defined in the paper and shows up as an identifier for the quasi-optimal Bellman operator (first appears in Section 3). It would be helpful if the usage of this parameter (is it a scalar? a stand-in for a sampling policy? etc..) was more precisely discussed when it is first introduced. For example, it's never discussed how $\\mu$ is chosen and what is used for that selection process. Since this construction is critical for defining the disjoint regions of admissible actions, this should be better handled throughout the paper.\n- Figure 1 is not very informative. There is not enough detail in the main body of the paper to round out the insights that the Figure is trying to reaffirm. For instance, it's unclear what actions are admissible since the volume under the curves primarily correspond to the $<2\\mu$ region. I was led to believe (based on the definition of the screening set) that the admissible actions--eg. the support set-- would satisfy the $\\leq2\\mu$ condition. The term \"screening intensity\" is undefined\". \n- The choice of policy class to be concavely quadratic seems to be a pretty strong assumption/constraint? It's unclear whether the component $\\alpha_*(s)$ functions are explicitly learned or not. Being more straightforward about this would be helpful, particularly if there's a difference in how the policies (and underlying value functions) are pragmatically used. There's clear benefit theoretically but...\n- The assumption that equation 10 can be \"solved\" with off-policy data also assumes that the data is self-contained and consistent, right? I mean, that the data provides full coverage and there is no stochasticity/confounding? Otherwise, a batch of data would need to be infinite in order to have the value function and corresponding policy fully converge, correct?\n- The use of the Kernel representation is not very well motivated or explained. This was perhaps the weakest point of the theoretical derivation of the full method in my eyes. I can understand it's utility and why the choice was made. But the follow through and justification (and then actual implementation) is missing from the paper.\n- The experiments are not adequately described nor introduced. I cannot ascribe any significance to the presented results without understanding how the specific experiments were designed to test the proposed method and how the chosen baselines are fairly compared to. I'm not dismissing the results in total but I am not overly enthused by them since it is clear that the DeepRL baselines aren't set up to be real great comparisons (due to dataset limitations as well as the overall design of the algorithms). None of these baseline approaches claim to work in the small data regime so it's no surprise that the observed variance of the results is as high as it is.\n- Continuing with the complaints I have about the experiments, it's unclear what the experimental procedure is. Were the algorithms trained in a fully offline, off-policy fashion? If not, why?\n- The biggest weakness of this paper in my mind and which causes hesitation for me to consider accepting this work is that it does not adequately frame its contributions among the decades of work in safe RL as well as work applying Reinforcement Learning to healthcare challenges. These omissions overlook relevant contributions made in the community recently that address similar technical considerations raised in this paper. I will describe two such papers that the authors should consider as well as point to a really through survey of the field (as of 2020). Beyond this there are several more papers that should be considered by the authors for framing and adequately justifying the development of algorithms and strategies motivated by and for use in healthcare.\n  - First, I'll point to Tang, et al (ICML 2019) \"Clinician-in-the-loop decision making\". Which constructs set-valued policies of nearly optimal actions, learning to ignore those regions of the action space that are not relevant. While this paper doesn't address continuous action-spaces the theoretical foundations and questions about \"quasi-optimal\" learning are very similar.\n   - Second, I'll point to Fatemi, et al (NeurIPS 2021) \"Medical Dead-ends\" which introduces a way to re-think learning value functions in data-limited safety-critical environments by assessing regions of risk and recommending that these actions be removed from the list of options an expert can consider. \n   - For the survey, I strongly recommend Yu, et al (2021) \"Reinforcement Learning in Healthcare: a survey\".", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "TfHZBWOhVp", "reviewer": "Reviewer_RoFn", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors address the problem of an unreliable policy caused by using a Gaussian distribution to represent it in a continuous control setting. They create a new quasi-optimal learning algorithm that uses the q-Gaussian distribution and show that it has provable convergence in off-policy settings. They also analyze other key theoretical aspects of the algorithm, such as sample complexity and consistency.\n\nIn a synthetic environment where there are bounded action regions, they show their method performs better than competitors. They additionally show that it performs better than others in an existing simulator for diabetes.", "summary_of_the_review": "The authors present a thorough and mathematical analysis of a new algorithm that claims to address issues that arise in continuous action RL, but it is hard to tell how practical or useful the algorithm is because of limited experimental comparisons.", "strength_and_weaknesses": "Strengths:\n1. The authors' arguments are comprehensively supported by theory.\n2. The empirical performance against other algorithms is strong, even deterministic ones that do not suffer from off-support bias.\n\nWeaknesses:\n1. The paper is closely related to offline RL, but there is not a clear discussion on off-policy evaluation or avoiding unwanted behaviors. The experiments appear to use only offline data, but use algorithms that were not developed for offline or safe RL (see #2). The method appears to also be applicable online—it is not clear whether there is also an advantage in online settings.\n2. There is no comparison in experiments with other safe RL approaches. Indeed, there is little discussion of safe RL research in the paper—it is unclear to me why this is. The motivation of the authors seems to lie at the intersection of offline and safe RL.\n3. The appendix gives the impression that the hyperparameters were tuned for the authors' method but not for the others.\n\nPost-response:\nThe authors have done substantial updates to the paper during the review period, including running new experiments. I am mainly satisfied with the paper as it stands and have updated my score.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0jW3Hezgic", "reviewer": "Reviewer_Y35L", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper considers policy learning in continuous treatment setting, particularly in optimal dose finding. The contributions of the paper includes (1) the development of the quasi-optimal Bellman operator to address the non-smoothness issue; (2) the use of q-Gaussian policy distribution to avoid off-policy support; (3) the development of a PAC learnable algorithm with appealing theoretical properties; (4) comprehensive numerical analysis in comparison to several baseline algorithms. ", "summary_of_the_review": "The paper considers policy learning in continuous treatment setting, particularly in optimal dose finding. The proposed algorithm is novel and statistically sound. It is also empirically verified based on simulation and real data applications. However, I have some questions and comments regarding the presentation and the proposed methodology. I hope the author(s) can help address my comments.", "strength_and_weaknesses": "The strengths and advances of the proposal includes:\n\n1. The development of the introduction of the quasi-optimal Bellman operator for policy learning in continuous action space;\n2. The use of q-Gaussian policy distribution to realize bounded support;\n3. A practical algorithm for quasi-optimal policy learning;\n4. Nice theoretical properties of the proposed algorithm;\n5. The use of real datasets to justify the finite-sample performance of the algorithm.\n\nSome suggestions, questions and comments:\n\n1. The paper considers estimating the optimal policy from an offline dataset. It would benefit from a discussion about existing offline RL algorithms and whether/how these algorithms are related to your proposal. \n2. Page 2 (minor) The notation paragraph. For the two sequences $(\\Psi(m))_m$ and $(\\Upsilon(m))_m$, I guess you require their elements to be strictly positive. \n3. Page 3 (minor) $\\Delta_{\\textit{convex}}(\\mathcal{A})$ is not defined. \n4. I found Section 3.1 a bit difficult to follow. For instance, does Equation (2) holds for any $\\rho$? When $\\rho=0$, it seems the right-hand-side equals the optimal value function. So why does the penalty term $\\rho$ exist on the right-hand-side? In addition, it seems the right-hand-side does not involve the state transition. So why is it referred as the Bellman operator? Equation (3), you may want to discuss the choice of $\\mu$. Is the proximal term used to smooth the policy class? Is $\\mu$ the degree of smoothness? A related question is, I was wondering if the Fenchel representation on the Bellman operator has been employed in some other papers. \n5. It seems Theorem 3.2 implicitly requires the action space to be one-dimensional. Do you have similar results for multi-dimensional action space? \n6. Section 3.3. It was mentioned that Equation (10) can be directly solved using the transition samples, so it overcomes the distributional shift issue in offline data. However, the distributional shift issue usually refers to the difference in the marginal state-action distribution instead of the state transitions. \n7. In additional to the kernel representation, can we use other functional approximators to solve the minimax optimization? Would you please discuss?\n8. Page 6, minimax optimization. It is not clear to me how the method based on average Bellman error solve the double sampling issue. Is it related to the couple estimation method (Antos, Szepesvari and Munos, 2008a; Farahmand et al., 2016)?\n9. Page 8, numerical experiments. The action space in the four environments are one-dimensional. Can we compare the proposed with method against existing policy learning algorithms with certain discretization? This could better support the claim in the introduction section. When the action space is $[0,1]$, you may want to equally define the interval into e.g., 5 or 10 subintervals. When the action space is unbounded, we could first apply certain transformation (e.g., based on the normal cdf) and then divide the interval. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster