SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yudong Luo $^{1,4}$ , Guiliang Liu $^{2}$ , Pascal Poupart $^{1,4}$ , Yangchen Pan $^{3}$
<sup>1</sup>University of Waterloo, <sup>2</sup>The Chinese University of Hong Kong, Shenzhen,
$^{3}$ University of Oxford,  $^{4}$ Vector Institute
yudong.luo@uwaterloo.ca, liuguiliang@cuhk.edu.cn,
ppoupart@uwaterloo.ca, yangchen.pan@eng.ox.ac.uk
Restricting the variance of a policy's return is a popular choice in risk-averse Reinforcement Learning (RL) due to its clear mathematical definition and easy interpretability. Traditional methods directly restrict the total return variance. Recent methods restrict the per-step reward variance as a proxy. We thoroughly examine the limitations of these variance-based methods, such as sensitivity to numerical scale and hindering of policy learning, and propose to use an alternative risk measure, Gini deviation, as a substitute. We study various properties of this new risk measure and derive a policy gradient algorithm to minimize it. Empirical evaluation in domains where risk-aversion can be clearly defined, shows that our algorithm can mitigate the limitations of variance-based risk measures and achieves high return with low risk in terms of variance and Gini deviation when others fail to learn a reasonable policy.
The demand for avoiding risks in practical applications has inspired risk-averse reinforcement learning (RARL). For example, we want to avoid collisions in autonomous driving [1], or avoid huge financial losses in portfolio management [2]. In addition to conventional RL, which finds policies to maximize the expected return [3], RARL also considers the control of risk.
Many risk measures have been studied for RARL, for instance, exponential utility functions [4], value at risk (VaR) [5], conditional value at risk (CVaR) [6, 7], and variance [8, 9]. In this paper, we mainly focus on the variance-related risk measures given their popularity, as variance has advantages in interpretability and computation [10, 11]. Such a paradigm is referred to as mean-variance RL. Traditional mean-variance RL methods consider the variance of the total return random variable. Usually, the total return variance is treated as a constraint to the RL problem, i.e., it is lower than some threshold [8, 9, 12]. Recently, [13] proposed a reward-volatility risk measure, which considers the variance of the per-step reward random variable. [13] shows that the per-step reward variance is an upper bound of the total return variance and can better capture the short-term risk. [14] further simplifies [13]'s method by introducing Fenchel duality.
Directly optimizing total return variance is challenging. It either necessitates double sampling [8] or calls for other techniques to avoid double sampling for faster learning [8, 9, 12]. As for the reward-volatility risk measure, [13] uses a complicated trust region optimization due to the modified reward's policy-dependent issue. [14] overcomes this issue by modifying the reward according to Fenchel duality. However, this reward modification strategy can possibly hinder policy learning by changing a "good" reward to a "bad" one, which we discuss in detail in this work.
To overcome the limitations of variance-based risk measures, we propose to use a new risk measure: Gini deviation (GD). We first review the background of mean-variance RL. Particularly, we explain the limitations of both total return variance and per-step reward variance risk measures. We then introduce
GD as a dispersion measure for random variables and highlight its properties for utilizing it as a risk measure in policy gradient methods. Since computing the gradient using the original definition of GD is challenging, we derive the policy gradient algorithm from its quantile representation to minimize it. To demonstrate the effectiveness of our method in overcoming the limitations of variance-based risk measures, we modify several domains (Guarded Maze [7], Lunar Lander [15], Mujoco [16]) where risk-aversion can be clearly verified. We show that our method can learn risk-averse policy with high return and low risk in terms of variance and GD, when others fail to learn a reasonable policy.
2 Mean-Variance Reinforcement Learning
In standard RL settings, agent-environment interactions are modeled as a Markov decision process (MDP), represented as a tuple  $(S, A, R, P, \mu_0, \gamma)$  [17].  $S$  and  $A$  denote state and action spaces.  $P(\cdot | s, a)$  defines the transition.  $R$  is the state and action dependent reward variable,  $\mu_0$  is the initial state distribution, and  $\gamma \in (0, 1]$  is the discount factor. An agent follows its policy  $\pi: S \times A \to [0, +\infty)$ . The return at time step  $t$  is defined as  $G_t = \sum_{i=0}^{\infty} \gamma^i R(S_{t+i}, A_{t+i})$ . Thus,  $G_0$  is the random variable indicating the total return starting from the initial state following  $\pi$ .
Mean-variance RL aims to maximize  $\mathbb{E}[G_0]$  and additionally minimize its variance  $\mathbb{V}[G_0][8,9,12]$ . Generally, there are two ways to define a variance-based risk. The first one defines the variance based on the Monte Carlo total return  $G_{0}$ . The second defines the variance on the per-step reward  $R$ . We review these methods and their limitations in the following subsections. We will refer to  $\pi$ ,  $\pi_{\theta}$  and  $\theta$  interchangeably throughout the paper when the context is clear.
2.1 Total Return Variance
Methods proposed by [8, 9, 12] consider the problem
$$
\max  _ {\pi} \mathbb {E} [ G _ {0} ], \text {s . t .} \mathbb {V} [ G _ {0} ] \leq \xi \tag {1}
$$
where  $\xi$  indicates the user's tolerance of the variance. Using the Lagrangian relaxation procedure [18], we can transform it to the following unconstrained optimization problem:  $\max_{\pi} \mathbb{E}[G_0] - \lambda \mathbb{V}[G_0]$ , where  $\lambda$  is a trade-off hyper-parameter. Note that the mean-variance objective is in general NP-hard [19] to optimize. The main reason is that although variance satisfies a Bellman equation, it lacks the monotonicity of dynamic programming [20].
Double Sampling in total return variance. We first show how to solve unconstrained mean-variance RL via vanilla stochastic gradient. Suppose the policy is parameterized by  $\theta$ , define  $J(\theta) = \mathbb{E}_{\pi}[G_0]$  and  $M(\theta) \coloneqq \mathbb{E}_{\pi}\left[(\sum_{t=0}^{\infty} \gamma^t R(S_t, A_t))^2\right]$ , then  $\mathbb{V}[G_0] = M(\theta) - J^2(\theta)$ . The unconstrained mean-variance objective is equivalent to  $J_{\lambda}(\theta) = J(\theta) - \lambda(M(\theta) - J^2(\theta))$ , whose gradient is
$$
\begin{array}{l} \nabla_ {\theta} J _ {\lambda} \left(\theta_ {t}\right) = \nabla_ {\theta} J \left(\theta_ {t}\right) - \lambda \nabla_ {\theta} \left(M (\theta) - J ^ {2} (\theta)\right) (2) \\ = \nabla_ {\theta} J (\theta_ {t}) - \lambda \left(\nabla_ {\theta} M (\theta) - 2 J (\theta) \nabla_ {\theta} J (\theta)\right) (3) \\ \end{array}
$$
The unbiased estimates for  $\nabla_{\theta}J(\theta)$  and  $\nabla_{\theta}M(\theta)$  can be estimated by approximating the expectations over trajectories by using a single set of trajectories, i.e.,  $\nabla_{\theta}J(\theta) = \mathbb{E}_{\tau}[R_{\tau}\omega_{\tau}(\theta)]$  and  $\nabla_{\theta}M(\theta) = \mathbb{E}_{\tau}[R_{\tau}^{2}\omega_{\tau}(\theta)]$ , where  $R_{\tau}$  is the return of trajectory  $\tau$  and  $\omega_{\tau}(\theta) = \sum_{t}\nabla_{\theta}\log \pi_{\theta}(a_{t}|s_{t})$ . In contrast, computing an unbiased estimate for  $J(\theta)\nabla_{\theta}J(\theta)$  requires two distinct sets of trajectories to estimate  $J(\theta)$  and  $\nabla_{\theta}J(\theta)$  separately, which is known as double sampling.
Remark. Some work claims that double sampling cannot be implemented without having access to a generative model of the environment that allows users to sample at least two next states [12]. This is, however, not an issue in our setting where we allow sampling multiple trajectories. As long as we get enough trajectories, estimating  $J(\theta)\nabla_{\theta}J(\theta)$  is possible.
Still, different methods were proposed to avoid this double sampling for faster learning. Specifically, [8] considers the setting  $\gamma = 1$  and considers an unconstrained problem:  $\max_{\theta}L_1(\theta) = \mathbb{E}[G_0] - \lambda g\big(\mathbb{V}[G_0] - \xi \big)$ , where  $\lambda > 0$  is a tunable hyper-parameter, and penalty function  $g(x) = (\max \{0,x\})^2$ . This method produces faster estimates for  $\mathbb{E}[G_0]$  and  $\mathbb{V}[G_0]$  and a slower updating for  $\theta$  at each episode, which yields a two-time scale algorithm. [9] considers the setting  $\gamma < 1$  and converts Formula 1 into an unconstrained saddle-point problem:  $\max_{\lambda}\min_{\theta}L_{2}(\theta ,\lambda) = -\mathbb{E}[G_{0}] + \lambda \big(\mathbb{V}[G_{0}] - \xi \big)$ , where  $\lambda$  is the dual variable. This approach uses perturbation method and smoothed function method to compute the gradient of value functions with respect to policy parameters. [12] considers the setting  $\gamma = 1$ , and introduces Fenchel duality  $x^{2} = \max_{y}(2xy - y^{2})$  to avoid the term  $J(\theta)\nabla_{\theta}J(\theta)$  in the gradient. The original problem is then transformed into  $\max_{\theta ,y}L_{3}(\theta ,y) = 2y\big(\mathbb{E}[G_{0}] + \frac{1}{2\lambda}\big) - y^{2} - \mathbb{E}[G_{0}^{2}]$ , where  $y$  is the dual variable.
Limitations of Total Return Variance. The presence of the square term  $R_{\tau}^{2}$  in the mean-variance gradient  $\nabla_{\theta}M(\theta) = \mathbb{E}_{\tau}[R_{\tau}^{2}\omega_{\tau}(\theta)]$  (Equation 2) makes the gradient estimate sensitive to the numerical scale of the return, as empirically verified later. This issue is inherent in all methods that require computing  $\nabla_{\theta}\mathbb{E}[G_0^2]$ . Users can not simply scale the reward by a small factor to reduce the magnitude of  $R_{\tau}^{2}$ , since when scaling reward by a factor  $c$ ,  $\mathbb{E}[G_0]$  is scaled by  $c$  but  $\mathbb{V}[G_0]$  is scaled by  $c^2$ . Consequently, scaling the reward may lead to different optimal policies being obtained.
2.2 Per-step Reward Variance
A recent perspective uses per-step reward variance  $\mathbb{V}[R]$  as a proxy for  $\mathbb{V}[G_0]$ . The probability mass function of  $R$  is  $\operatorname*{Pr}(R = x) = \sum_{s,a}d_{\pi}(s,a)\mathbb{I}_{r(s,a) = x}$ , where  $\mathbb{I}$  is the indicator function, and  $d_{\pi}(s,a) = (1 - \gamma)\sum_{t = 0}^{\infty}\gamma^{t}\operatorname*{Pr}(S_{t} = s,A_{t} = a|\pi ,P)$  is the normalized discounted state-action distribution. Then we have  $\mathbb{E}[R] = (1 - \gamma)\mathbb{E}[G_0]$  and  $\mathbb{V}[G_0]\leq \frac{\mathbb{V}[R]}{(1 - \gamma)^2}$  (see Lemma 1 of [13]). Thus, [13] considers the following objective
$$
\hat {J} _ {\lambda} (\pi) = \mathbb {E} [ R ] - \lambda \mathbb {V} [ R ] = \mathbb {E} [ R - \lambda (R - \mathbb {E} [ R ]) ^ {2} ] \tag {4}
$$
This objective can be cast as a risk-neutral problem in the original MDP, but with a new reward function  $\hat{r}(s,a) = r(s,a) - \lambda\big(r(s,a) - (1 - \gamma)\mathbb{E}[G_0]\big)^2$ . However, this  $\hat{r}(s,a)$  is nonstationary (policy-dependent) due to the occurrence of  $\mathbb{E}[G_0]$ , so standard risk-neutral RL algorithms cannot be directly applied. Instead, this method uses trust region optimization [21] to solve.
[14] introduces Fenchel duality to Equation 4. The transformed objective is  $\hat{J}_{\lambda}(\pi) = \mathbb{E}[R] - \lambda \mathbb{E}[R^2] + \lambda \max_y(2\mathbb{E}[R]y - y^2)$ , which equals to  $\max_{\pi ,y}J_{\lambda}(\pi ,y) = \sum_{s,a}d_{\pi}(s,a)\big(r(s,a) - \lambda r(s,a)^2 +2\lambda r(s,a)y\big) - \lambda y^2$ . The dual variable  $y$  and policy  $\pi$  are updated iteratively. In each inner loop  $k$ ,  $y$  has analytical solution  $y_{k + 1} = \sum_{s,a}d_{\pi_k}(s,a)r(s,a) = (1 - \gamma)\mathbb{E}_{\pi_k}[G_0]$  since it is quadratic for  $y$ . After  $y$  is updated, learning  $\pi$  is a risk-neutral problem in the original MDP, but with a new modified reward
$$
\hat {r} (s, a) = r (s, a) - \lambda r (s, a) ^ {2} + 2 \lambda r (s, a) y _ {k + 1} \tag {5}
$$
Since  $\hat{r}(s, a)$  is now stationary, any risk-neutral RL algorithms can be applied for policy updating.
Limitations of Per-step Reward Variance. 1)  $\mathbb{V}[R]$  is not an appropriate surrogate for  $\mathbb{V}[G_0]$  due to fundamentally different implications. Consider a simple example. Suppose the policy, the transition dynamics and the rewards are all deterministic, then  $\mathbb{V}[G_0] = 0$  while  $\mathbb{V}[R]$  is usually nonzero unless all the per-step rewards are equal. In this case, shifting a specific step reward by a constant will not affect  $\mathbb{V}[G_0]$  and should not alter the optimal risk-averse policy. However, such shift can lead to a big difference for  $\mathbb{V}[R]$  and may result in an invalid policy as we demonstrated in later example. 2) Reward modification hinders policy learning. Since the reward modifications in [13] (Equation 4) and [14] (Equation 5) share the same issue, here we take Equation 5 as an example. This modification is likely to convert a positive reward to a much smaller or even negative value due to the square term, i.e.  $-\lambda r(s,a)^2$ . In addition, at the beginning of the learning phase, when the policy performance is not good,  $y$  is likely to be negative in some environments (since  $y$  relates to  $\mathbb{E}[G_0]$ ). Thus, the third term  $2\lambda r(s,a)y$  decreases the reward value even more. This prevents the agent to visit the good (i.e., rewarding) state even if that state does not contribute any risk. These two limitations raise a great challenge to subtly choose the value for  $\lambda$  and design the reward for the environment.
Empirical demonstration of the limitations. Consider a maze problem (a modified version of Guarded Maze [7]) in Figure 1. Starting from the bottom left corner, the agent aims to reach the green goal state. The gray color corresponds to walls. The rewards for all states are deterministic (i.e.,  $-1$ ) except for the red state whose reward is a categorical distribution with mean  $-1$ . The reward for visiting the goal is a positive constant value. To reach the goal, a risk-neutral agent prefers the path at the bottom that goes through the red state, but  $\mathbb{V}[G_0]$  will be nonzero. A risk-averse agent prefers the white path in the figure even though  $\mathbb{E}[G_0]$  is slightly lower, but  $\mathbb{V}[G_0] = 0$ . Per-step reward variance methods aim to use  $\mathbb{V}[R]$  as a proxy of  $\mathbb{V}[G_0]$ . For the risk-averse policy leading to the white path, ideally, increasing the goal reward by a constant will not effect  $\mathbb{V}[G_0]$ , but will make a big difference to  $\mathbb{V}[R]$ . For instance, when the goal reward is  $10$ ,  $\mathbb{V}[R] = 10$ . When goal reward is  $20$ ,  $\mathbb{V}[R] \approx 36.4$ ,
which is much more risk-averse. Next, consider the reward modification (Equation 5) for the goal
Figure 1: A modified Guarded Maze [7]. Red state returns an uncertain reward (details in text).
reward when it is 20. The square term in Equation 5 is  $-400\lambda$ . It is very easy to make the goal reward negative even for small  $\lambda$ , e.g., 0.1. We do find this reward modification prevents the agent from reaching the goal in our experiments.
3 Gini Deviation as an Alternative of Variance
To avoid the limitations of  $\mathbb{V}[G_0]$  and  $\mathbb{V}[R]$  we have discussed, in this paper, we propose to use Gini deviation as an alternative of variance. Also, since GD has a similar definition and similar properties as variance, it serves as a more reasonable proxy of  $\mathbb{V}[G_0]$  compared to  $\mathbb{V}[R]$ .
3.1 Gini Deviation: Definition and Properties
GD [22], also known as Gini mean difference or mean absolute difference, is defined as follows. For a random variable  $X$ , let  $X_{1}$  and  $X_{2}$  be two i.i.d. copies of  $X$ , i.e.,  $X_{1}$  and  $X_{2}$  are independent and follow the same distribution as  $X$ . Then GD is given by
$$
\mathbb {D} [ X ] = \frac {1}{2} \mathbb {E} [ | X _ {1} - X _ {2} | ] \tag {6}
$$
Variance can be defined in a similar way as  $\mathbb{V}[X] = \frac{1}{2}\mathbb{E}[(X_1 - X_2)^2]$ .
Given samples  $\{x_i^1\}_{i = 1}^n$  from  $X_{1}$  and  $\{x_j^2\}_{j = 1}^n$  from  $X_{2}$ . The unbiased empirical estimations for GD and variance are  $\hat{\mathbb{D}} [X] = \frac{1}{2n^2}\sum_{i = 1}^{n}\sum_{j = 1}^{n}|x_i^1 -x_j^2 |$  and  $\hat{\mathbb{V}} [X] = \frac{1}{2n^2}\sum_{i = 1}^{n}\sum_{j = 1}^{n}(x_i^1 -x_j^2)^2.$
Both risk profiles aim to measure the variability of a random variable and share similar properties [23]. For example, they are both location invariant, and can be presented as a weighted sum of order statistics. [23] argues that the GD is superior to the variance as a measure of variability for distributions far from Gaussian. We refer readers to this paper for a full overview. Here we highlight two properties of  $\mathbb{D}[X]$  to help interpret it. Let  $\mathcal{M}$  denote the set of real random variables and let  $\mathcal{M}^p$ ,  $p \in [1,\infty)$  denote the set of random variables whose probability measures have finite  $p$ -th moment, then
-  $\mathbb{V}[X] \geq \sqrt{3} \mathbb{D}[X]$  for all  $X \in \mathcal{M}^2$ .
-  $\mathbb{D}[cX] = c\mathbb{D}[X]$  for all  $c > 0$  and  $X \in \mathcal{M}$ .
The first property is known as Glasser's inequality [24], which shows  $\mathbb{D}[X]$  is a lower bound of  $\mathbb{V}[X]$  if  $X$  has finite second moment. The second one is known as positive homogeneity in coherent measures of variability [25], and is also clear from the definition of GD in Equation 6. In RL, considering  $X$  is the return variable, this means GD is less sensitive to the reward scale compared to variance, i.e., scaling the return will scale  $\mathbb{D}[X]$  linearly, but quadratically for  $\mathbb{V}[X]$ . We also provide an intuition of the relation between GD and variance from the perspective of convex order, as shown in Appendix 7. Note also that while variance and GD are both measures of variability, GD is a coherent measure of variability [25]. Appendix 12 provides a discussion of the properties of coherent measures of variability, while explaining the differences with coherent measures of risk such as conditional value at risk (CVaR).
3.2 Signed Choquet Integral for Gini Deviation
This section introduces the concept of signed Choquet integral, which provides an alternative definition of GD and makes gradient-based optimization convenient. Note that with the original definition (Equation 6), it can be intractable to compute the gradient w.r.t. the parameters of a random variable's density function through its GD.
The Choquet integral [26] was first used in statistical mechanics and potential theory and was later applied to decision making as a way of measuring the expected utility [27]. The signed Choquet integral belongs to the Choquet integral family and is defined as:
Definition 1 ([28], Equation 1). A signed Choquet integral  $\Phi_h: X \to \mathbb{R}, X \in \mathcal{L}^\infty$  is defined as
$$
\Phi_ {h} (X) = \int_ {- \infty} ^ {0} \left(h (\Pr (X \geq x)) - h (1)\right) d x + \int_ {0} ^ {\infty} h (\Pr (X \geq x)) d x \tag {7}
$$
where  $\mathcal{L}^{\infty}$  is the set of bounded random variables in a probability space,  $h$  is the distortion function and  $h\in \mathcal{H}$  such that  $\mathcal{H} = \{h:[0,1]\to \mathbb{R},h(0) = 0,h$  is of bounded variation}.
This integral has become the building block of law-invariant risk measures  ${}^{1}$  after the work of [29, 30]. One reason for why signed Choquet integral is of interest to the risk research community is that it
$^{1}$ Law-invariant property is one of the popular "financially reasonable" axioms. If a functional returns the same value for two random variables with the same distribution, then the functional is called law-invariant.
is not necessarily monotone. Since most practical measures of variability are not monotone, e.g., variance, standard deviation, or deviation measures in [31], it is possible to represent these measures in terms of  $\Phi_h$  by choosing a specific distortion function  $h$ .
Lemma 1 ([28], Section 2.6). Gini deviation is a signed Choquet integral with a concave  $h$  given by  $h(\alpha) = -\alpha^2 + \alpha, \alpha \in [0,1]$ .
This Lemma provides an alternative definition for GD, i.e.,  $\mathbb{D}[X] = \int_{-\infty}^{\infty} h\big(\operatorname*{Pr}(X \geq x)\big) dx$ ,  $h(\alpha) = -\alpha^2 + \alpha$ . However, this integral is still not easy to compute. Here we turn to its quantile representation for easy calculation.
Lemma 2 ([28], Lemma 3).  $\Phi_h(X)$  has a quantile representation. If  $F_X^{-1}$  is continuous, then  $\Phi_h(X) = \int_1^0 F_X^{-1}(1 - \alpha) dh(\alpha)$ , where  $F_X^{-1}$  is the quantile function (inverse CDF) of  $X$ .
Combining Lemma 1 and 2,  $\mathbb{D}[X]$  can be computed alternatively as
$$
\mathbb {D} [ X ] = \Phi_ {h} (X) = \int_ {0} ^ {1} F _ {X} ^ {- 1} (1 - \alpha) d h (\alpha) = \int_ {0} ^ {1} F _ {X} ^ {- 1} (\alpha) (2 \alpha - 1) d \alpha \tag {8}
$$
With this quantile representation of GD, we can derive a policy gradient method for our new learning problem in the next section. It should be noted that variance cannot be directly defined by a  $\Phi_h$ -like quantile representation, but as a complicated related representation:  $\mathbb{V}[X] = \sup_{h\in \mathcal{H}}\left\{\Phi_h(X) - \frac{1}{4}\| h' \|_2^2\right\}$ , where  $\| h' \|_2^2 = \int_0^1 (h'(p))^2 dp$  if  $h$  is continuous, and  $\| h' \|_2^2 \coloneqq \infty$  if it is not continuous (see Example 2.2 of [32]). Hence, such representation of the conventional variance measure is not readily usable for optimization.
4 Policy Gradient for Mean-Gini Deviation
In this section, we consider a new learning problem by replacing the variance with GD. Specifically, we consider the following objective
$$
\max  _ {\pi} \mathbb {E} [ G _ {0} ] - \lambda \mathbb {D} [ G _ {0} ] \tag {9}
$$
where  $\lambda$  is the trade-off parameter. To maximize this objective, we may update the policy towards the gradient ascent direction. Computing the gradient for the first term has been widely studied in risk-neutral RL [3]. Computing the gradient for the second term may be difficult at the first glance from its original definition, however, it becomes possible via its quantile representation (Equation 8).
4.1 Gini Deviation Gradient Formula
We first give a general gradient calculation for GD of a random variable  $Z$ , whose distribution function is parameterized by  $\theta$ . In RL, we can interpret  $\theta$  as the policy parameters, and  $Z$  as the return under that policy, i.e.,  $G_{0}$ . Denote the Probability Density Function (PDF) of  $Z$  as  $f_{Z}(z;\theta)$ . Given a confidence level  $\alpha \in (0,1)$ , the  $\alpha$ -level quantile of  $Z$  is denoted as  $q_{\alpha}(Z;\theta)$ , and given by
$$
q _ {\alpha} (Z; \theta) = F _ {Z _ {\theta}} ^ {- 1} (\alpha) = \inf  \left\{z: \Pr \left(Z _ {\theta} \leq z\right) \geq \alpha \right\} \tag {10}
$$
For technical convenience, we make the following assumptions, which are also realistic in RL.
Assumption 1.  $Z$  is a continuous random variable, and bounded in range  $[-b,b]$  for all  $\theta$ .
Assumption 2.  $\frac{\partial}{\partial\theta_i} q_{\alpha}(Z;\theta)$  exists and is bounded for all  $\theta$ , where  $\theta_{i}$  is the  $i$ -th element of  $\theta$ .
Assumption 3.  $\frac{\partial f_Z(z;\theta)}{\partial\theta_i} /f_Z(z;\theta)$  exists and is bounded for all  $\theta ,z.\theta_{i}$  is the  $i$  -th element of  $\theta$
Since  $Z$  is continuous, the second assumption is satisfied whenever  $\frac{\partial}{\partial \theta_i} f_Z(z; \theta)$  is bounded. These assumptions are common in likelihood-ratio methods, e.g., see [33]. Relaxing these assumptions is possible but would complicate the presentation.
Proposition 1. Let Assumptions 1, 2, 3 hold. Then