SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lennert De Smet  
KU Leuven
Emanuele Sansone  
KU Leuven
Pedro Zuidberg Dos Martires  
Örebro University
Categorical random variables can faithfully represent the discrete and uncertain aspects of data as part of a discrete latent variable model. Learning in such models necessitates taking gradients with respect to the parameters of the categorical probability distributions, which is often intractable due to their combinatorial nature. A popular technique to estimate these otherwise intractable gradients is the Log-Derivative trick. This trick forms the basis of the well-known REINFORCE gradient estimator and its many extensions. While the Log-Derivative trick allows us to differentiate through samples drawn from categorical distributions, it does not take into account the discrete nature of the distribution itself. Our first contribution addresses this shortcoming by introducing the CatLog-Derivative trick—a variation of the Log-Derivative trick tailored towards categorical distributions. Secondly, we use the CatLog-Derivative trick to introduce IndeCateR, a novel and unbiased gradient estimator for the important case of products of independent categorical distributions with provably lower variance than REINFORCE. Thirdly, we empirically show that IndeCateR can be efficiently implemented and that its gradient estimates have significantly lower bias and variance for the same number of samples compared to the state of the art.
Categorical random variables naturally emerge in many domains in AI, such as language modelling, reinforcement learning and neural-symbolic AI [10]. They are compelling because they can faithfully represent the discrete concepts present in data in a sound probabilistic fashion. Unfortunately, inference in probabilistic models with categorical latent variables is usually computationally intractable due to its combinatorial nature. This intractability often leads to the use of sampling-based, approximate inference techniques, which in turn poses problems to gradient-based learning as sampling is an inherently non-differentiable process.
In order to bypass this non-differentiability, two main classes of gradient estimators have been developed. On the one hand, there is a range of unbiased estimators based on the Log-Derivative trick and the subsequent REINFORCE gradient estimator [40]. On the other hand, we have biased estimators that use continuous relaxations to which the reparametrisation trick [33] can be applied, such as the Gumbel-Softmax trick [16, 22].
A clear advantage of the REINFORCE estimator over relaxation-based estimators is its unbiased nature. However, REINFORCE tends to be sample-inefficient and its gradient estimates exhibit high variance in practice. To resolve these issues, methods have been proposed that modify REINFORCE by, for instance, adding control variates [31, 37]. These modified estimators have been shown to deliver more robust gradient estimates than standard REINFORCE.
Instead of modifying REINFORCE, we take a different approach and modify the Log-Derivative trick by explicitly taking into account that we are working with multivariate categorical distributions.
We call this first contribution the CatLog-Derivative trick. Interestingly, we show that our CatLog-Derivative trick leads to Rao-Blackwellised estimators [4], immediately giving us a guaranteed reduction in variance. The CatLog-Derivative trick can also be seen as a generalisation of the Local Expectation Gradients (LEG) [36] capable of exploiting more structural, distributional properties. This connection to LEG will be clarified throughout the paper.
As a second contribution, we propose IndeCateR (read as 'indicator'), a gradient estimator for the special case of independent categorical random variables. IndeCateR is a hyperparameter-free estimator that can be implemented efficiently by leveraging parallelisation on modern graphical processing units (GPUs). Thirdly, we empirically show that IndeCateR is competitive with comparable state-of-the-art gradient estimators on a range of standard benchmarks from the literature.
Throughout this paper, we consider expectations with respect to multivariate categorical probability distributions of the form
$$
\mathbb {E} _ {\mathbf {X} \sim p (\mathbf {X})} [ f (\mathbf {X}) ] = \sum_ {\mathbf {x} \in \Omega (\mathbf {X})} p (\mathbf {x}) f (\mathbf {x}), \tag {2.1}
$$
where we assume this expectation to be finite. The symbol  $\mathbf{X}$  denotes a random vector  $(X_{1},\ldots ,X_{D})$  of  $D$  categorical random variables while  $p(\mathbf{X})$  denotes a multivariate probability distribution. The expression  $\mathbf{X}\sim p(\mathbf{X})$  indicates that the random vector  $\mathbf{X}$  is distributed according to  $p(\mathbf{X})$ . On the right-hand side of Equation (2.1) we write the expectation as an explicit sum over  $\Omega (\mathbf{X})$ , the finite sample space of the random vector  $\mathbf{X}$ , using  $\mathbf{x} = (x_{1},\dots,x_{D})$  for the specific assignments of the random vector  $(X_{1},\ldots ,X_{D})$ .
Given an order of the random variables in  $\mathbf{X}$ , we can induce a factorisation of the joint probability distribution as follows
$$
p (\mathbf {X}) = \prod_ {d = 1} ^ {D} p \left(X _ {d} \mid \mathbf {X} _ {<   d}\right). \tag {2.2}
$$
Here,  $\mathbf{X}_{<d}$  denotes the ordered set of random variables  $(X_{1},\ldots,X_{d-1})$ . Similarly,  $\mathbf{X}_{>d}$  will denote the ordered set  $(X_{d+1},\ldots,X_{D})$  in subsequent sections.
When performing gradient-based learning, we are interested in partial derivatives of the expected value in (2.1), i.e.,  $\partial_{\lambda}\mathbb{E}_{\mathbf{X}\sim p(\mathbf{X})}[f(\mathbf{X})]$ . Here, we take the partial derivative of the expectation with respect to the parameter  $\lambda$  and assume that the distribution  $p(\mathbf{X})$  and the function  $f(\mathbf{X})$  depend on a set of parameters  $\Lambda$  with  $\lambda \in \Lambda$ . For probability distributions to which the reparametrisation trick does not apply, we can rewrite the partial derivative using the Log-Derivative trick.
Theorem 2.1 (Log-Derivative Trick [40]). Let  $p(\mathbf{X})$  be a probability distribution and  $f(\mathbf{X})$  such that its expectation is finite, with both functions depending on a set of parameters  $\Lambda$ . Then, it holds that
$$
\partial_ {\lambda} \mathbb {E} _ {\mathbf {X} \sim p (\mathbf {X})} [ f (\mathbf {X}) ] = \mathbb {E} _ {\mathbf {X} \sim p (\mathbf {X})} [ \partial_ {\lambda} f (\mathbf {X}) ] + \mathbb {E} _ {\mathbf {X} \sim p (\mathbf {X})} [ f (\mathbf {X}) \partial_ {\lambda} \log p (\mathbf {X}) ]. \tag {2.3}
$$
In general, both expectations in Equation (2.3) are intractable and often estimated with a Monte Carlo scheme. The most immediate such estimation is provided by the REINFORCE gradient estimator [40]
$$
\partial_ {\lambda} \mathbb {E} _ {\mathbf {X} \sim p (\mathbf {X})} [ f (\mathbf {X}) ] \approx \frac {1}{N} \sum_ {n = 1} ^ {N} \left(\partial_ {\lambda} f \left(\mathbf {x} ^ {(n)}\right) + f \left(\mathbf {x} ^ {(n)}\right) \partial_ {\lambda} \log p \left(\mathbf {x} ^ {(n)}\right)\right). \tag {2.4}
$$
The superscript on  $\mathbf{x}^{(n)}$  denotes that it is the  $n^{\mathrm{th}}$  sample vector drawn from  $p(\mathbf{X})$ .
A well-known problem with the REINFORCE gradient estimator is the high variance stemming from the second term in Equation (2.4). A growing body of research has been tackling this problem by proposing variance reduction techniques [11, 31, 37, 39]. In what follows we will focus on estimating this second term and drop the first term, since it can be assumed to be unproblematic.
3 The CatLog-Derivative Trick
The standard Log-Derivative trick and its corresponding gradient estimators are applicable to both discrete and continuous probability distributions. However, this generality limits their usefulness when it comes to purely categorical random variables. For example, the REINFORCE gradient estimator suffers from high variance when applied to problems involving high-dimensional multivariate categorical random variables. In such a setting there are exponentially many possible states to be sampled, which makes it increasingly unlikely that a specific state gets sampled. We now introduce the CatLog-Derivative trick that reduces the exponential number of states arising in a multivariate categorical distribution by exploiting the distribution's factorisation.
Theorem 3.1 (CatLog-Derivative Trick). Let  $p(\mathbf{X})$  be a multivariate categorical probability distribution that depends on a set of parameters  $\Lambda$  and assume  $\mathbb{E}_{\mathbf{X} \sim p(\mathbf{X})}[f(\mathbf{X})]$  is finite. Then, it holds that  $\partial_{\lambda}\mathbb{E}_{\mathbf{X} \sim p(\mathbf{X})}[f(\mathbf{X})]$  is equal to
$$
\sum_ {d = 1} ^ {D} \sum_ {x _ {\delta} \in \Omega (X _ {d})} \mathbb {E} _ {\mathbf {X} _ {<   d} \sim p (\mathbf {X} _ {<   d})} \left[ \partial_ {\lambda} p \left(x _ {\delta} \mid \mathbf {X} _ {<   d}\right) \mathbb {E} _ {\mathbf {X} _ {> d} \sim p \left(\mathbf {X} _ {> d} \mid x _ {\delta}, \mathbf {X} _ {<   d}\right)} \left[ f \left(\mathbf {X} _ {\neq d}, x _ {\delta}\right) \right] \right] \tag {3.1}
$$
Proof. We start by applying the standard Log-Derivative trick and fill in the product form of the categorical distribution followed by pulling this product out of the logarithm and the expectation
$$
\begin{array}{l} \mathbb {E} _ {\mathbf {X} \sim p (\mathbf {X})} [ f (\mathbf {X}) \partial_ {\lambda} \log p (\mathbf {X}) ] = \mathbb {E} _ {\mathbf {X} \sim p (\mathbf {X})} \left[ f (\mathbf {X}) \partial_ {\lambda} \log \prod_ {d = 1} ^ {D} p \left(X _ {d} \mid \mathbf {X} _ {<   d}\right) \right], (3.2) \\ = \sum_ {d = 1} ^ {D} \mathbb {E} _ {\mathbf {X} \sim p (\mathbf {X})} [ f (\mathbf {X}) \partial_ {\lambda} \log p (X _ {d} \mid \mathbf {X} _ {<   d}) ]. (3.3) \\ \end{array}
$$
To continue, we write out the expectation explicitly and write the sum for the random variable  $X_{d}$  separately, resulting in
$$
\sum_ {d = 1} ^ {D} \sum_ {\mathbf {x} \in \Omega (\mathbf {X} _ {\neq d})} \sum_ {x _ {\delta} \in \Omega (X _ {d})} p \left(\mathbf {x} _ {\neq d}, x _ {\delta}\right) f \left(\mathbf {x} _ {\neq d}, x _ {\delta}\right) \partial_ {\lambda} \log p \left(x _ {\delta} \mid \mathbf {x} _ {<   d}\right). \tag {3.4}
$$
Next, we factorize the joint probability  $p(\mathbf{x}_{\neq d}, x_{\delta})$  as  $p(\mathbf{x}_{>d} \mid x_{\delta}, \mathbf{x}_{<d}) p(x_{\delta} \mid \mathbf{x}_{<d}) p(\mathbf{x}_{<d})$ . Multiplying the second of these factors with  $\partial_{\lambda} \log p(x_{\delta} \mid \mathbf{x}_{<d})$  gives us  $\partial_{\lambda} p(x_{\delta} \mid \mathbf{x}_{<d})$ . Finally, plugging  $\partial_{\lambda} p(x_{\delta} \mid \mathbf{x}_{<d})$  into Equation (3.4) gives the desired expression.
Intuitively, the CatLog-Derivative trick decomposes the Log-Derivative trick into an explicit sum of multiple Log-Derivative tricks, one for each of the categorical random variables present in the multivariate distribution. Next, we use the CatLog-Derivative trick to define a novel gradient estimator that exploits the structure of a probability distribution by following the variable ordering of the distribution's factorisation and effectively Rao-Blackwellises [4] REINFORCE.
Definition 3.2 (The SCateR gradient estimator). We define the Structured Categorical REINFORCE (SCateR) estimator via the expression
$$
\partial_ {\lambda} \mathbb {E} _ {\mathbf {X} \sim p (\mathbf {X})} [ f (\mathbf {X}) ] \approx \sum_ {d = 1} ^ {D} \sum_ {x _ {\delta} \in \Omega (X _ {d})} \frac {1}{N} \sum_ {n _ {d} = 1} ^ {N} \partial_ {\lambda} p \left(x _ {\delta} \mid \mathbf {x} _ {<   d} ^ {(n _ {d})}\right) f \left(\mathbf {x} _ {<   d} ^ {(n _ {d})}, x _ {\delta}, \mathbf {x} _ {> d} ^ {(n _ {d})}\right), \tag {3.5}
$$
where the sample  $\mathbf{x}_{>d}^{(n_d)}$  is drawn while conditioning on  $x_{\delta}$  and  $\mathbf{x}_{<d}^{(n_d)}$ . The subscript on  $n_d$  indicates that different samples can be drawn for every  $d$ .
Proposition 3.3. The SCateR estimator Rao-Blackwellises REINFORCE.
Proof. We start from Equation (2.4) for the REINFORCE estimator, where we ignore the first term and factorize the probability distribution similar to Equation (3.3)
$$
\partial_ {\lambda} \mathbb {E} _ {\mathbf {X} \sim p (\mathbf {X})} [ f (\mathbf {X}) ] \approx \frac {1}{N} \sum_ {n = 1} ^ {N} f \left(\mathbf {x} ^ {(n)}\right) \partial_ {\lambda} \log p \left(\mathbf {x} ^ {(n)}\right) \approx \sum_ {d = 1} ^ {D} \frac {1}{N} \sum_ {n = 1} ^ {N} f \left(\mathbf {x} ^ {(n)}\right) \partial_ {\lambda} \log p \left(x _ {d} ^ {(n)}\right). \tag {3.6}
$$
For notational conciseness, we drop the subscript on  $n_d$  and simply use  $n$  to identify single samples. Now we compare Equation (3.5) and Equation (3.6) term-wise for  $N \to \infty$
$$
\sum_ {n = 1} ^ {N} \mathbb {E} _ {X _ {d} \sim p (X _ {d} | \mathbf {x} _ {<   d} ^ {(n)})} \left[ f (\mathbf {x} _ {<   d} ^ {(n)}, X _ {d}, \mathbf {x} _ {> d} ^ {(n)}) \partial_ {\lambda} \log p (X _ {d} | \mathbf {x} _ {<   d} ^ {(n)}) \right] = \sum_ {n = 1} ^ {N} f (\mathbf {x} _ {<   d} ^ {(n)}, x _ {d} ^ {(n)}, \mathbf {x} _ {> d} ^ {(n)}) \partial_ {\lambda} \log p (x _ {d} ^ {(n)}).
$$
In the equation above, we see that SCateR takes the expected value for  $X_{d}$  (left-hand side) and computes it exactly using an explicit sum over the space  $\Omega(X_{d})$ , whereas REINFORCE (right-hand side) uses sampled values. This means, in turn, that the left-hand side is a Rao-Blackwellised version of the right-hand side. Doing this for every  $d$  gives us a Rao-Blackwellised version for REINFORCE, i.e., the SCateR estimator.
Corollary 3.4 (Bias and Variance). The SCateR estimator is unbiased and its variance is upper-bounded by the variance of REINFORCE.
Proof. This follows immediately from Proposition 3.3, the law of total expectation and the law of total variance [3, 29].  $\square$
Computational complexity. Consider Equation (3.5) and observe that none of the random variables has a sample space larger than  $K = \max_d(|\Omega(X_d)|)$ . Computing our gradient estimate requires performing three nested sums with lower bound 1 and upper bounds equal to  $D$ ,  $K$  and  $N$ , respectively. These summations result in a time complexity of  $\mathcal{O}(D \cdot K \cdot N)$ . Leveraging the parallel implementation of prefix sums on GPUs [19], a time complexity of  $\mathcal{O}(\log D + \log K + \log N)$  can be obtained, which allows for the deployment of SCateR in modern deep architectures.
Function evaluations. The main limitation of SCateR is the number of function evaluations it requires to provide its estimates. Indeed, each term in the three nested sums of Equation (3.5) involves a different function evaluation, meaning an overall number of at most  $D \cdot \max_d(|\Omega(X_d)|) \cdot N$  function evaluations is necessary for every estimate. Fortunately, these evaluations can often be parallelised in modern deep architectures, leading to a positive trade-off between function evaluations and performance in our experiments (Section 6).
Connection to Local Expectation Gradients (LEG). LEG [36] was proposed as a gradient estimator tailored for variational proposal distributions. Similar to the CatLog-Derivative trick, it also singles out variables from a target expectation by summing over the variables' domains. However, it does so one variable at a time by depending on the Markov blanket of those variables. This dependency has the disadvantage of inducing a weighted expression, as variables downstream of a singled-out variable  $X_{i}$  are sampled independently of the values assigned to  $X_{i}$ . In this sense, LEG can intuitively be related to Gibbs sampling where a variable is marginalised given possible assignments to its Markov blanket instead of being sampled.
In contrast, the CatLog-Derivative trick is applicable to any probability distribution with a known factorisation, even in the case where different factors share parameters. It sums out variables by following the variable ordering of this factorisation rather than depending on the Markov blanket. Following the topology induced by the distribution's factorisation naturally avoids the disconnect between summed-out and downstream variables for which LEG required a weighting term and makes the CatLog-Derivative trick more related to ancestral sampling. Moreover, computing these LEG weights adds additional computational complexity. A more technical and formal comparison between LEG and the CatLog-Derivative trick can be found in the appendix.
4 The IndeCateR Gradient Estimator
Using the CatLog-Derivative trick derived in the previous section we are now going to study a prominent special case of multivariate categorical distributions. That is, we will assume that our probability distribution admits the independent factorisation  $p(\mathbf{X}) = \prod_{d=1}^{D} p_d(X_d)$ . Note that all  $D$  different distributions still depend on the same set of learnable parameters  $\Lambda$ . Furthermore, we subscript the individual distributions  $p_d$  as they can no longer be distinguished by their conditioning sets. Plugging in this factorisation into Theorem 3.1 gives us the Independent Categorical REINFORCE estimator, or IndeCateR for short.
Proposition 4.1 (IndeCateR). Let  $p(\mathbf{X})$  be a multivariate categorical probability distribution that depends on a set of parameters  $\Lambda$  and factorises as  $p(\mathbf{X}) = \prod_{d=1}^{D} p_d(X_d)$ , then the gradient of a finite expectation  $\mathbb{E}_{\mathbf{X} \sim p(\mathbf{X})} [f(\mathbf{X})]$  can be estimated with
$$
\sum_ {d = 1} ^ {D} \sum_ {x _ {\delta} \in \Omega \left(X _ {d}\right)} \partial_ {\lambda} p _ {d} \left(x _ {\delta}\right) \frac {1}{N} \sum_ {n _ {d} = 1} ^ {N} f \left(\mathbf {x} _ {\neq d} ^ {(n _ {d})}, x _ {\delta}\right), \tag {4.1}
$$
where  $\mathbf{x}_{\neq d}^{(n_d)}$  are samples drawn from  $p(\mathbf{X}_{\neq d})$
Proof. We start by looking at the expression in Equation (3.1). Using the fact that we have a set of independent random variables, we can simplify  $p(x_{\delta} \mid \mathbf{X}_{<d})$  to  $p_d(x_{\delta})$ . As a result, the gradient of the expected value can be rewritten as
$$
\begin{array}{l} \partial_ {\lambda} \mathbb {E} _ {\mathbf {X} \sim p (\mathbf {X})} [ f (\mathbf {X}) ] = \sum_ {d = 1} ^ {D} \sum_ {x _ {\delta} \in \Omega (X _ {d})} \mathbb {E} _ {\mathbf {X} _ {<   d} \sim p (\mathbf {X} _ {<   d})} \left[ \partial_ {\lambda} p _ {d} \left(x _ {\delta}\right) \mathbb {E} _ {\mathbf {X} _ {> d} \sim p (\mathbf {X} _ {> d})} \left[ f \left(\mathbf {X} _ {\neq d}, x _ {\delta}\right) \right] \right] (4.2) \\ = \sum_ {d = 1} ^ {D} \sum_ {x _ {\delta} \in \Omega \left(X _ {d}\right)} \partial_ {\lambda} p _ {d} \left(x _ {\delta}\right) \mathbb {E} _ {\mathbf {X} _ {\neq d} \sim p \left(\mathbf {X} _ {\neq d}\right)} \left[ f \left(\mathbf {X} _ {\neq d}, x _ {\delta}\right) \right] (4.3) \\ \end{array}
$$
Drawing  $N$  samples for the  $D - 1$  independent random variables  $\mathbf{X}_{\neq d}$  and for each term in the sum over  $d$  then gives us the estimate stated in the proposition.
Example 4.2 (Independent Factorisation). Let us consider a multivariate distribution involving three 3-ary independent categorical random variables. Concretely, this gives us
$$
p \left(X _ {1}, X _ {2}, X _ {3}\right) = p _ {1} \left(X _ {1}\right) p _ {2} \left(X _ {2}\right) p _ {3} \left(X _ {3}\right), \tag {4.4}
$$
where  $X_{1}$ ,  $X_{2}$  and  $X_{3}$  can take values from the set  $\Omega(X) = \{1, 2, 3\}$ . Taking this specific distribution and plugging it into Equation (4.1) for the IndeCateR gradient estimator now gives us
$$
\sum_ {d = 1} ^ {3} \sum_ {x _ {\delta} \in \{1, 2, 3 \}} \partial_ {\lambda} p _ {d} \left(x _ {\delta}\right) \frac {1}{N} \sum_ {n _ {d} = 1} ^ {N} f \left(\mathbf {x} _ {\neq i} ^ {(n _ {d})}, x _ {\delta}\right). \tag {4.5}
$$
In order to understand the difference between the Log-Derivative trick and the CatLog-Derivative trick, we are going to look at the term for  $d = 2$  and consider the single-sample estimate
$$
\partial_ {\lambda} p _ {2} (1) f \left(x _ {1}, 1, x _ {3}\right) + \partial_ {\lambda} p _ {2} (2) f \left(x _ {1}, 2, x _ {3}\right) + \partial_ {\lambda} p _ {2} (3) f \left(x _ {1}, 3, x _ {3}\right), \tag {4.6}
$$
where  $x_{1}$  and  $x_{3}$  are sampled values for the random variables  $X_{1}$  and  $X_{3}$ . These samples could be different ones for  $d \neq 2$ . The corresponding single sample estimate using REINFORCE instead of IndeCateR would be  $\partial_{\lambda}p_2(x_2)f(x_1,x_2,x_3)$ .
We see that for REINFORCE we sample all the variables whereas for IndeCateR we perform the explicit sum for each of the random variables in turn and only sample the remaining variables.
In Section 6, we demonstrate that performing the explicit sum, as shown in Example 4.2, leads to practical low-variance gradient estimates. Note how, in the case of  $D = 1$ , Equation (4.1) reduces to the exact gradient. With this in mind, we can interpret IndeCateR as computing exact gradients for each single random variable  $X_{d}$  with respect to an approximation of the function  $\mathbb{E}_{\mathbf{X}_{\neq d}\sim p(\mathbf{X}_{\neq d})}[f(\mathbf{X}_{\neq d},X_d)]$ .
LEG in the independent case. The gap between the LEG and CatLog-Derivative trick gradient estimates grows smaller in the case the underlying distributions factorises into independent factors. The additional weighting function for LEG collapses to 1, resulting in similar expressions. However, two main differences remain. First, LEG does not support factors with shared parameters, although this problem can be mitigated by applying the chain rule. Second, and more importantly, LEG does not allow for different samples to be drawn for different variables. While this difference seems small, we will provide clear evidence that it can have a significant impact on performance (Section 6.4).
Apart from LEG, another closely related work is the RAM estimator [38]. RAM starts by reparametrising the probability distribution using the Gumbel-Max trick followed by a marginalisation. We show in Section 3 that this reparametrisation step is unnecessary, allowing SCateR to explicitly retain the conditional dependency structure. Here we also see the different objectives of our work and Tokui and Sato's. While they proposed RAM in order to theoretically study the quality of control variate techniques, we are interested in building practical estimators that can exploit the structure of specific distributions.
Moreover, just like LEG, Tokui and Sato [38] did not study the setting of a shared parameter space  $\Lambda$  between distributions, although this being the most common setting in modern deep-discrete and neural-symbolic architectures. Hence, an experimental evaluation for this rather common setting is missing in Tokui and Sato [38]. By providing such an evaluation, we show that efficiently implemented Rao-Blackwellised gradient estimators for categorical random variables are a viable option in practice when compared to variance reduction schemes based on control variates.
Such variance reduction methods for REINFORCE aim to reduce the variance by subtracting a mean-zero term, called the baseline, from the estimate [2, 28]. Progress in this area is mainly driven by multi-sample, i.e., sample-dependent, baselines [11, 12, 13, 25, 30, 37, 39] and leave-one-out baselines [17, 18, 26, 31, 34, 35]. Variance can further be reduced by coupling multiple samples and exploiting their dependencies [6, 7, 8, 9, 43, 44, 45]. These latter methods usually introduce bias, which has to be resolved by, for instance, an importance weighing step [8].
A general drawback of baseline variance reduction methods is that they often involve a certain amount of computational overhead, usually in the form of learning optimal parameters or computing statistics. This computational overhead can be justified by assuming that the function in the expectation is costly to evaluate. In such cases, it might pay off to perform extra computations if that means the expensive function is evaluated fewer times.