CARMS: Categorical-Antithetic-REINFORCE Multi-Sample Gradient Estimator

1 Introduction . When optimizing an expectation based objective of the form Ez∼qϕ ( z ) [ f ( z ) ] , we sometimes require the gradients with respect to the parameters ϕ of the distribution . This is challenging for discrete variables , because the commonly used reparameterization gradient does not directly work , unless the discrete distribution is approximated by a continuous and reparameterizable one [ Jang et al. , 2017 , Maddison et al. , 2017 ] . A significant part of this field has thus been score function ( REINFORCE ) based estimators [ Glynn , 1990 , Williams , 1992 , Fu , 2006 ] , which are general and do not require f to be the differentiable . In this paper , we focus on the case when z is a high dimensional categorical variable with logits ϕ . An common example of this form is the evidence lower bound ( ELBO ) [ Jordan et al. , 1998 ] , which arises in variational inference and is used for training variational autoencoders [ Kingma and Welling , 2014 , Rezende et al. , 2014 ] . Because their latent space consists of a large number of categorical variables , they require Monte Carlo gradients with respect to the parameters of the stochastic distribution , but have excellent performance , as a categorical VAE has in practice achieved state-of-the-art zero-shot image generation [ Ramesh et al. , 2021 ] . Our main contribution is a novel unbiased and low-variance gradient estimator for categorical variables . The Categorical-Antithetic-REINFORCE-Multi-Sample ( CARMS ) estimator uses a copula to generate any number of antithetic ( mutually negatively correlated ) [ Owen , 2013 ] categorical samples , and constructs an unbiased estimator by combining them into a baseline for variance reduction . This approach is inspired by the ARMS estimator [ Dimitriev and Zhou , 2021 ] , which also uses multiple antithetic samples , but only works for binary variables . For two categories , CARMS 1https : //github.com/alekdimi/carms 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . reduces to it , while for independent samples , CARMS reduces to the leave-one-out-REINFORCE ( LOORF ) estimator . Our approach achieves higher ELBO with VAE models than the state of the art , as well as higher log likelihood for conditional image completion . Related work One widely used group of gradient estimators for categorical variables is based on trading off bias for lower variance . This includes the straight through ( ST ) estimator [ Bengio et al. , 2013 ] , the direct argmax [ Lorberbom et al. , 2019 ] , as well as the concurrently developed equivalent Gumbel-Softmax ( GS ) [ Jang et al. , 2017 ] or Concrete [ Maddison et al. , 2017 ] that uses a continuous relaxation . These were further improved by combining them with REINFORCE to obtain unbiased estimators , which includes REBAR [ Tucker et al. , 2017 ] , as well as RELAX [ Grathwohl et al. , 2018 ] , which uses a free-form neural network . In practice , REINFORCE is almost always augmented by baselines , e.g. , in variational inference [ Mnih and Gregor , 2014 , Ranganath et al. , 2014 , Paisley et al. , 2012 , Ruiz et al. , 2016 , Kucukelbir et al. , 2017 ] . MuProp [ Gu et al. , 2016 ] uses a first order mean-field Taylor approximation , but requires f to be differentiable . Other estimators apply Rao-Blackwellization [ Liu et al. , 2019 , Kool et al. , 2020 ] , e.g. , to one latent variable at a time [ Titsias and Lázaro-Gredilla , 2015 ] , to the ST estimator [ Paulus et al. , 2021 ] , or to the reparameterized Dirichlet vector in ARSM [ Dong et al. , 2021 ] . Besides REINFORCE and reparameterization type gradients , there is also the measure valued gradient [ Rosca et al. , 2019 ] and finite differences [ Fu , 2006 ] , but are less common in practice . A comprehensive review can be found in Mohamed et al . [ 2020 ] . The more recent approaches for categorical variables are unbiased and REINFORCE based , building on the idea of using multiple samples to construct a baseline for variance reduction . One such baseline is LOORF [ Kool et al. , 2019a ] , originally introduced in Salimans and Knowles [ 2014 ] and also known as VarGrad [ Richter et al. , 2020 ] , where its theoretical properties are further analyzed . VIMCO [ Mnih and Rezende , 2016 ] has a similar form , but is specific to the importance weighted multi sample bound [ Burda et al. , 2016 ] . A different approach is ARSM [ Yin et al. , 2019 ] , which reparameterizes the gradient with a Dirichlet distribution and uses swaps to obtain multiple correlated samples . However , it adds some amount of variance due to the continuous reparameterization , and can require up to C ( C − 1 ) /2 function evaluations per step , which can be computationally expensive . More recently , the unordered set estimator ( UNORD ) [ Kool et al. , 2020 ] uses the Gumbel top-k trick to sample without replacement , and then constructs a baseline using a multiplicative term to preserve its unbiasedness . 2 Background . Let z = ( z1 , ... , zD ) denote D independent categorical variables , where zd is a one hot encoded categorical sample from zd ∼ qϕd ( zd ) = Cat ( σ ( ϕd ) ) , with σ ( x ) i = exi/ ∑ j e xj being the softmax function . Let also ı̂ be the basis vector with its ith coordinate set to one , and otherwise zero , such that P ( z = ı̂ ) = σ ( ϕ ) i , or equivalently E [ z ] = σ ( ϕ ) . Unless otherwise stated , superscripts denote the dimension , and subscripts denote different samples , with vectors and matrices being bold lowercase and bold uppercase symbols , respectively . We are interested in optimizing the following objective with respect to the logits ϕ = ( ϕ1 , ... , ϕD ) : L ( ϕ ) = Ez∼qϕ ( z ) [ f ( z ) ] , qϕ ( z ) = D∏ d=1 qϕd ( zd ) . Although the score function gradient contains two terms : ∇ϕL ( ϕ ) = Ez∼qϕ ( z ) [ ∇ϕf ( z ) + f ( z ) ∇ϕ ln qϕ ( z ) ] the first term is easily estimated , so we omit the subscript in f for notational clarity , and we focus on the latter term in this work . Since CARMS generalizes the multisample LOORF estimator beyond independent samples , we review LOORF here . We also review the ARMS estimator , which uses a copula to generate antithetic samples , but is restricted to binary variables . 2.1 LOORF . Leave-one-out-REINFORCE ( LOORF ) [ Salimans and Knowles , 2014 , Kool et al. , 2019a ] , also known as VarGrad [ Richter et al. , 2020 ] , is a general score function based gradient estimator that uses N i.i.d . samples to construct a baseline for variance reduction , and is competitive with state of the art estimators . Its theoretical properties have recently been analyzed in Richter et al . [ 2020 ] , where the same estimator results from minimizing the variance of the log ratio between the posterior and approximating distribution in variational inference . The only requirements for LOORF are the ability to sample z ∼ qϕ ( z ) and evaluate f ( z ) . Given N samples z1 , ... , zN iid∼ Cat ( σ ( ϕ ) ) , it has the form : gLOORF = 1 N−1 N∑ n=1 ( f ( zn ) − f̄ ( z ) ) ∇ϕ ln qϕ ( zn ) = 1N−1 N∑ n=1 ( f ( zn ) − f̄ ( z ) ) ( zn − σ ( ϕn ) ) , ( 1 ) where f̄ ( z ) = 1N ∑N n=1 f ( zn ) . It is commonly used due to its simplicity and strong performance . 2.2 ARMS . The Antithetic-REINFORCE-MultiSample ( ARMS ) [ Dimitriev and Zhou , 2021 ] estimator is a recent work that uses any number N of mutually negatively correlated samples . However , although unbiased , it is limited to binary variables , which motivated us to extend it to the categorical case . Since any antithetic copula in two dimensions reduces to the pair ( u , 1 − u ) , it generalizes DisARM [ Dong et al. , 2020 ] , independently discovered as unbiased uniform gradient ( U2G ) [ Yin et al. , 2020 ] , which use N = 2 samples . ARMS achieves this generalization by using a copula [ Trivedi and Zimmer , 2007 ] , which is any multivariate distribution whose marginals are uniform random variables : u = ( u1 , ... , uN ) ∼ CN , ∀i : ui ∼ Unif ( 0 , 1 ) . ARMS uses a Dirichlet or Gaussian copula with strong negative dependence between each dimension of u . However any copula can be used instead , with the only two requirements being the ability to generate samples easily , as well as being able to evaluate the bivariate CDF Φ ( ui , uj ) of the copula , so that the debiasing term can be calculated . Given a copula sample u , it uses inverse CDF sampling to convert it into N antithetic Bern ( p ) samples , which are simply bi = 1ui < p , ∀i . For a set of N antithetic D-dimensional Bernoulli variables b1 , ... , bN with probabilities σ ( ϕ ) , the N -sample unbiased estimator has the following simple form : gARMS = 1 N − 1 N∑ n=1 ( f ( bn ) − 1 n N∑ m=1 f ( bm ) ) bn − σ ( ϕ ) 1− ρ , ( 2 ) with ρ = ( ρ1 , ... , ρD ) and ρd = corr ( bdi , b d j ) , which is simple to compute given the bivariate CDF of the copula . 3 CARMS . The CARMS estimator has a similar form to LOORF , but requires a multiplicative term to remain unbiased . It has two requirements : an easy way to sample antithetic categorical variables , and being able to compute the bivariate probability mass function ( PMF ) , which should be identical for any pair . For clarity , we begin by assuming the ability to do this , and derive the univariate version for two samples , which we then extend to N samples . Next , we generalize CARMS to any number of categorical variables . Lastly , in Section 3.2 , we show two different ways of sampling that satisfy both conditions : inverse CDF sampling with an easily computable analytical bivariate PMF , and the Gumbel max trick combined with an empirical estimate of the PMF . The two sample version of CARMS can be obtained by replacing the two i.i.d . samples in 2-LOORF with an arbitrarily correlated pair of categorical variables and an added debiasing term . For N = 2 samples z , z′ ∼ Cat ( σ ( ϕ ) ) , LOORF has the following simple form : gLOORF ( z , z ′ ) = 1 2 ( f ( z ) − f ( z′ ) ) ( z − z′ ) , ( 3 ) which , importantly , is unbiased [ Kool et al. , 2019a , Dimitriev and Zhou , 2021 ] . This means that using a simple importance weight preserves its unbiasedness , for an arbitrary bivariate categorical distribution ( z , z′ ) ∼ Cat ( σ ( ϕ ) , σ ( ϕ ) ) : E [ gLOORF ( z , z ′ ) P ( z = ı̂ ) P ( z′ = ȷ̂ ) P ( z = ı̂ , z′ = ȷ̂ ) ] = ∑ i , j P ( z = ı̂ , z′ = ȷ̂ ) P ( z = ı̂ ) P ( z′ = ȷ̂ ) P ( z = ı̂ , z′ = ȷ̂ ) gLOORF ( ̂ı , ȷ̂ ) = ∑ i , j P ( z = ı̂ ) P ( z′ = ȷ̂ ) gLOORF ( ̂ı , ȷ̂ ) = E z , z′ iid∼ Cat ( σ ( ϕ ) ) [ gLOORF ( z , z ′ ) ] = ∇ϕL ( ϕ ) . We summarize the derivation of the two sample version of CARMS , which we denote as the Categorical-Antithetic-REINFORCE-Two-Sample ( CARTS ) estimator in the following theorem . Theorem 1 Let ( z , z′ ) be a sample from an arbitrary bivariate Categorical distribution with marginal distributions z , z′ ∼ Cat ( σ ( ϕ ) ) , and a known bivariate PMF . An unbiased estimator of ∇ϕE [ f ( z ) ] is : gCARTS ( z , z ′ ) = 1 2 ( f ( z ) − f ( z′ ) ) ( z − z′ ) zTRz′ , Rij = σ ( ϕ ) iσ ( ϕ ) j P ( z = ı̂ , z′ = ȷ̂ ) . ( 4 ) The intuition behind using negatively correlated variables is that we want to avoid the case when z = z′ , because the sample is then “ wasted. ” We formalize this intuition below , and defer the proof to the Appendix . Theorem 2 Let ( z , z′ ) ∼ Cat ( σ ( ϕ ) , σ ( ϕ ) ) . If the bivariate PMF satisfies : ∀i ̸= j : P ( z = ı̂ , z′ = ȷ̂ ) ≥ P ( z = ı̂ ) P ( z′ = ȷ̂ ) , with strict inequality for at least one pair , then Var [ gCARTS ( z , z′ ) ] < Var [ gLOORF ( z , z′ ) ] . We now extend CARTS to N samples , using the following identity , the proof of which can be found in Dimitriev and Zhou [ 2021 ] . It states that N -sample LOORF is equivalent to averaging 2-sample LOORF over all ( N 2 ) pairs : gLOORF ( z1 , .. , zN ) = 1 N N∑ n=1 ( f ( zn ) − 1 N N∑ m=1 f ( zm ) ) ( zn − σ ( ϕ ) ) = 1N ( N−1 ) ∑ n ̸=m 1 2 ( f ( zn ) − f ( zm ) ) ( zn − zm ) = 1N ( N−1 ) ∑ n ̸=m gLOORF ( zn , zm ) . With the above identity it easily follows that given N antithetic categorical samples Z = [ z1 , ... , zN ] T , applying CARTS to all pairs results in an unbiased estimator , due to linearity of expectations : E [ gCARMS ( Z , R ) ] = E [ 1 N ( N−1 ) ∑ n ̸=m gCARTS ( zn , zm , R ) ] = E [ 1 N ( N−1 ) ∑ n ̸=m gLOORF ( zn , zm ) ] = E [ gLOORF ( Z ) ] = ∇ϕL ( ϕ ) . We summarize CARMS in the next theorem and also rewrite it in a simpler matrix form used in our implementation . The matrix form can be obtained after some algebra , which can be found in the Appendix . Theorem 3 Let Z = [ z1 , ... , zN ] T be a sample from an arbitrary N -variate Categorical distribution with identical marginal and bivariate distributions , such that zi ∼ Cat ( σ ( ϕ ) ) , and Rij = σ ( ϕ ) iσ ( ϕ ) j/P ( zn = ı̂ , zm = ȷ̂ ) . An unbiased estimator of ∇ϕE [ f ( z ) ] is : gCARMS ( Z ) = 1 N ( N−1 ) ∑ n ̸=m 1 2 ( f ( zn ) − f ( zm ) ) ( zn − zm ) zTnRz′m Lemma 4 Let f ( Z ) = [ f ( z1 ) , ... , f ( zN ) ] T , ◦ denote the Hadamard product , 1N×N a matrix of ones , and IN the identity matrix . Define O = 1N−1 ( 1NxN − IN ) ◦ ( ZRZ T ) , and D = diag ( O1N ) , to be a diagonal matrix . The CARMS estimator can equivalently be written in the following form : gCARMS ( Z ) = 1 N f ( Z ) T ( D −O ) ( Z − 1Nσ ( ϕ ) T ) . ( 5 ) Furthermore , for independent samples , ZRZT = 1N×N and LOORF has the form : gLOORF ( Z ) = 1 N f ( Z ) T ( IN×N − 1N−1 ( 1NxN − IN ) ) ( Z − 1Nσ ( ϕ ) T ) ( 6 )

Summary: This paper tackles the problem of differentiating expectations of discrete random variables. Specifically, this paper proposes a generalization of the recent ARMS approach to more than two categories. The CARMS gradient estimator uses multiple samples from a categorical distribution. Unlike e.g. LOORF, CARMS uses samples that are dependent (typically negatively correlated), hoping to further reduce estimator variance. The authors prove how to construct an unbiased gradient from these dependent samples using importance sampling. The authors propose several joint distributions (copulas) to actually generate such dependent samples. Experimental results are provided on several problems such as a categorical latent VAE.