Sparse Communication via Mixed Distributions

1 INTRODUCTION . Historically , discrete and continuous domains have been considered separately in machine learning , information theory , and engineering applications : random variables ( r.v . ) and information sources are chosen to be either discrete or continuous , but not both ( Shannon , 1948 ) . In signal processing , one needs to opt between discrete ( digital ) and continuous ( analog ) communication , whereas analog signals can be converted into digital ones by means of sampling and quantization . Discrete latent variable models are appealing to facilitate learning with less supervision , to leverage prior knowledge , and to build more compact and interpretable models . However , training such models is challenging due to the need to evaluate a large or combinatorial expectation . Existing strategies include the score function estimator ( Williams , 1992 ; Mnih & Gregor , 2014 ) , pathwise gradients ( Kingma & Welling , 2014 ) combined with a continuous relaxation of the latent variables ( such as the Concrete distribution , Maddison et al . ( 2017 ) ; Jang et al . ( 2017 ) ) , and sparse parametrizations ( Correia et al. , 2020 ) . Pathwise gradients , in particular , require continuous approximations of quantities that are inherently discrete , sometimes requiring proxy gradients ( Jang et al. , 2017 ; Maddison et al. , 2017 ) , sometimes giving the r.v . different treatment in different terms of the same objective ( Jang et al. , 2017 ) , sometimes creating a discrete-continuous hybrid ( Louizos et al. , 2018 ) . Since discrete variables and their continuous relaxations are so prevalent , they deserve a rigorous mathematical study . Throughout , we will use the name mixed variable to denote a hybrid variable that takes on both discrete and continuous outcomes . This work takes a first step into a rigorous study of mixed variables and their properties . We will call communication through mixed variables sparse communication : its goal is to retain the advantages of differentiable computation but still be able to represent and approximate discrete symbols . Our main contributions are : • We provide a direct sum measure as an alternative to the Lebesgue and counting measures used for continuous and discrete variables , respectively ( Halmos , 2013 ) . The direct sum measure hinges on a face lattice stratification of polytopes , including the probability simplex , avoiding the need for Dirac densities when expressing densities with point masses in the boundary of the simplex ( §3 ) . • We use the direct sum measure to formally defineK th-dimensional mixed random variables . We provide extrinsic ( “ sample-and-project ” ) and intrinsic ( based on face stratification ) characterizations of these variables , leading to several new distributions : the K-D Hard Concrete , the GaussianSparsemax , and the Mixed Dirichlet ( summarized in Table 1 ) . See Figure 1 for an illustration . • We propose a new direct sum entropy and Kullback-Leibler divergence , which decompose as a sum of discrete and continuous ( differential ) entropies/divergences . We provide an interpretation in terms of optimal code length , and we derive an expression for the maximum entropy ( §4 ) . • We illustrate the usefulness of our framework by learning mixed latent variable models in an emergent communication task and with VAEs to model Fashion-MNIST and MNIST data ( §5 ) . 2 BACKGROUND . We assume throughout an alphabet with K ≥ 2 symbols , denoted [ K ] = { 1 , . . . , K } . Symbols can be encoded as one-hot vectors ek . RK denotes the K-dimensional Euclidean space , RK > 0 its strictly positive orthant , and4K−1 ⊆ RK the probability simplex,4K−1 : = { y ∈ RK | y ≥ 0 , 1 > y = 1 } , with vertices { e1 , . . . , eK } . Each y ∈ 4K−1 can be seen as a vector of probabilities for the K symbols , parametrizing a categorical distribution over [ K ] . The support of y ∈ 4K−1 is the set of nonzero-probability symbols supp ( y ) : = { k ∈ [ K ] | yk > 0 } . The set of full-support categoricals corresponds to the relative interior of the simplex , ri ( 4K−1 ) : = { y ∈ 4K−1 | supp ( y ) = [ K ] } . 2.1 TRANSFORMATIONS FROM RK TO 4K−1 . In many situations , there is a need to convert a vector of real numbers z ∈ RK ( scores for the several symbols , often called logits ) into a probability vector y ∈ 4K−1 . The most common choice is the softmax transformation ( Bridle , 1990 ) : y = softmax ( z ) ∝ exp ( z ) . Since the exponential function is strictly positive , softmax reaches only the relative interior ri ( 4K−1 ) , that is , it never returns a sparse probability vector . To encourage more peaked distributions ( but never sparse ) it is common to use a temperature parameter β > 0 , by defining softmaxβ ( z ) : = softmax ( β−1z ) . The limit case β → 0+ corresponds to the indicator vector for the argmax , which returns a one-hot distribution indicating the symbol with the largest score . While the softmax transformation is differentiable ( hence permitting end-to-end training with the gradient backpropagation algorithm ) , the argmax function has zero gradients almost everywhere . With small temperatures , numerical issues are common . A direct sparse probability mapping is sparsemax ( Martins & Astudillo , 2016 ) , the Euclidean projection onto the simplex : sparsemax ( z ) : = arg miny∈4K−1 ‖y−z‖ . Unlike softmax , sparsemax reaches the full simplex4K−1 , including the boundary , often returning a sparse vector y , without sacrificing differentiability almost everywhere . With K = 2 and parametrizing z = ( z , 1 − z ) , sparsemax becomes a “ hard sigmoid , ” [ sparsemax ( ( z , 1 − z ) ) ] 1 = max { 0 , min { 1 , z } } . We will come back to this point in §3.3 . Other sparse transformations include α-entmax ( Peters et al. , 2019 ; Blondel et al. , 2020 ) , top-k softmax ( Fan et al. , 2018 ; Radford et al. , 2019 ) , and others ( Laha et al. , 2018 ; Sensoy et al. , 2018 ; Kong et al. , 2020 ; Itkina et al. , 2020 ) . 2.2 DENSITIES OVER THE SIMPLEX . Let us now switch from deterministic to stochastic maps . Denote by Y a r.v . taking on values in the simplex4K−1 with probability density function pY ( y ) . The density of a Dirichlet r.v . Y ∼ Dir ( α ) , with α ∈ RK > 0 is pY ( y ; α ) ∝ ∏K k=1 y αk−1 k . Sampling from a Dirichlet produces a point in ri ( 4K−1 ) , and , although a Dirichlet can assign high density to y close to the boundary of the simplex when α < 1 , a Dirichlet sample can never be sparse . A Logistic-Normal r.v . ( Atchison & Shen , 1980 ) , also known as Gaussian-Softmax by analogy to other distributions to be presented , is given by the softmax-projection of a multivariate Gaussian r.v . with mean z and covariance Σ : Y = softmax ( z + Σ 1 2N ) with Nk ∼ N ( 0 , 1 ) . Since the softmax is strictly positive , the Logistic-Normal places no probability mass to points in the boundary of4K−1 . A Concrete ( Maddison et al. , 2017 ) , or Gumbel-Softmax ( Jang et al. , 2017 ) , r.v . is given by the softmax-projection of K independent Gumbel r.vs. , each with mean zk : Y = softmaxβ ( z+G ) with Gk ∼ Gumbel ( 0 , 1 ) . Like in the previous cases , a Concrete draw is a point in ri ( 4K−1 ) . When the temperature β approaches zero , the softmax approaches the indicator for argmax and Y becomes closer to a categorical r.v . ( Luce , 1959 ; Papandreou & Yuille , 2011 ) . Thus , a Concrete r.v . can be seen as a continuous relaxation of a categorical . 2.3 TRUNCATED UNIVARIATE DENSITIES . Binary Hard Concrete . For K = 2 , a point in the simplex can be represented as y = ( y , 1− y ) and the simplex is isomorphic to the unit interval,41 ' [ 0 , 1 ] . For this binary case , Louizos et al . ( 2018 ) proposed a Hard Concrete distribution which stretches the Concrete and applies a hard sigmoid transformation ( which equals the sparsemax with K = 2 , per §2.1 ) as a way of placing point masses at 0 and 1 . These “ stretch-and-rectify ” techniques enable assigning probability mass to the boundary of41 and are similar in spirit to the spike-and-slab feature selection method ( Mitchell & Beauchamp , 1988 ; Ishwaran et al. , 2005 ) and for sparse codes in variational auto-encoders ( Rolfe , 2017 ; Vahdat et al. , 2018 ) . We propose in §3.3 a more general extension to K ≥ 2 . Rectified Gaussian . Rectification can be applied to other continuous distributions . A simple choice is the Gaussian distribution , to which one-sided ( Hinton & Ghahramani , 1997 ) and two-sided rectifications ( Palmer et al. , 2017 ) have been proposed . Two-sided rectification yields a mixed r.v . in [ 0 , 1 ] . Writing y = ( y , 1− y ) and z = ( z , 1− z ) , this distribution has the following density : pY ( y ) = N ( y ; z , σ2 ) + 1− erf ( z/ ( √ 2σ ) ) 2 δ0 ( y ) + 1 + erf ( ( z − 1 ) / ( √ 2σ ) ) 2 δ1 ( y ) , ( 1 ) where δs ( y ) is a Dirac delta density . Extending such distributions to the multivariate case is nontrivial . For K > 2 , a density expression with Diracs would be cumbersome , since it would require a combinatorial number of Diracs of several “ orders , ” depending on whether they are placed at a vertex , edge , face , etc . Another annoyance is that Dirac deltas have −∞ differential entropy , which prevents information-theoretic treatment . The next section shows how we can obtain densities that assign mass to the full simplex while avoiding Diracs , by making use of the face lattice and a new base measure . 3 FACE STRATIFICATION AND MIXED RANDOM VARIABLES . 3.1 THE FACE LATTICE . Let P be a convex polytope whose vertices are bit vectors ( i.e. , elements of { 0 , 1 } K ) . Examples are the probability simplex 4K−1 , the hypercube [ 0 , 1 ] K , and marginal polytopes of structured variables ( Wainwright & Jordan , 2008 ) . The combinatorial structure of a polytope is determined by its face lattice ( Ziegler , 1995 , §2.2 ) , which we now describe . A face of P is any intersection of P with a closed halfspace such that none of the interior points of P lie on the boundary of the halfspace ; we denote by F ( P ) the set of all faces of P and by F̄ ( P ) : = F ( P ) \ { ∅ } the set of proper faces . We denote by dim ( f ) the dimension of a face f ∈ F̄ ( P ) . Thus , the vertices of P are 0-dimensional faces , and P itself is a face of the same dimension as P , called the “ maximal face ” . Any other face of P can be regarded as a lower-dimensional polytope . The set F ( P ) has a partial order induced by set inclusion , that is , it is a partially ordered set ( poset ) , and more specifically a lattice . The full polytope P can be decomposed uniquely as the disjoint union of the relative interior of its faces , which we call face stratification : P = ⊔ f∈F̄ ( P ) ri ( f ) . For example , the simplex 42 is composed of its face ri ( 42 ) ( i.e. , excluding the boundary ) , three edges ( excluding the vertices in the corners ) , and three vertices ( the corners ) . This is represented schematically in Figure 2 . Likewise , the square [ 0 , 1 ] 2 is composed of its maximal face ( 0 , 1 ) 2 , four edges ( excluding the corners ) and four vertices ( the corners ) . The partition above implies that any subset A ⊆ P can be represented as a tuple A = ( Af ) f∈F̄ ( P ) , where Af = A ∩ ri ( f ) ; and the sets Af are all disjoint .

Summary: In this paper, the authors build rigorous theoretical foundations for mixed random variables. They first define a natural measure for mixed random variables, which looks like a direct sum. Then they define the entropy and KL divergence based on the proposed measure. They also give two strategies for representing and sampling mixed random variables. At last, they conduct some experiments to illustrate the usefulness of their framework.