Sparse Communication via Mixed Distributions

1 INTRODUCTION . Historically , discrete and continuous domains have been considered separately in machine learning , information theory , and engineering applications : random variables ( r.v . ) and information sources are chosen to be either discrete or continuous , but not both ( Shannon , 1948 ) . In signal processing , one needs to opt between discrete ( digital ) and continuous ( analog ) communication , whereas analog signals can be converted into digital ones by means of sampling and quantization . Discrete latent variable models are appealing to facilitate learning with less supervision , to leverage prior knowledge , and to build more compact and interpretable models . However , training such models is challenging due to the need to evaluate a large or combinatorial expectation . Existing strategies include the score function estimator ( Williams , 1992 ; Mnih & Gregor , 2014 ) , pathwise gradients ( Kingma & Welling , 2014 ) combined with a continuous relaxation of the latent variables ( such as the Concrete distribution , Maddison et al . ( 2017 ) ; Jang et al . ( 2017 ) ) , and sparse parametrizations ( Correia et al. , 2020 ) . Pathwise gradients , in particular , require continuous approximations of quantities that are inherently discrete , sometimes requiring proxy gradients ( Jang et al. , 2017 ; Maddison et al. , 2017 ) , sometimes giving the r.v . different treatment in different terms of the same objective ( Jang et al. , 2017 ) , sometimes creating a discrete-continuous hybrid ( Louizos et al. , 2018 ) . Since discrete variables and their continuous relaxations are so prevalent , they deserve a rigorous mathematical study . Throughout , we will use the name mixed variable to denote a hybrid variable that takes on both discrete and continuous outcomes . This work takes a first step into a rigorous study of mixed variables and their properties . We will call communication through mixed variables sparse communication : its goal is to retain the advantages of differentiable computation but still be able to represent and approximate discrete symbols . Our main contributions are : • We provide a direct sum measure as an alternative to the Lebesgue and counting measures used for continuous and discrete variables , respectively ( Halmos , 2013 ) . The direct sum measure hinges on a face lattice stratification of polytopes , including the probability simplex , avoiding the need for Dirac densities when expressing densities with point masses in the boundary of the simplex ( §3 ) . • We use the direct sum measure to formally defineK th-dimensional mixed random variables . We provide extrinsic ( “ sample-and-project ” ) and intrinsic ( based on face stratification ) characterizations of these variables , leading to several new distributions : the K-D Hard Concrete , the GaussianSparsemax , and the Mixed Dirichlet ( summarized in Table 1 ) . See Figure 1 for an illustration . • We propose a new direct sum entropy and Kullback-Leibler divergence , which decompose as a sum of discrete and continuous ( differential ) entropies/divergences . We provide an interpretation in terms of optimal code length , and we derive an expression for the maximum entropy ( §4 ) . • We illustrate the usefulness of our framework by learning mixed latent variable models in an emergent communication task and with VAEs to model Fashion-MNIST and MNIST data ( §5 ) . 2 BACKGROUND . We assume throughout an alphabet with K ≥ 2 symbols , denoted [ K ] = { 1 , . . . , K } . Symbols can be encoded as one-hot vectors ek . RK denotes the K-dimensional Euclidean space , RK > 0 its strictly positive orthant , and4K−1 ⊆ RK the probability simplex,4K−1 : = { y ∈ RK | y ≥ 0 , 1 > y = 1 } , with vertices { e1 , . . . , eK } . Each y ∈ 4K−1 can be seen as a vector of probabilities for the K symbols , parametrizing a categorical distribution over [ K ] . The support of y ∈ 4K−1 is the set of nonzero-probability symbols supp ( y ) : = { k ∈ [ K ] | yk > 0 } . The set of full-support categoricals corresponds to the relative interior of the simplex , ri ( 4K−1 ) : = { y ∈ 4K−1 | supp ( y ) = [ K ] } . 2.1 TRANSFORMATIONS FROM RK TO 4K−1 . In many situations , there is a need to convert a vector of real numbers z ∈ RK ( scores for the several symbols , often called logits ) into a probability vector y ∈ 4K−1 . The most common choice is the softmax transformation ( Bridle , 1990 ) : y = softmax ( z ) ∝ exp ( z ) . Since the exponential function is strictly positive , softmax reaches only the relative interior ri ( 4K−1 ) , that is , it never returns a sparse probability vector . To encourage more peaked distributions ( but never sparse ) it is common to use a temperature parameter β > 0 , by defining softmaxβ ( z ) : = softmax ( β−1z ) . The limit case β → 0+ corresponds to the indicator vector for the argmax , which returns a one-hot distribution indicating the symbol with the largest score . While the softmax transformation is differentiable ( hence permitting end-to-end training with the gradient backpropagation algorithm ) , the argmax function has zero gradients almost everywhere . With small temperatures , numerical issues are common . A direct sparse probability mapping is sparsemax ( Martins & Astudillo , 2016 ) , the Euclidean projection onto the simplex : sparsemax ( z ) : = arg miny∈4K−1 ‖y−z‖ . Unlike softmax , sparsemax reaches the full simplex4K−1 , including the boundary , often returning a sparse vector y , without sacrificing differentiability almost everywhere . With K = 2 and parametrizing z = ( z , 1 − z ) , sparsemax becomes a “ hard sigmoid , ” [ sparsemax ( ( z , 1 − z ) ) ] 1 = max { 0 , min { 1 , z } } . We will come back to this point in §3.3 . Other sparse transformations include α-entmax ( Peters et al. , 2019 ; Blondel et al. , 2020 ) , top-k softmax ( Fan et al. , 2018 ; Radford et al. , 2019 ) , and others ( Laha et al. , 2018 ; Sensoy et al. , 2018 ; Kong et al. , 2020 ; Itkina et al. , 2020 ) . 2.2 DENSITIES OVER THE SIMPLEX . Let us now switch from deterministic to stochastic maps . Denote by Y a r.v . taking on values in the simplex4K−1 with probability density function pY ( y ) . The density of a Dirichlet r.v . Y ∼ Dir ( α ) , with α ∈ RK > 0 is pY ( y ; α ) ∝ ∏K k=1 y αk−1 k . Sampling from a Dirichlet produces a point in ri ( 4K−1 ) , and , although a Dirichlet can assign high density to y close to the boundary of the simplex when α < 1 , a Dirichlet sample can never be sparse . A Logistic-Normal r.v . ( Atchison & Shen , 1980 ) , also known as Gaussian-Softmax by analogy to other distributions to be presented , is given by the softmax-projection of a multivariate Gaussian r.v . with mean z and covariance Σ : Y = softmax ( z + Σ 1 2N ) with Nk ∼ N ( 0 , 1 ) . Since the softmax is strictly positive , the Logistic-Normal places no probability mass to points in the boundary of4K−1 . A Concrete ( Maddison et al. , 2017 ) , or Gumbel-Softmax ( Jang et al. , 2017 ) , r.v . is given by the softmax-projection of K independent Gumbel r.vs. , each with mean zk : Y = softmaxβ ( z+G ) with Gk ∼ Gumbel ( 0 , 1 ) . Like in the previous cases , a Concrete draw is a point in ri ( 4K−1 ) . When the temperature β approaches zero , the softmax approaches the indicator for argmax and Y becomes closer to a categorical r.v . ( Luce , 1959 ; Papandreou & Yuille , 2011 ) . Thus , a Concrete r.v . can be seen as a continuous relaxation of a categorical . 2.3 TRUNCATED UNIVARIATE DENSITIES . Binary Hard Concrete . For K = 2 , a point in the simplex can be represented as y = ( y , 1− y ) and the simplex is isomorphic to the unit interval,41 ' [ 0 , 1 ] . For this binary case , Louizos et al . ( 2018 ) proposed a Hard Concrete distribution which stretches the Concrete and applies a hard sigmoid transformation ( which equals the sparsemax with K = 2 , per §2.1 ) as a way of placing point masses at 0 and 1 . These “ stretch-and-rectify ” techniques enable assigning probability mass to the boundary of41 and are similar in spirit to the spike-and-slab feature selection method ( Mitchell & Beauchamp , 1988 ; Ishwaran et al. , 2005 ) and for sparse codes in variational auto-encoders ( Rolfe , 2017 ; Vahdat et al. , 2018 ) . We propose in §3.3 a more general extension to K ≥ 2 . Rectified Gaussian . Rectification can be applied to other continuous distributions . A simple choice is the Gaussian distribution , to which one-sided ( Hinton & Ghahramani , 1997 ) and two-sided rectifications ( Palmer et al. , 2017 ) have been proposed . Two-sided rectification yields a mixed r.v . in [ 0 , 1 ] . Writing y = ( y , 1− y ) and z = ( z , 1− z ) , this distribution has the following density : pY ( y ) = N ( y ; z , σ2 ) + 1− erf ( z/ ( √ 2σ ) ) 2 δ0 ( y ) + 1 + erf ( ( z − 1 ) / ( √ 2σ ) ) 2 δ1 ( y ) , ( 1 ) where δs ( y ) is a Dirac delta density . Extending such distributions to the multivariate case is nontrivial . For K > 2 , a density expression with Diracs would be cumbersome , since it would require a combinatorial number of Diracs of several “ orders , ” depending on whether they are placed at a vertex , edge , face , etc . Another annoyance is that Dirac deltas have −∞ differential entropy , which prevents information-theoretic treatment . The next section shows how we can obtain densities that assign mass to the full simplex while avoiding Diracs , by making use of the face lattice and a new base measure . 3 FACE STRATIFICATION AND MIXED RANDOM VARIABLES . 3.1 THE FACE LATTICE . Let P be a convex polytope whose vertices are bit vectors ( i.e. , elements of { 0 , 1 } K ) . Examples are the probability simplex 4K−1 , the hypercube [ 0 , 1 ] K , and marginal polytopes of structured variables ( Wainwright & Jordan , 2008 ) . The combinatorial structure of a polytope is determined by its face lattice ( Ziegler , 1995 , §2.2 ) , which we now describe . A face of P is any intersection of P with a closed halfspace such that none of the interior points of P lie on the boundary of the halfspace ; we denote by F ( P ) the set of all faces of P and by F̄ ( P ) : = F ( P ) \ { ∅ } the set of proper faces . We denote by dim ( f ) the dimension of a face f ∈ F̄ ( P ) . Thus , the vertices of P are 0-dimensional faces , and P itself is a face of the same dimension as P , called the “ maximal face ” . Any other face of P can be regarded as a lower-dimensional polytope . The set F ( P ) has a partial order induced by set inclusion , that is , it is a partially ordered set ( poset ) , and more specifically a lattice . The full polytope P can be decomposed uniquely as the disjoint union of the relative interior of its faces , which we call face stratification : P = ⊔ f∈F̄ ( P ) ri ( f ) . For example , the simplex 42 is composed of its face ri ( 42 ) ( i.e. , excluding the boundary ) , three edges ( excluding the vertices in the corners ) , and three vertices ( the corners ) . This is represented schematically in Figure 2 . Likewise , the square [ 0 , 1 ] 2 is composed of its maximal face ( 0 , 1 ) 2 , four edges ( excluding the corners ) and four vertices ( the corners ) . The partition above implies that any subset A ⊆ P can be represented as a tuple A = ( Af ) f∈F̄ ( P ) , where Af = A ∩ ri ( f ) ; and the sets Af are all disjoint .

Summary: This paper presents multidimensional extensions for mixed random variables originating from discrete-continuous hybrids based on truncation and rectification, which have been proposed for univariate distributions. The proposed extension replaces truncation by sparse projections to the simplex. The authors also propose a direct sum base measure definition on the face lattice of the probability simplex and intrinsic sampling strategies motivated by “manifold stratification”. Based on these introductions, new entropy and Kullback-Leibler divergence functions that subsume the discrete and differential cases and have interpretations in terms of code optimality, are presented.