Sparse Communication via Mixed Distributions
Authors: António Farinhas
Wilker Aziz
Vlad Niculae
Andre Martins
Venue: ICLR 2022 Oral (2022)

Abstract: Neural networks and other machine learning models compute continuous representations, while humans communicate mostly through discrete symbols. Reconciling these two forms of communication is desirable for generating human-readable interpretations or learning discrete latent variable models, while maintaining end-to-end differentiability. Some existing approaches (such as the Gumbel-Softmax transformation) build continuous relaxations that are discrete approximations in the zero-temperature limit, while others (such as sparsemax transformations and the Hard Concrete distribution) produce discrete/continuous hybrids. In this paper, we build rigorous theoretical foundations for these hybrids, which we call "mixed random variables.'' Our starting point is a new "direct sum'' base measure defined on the face lattice of the probability simplex. From this measure, we introduce new entropy and Kullback-Leibler divergence functions that subsume the discrete and differential cases and have interpretations in terms of code optimality. Our framework suggests two strategies for representing and sampling mixed random variables, an extrinsic ("sample-and-project'’) and an intrinsic one (based on face stratification). We experiment with both approaches on an  emergent communication benchmark and on modeling MNIST and Fashion-MNIST data with variational auto-encoders with mixed latent variables.

emergent communica- Published as a conference paper at ICLR 2022
SPARSE COMMUNICATION VIA MIXED DISTRIBUTIONS
António Farinhas Wilker Aziz 2, Vlad Niculae " André F. T. Martins 1,4 I Instituto de Telecomunicações, Instituto Superior Técnico (Lishon ELLIS Unit), ILLL, University of Amsterdam, VIvI University of Amsterdam, UUnabeel (antonic. fari andre. www.ttt.atti..... tecnico..... . .uw. ulisboa.pt, (H.azi2, v.alculae @uva.n1
ABSTRACT
Neural networks and other machine learning models compute continuous represen- tations, while humans communicate mostly through discrete symbols. Reconciling these two forms of communication is desirable for generating human reaablee interpretations or learning discrete latent variable models, while maintaining end- to-end differentiability. Some existing approaches (such the Sumbel-SSoftmax transformation) build continuous relaxations that are discrete approximations in the eero-temperature limit, while others (such as sparsemax transformations and the Hard Concrete distribution) produce discrete/continuous hybrids. In this paper, we build rigorous theoretical foundations for these hybrids, which we call "mixed random variables." Our starting point is new "direct sum" base measure defined on the face lattice of the probability simplex. From this measure, we introduce new entropy and Kullback-Leible divergence functions that subsume the discrete and differential cases and have interpretations in terms of code optimality. Our framework suggests two strategies for representing and sampling mixed random variables, an extrinsic ("sample-anc and-project") and an intrinsic one (based on face stratification). We experiment with both approaches tion benchmark and on modeling MNIST and Fashion-MNIST data with variational auto-encoders with mixed latent variables. Our code is publicly available
INTRODUCTION
Historically, discrete and continuous domains have been considered separately in machine learning, information theory, and engineering applications: random variables (I.v.) and information sources are chosen to be either discrete or continuous, but not both (Shannon 1948). In signal processing, one needs to opt between discrete (digital) and continuous (analog) communication, whereas analog signals can be converted into digital ones by means of sampling and quantization.
Discrete latent variable models are appealing to facilitate learning with less supervision, to leverage prior knowledge, and to build more compact and interpretable models. However, training such models challenging due the need evaluate a large or combinatorial expectation. Existing strategies
include the score function estimator (Williams 1992, Mnih & Gregor 2014). pathwise gradients (Kingma & Welling 2014) combined with a continuous relaxation of the latent variables (such as the
Concrete distribution, Maddison et al. :ang et al. (2017)), and sparse parametrizations (Correia) al. 2020). Pathwise gradients, in particular, require continuous approximations of quantities that are inherently discrete, sometimes requiring proxy gradients (Jang et al.) [017]] Maddison et al.) [017)] sometimes giving the r.v. different treatment different terms the same objective (Jang et al. 2017), sometimes creating a discrete-continuous hybrid (Louizos et al. 2018).
Since discrete variables and their continuous relaxations are so prevalent, they deserve a rigorous mathematical study. Throughout, we will use the name mixed variable to denote hybrid variable that takes on both discrete and continuous outcomes. This work takes a first step into a rigorous study of mixed variables and their properties. We will call communication through mixed variables sparse communication: its goal is to retain the advantages of differentiable computation but still be able to represent and approximate discrete symbols. Our main contributions are:
We provide a direct sum measure as an alternative to the Lebesgue and counting measures used for continuous and discrete variables, respectively (Halmos) 2013). The direct sum measure hinges

---

Published as a conference paper at ICLR 2022
Table 1: Discrete, continuous, and mixed distributions considered in this work, all involving the probability simplex AK-1. For each distribution we indicate if it assigns probability mass to all faces of the simplex or only some, if is multivariate (K 2 2), and, for mixed distributions, if characterized extrinsically (sample-and-prog -rojecc) or intrinsically (based on face stratification).
Distribution
All faces?
Multivariate?
Intrinsic?
Categorical
Discrete x
Yes v
Dirichlet, Logistic-Normal, Concrete
Continuous x
Yes /
Hard Concrete, Rectified Gaussian
Mixed V
No X
Extrinsic X
K-D Hard Concrcte, Gaussian- Spaasssaap (this paper) Mixed Dirichlet (this paper)
Mixed /
Yes /
Extrinsic X
Mixed /
Yes /
Intrinsic /
on a face lattice stratification of polytopes, including the probability simplex, avoiding the need for Dirac densities when expressing densities with point masses in the boundary of the simplex ($
We use the direct sum measure to formally define - mixed random variables. We provide extrinsic ('sample-and-pep projjetccct"" and intrinsic (based face stratification) characterizations of these variables, leading to several new distributions: the K-D Hard Concrete, the Gaussian- Sparsemax, and the Mixed Dirichlet (summarized in Table See Figure for an illustration.
We propose new direct sum entropy and Kullback-Leein sum of discrete and continuous (differential) entropies/divergence terms of optimal code length, and we derive an expression for the maximum entropy ($4).
divergence, which decompose as We provide an interpretation in
We illustrate the usefulness of our framework by learning mixed latent variable models in an emergent communication task and with VAEs model Fashion-MNIST and MNIST data ($5).
BACKGROUND
.1......K) KJ. Symbols can be encoded as one-hot vectors eg. RK denotes the K K-dimensional Euclidean space, R its strictly positive orthant, and AK-1 K the probability simplex, Ax- : {y E TR y 2 0, 1 1 1}. with vertices (ei,..., ex Each y E A can be seen as a vector of probabilities for the K symbols, parametrizing a categorical distribution over [K. The support of y E A is the set of ponzero-probability symbols supp(y) := {k E [K] V/ 0}. The set of full-support categoricals corresponds to the relative interior of the simplex, ri (-(Ak-1) == {y E AK-1 supp(y) = [K]]-
We assume throughout an alphabet with K 2 2 symbols, denoted [K] = f1.....
2.1 TRANSFORMATIONS FROM K TO AK-
In many situations, there a need to convert vector of real numbers z E B (scores for the several symbols, often called logits) into a probability vector y AK-1- The most common choice the softmax transformation (Bridle 1990): y softmax( с exp(z) Since the exponential function strictly positive, softmax reaches only the relative interior ri [ k-1,, that is, it never returns sparse probability vector. To encourage more peaked distributions (but never sparse) it is common to use temperature parameter > 0, by defining softmax;(2) := softmax (f-maxx((112) 2). The limit case 3 + 0+, corresponds to the indicator vector for the argmax, which returns one-hot distribution indicating the symbol with the largest score. While the softmax transformation differentiable (hence permitting end-to-end training with the gradient backpropagation algorithm). the argmax function has zero gradients almost everywhere. With small temperatures, numerical issues are common.
A direct sparse probability mapping is sparsemax Marrinss & Astudillo, 2016), the Euclidean projection onto the simplex: sparsemax(z) := arg m - - Unlike softmax, sparsemax reaches the full simplex A x-1 including the boundary, often returning sparse vector y, without sacrificing differeetiabillity almost everywhere. With K 2 and parametrizing z = (2, 2).
sparsemax becomes a "hard sigmoid," (sparsemax (2, - =)) = max(0, min 2))- We will come back this point in $3.3 Other sparse transformations include a-entmax (Peters al. 2019 Blondel et al. (2020). top-k softmax (Fan et al. (2018: Radford et al.) 2019). and others (Laha et al. 2018 Sensoy et al. 2018 Kong et al. 2020 Itkina et al. 2020).
2

---

Published as a conference paper at ICLR 2022
(0,0.
03
0.
0.6
0.5
688
1.0
11, 0,
10 01
0.
10.
Figure I: Multivariate distributions over -I
1. Standard distributions, like the Logistic-Normal (left), assign zero probability all faces but ri) Our mixed distributions support assigning probability to the full simplex, including its boundary: the Gaussian Gaussian-Sparsemaaa Sparsemax (right) induces
distribution over the 1-dimensional edges (shown as a histogram), and assigns Pr[ 0, 0)} .022.
2.2 DENSITIES OVER THE SIMPLEX
Let us now switch from deterministic stochastic maps. Denote by Y a n.v. taking on values in the simplex A with probability density function (t).
The density of Dirichlet r.v. Y ~ Dir(a.), with RK is py (y: a.) I 0% p Sampling from a Dirichlet produces a point in ri(Ax-1), and, although a Dirichlet can assign high density to y close to the boundary of the simplex when or 1, Dirichlet sample
never be sparse
Logistic-Normal r.v. (Atchison & Shen 1980). also known as Gaussian-Softmar by analogy to other distributions to presented, is given by the softmax-projjp cetoax--roooection of a multivariate Gaussian r.v. with mean and covariance Σ: softmax(2 + Σ N) with N₂. NO00 1). Since the softmax is strictly positive, the Logistic- Normal places no probability mass to points in the boundary of A
Concrete (Maddison al.) 2017). or Gumbel-Sootmax (Jang et al., 2017). n.v. is given by the softmax- softmax-projection of K independent Gumbel r.vs., each with mean 2
softmax (z + G') with G3 Gumbel(O, 1). Like in the previous cases, a Concrete draw is a point in ri(Ax-1). When the temperature 3 approaches zero, the softmax approaches the indicator for argmax and Y becomes closer to a categorical r.v. (Luce 1959 Papandreeu & Yuille 2011). Thus, Concrete r.v. can be seen as continuous relaxation of categorical.
2.3 TRUNCATED UNIVARIATE DENSITIES
Binary Hard Concrete. For K = 2, a point in the simplex can be represented as y
(v.
and the simplex is isomorphic to the unit interval, A₁ ~ [0. 1]. For this binary case, Louizos et al. (2018) proposed a Hard Concrete distribution which stretches the Concrete and applies hard sigmoid transformation (which equals the sparsemax with K = 2. per 1 as way of placing point masses at and 1. These "stretch-and-rectif" techniques enable assigning probability mass to the boundary of A and are similar spirit to the spike-and-slas feature selection method (Mitchell & Beauchamp 1988 Ishwaran et al., (005) and for sparse codes in variational auto-encoders (Rolfe 2017 Vahdat
et al. 2018). We propose in 33.3 a more general extension K 2 2.
Rectification can be applied to other continuous distributions. A simple choice the Gaussian distribution, to which one-sided (Hinton & Ghahramani 1997) and two-sided rectifications (Palmer et al. 2017) have been proposed. Two-sided rectification yields a mixed r.v. in [0, 1]. Writing (y, y) and z (2, 2)), this distribution has the following density:
Rectified Gaussian.
1 - crf(=/(V2o)) 80(v) + crferr((( - 10)(11))))) (1()),
py (v) NNy 2, 0 +
(1)
2
where dv()) is a Dirac delta density. Extending such distributions to the multivariate case is non- trivial. For K > 2, a density expression with Diracs would be cumbersome, since it would require combinatorial number Diracs of several "orders." depending on whether they are placed at a vertex, edge, face, etc. Another annoyance is that Dirac deltas have -oo differential entropy. which prevents informatiinn-hhoreee treatment. The next section shows how we can obtain densities that assign mass to the full simplex while avoiding Diraes, by making use of the face lattice and base measure.
3

---

Published as a conference paper at ICLR 2022
3 FACE STRATIFICATION AND MIXED RANDOM VARIABLES
3.1 THE FACE LATTICE
Let P be a convex polytope whose vertices are bit vectors (i.e., elements of {0, 1) Examples
the probability simplex K- the hypereube [0, 1]K and marginal polytopes of structured variables (Wainwright & Jordan 2008). The combinatorial structure of polytope is determined by its face lattice (Ziegler] 1995 $2.2), which we now describe. face of P is any intersection of P with a closed halispace such that none of the interior points of P lie on the boundary of the halfspace; we denote by F(P) the set of all faces of P and by F(P) == F(P) {Ø) the set of proper faces. We denote by dim(f) the dimension of face E F(P). Thus, the vertices of P are O-dimensional faces, and P itself face". Any other face of P can be regarded partial order induced by set inclusion, that is, it is a partially ordered set (poset), and more specifically lattice. The full polytope P can be decomposed uniquely as the disjoint union of the relative interior of its faces, which we call face stratification: p = UI ri(f). For example, the simplex A₂ is composed of its face ri(A2) (i.e., excluding the boundary), three edges (excluding the vertices in the corners), and three vertices (the corners). This is represented schematically in Figure 2 Likewise, the square [0, 1J is composed of its maximal face (0, 1) four edges (excluding the corners) and four vertices (the corners). The partition above implies that any subset A S P can be represented as tuple A (EFP) where A,
face of the same dimension as P. called the "maximal lower-dimensional polytope. The set F(P) has
A n nri(f): and the sets A are all disjoint.
(3)
2,3)
Simplex and hypereube. P the sim- plex A- each face corresponds to an in- dex set I C [K] i.e., it can be expressed as fz = {p AK-1 supp(p) C Z), with dimension dim(fz) = [Z] 1: the set of distributions assigning zero probability mass outside I. The set F(Ax-1) has 2 elements. Since A [0, the hyper- cube [0, 1] can be regarded as a product of K binary probability simplices. It has 3 nonempty faces for each dimension we choose between {0], {1}. and [0. 1]. We ex- periment with P E (Ax-1. [0, 1]kh in 5
(1.2)
(2.3)
(1.3)
(L.3)
(2.3)
24, a}
(2)
(1)
(1)
(1,2)
(2)
Figure 2: Left: Decomposition of simplex as the disjoint union of the relative interior of its faces. Right: Hasse diagram of the face lattice DAG where points represent faces, and directed path from point to another represents face inclusion.
3.2 MIXED RANDOM VARIABLES
Categorical distributions assign probability only to the vertices of AK- 1. In the opposite extreme, the densities listed in 22. assign probability mass to the maximal face only, that is, Pr(y E fz} J1z Py (y) = 0 for any I 7 [ J. Any proper density (without Diracs) has this limitation, since non-
maximal faces have zero Lebesgue measure in K-1 While for K = 2 it is possible to circumvent this by defining densities that contain Dirac functions (as in 2.3). this becomes cumbersome for K > 2. Fortunately, there is a more elegant construction that does not require generalized functions.
The key to replace the Lebesgue measure by a measure inspired by face stratification.
Definition 1 (Direct sum measure). The direct sum measure on a polytope P is
n (A)= = H((Anri(f))
SEF(P)
where p the dim((f)-dimensoonal Lebesgue measure for dim(f) > and the counting measure for dim(s) = 0.
We show in App. A that μ is a valid measure on P under the product r-algebra of its faces. We can then define probability densities (gy) w.r.t. this base measure and use them to compute probabilities of measurable subsets of P. Such distributions can equivalently be defined as follows: (i) define a probability mass function Pr (f) on F(P), and (ii) for each face f E F(P), define probability
4

---

Published as a conference paper at ICLR 2022
density Pyje(y) f) over ri(f). Random variables with a distribution of this form have discrete part and a continuous part, thus we call them mixed random variables. This is formalized as follows.
Definition 2 (Mixed random variable), A mixed e.v. is a E. Y over a polytope P. including the boundary. Let F be the corresponding discrete over F(P), with Pr(J) Pr(y ri(f)- Since the mapping Y - F is deterministic, have py (v) Pr(f)PyIF( f) ri(f). The probability of a set _ P is given by:
/ py (y)dp
E PF(f) S prirly SEF(P)
Pr(y E A}
Ammict)
(3)
Equation may be regarded as a manifestation of the law of total probability mixing discrete and continuous variables. Using (3), write expectations over mixed r.v.
E. py (Y() = E.p [Bpy (o()) F =]] where g P - R.
(4)
Both discrete and continuous distributions are recovered with our definition: If Pr(J) = 0 for dim(f) > O, we have a categorical distribution, which only assigns probability to the O-faces. In the other extreme, if (P(P) 1, we have a continuous distribution confined to ri(P). That is, mixed random variables include purely discrete and purely continuous r.vs. particular cases. To parametrize distributions of eigh-dimensional mixed r.vs., it is not efficient to consider all degrees of freedom suggested Definition [ since there can be exponentially many faces. Instead, we need to derive parametrizations that exploit the lattice structure of the faces. We next build upon this idea.
3.3 EXTRINSIC AND INTRINSIC CHARACTERIZATION
There are two possible characterizations of mixed random variables: an extrinsic one, where one starts with a distribution over the ambient space (e.g. Rk) and then applies deterministic, non- invertible, transformation that projects it to P; and an intrinsic one, where one specifies a mixture of distributions directly over the faces of P, by specifying Pp and py F for each f E F(P). We next provide constructions for both cases: We extend the Hard Concrete and Rectified Gaussian distributions reviewed in 22.3 which are instances of the extrinsic characterization, to K 2 and we present new Mixed Dirichlet distribution which is an instance of the intrinsic characterization.
K-D Hard Concrete. We define the K-D Hard Concrete as the following generative story:
Y' (0nccete(z,:3) 3), Y sparsemax(XY") with À 2 1.
When K = 2, sparsemax becomes hard sigmoid and recover the binary Hard Concrete ($2.)) For 2 this is a projection of a "stretched" Concrete r.v. onto the simplex the larger X, the higher the tendency of this projection to hit non-maximal face of the simplex and induce sparsity.
Gaussinn-Sarrsmmax. A similar idea (but without any stretching required) can be used to obtain a sparsemax counterpart of the Logistic-Normal in 2.2 which we call "aauusiaannSparsmmaa'
N~ N(0.I. Y sparsemax(= + E1/2N).
Unlike the Logistic-Nou Normal, the Gaussian-S Gaussian-Sparsemas
assign nonzero probability mass to the boundary of the simplex. When 2, we recover the double-sided rectified Gaussian described in $2.3 In that case, using Dirac deltas, the density with respect to the Lebesgue measure in R has the form in [I. With 6g = 1-en((=/([2oo), 01 [+er((s-11)/(VZB) and 0g = 1 - 00 01, the same
2
distribution can be expressed intrinsically via the density p (v) Py (f)))rrr f)
P=((0)) = 0o, Pr((1)) = 0,, PF 1)) = Ocr PrIe (y) F = [0, 1J) NV(gi.ot)/e
For K 2. expressions for PF and pr (i.e., an intrinsic representation) are less direct; we express those distributions as a function of the orthant probability of multivariate Gaussians in App. B
Mixed Dirichlet. We now propose an intrinsic mixed distribution over - the Mixed Dirichlet, whose generative story is as follows. First, a face F = fz is sampled with probability
(fz w) = exp(w (op(w)((f))) log Z(w)), Φx(fz) (-1) (-1)1-(AEZ)
(8)
5

---

Published as a conference paper at ICLR 2022
where E IR is the natural parameter (a.k.a. log-potentials), ((fz) E [-1,1] is the sufficient statistic, and log (ww is the log normalizer. We then parametrize a Dirichlet distribution over the relative interior of fi, that is, Y|F fr (ir( where ax()) E R For compact parametrization. we have a single K- d-dimensional vector a of concentration parameters. one parameter per vertex, and (I) gathers the coordinates of associated with the vertices in f. The normalizer of (8) can be evaluated in time (K) via the forward algorithm (Baum & Eagon, 1967) on a directed path. Similarly, we can draw independent samples by stochastic traversals through this DAG. The graph needed for this construct and the associated algorithms are detailed in App.
acyclic graph (DAG) that encodes each non-empty corner I C [K] of the face lattice
INFORMATION THEORY FOR MIXED RANDOM VARIABLES
Now that we have the theoretical pillars for mixed random variables, we proceed to defining informa- tion theoretic quantities for them: their entropy and Kullback-Leibler divergence.
Direct sum entropy and KL divergence. The entropy of a n.v. X with respect to a measure y is:
H²(X) Px (z)du(x))
where Px (x) is a probability density satisfying Jx Px (x) 1. When X' is finite and μ is the counting measure, the integral becomes a sum and we recover Shannon's discrete entropy. which non-negative and upper bounded by log (2)] When X' C IR* is continuous and μ is the Lebesgue measure, we recover the differential entropy. which can be negative and, for compact X, is upper bounded by the logarithm of the volume of X. When relaxing discrete r.v. to a continuous one in a variational model (e.g., using the Concrete distribution), correct variational lower bounds require switching to differential entropy (Maddison et al.) (017) (although discrete entropy sometimes used (Jang et al.] (017)). This is problematic since the differential entropy is not a limit case of the discrete entropy (Cover & Thomas 2012). Our direct sum entropy, defined below, obviates this. The key idea is to plug in (2) the direct sum measure 2 Since Y - F is deterministic, we have H(Y, F) = (Y). This leads to:
Definition 3 (Direct sum entropy and KL divergence). The direct sum entropy of mixed n.v. Y is
Ha (V) = H(F) + H(Y P))
(10)
Σ PF(l) (-S Mollly/oopy111111
= Σ Pe(f)logPE(T) SEF(P)
(EF(P)
differencial entraprs
discrete estropy
The KL divergence between distributions p = (Pr.PrIF) and o8 = =QF,qr|P) is:
De (r.l1l19))) = DKL((r] Esrry [DKL(PRIE F f)lla1s( F f)]
(11)
Σ (EF(P)
Pr(J) log Qr(f) SET(P)
Pf(f)
Σ ppl)) ( pprieeo log Prir(y f) or rly v
continamus
discette Kl.
As shown in Definition the direct sum entropy and the KL divergence have two components: discrete one over faces and an expectation of a continuous one over each face. The KL divergence always non-negative and it becomes +0o if supp (P) g supp(QF) if there is some face where supp(py(e=,) & supp(gy] F=)) App. provides more information theoretic extensions.
The direct sum entropy and KL divergence have an interpretation in terms of optimal coding, described in Proposition for the case P Ax - and proven in App. D In words, the direct sum entropy is the average length of the optimal code where the sparsity pattern of E A K-1 must be encoded losslessly and where there a predefined bit precision for the fractional
Relation to optimal codes.
In particular, this means that mixed distributions shall not he used as relaxation in VAEs with purely discrete priors using the ELBO rather, the prior should be also mixed.

---

Published as a conference paper at ICLR 2022
entries of y. On the other hand, the KL divergence between PY and o expresses the additional average code length if we encode variable Y (3) with code that is optimal for distribution qy (3). and it is independent of the required bit precision.
Proposition 1. Let Y be mixed r.v. in A In order encode the face of Y losslessly and to ensure an bit precise encoding of Y in that face we need the following bits on average:
K
(Y = H (Y)
1-
Σ
Pr(f).
(12)
(EP(A--)) dim(s)b-11
Entropy of GaussianSparreenxx. Revisiting the Gaussian Gaussian-Sparseman Sparsemax with K = 2 ($3.3 and using the intrinsic representation 7
the direct sum entropy becomes
[ f'N(vrz, log NogN(w;; o (13)
H(Y) H(F)+ = Po Po P1 log P1
This leads to simple expressions for the entropy and KL divergence, detailed in App. E We use these expressions in our experiments with mixed bit-vector VAEs in $5)
Entropy of Mixed Dirichlet. The intrinsic representation of the Mixed Dirichlet ($3.) al- lows for ((K) computation of (F) and (KL(PFl via dynamic programming, necessary for H (Y) and D KL (p (l (see App. C for details). The continuous parts H and Ep, (DrT.DPPPP (PY))=s F= F-f)l require computing terms (one per proper face) and can be approximated with an MC estimate by sampling from Pr and assessing closed-form the differential entropy of Dirichlet distributions over the sampled faces.
expectation with an exponential number of
Maximum entropy density in the full simplex. An important question to characterize maximum entropy mixed distributions. If we consider only continuous distributions, confined to the maximal face ri --1), the answer is the flat distribution, with entropy - log ((K 1 1)!). corresponding to a deterministic F which puts all probability mass in this maximal face. But constraining ourselves to single face is quite a limitation, and in particular knowing this constraint provides valuable information that intuitively should reduce entropy What if consider densities that assign probability the boundaries? This is answered by the next proposition, proved in App.
Proposition (Maxent mixed distribution on simplex). Let Y be mixed r.v. on AK- with PF(f) & dim(f) direct sum entropy @ @.aax(
(d and Pr (y) f) uniform for each f E F(Ax-1). Then, Y has maximal (Y). The value of the entropy is:
K (K)2
H maxe (Y) log Σ
(I: ( 1)! log 1-2N(-22)
(14)
where L( (z) denotes the generalized Laguerre polynomial
Sonine 1880
For example, for K = 2, the maximal entropy distribution is Pr((0)) = P=((1)) 1/2 + ) Pr([0, 1)) = 22*), with H mmaee (Y) log(2 + 2 Therefore, in the worst case, we need
logz (2 + ) bits on average to encode Y E A with bit precision N. For K = 3, H max (Y)
log(3 3- 2 22. A plot of the entropy as a function of K is shown in Figure App. F
EXPERIMENTS
We report experiments in three representation learning tasks an emergent communication game, where we assess the ability of mixed latent variables over - to induce sparse communication
2 At the opposite extreme, if only assign prohability to pure vertices, i.e., are constrained minimal faces, the maximal discrete entropy is log K. We will see that looking all faces further increases entropy. Appendix G.4 contains fourth experiment-regreee Mixed Dirichlet as likelihood function in generalized linear model.
towards voting proportions-whee we use
7

---

Published as a conference paper at ICLR 2022
Table 2: Test results. Left: Emergent communication success average and standard error over 10 runs. Random guess baseline: 6.25%. Right: Fashion-MNIST bit-vector VAE NLL (bits/dim, lower better). Entropy column legend: continuous/dicrrtte/mixe,, se estimated, =: exact.
Method
Success (%) Nonzeros
Method
Entropy NLL Sparsity (%)
Gumbel-Softmax
78.84 18.00
256
Binary Concrete
C =
3.60
0
Gumbel-Softmax ST 49.96 19.51
Gumbel-Softmax
D
3.49
0
Gumbel-Softmax ST
D =
3.57
100
K-D Hard Concrete
76.07 #7.76
21.43 #17.56
Gaussian- Gaussian-Spaseemaa 80.88 +0.50
1.57 +0.02
Hard Concrete
X
3.57
45.64
Gaussian-Sparsemen
X F 3.53
82.82
Gausiaan-Spasseaax
X =
3.49
73.83
between two agents; bit-vector VAE modeling BusshonnMNISSI images (Xiao et al. 2017). where we compare several mixed distributions over [0, 1] and study the impact of the direct sum entropy ($ and mixed-latent VAE, where experiment with the Mixed Dirichlet over 4 K-1 model MNIST images (LeCun et al. 2010). Full details about each task and the experimental setup are reported in App. G Throughout, we report our three mixed distributions described in $3.3 alongside the following distributions:
Concrete (Maddison et al. (2017); purely continuous density). Gumbel- Softmax (Jang al. (2017); like above, but using discrete KI. term in VAF, which introduces inconsistency), Gumbel-Softmax ST (Jang et al. (2017); categorical latent variable, but using Concrete straight-through gradients), and Dirichlet and Gaussian, when applicable.
Emergent communication. This is a cooperative game between two agents: a sender sees image and emits a single-symbol message from a fixed vocabulary [K] a receiver reads the symbol and tries to identify the correct image out of a set. We follow the architecture in Lazaridou & Baroni (2020); Havrylov & Titov (2017). choosing a vocabulary size of 256 and 16 candidate images. For the mixed variable models, the message is "mixed symbol", i.e., (sparse) point in A Æ Table 2 reporrs communication success (accuracy of the receiver), along with the average number of nonzero entries in the samples test time. Gaussian-Sparsemax learns to become very sparse, attaining the best overall communication success, exhibiting in general a better trade-off than the K-D Hard Concrete.
Bit- Vector VAE. We model Fashion-MNIS" following Correia et al. 2020)) with 128 binary latent bits, maximizing the ELBO. Each latent bit has a uniform prior for purely discrete and continuous models, and the maxent prior (Prop. 2 which assigns probability 1// to each face of A in the mixed case. For Hard Concrete (Louizos et al. 2018) and variables and our direct sum entropy, yielding a coherent objective and unbiased gradients. For the Hard Concrete, we estimate H(Y X x) via MC; for
we use mixed latent
we consider both cases: using an MC estimate and computing the entropy exactly using the expression derived in $3.3 In Table 2 (right) we report an importance sampling estimate (1024 samples) of negative log-like leelikklihood (NLL) on test data, normalized by the number of pixels. For Gaussian-SSpaseena exact entropy computation improves performance, matching Gumbel-Softmn while being sparse.
Mixed-Latent VAE on MNIST. We model the bina- rized MNIST dataset using mixed-latent VAE, with E A10-1, a maximum-entrop prior on Y, and a factorized decoder. We use forwadd encoder with a single-hidden-llyerr to predict the variational distributions (e.g., log-potentials w and concentra- tions a, for Mixed Dirichlet). For Mixed Dirich- let, gradients are estimated via combination of im- plicit reparametrization (Figurnov et al., 2018) and the score function estimator (Mnih & Gregor 2014)) see App-[Cior details. We report a single-sample MC estimate distortion (D) and rate (R) as well as a 1000-samples importance sampling estimate of NLL. The Gaussian prior has access to all of the I0 whereas all other priors are constrained to the simplex A10-1, namely, its relative interior (Dirichlet), its vertices (Gumbel-Softmax ST. and Categorical), or all of it (Mixed Dirichlet). The Dirichlet model clusters digits well as a Gaussian VAE does, while purely discrete models struggle to discover structure. App. G lists qualitative evidence. Compared to Dirichlet, the Mixed Dirichlet makes some plausible confusions (Fig.
Table 3: MNIST test results (avg. of 5 runs). Categorical marginalized exactly.
Method
D
R NLI.
Gaussian (R.9)
76.67
19.94
91.12
Dirichlet
78.62
19.94
93.81
Categorical
164.72
2.28
166.95
Gumbel-Softman ST
171.76
1.70
168.50
Mixed Dirichlet
90.34
19.39 106.59
8

---

Published as a conference paper at ICLR 2022
left), but, crucially, it solves part of the problem by allocating digits to specific faces (Fig. 3 right). Even though we employed a sampled self-critic, Mixed Dirichlet seems to suffer from vanance of SFE, which may explain why Mixed Dirichlet under-perform. in terms of NLL.
RELATED WORK
Existing strategies to learn discrete latent vari- able models include the score function estimator (Williams) 1992, Mnih & Gregor 2014) and
pathwise gradients combined with Concrete relaxation (Maddison et al., 2017: Jang et al.) 2017). The latter is often used with straight- through gradients and by combining continu- ous latent variable with a discrete entropy, which theoretically not sound. Besides the Concrete, continuous versions of the Bernoulli and Cate- gorical have also been proposed Sloaiza-Ganen & Cunningham 2019; Corddnn-Rodriguee et al., 2020). but they are also purely continuous den- sittes, assigning zero probability mass to the boundary of the simplex.
Figure 3: tSNE plots of: posterior samples y AK- (left), predicted log-potentials w E IR (right). Colors encode digit label (not available to models). Clusters are formed in latent space and in how digits are assigned to faces (recall that pa- rameterizes a Gibbs distribution over F(AK-1)).
Our approach is inspired by discrete-contimuous hybrids based on truncation and rectification (Hinton Ghahramani 1997| Palmer et al.] 2017 [ouizos et al.) 2018), which have been proposed for univariate distributions. We generalize this idea to arbitrary dimensions replacing truncation by sparse projections to the simplex. The direct sum measure and our proposed intrinsic sampling strategies related the concept of "manifold stratification" proposed in the statistical physics literature (Holmes-Cerfon 2020). Other mixed r.v.. have also been recently considered (for a few special cases) in (Bastings al. 2019 Murady 2020 van der Well 2020). For instance, Burkhardt & Kramer (209)) induce sparsity in hierarchical models by deterministically masking the concentration parameters of Dirichlet distribution yielding special case of Mixed Dirichlet with degenerate Pr, trained with the straight- thrruuh estimator. Our paper generalizes these attempts, providing solid theoretical support for manipulating mixed distributions higher dimensions.
Discrete and continuous representations in the context of emergent communication have been dis- cussed in Foerster et al. (2016): Lazaridou & Baroni (2020). Discrete communication computa- tionally more challenging because it prevents direct gradient backpropagation (Foerster et al. 2016 Havrylov Titov| 2017), but it hypothesized that this "discrete bottleneck" forces the emergence of symbolic protocols. Our mixed variables bring new perspective into this problem, leading to sparse communication, which lies in between discrete and continuous communication and supports gradient backpropagation without the need for straight-through gradients.
CONCLUSIONS
We presented mathematical framework for handling mixed random variables, which are dis- crete/continuous hybrids. Key our framework is the use of direct sum measure as alternative to the Lebesgue-Bord and the counting measures, which considers all faces of the simplex. We developed generalizations of information theoretic concepts for mixed symbols, and we experimented on emergent communication and on variational modeling of MNIST and Fashion-MNIST images.
We believe the framework described here is only scratching the surface. For example, the Mixed Dirichlet is just one example of an intrinsic mixed distribution; more effective intrinsic parametriza- tions may exist and are a promising avenue. While our main focus was on the probability simplex and hypercube, mixed structured variables are another promising direction, enabled by theoretical characterization of direct sum measures, which can defined for any polytope via their face lattice (Ziegler 1995 Grünbaum| 2003). Current methods for structured variables include perturbations (continuous, Corro & Titov| (2019); Berthet et al. (2020): Paulus et al.] (2020)), and sparsemax (discrete,/Niculaee extensions (discrete, Niculae al. (2018); Correia (2200)) lacking tractable densities.

---

Published as a conference paper at ICLR 2022
Ethics statement. We highlight indirect impact of our work through applications such as generation and explainable AL, where improved performance must be carefully scrutinized. Our proof-of-concee emergent communication, like previous work, uses ImageNet, whose construction exhibits societal biases, including racism and sexism (Crawford & Paglen 2019) that communicating agents may learn. Even in unsupervised settings, biases may be learned by communicating agents.
Reproducibility statement. We now discuss the efforts that have been made to ensure repro- ducibility our work. We state the full set assumptions of our theoretical results and include complete proofss in App. A App. B App. C App. D App. and App. FF) Additionally, code and instructions to reproduce our experiments are available at ts peattts://githeee sparse- commun i tonn We report the standard error over 10 runs for the emergent communi- cation experiment due to the high variance of results across seeds. We include the type of computing resources used in our experiments in App. G.5
com/ seeep-ppin,
ACKNOWLEDGMENTTS
We would like to thank Mário Figueiredo, Gonçalo Correia, and the DeepSPIN team for helpful discussions, Tim Vieira, who answered several questions about order statistics, Sam Power, who pointed out to manifold stratification, and Juan Rivas, who suggested the name "mixed random variables." This work was built on open-source software; we acknowledge Van Rossum & Drake (2009); ; Oliphant [Vrtt)]Virtanenn al. (2019). AF and AM are supported by the P2020 program MAIA (LISBOA-01-0244 111-0227-FEDRR FEDER- 045909), the European Research Council (ERC StG DeepSPIN 758969), and by the Fundação para Ciência Tecnologia through project PTDC/CCI-NF/4700200
Virtanen et al. (2020): Walt et al. (201]) Pedregosa et al. (2011). and Paszke
(PRELUÑA) and contract UIDB/50008/2020, WA received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 825299 (GoURMET). VN is partially supported by the Hybrid Intelligence Centre, a 10-year programme funded by the Dutch Ministry of Education. Culture and Science through the Netherlands Organisation for Scientific Research (nttps:// sorrddinnteliin wen.gennnneeeeeeeeeeeeeeeee centre.
REFERENCES
J Atchison and Sheng M Shen. Logistic-normal distributions: Some properties and uses. Biometrika, 67(2):261-22 272, 1980.
Jasmijn Bastings, Wilker Aziz, and Ivan Titov. Interpretable neural predictions with differentiable binary variables. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. pp. 2963-2977, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 18.18653/v 1///5//v1//111-.1 199-444444
wttp://ooooooo...ooooo1oo.. 1284
Leonard E. Baum and J. A. Eagon. An inequality with applications to statistical estimation for probabilistic functions of Markov processes and a model for ecology. Bulletin of the American Mathematical Society, 73(3):360 - 363, 1967. doi: bams/1 (amms/11835221 183528841.
Quentin Berthet, Mathieu Blondel, Olivier Teboul, Marco Cuturi, Phin-ppil Vert, and Francis Bach. Learning with differentiable pertubed optimizers. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. ht//://ar22000/11eeee//662226222 CC/
Curran Associates, Inc., 2020. URL htt.ttts//prrcceeisssss.eeee
Mathieu Blondel, André F.T. Martins, and Vlad Niculae. Learning with fenchel- losses. Journal Journal of Machine Learning Research, 21(35):1-69, 2020. URL BRLLhttp::/// httt:::://imlr.oog///peeee
19- 021.0tmmmm........
Diane Bouchacourt and Marco Baroni. How agents see things: On visual representations in an the emergent language game. In Proceedings the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 981-985, Brussels, Belgium, October-Novembee 2018. Association for Computational Linguistics. doi: 10.18653/v 1/D/s/////////// 1119. URL https: //ttps://accee http:://aaaathhhhio
prg/D18-1111
10

---

Published as a conference paper at ICLR 2022
John S. Bridle. Probabilistic interpretation of feedforward classification network outputs, with relationships to statistical pattern recognition. In Françoise Fogelman Soulié and Jeanny Hérault (eds.), Neurocomputingg pp. 227-236, Berlin, Heidelberg, 1990. Springer Berlin Heidelberg. ISBN 978-3-642-77153-9)
Sophie Burkhardt and Stefan Kramer. Decoupling sparsity and smoothness in the dirichlet variational autoencoder topic model. Journal of Machine Learning Research, 20(131):1-27, 2019. URL
569.htm.
John B Conway. course in functional analysis, volume 96. Springer, 2019.
Gonçalo Correia, Vlad Niculae, Wilker Aziz, and André Martins. Efficient marginalization of discrete and structured latent variables via sparsity. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 11789-11802 1802.. Curran Associates, Inc., 2020. URL URLittpp::/prrceecdinng newriiss cc/ paper /tteer/222//0
Caio Corro and Ivan Titov. Diffcrentiable perturb-and-paase: Semi-supervised parsing with a structured variational autoencoder. International Conference on Learning Representations, 2019. URL
Thomas M Cover and Joy A Thomas. Elements of Information Theory. John Wiley & Sons, 2012.
Kate Crawford and Trevor Paglen. Excavating AI: The politics of images in machine learning training sets. Accessed 28 May 2021, <https://excavat:
alp, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009. doi: 10. 109/COPR. 200.5555800...0888.....
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In Proceedings of the 56th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 889-898, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18833v1/1111 1082. Utt.https: //acl anthol Molhoooogyy-.. ogy. ogy.oog/p18-1100
Mikhail Figurnov, Shakir Mohamed, and Andriy Mnih. Implicit reparameterizatioo gradi- ents. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Asso- ciates, Inc., 2018 URL Bttpp::/pproceecingg neuripss...e......eeee......... httpp//cppppppp///LLLLL 322299444T7117771111000333222-23
pdf
Jakob Foerster, Ioannis Alexandros Assael, Nando de Freitas, and Shimon Whiteson. Learn- ing to communicate with deep multi agett reinforcement learning. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc., 2016. URL http::tt///// htattps::/pproceeiingge wewpeeiiiiinnnuniiiiii.e nww.ips.i.p.... cc/ paa
Alan Genz. Numerical computation of multivariate normal probabilities. Journal of computational and graphical statistics, 1(2):141-149, 1992.
Elliott Gordon-Rodriguez, Gabriel Loaiza- Ganem, and John Cunningham. The continuous categorical: novel simplex-valued exponential family. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning. volume 119 of Proceedings of Machine Learning Research, pp. 3637-3647. PMLR, 13-18 Jul 2020. URL WtRLitttp:///poceeeei mlr
quez html
Branko Grünbaum. Convex polytopes, volume 221. Springer, Graduate Texts Mathematics, 2003.
Paul R Halmos. Measure Theory, volume 18. Springer, 2013.
Serhii Havrylov and Ivan Titov. Emergence of language with multi-agent games: Learning to communicate with sequences of symbols. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL wtt.tttts::/ppooeeiiiiicc neurips

---

Published as a conference paper at ICLR 2022
Geoffrey E Hinton and Zoubin Ghahramani. Generative models for discovering sparse distributed representations. Philosophical Transactions of the Royal Society of London, Series B: Biological Sciences, 352(11588:1177711199 1997.
Miranda Holmes-Cerfon. Simulating sticky particles: A Monte Carlo method to sample a stratification. The Journal of Chemical Physics, 153(16): (2116)::641122 2020.
Hemant Ishwaran, J Sunil Rao, et al. Spike and slab variable selection: frequentist and bayesian strategies. Annals of Statisties, 33(2):730-773, 2005.
Masha Itkina, Boris Ivanovic, Ransalu Senanayake, Mykel Kochenderfer, and Marco Pavone. Evidential sparsification of multimodal latent spaces in conditional variational autoencoders. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 10235-10246. Curran Asso- ciates, Inc., 2020. URL 73 4 73881
neurips cc/ cc/papper/222//E11ee
Eric Jang. Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In 5th International Conference on Learning Representations, ICIR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL /ttpss:/openzeview net ftr:/formm?22 id IKE
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff. AB, Canada, April 14-16, 2014, Conference Track Proceedings. 2014. URL https:
bttps::///ppeeeeee //tp///pppeee...///e//e/ee net / forummimmi
33X9Ed2- - 2F
Weiwei Kong, Walid Krichene, Nicolas Mayoraz, Steffen Rendle, and Li Zhang. Rankmax: An adaptive projection alternative to the softmax function. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 633-643. Curran Associates, Inc., 2020. URL (0..0///fleee/ 01wlll/ee0000b0000000000000 04833378442288812222222 Paper...p-ppp.ppp...
http://rrcceddnngg...i...rss.sss neurips cc/
Anirban Laha, Saneem Ahmed Chemmengath, Priyanka Agrawal, Mitesh Khapra, Karthik Sankaranarayanan, and Harish G Ramaswamy. On controllable sparse alternatives to soft- max. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 31, Curran Asso- ciates, Inc., 2018. URL htttps://proceeeeee Ba465552dee0111acaaaffaaaaa3 Papes pdf
https//pss.scccpppee/////// ccc/paper//018/f1ee.
Angeliki Lazaridou and Marco Baroni. Emergent multi-agent communication in the deep learning era. preprint wwXiv:2006.02419 2020.
Yann LeCun, Corinna Cortes, and CJ Burges. MNIST handwritten digit database. Available online: <http: 1ew.raannnnnenn.mmmmm lecun. com/ exdb/ mn st'. 2010.
Gabriel Loaiza-Ganem Ganem and John P Cunningham. The continuous bernoulli: fixing a pervasive error in variational autoencoders. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché- -uch E. Fox, and Garnett (eds.), Advances in Neural Information Processing Systems, volume 32, Curran As- sociates, Inc., 2019. URL https: http:///occeddd/ecoeooo ngs. neurips. cc/pppper/211 9/filel E82798ec8 0eteee35900222355666665658 3eee33355672222225663322
Paaprr.ppppp.. pdf
Christos Louizos, Max Welling, and Diederik P. Kingma. Learning sparse neural networks through lo regularization. In International Conference on Learning Representations, 2018. URL https
//openreview.oo net htt:://rrmmmmmoommmm/oo ootp:nneet/oooommmdmm Y8hhggg
Dunean Luce. Individual choice behavior: A theoretical analysis. New York: Wiley, 1959, 1959.
Chris J. Maddison, Andriy Maih, and Yee whye Teh. The concrete distribution: A continuous relax- ation of discrete random variables. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26. 2017, Conference Track Proceedings. OpenReview.net, 2017. URL htt.hhttp///pppeeee.
net. wtwe/eoooumtttt www.:/orrmmmmimmmm d= 1F.5.5
12

---

Published as a conference paper at ICLR 2022
Andre Martins and Ramon Astudillo. From softmax to sparsemax: A sparse model of attention and multi-label classification, In Maria Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 1614-1623, New York, New York, USA, 20-22 Jun 2016. PMLR. URL ml www.s /w..ss/v44// iss ht.l
nttps::/proceedin.
Toby Mitchell and John J Beauchamp. Bayesian variable selection in linear regression. Journal of the American Statistical Association, 83(404):1023-1033) 1988.
Andriy Mnih and Karol Gregor. Neural variational inference and learning in belief networks. In Proceedings of the 3 International Conference on International Conference on Machine Learning Volume 32, ICML. 14, pp. 11-1791-11-1799 JMI..org. .org, 2014.
Lina Murady. Probabilistic models for joint classification and rationale extraction. Master's thesis, University of Amsterdam, 2020.
Vlad Niculae, Andre Martins, Mathieu Blondel, and Claire Cardie. SparseMAP: Differentiable sparse structured inference. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 3799-3805 PMLR. 10-15 Jul 2018. URL https::/prroeeedinggs...
Cxess/v80/nic.ue
html
Frank Nielsen and Richard Nock. Entropies and cross-entropies of exponential families. In 2010 IEEE International Conference on Image Processing, pp. 3621-3624, 2010. doi: 10. 109/CCIP.
2010.5652054.
Travis E Oliphant. A guide to NumPy, volume 1. Trelgol Publishing USA, 2006.
Andrew W Palmer, Andrew J Hill, and Steven J Scheding, Methods for stochastic collection and replenishment (scar) optimisation for persistent autonomy. Robotics and Autonomous Systems, 87: 51 51-65, 2017.
George Papandreou and Alan L. Yuille. Beturb-and-maa random fields: Using discrete optimization to Iearn and sample from energy models. In 2011 International Conference on Computer Vision, pp. 193-200, 2011. doi: 10.1100//C 12201..1226222
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, hegh-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Bin -ucé E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024 8035. Curran Associates, Inc., 2019.
Max Paulus, Dami Choi, Daniel Tarlow, Andreas Krause, and Chris Maddison. Gradient estimation with stochastic softmax tricks. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances Neural Information Processing Systems, volume 33, pp. 5691-5676 Curran As- sociates, Inc., 2020. URL 3df80a 3 f
cew.:/eeeeeeeinee...eei.e...o..p newrips cc pttpp///////1ee...e/ pdf
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830. 2011.
Ben Peters, Vlad Niculae, and André F. T. Martins. Sparse sequence-to-sequenne models. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1504-1519, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/ v1/P19- 1146. URL htLpsttps:
1146
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models unsupervised multitask learners. OpenAl blog, 188)9, 2019.
13

---

Published as a conference paper at ICLR 2022
Jason Tyler Rolfe. Discrete variational autoencoders. In 5th International Conference on Learning Representations. ICLR 2017, Toulon. France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL
Murat Sensoy, Lance Kaplan, and Melih Kandemir. Evidential deep learning quantify classifi- cation uncertainty. In S. Bengio. H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran As- sociates, Inc., 2018. URL htttppppp//pppcceddngttoooooo/et neuripss.nn.nnn............... http:/pppppppppppp////////////// a981 41011120088000777741741444422444 - pdd
neurips
Claude Shannon, A mathematical theory of communication. The Bell system technical journal, 27 (3):379 423, 1948.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Yoshua Bengio and Yann LeCun (eds.). 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Truck Proceedings, 2015. URL http:
orq/ abs 1109 1556
N Sonine. Recherches sur les fonctions cylindriques et le développement des fonctions continues en séries. Mathematische Annalen, 16(1):1-8)) 1880.
Arash Vahdat, William Macready, Zhengbing Bian, Amir Khoshaman, and Evgeny Andriyash.
DVAE++:
Discrete variational autoencoders with overlapping transformations. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 5035-5044. PMLR, 10-15 Jul 2018. www.ss........sseee.e.e
URL https:
WtRLhttps:///prceeeeeee http:///oceeeeeeeoooo igg m www.press/eeee peress/v80/vahheee 18a.html
Eelco van der Wel. Improving controllable generation with semi-supervised deep generative models. Master's thesis, University of Amsterdam, 2020.
Guido Van Rossum and Fred L. Drake. Python Reference Manual, CreateSpace, Scotts Valley, CA, 2009. ISBN 1441412697.
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. Nelson, Eric Jones, Robert Kern, Eric Larson, CJ Carey, Ilhan Polat, Yu Feng, Eric W. Moore, Jake Vand erPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, A. Quintero, Charles R Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa. Paul van Mulbregt, and SciPy 1. O Contributors. SciPyy 1.0: Fundamental Algorithms for Scientific Computing Python Nature Methods, 2020. doi: https://doi.org// 1ttp://o......111110//1/44/11110
Martin J Wainwright and Michael Jordan. Graphical models, exponential families, and variational inference. Foundations and Trends@ in Machine Learning. 1(1-2):1-30x 2008
Stéfan van der Walt, Chris Colbert, and Gael Varoquaux. The NumPy array: structure for efficient numerical computation Computing in Science & Engineering, 13(2):22-30, 2011.
Ronald Williams. Simple statistical fralient algorithmms for connectionist reinforcement learning. Machine Learning, 8(3):229-256, 1992. doi: 10. 10.10077BF000992669 URL https
1007/BF00992695
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion- a novel image dataset for bench marking machine learning algorithms preprint arXlv:1708.07747, 2017.
Günter M Ziegler. Lectures on polytopes, volume 152. Springer, Graduate Texts in Mathematics, 1995.
14

---

Published as a conference paper at ICLR 2022
A PROOF OF WELL-DEFINEDNESS OF DIRECT SUM MEASURE
We start by recalling the definitions of a-algebras, measures, and measure spaces. A o-algebra on a X is collection of subsets, 52 2*,, which is closed under complements and under countable unions. A measure on (X, Xt) is a function from S to I U { satisfying (i) p((A) > 0 for all A E Ω (ii) p(2)= = O, and (iii) the o-additivity property: ((L)SEAA)) countable collections {AA)jeN S of pairwise disjoint sets in Ω. measure space is triple (X, where X is set, A is o-algebra X and is measure on (X, An example is the Euclidean space X R endowed with the Lebesgue measure, where St is the Borel algebra generated by the open sets (i.e. the set f2 which contains these open sets and countably many Boolean operations over them).
EJEN ((A)) for every
The well-definedneet of the direct sum measure t comes from the following more general result, which appears (without proof) as exercise I.6 Conway (2019).
Lemma 1. Let (X&, S2z (4R) be measure spaces for k = 1....,K K. Then, (X, K,S. n.) is also measure space, with X + Xx -I X (the direct sum Cartesian product of sets Xxl, -= (A (AS X An XA E Sk Ak E [K]), and p(A) = (Ann
Proof. First, show that Q a o-algebra. We need to show that (i) if A e Ω, then A Q and (ii) if A; E S for each i E N then ULeN A; E Ω. For (i), we have that, if A E S2 then we must have
A Xx E Pr for every k, and therefore A n X Xx A= X2 \ (A Xx. E Ω2. since Qr is a o-algebra on X₁. This implies that A E For (ii), we have that, if A; E Ω2 then we must have N and & [K]. and therefore (ULeN A.) nXx (, e 2u. since 24 is closed under countable unions. This implies that ULEN A, E f. Second, we show that is a measure. We clearly have μ(A) Σx= u 2 O, since each ui is a measure itself, and hence it is non-negative. We also have μ(Ø) Σx= (n Xx) Σk= mx( O. Finally, if {Ay)ieN S Q is a countable collection of disjoint sets, we have p(LIJENA)) Σ me (L)JEe(())))) n Xx)) = ER EjEN (A, Xx) = EjEN E (A(A, 7 Xx) = EjEN ((A)).
A, Ax E D for every
We have seen in that that the simplex AK- can be decomposed interior of its faces. Each of these relative interiors is an open subset an affine subspace isomorphic to R- for & [K], which is equipped with the Lebesgue measure for k >> and the counting measure for & = 1. Lemma then guarantees that we can take the direct sum of all these affine spaces as measure space with the direct sum measure μ th of Definition
disjoint union of the relative
B K-DIMENSIONAL GAUSSIAN-SPARSENN
We derive expressions for the density Py.r(y for the sample-and-prone projeete-ppe-tttt case (stochastic sparse- max). We assume without loss of generality that f = {1, 2, 2....,8], and that we want to compute the density PYa.x (y2,..., Ms. ,,, for given , Us such that y := 1 -
0.
The process that generates the data as follows: first, 2; pz, (z:) independently for i E [K]. Then, is obtained deterministically from 21:K as sparsemax(21 (2:K)) ). We assume here that each Pz, (2:) is univariate Gaussian distribution with mean μ4 and variance o7.
We make use of the following well-known properties of multivariate Gaussians:
Lemma 2. Let 7 M(,,E) E). Then, for any matrix A, not necessarily square we have AZ ~ N(Au, AEAT). Furthermore, splitting
2 =
X Y
Pz
Σgz T ay
Eny Eny
(15)
Bl
we have the following expression for the marginal distribution (x(x)):
X ~ ~Noo [ Egg)
(16)
15

---

Published as a conference paper at ICLR 2022
and the following expression for the conditional distribution Px Y y):
E-1ΣT X x = ~ N(pz + Eay Eny Myy; Eaz Eay Bny Ezy).
(17)
We start by picking one index in the support of y we assume without loss of generality that this pivot index is 1 and that the support set is {I,..., 8} as stated above and introducing new random variables U; = Zi Z1 for E {2,.... Kj. Note that these new random variables are not independent since they all depend on Z1. Then, from the change of variable formula for non- non-invertible transformations, we have, for given y/2, ................................ > 0 such that y1 := 1 Σj=2 Uj > 0:
PYa. (12,.... ... .......)) PUni (u2:s) x |B|x
[
(aenDaalUna(((ttt)) U1: u1:s) u1:))dus+1 duk.
(18)
where u, 1:-:--y11 11 for i 2,..., 8, which written u2:s Byz. +o B 7--1
11
and c = -1,-1- The determinant B is simply B = 8.
Note that U2. = AZı.K, where A
K
Ix-I]
R (1K-11 1)xK that is
0
0
1
0
A
(19)
0
0
1
From Lemma] 2 we have that
(u) = NNu Au, ADiag(o?)
=N(u; M2lk-1, Diag(o 2 + +o
(20)
and the marginal distribution PU2., (w2:s) is
PUzza (u2:s)
(u2:s P2:s M1 1,- Diag (22.) -
NNy:s 11 P2:5 in 1, Diag( + of1--111-1).
(21)
Note that this is a multivariate distribution whose covariance matrix is the sum of diagonal matrix with constant matrix.
We now calculate the conditional distribution of the variables which are not in the support conditioned on the ones which are in the support. We have according the notation in Lemma
Exz Diag o (++1):k) =o
Egy = Diag )2.0) + 021.-11.-
Eny = o
(22)
Using the Sherman-Morrison formula, obtain
(0 -
Σ Diag(oz.3)
02.6 2:8
(23)
0 + 2 0,
from which we get
= Pz + En (y pby
-M(+1)) MIlk-
-
Ix (x=2(0 (u2:3 12:s + un 1=11s
oi + Ei o;
1
=Mo+1)K 11l--
1k-s(022)
(y2.
in U1 11:0 + M1 M1ls-1)
Σi
-= o, -2 (yi ui)
=M(41): - 1
-2
1-
(24)
2im10 0,
16

---

Published as a conference paper at ICLR 2022
and
= Exe Exy 20
2y May
ofl T
Diag(o + tolk-slkk
01 1=
of 0,
= Diag(o fflk
i=
-K-slk
01
2i= 0
= Diag(o top (1
Eiz
(
1K-slK-s
01
2
Ei=2 0
= Diag( [a (++1): +
K- kKolk
01 Ei=z
2
Diag(o (++1):: ()+1):K
+
k -K-sk--
(25)
Σ Ei=10
Finally, also from Lemma we get
PU pe (Us (3(s+ U1: = u1:.) (()
(26)
Note that this is again multivariate Gaussian distribution with a covariance matrix Σ which the sum of a diagonal and constant matrix.
Putting everything together, we get
PVa. (y2,.... Ma. .,....,.))................ 0) NN(122 91 1-1111111111 1111-11122:0 1111,-1, Diag(o2.s) + ofl.-113-11
F ((s+t):x+ Ei o, 2 M4)
Ei=ro IK-si 5)
(27)
where = So (O, E) is the negative orthant cumulative distribution of multivariate Gaussian with mean v and covariance E. Efficient Monte Carlo approximations of this integral have been proposed by Genz (992).. Using again Lemma direction, the function F(u; Σ), setting ((s-1):K + Ei= 10: p.) unidimensional integral:
2 but now in the reverse
Ei= K-ss be reduced to
1 t: Li- of (m Wi)
K
F(u; E) = N
drN(ripu,
Ei= 0; -2
Eisl 0;
22
i=
00
Σi Oi (t + +. Hki)
K
t 44i
dt ④
Sb
-00
Ei=10 oi
1=++1
du
II + j=e-1
④ - (w)
Σ3 Oi 211
M4 + Mq)
(28)
Ei= 0; -2
where ((x) and (xx) are the p.d.f. and c.d.f. of a standard Gaussian distribution, re- spectively, and we applied the change of variables formula in the last line, with u = Ei (t M. - mi)\ V the Jacobian of this transformation cancels with the d-term. We can compute the quantities above via the erf function and its inverse e as ((z) = 1 + erf and Φ < = - 1).
*-) E: ( / Vi
whose inverse
Σi_
17

---

Published as a conference paper at ICLR 2022
When the variance is constant, 0, Vi E [K], and since Σi= V simplify to
1, the expressions above
F(u;E) = dt N t -1+
M4
0²
II
K
drN(r;
[s|
|8]
1=
++ s/i -
dt
Φ
1==1
Ui
ov 8
1=5+1
K
② Φ-
Mi
du
+
14j
+
(29)
j=s+1
C SAMPLING FROM THE MIXED DIRICHLET DISTRIBUTION
PyTorch code for a batched implementation of the procedures discussed in this section will be made available online.
Distribution over Face Lattice. The distribution PF is a discrete exponential family whose support is the set of all 2* 1 proper faces of the simplex. The computation of the natural parameter in Equation 8 factorizes over vertices, but, because the empty face is not in the support of Pp. the computation of the log-normalizer requires special attention. The obvious strategy is to enumerate all assignments to (Fz, w ((Fz)), one per proper face. This strategy is not feasible for even moderately large K. We can, however, exploit a compact representation of the set of proper faces by encoding the face lattice in directed acyclic graph (DAG) where each proper face is associated with a complete path from the DAG's source to the DAG's sink.
This DAG G = A) is a collection of states E Q C (0,...,KIII x {0.1) x 10, and ares (u, v) E A C Q x Q. A path that ends in (k: b, 8) E Q corresponds to a non-empty subset of vertices if, and only if, s 1. A path that contains (k, E Q corresponds to face that includes the kth vertex if, and only if, b = 1. If u = (ki, b1, $1) E Q and v = (k2, b2. $2) E Q then u < e iff k S k2. The state 0, is the DAG's unique source, and the state (K + 1, 1) is the DAG's unique sink. Because = for the sink, no complete path (i.e., from source to sink) will correspond to an empty face. An are is pair of states (u, r), where u is the origin and is the destination. For every state (ke, b, s) such that k < K, we have arcs ((k, b, s), (K + 1,1,1) and ((k, b, s), (k + 1, 0, s)). For every state (K, b, 1), we have an are the final state (K + 1,0,1) By construction, the number of states (and arcs) in the DAG is proportional to K. A complete path (from source sink) has length K + 1. and it uniquely identifies a proper face. To compute the log-normalizer of Equation (8) we run the forward algorithm through G, with an arc's weight given by (-11-b)w; the arc's destination the state (k: b, 8) or O if the arc's destination the DAG's sink. Running the backward algorithm through G (that is, running the forward algorithm from sink to source) evaluates the marginal probabilities required for ancestral sampling.
Entropy of Mixed Dirichlet. The intrinsic representation of a Mixed Dirichlet distribution allows for efficient computation of (F)) and DKI.(Prl1Q) necessary for H*(Y) and D (py ( require solving expectation with an exponential number of terms (one per proper face). The distribution Pp is a discrete
The continuous parts FY)F) and Eyp (py)==)llyyl)) [ necessarry
exponential family indexed by the natural parameter w R² its discrete entropy is H (F)
log (ww) - (w, V.. log Z ((w),, where V.. log Z(w) = [[}(()). KL divergence from a member QP of the same family but with parameter v given by
DRI.(Prl(0
log 7(v) log (ww) (u- w. V.. w log Z(w))
(30)
The sufficient statistic) in single pass through DAG of size O(K). For results concerning entropy, cross-entropy, and relative entropy of exponential families see for example Nielsen & Nock (2010).
algorithm computes both the log-normalizer and its gradient (the expected
The continuous part FY is the expectation of the differential entropy of Y|F f. each Dirichlet distribution over the face f. under Pp. For small K we can enumerate the 2 terms in this expectation, since the entropy of each Dirichlet is known in closed-form. In general, for large K,
18

---

Published as a conference paper at ICLR 2022
obtain an MC estimate by sampling independently from Pp and assessing the differential entropy of Y | = f only for sampled faces. For Ep, [DLL (pvir= | the situation is similar, since KL for two Dirichlet distributions the same face f is known in closed-form.
Gradient Estimation. Parametrizing a Mixed Dirichlet variable takes two parameter vectors, namely, and In VAE, those predicted from given data point (e.g., an MNIST digit) by an inference network. To update the parameters of the inference network we need a Monte Carlo estimate of the gradient V V a B |r= [[(]]] with respect w and a of loss function & computed in expectation under the Mixed Dirichlet distribution. By chain rule, we can rewrite the gradient as follows:
Br/ [By]==fa((()) log (F(flw) + VaBy)r=saa((()]]
(31)
Given sampled face F= f. we can MC estimate both contributions to the gradient, the first takes MC estimation Fy a [[(y)) which is straightforward, the second can be done via implicit reprametrization (Figurnov et al., (2018). To reduce the variance of the score function estimator (first term), we employ a sampled self-critic baseline, i.e,, an MC estimate of Ey|F=f (<(y)) given an independently sampled face f".
D INFORMATION THEORY FOR MIXED RANDOM VARIABLES
D.1 MUTUAL INFORMATION FOR MIXED RANDOM VARIABLES
Besides the direct sum entropy and the Kullback-Leibbe divergence for mixed distributions, we can also define mutual information for mixed random variables follows:
Definition 4 (Mutual information). For mixed random variables Y and Z. the mutual information between and Z
= = - (Y Z)
=H(F) H( F) H(F H(Y F,Z) =((F + T(Y; Z F) 2.
(32)
With these ingredients it is possible to provide counterparts for channel coding theorems by combining
Shannon's 's discrete and continuous channel theorems (Shannon 1948)
D.2 PROOF OF PROPOSITION I (CODE OPTIMALITY)
The discrete entropy of random variable representing an alphabet symbol corresponds to the average length of the optimal code for the symbols the alphabet, in a lossless compression setting. Besides, is known (Cover) & Thomas 2012) that the optimal number of bits to encode a D-dimensional continuous random variable with N bits of precision equals its differential entropy (in bits) plus N D Therefore, the direct sum entropy (TO) is the average length of the optimal code where the sparsity pattern of y A k-1 must be encoded losslessly and where there a predefined bit precision for the fractional entries of Eq. 12 follows from these facts and the definition of direct sum entropy (Definition
Proposition 1 is consequence of the following facts (Shannon, 1948):
Cover & Thomas 2012 extend the argument for D > 1.
Theorem 9.3.1) provide informal proof for 12 = 1, but it is straightforward to
19

---

Published as a conference paper at ICLR 2022
E DIRECT SUM ENTROPY AND KULLBACK-LEIBLER DIVERGENCE OF 2D GAUSSIAN-PPARSMAAX
From ① and Definition 3
the direct sum entropy of the 2D Gaussian sparsemax is:
1Y) H(F) + H(Y FF)
-P)) 1." NNyi
Nyy 2, o²)
H([P, Pı, 1 - Po-Pi)) (1 Po Po-10 Pi)
PP)-P))
log
Po Pi dy
Po P1
-Polog Po - Pi log P1 - f Nog 2, d²)dy,
log.N
(33)
where we have Po ]-er[t=/[V20)) Pi (+er((2-11/(v220)) and
-f N(1 o²) Noy ddy
[" N(y; (1og(v + (y - 2)2
dy
J
20²
1-2
(1 Po - -P)) sog(vV2ni 2
+
N(t;0,1)Par
(1
Po Po) v 2TT22
erf 1-z
V20²
erf
V2o²
1
2 N
(1 21-2
2 :0.1 -
2 2x (-2.0.1)) (34)
+
2
2
σ σ
o
which leads closed form for the entropy. As for the KI. divergence, we have:
D& (2 :- (Re((PF((FF) + BynPe [DRI,PYIE(:1 [ F=
Σ PF(J) log Pp(J)
E Pr(S) ( (fmm
folo PriF(1) f)
Qr(f)
(y) lf)
SEF(P)
FEF(P)
Po
P + f N(y (() log Nly 11::2..77 0z.
= Po log
+ P log
(35)
Qo
Q
No zq, 0z)
20

---

Published as a conference paper at ICLR 2022
where we have Po- 1-orfisp/(v2rp)
1oor[[(xp (ati((sp-)/(((2)) 11 Qo = 1-arf(zg / ((v/mg))))))))))
Pi
2
2
2
Q 1+edil(sg 11)((v [)/oV) and
Niy 2p, op) N(y: Zp, 2²) log
dy=
N1rizQ 0Q)
log or σ
(U-2p)
(y ==2)²
+
dy
20
202
1-1Q
=-(1- Po (b-P))log Po) log
O
dp
0(t;0,1)&&
0Q
N(t;0,1)&a:
0Q
sp
-1q
op
OP
-11 - Po - P1) log
σQ
orf
1-ap
crf
dP
220z.
22p
1 EP N
(1-p 0p
(1 2P
:0
2P
N
(-33:00
0,
2
0p
0p
0r
erf
1-20
erf
(-
24
+ 2
σQ
220
2
1 2Q
=QN
N
1 (1-22;0.1) :011)
22 =2N (-20:0; 0,
0Q
0Q
0Q
(36)
F MAXIMUM DIRECT SUM ENTROPY OF MIXED DISTRIBUTIONS
Start by noting that the differential entropy of a Dirichlet random variable Y ~ Dir(a) is
(Y) = log B(a)) + (on K))(a))
(on - 1)p(ax),
(37)
A==1
where 0g- = DE or and ve is the digamma function. When a = 1, this becomes a flat (uniform) density and the entropy attains its maximum value:
H(Y) = log B(a:) = - log((K K - 1)!.
(38)
This value is negative for K > 2; it follows that the differential entropy of any distribution in the simplex is negative. We next determine the distribution p (y) with the largest direct sum entropy.
Considering only the maximal face, which corresponds to ri A Ax ) the distribution with the largest entropy is the flat distribution, whose entropy is given in 388 In our definition of entropy in (10)
this corresponds to a deterministic F which puts all probability mass in this maximal face. At the opposite extreme, if we only assign probability to pure vertices, i.e., we constrain to minimal faces, a uniform choice leads to (Shannon) entropy of log K. We will show that looking at all faces further increases entropy.
Looking at (10). we see that the differential entropy term H(Y F f) can be maximized separately for each f. the solution being the flat distribution on face f A which has entropy log((k - 1)!, where 1 < k < K. By symmetry, all faces of the same dimension k - 1 look the same, and there are (k) of them. Therefore the maximal entropy distribution is attained with Pr of the form Pr(f) = g(k))(( (k) where g [K] + R, is a function satisfying Σk= g(k) = (which can be regarded as categorical probability mass function). If we choose precision of N bits, this
21

---

Published as a conference paper at ICLR 2022
mixed N=0
mixed N=1
mixed N=2
17.5
mixed N=3
mixed N=4
15.0
discrete
continuous N
MaxEnt
12.5
10.0
7.5
5.0
0.0
Figure 4: Maximum entropies for mixed distributions for several values of bit precision N, as
function of the simplex dimensionality K K-1 1. Shown are also the maximum entropies for the corresponding discrete and continuous cases, for comparison.
leads to:
Σ Pe(f) (-/ prirly log ( lf) FEF(AR-1)
H (Y)
Σ
lo(f)logPi(f+
SEF(AR-1)
+ (k - (k)
(39)
K
g(k)
g(k) log
(kk -1)! 1)! N(k: 1))
(K)
k=1
k=1
(k)2N((-1) (I: 1)!
(kk) log (k)
(kk) log
(40)
Note that the first term the entropy of g(-) and the second term is a linear function of g(-), that is, (40) is entropy regula-regu argmax problem, hence the g(.) that maximizes this objective is the softmax transformation of the vector with components log (K-1)) that is:
(
(()2
(K
g(k)
(41)
12N(111)
1= (1-1))
and the maximum entropy value is
( (K)2N(k-1)
H N.
max (Y)= = log
(k (k- 1)! log (1)
_ (-2N(-2N),
(42)
where [()) (xx) denotes the generalized Laguerre polynomial (Sonine, 1880), as stated in Proposition
Figure shows how the largest direct sum entropy varies with K, compared to the discrete and
continuous entropies. For example, for = 2, we obtain g(1) = 242N, g(2) 2 and
2
N
H (Y) log(2 + 2N), therefore, in the worst case, need most log, (2 + 2 bits to encode
22

---

Published as a conference paper at ICLR 2022
Y E Ai with bit precision N. This is intuitive: the faces of A1 are the two vertices {(0, 1)} and {(1,0) and the line segment [0, 1]. The first two faces have a probability of 2428 and the last one
2
have probability 242 To encode point in the simplex we first need to indicate which of these three 2 faces it belongs to (which requires 2+2 1ogg(2 + 2*) + 242 log2
2+2 logg(2 + 2*)
N2
2
242
2 bits), and with 242 probability we need to encode point uniformly distributed in the segment [0, 1] .22 with NN bit precision, which requires extra 2422
bits on average. Putting this all together, the total
number of bits is 12 2%), expected.
G EXPERIMENTAL DETAILS
In this section, we give further details on the architectures and hyperparameters used in our main experiments (App. G.I App. G.2, and App G.3). which are focused on learning sparse representations, in App. G.4 we experiment with the Mixed Dirichlet likelihood function model simplex-valued data. Finally, in App G.5 we describe our computing infrastructure.
G.1 EMERGENT COMMUNICATION GAME
Let V = ( } be the collection of images from which the sender sees a single image dy- We consider a latent variable model with observed variables æ (V.j) E and latent stochastic variables y chosen from a vocabulary set y We let the sender correspond to the probability model π (y)az, 0) pyy 0y.0%) of the latent variable and the receiver be modeled as p(j V.y. y. fe). The overall fit to the dataset D is EneD (0), where we marginalize the latent variable y to compute the loss each observation, that is
C (0) = En [((2, v; = Σ Tul 0) E(x, y:: 0),
(43)
DEy
where the receiver is used define the downstream loss, ((r, y; log p(j V. v. 0z). Notably, we do not add a (discrete) entropy regularization term of T xr 0) to (43) with a coefficient as an hyperparameter as in Correia et al. (2020).
Data. The dataset consists of subset of ImageNet (Deng et al., 2009 containing 463,000 images that are then passed through a pretrained VGG (Simonyan Zisserman, 2013), from which the representations at the second-to-last fully connected layers are saved and used as input to the sender and the receiver. To get the dataset visit https: /ttptubb.b///////// com/ /iw.aeeeeoocccooutttttttte SignalingGart (Bouchacourt & Baroni 2018).
Architecture and hyperparameters. We follow the experimental procedure described in Correia al. (2020): the architecture of both the sender and the receiver are identical theirs, set the size of the collection of images [V] to 16, the size of the vocabulary of the sender to 256, the hidden size 512, and the embedding size to 256. We choose the best hyperparameter configuration by doing grid search on the learning rate (0.01, 0.005, 0.001) and, for each configuration, evaluating the communication success on the validation set. For the Gumbel-Softmax models, the temperature is annealed using the schedule
max(0.5, exp -rt), where r = le 5 and is updated every N 1000 steps. For the K-D Hard Concrete we use a scaling constant = 1. and for Gaussian- Sparsemax we set Σ = I. All models were trained for 500 epochs using the Adam optimizer with batch size of 64.
G.2 BIT- VECTOR VAE
In this experiment, the training objective is to minimize the negative ELBO: rewrite (43) with (xx, y: 0z) log p(z.vl) z. x where we decompose the approximate posterior as q(v xx A)
mh g(ve x. A), with D = 128 being the number of binary latent variables yt- For each bit, we
Available for research under the terms and conditions described in honnhtte httpt:://////.tt //ww.. image net o
down o
23

---

Published as a conference paper at ICLR 2022
consider a uniform prior p(y) when the approximate posterior is purely discrete continuous, and the maxent mixed distribution (Prop. for the mixed cases.
Data. We Fashion-MNIST (iio et al. 2017 comprising of 28 x 28 grayscale images of fashion products from different categories.
Architecture and hyperparameters. We follow the architecture and experimental procedure de- scribed in Correia et al. (2020), where the inference and generative network consist of one 128-node hidden layer with ReLU activation functions. We choose the best hyperparameter configuration by do- ing a grid search on the learning rate (0.0005, 0.001. 0.002) and choosing the best model based on the value of the negative ELBO on the validation set. For the Gumbel-Softmn max(0.5, exp -rt), where search E {le 5, le - 4] and t
models, the temperature
is annealed using the schedule is updated every N = 1000 steps; also, for these models, we relax the sample into the continuous space but assume discrete distribution when computing the entropy of mt xr f), leading to an incoherent evaluation for Gumbel-Softmax without ST. For the Hard Concrete distribution we follow Louizos al. (2018) and stretch the concrete distribution to the (-0.1, 1.1) interval and then apply hard-sigmoid on its random samples, For the
we use o² = 1. All models were trained for 100 epochs using the Adam optimizer with batch size of 64.
G.3 MIXED-LATENT VAE ON MNIST
Data. We use stochastically binarized MNIST (LeCun al. 2010 Thee first 55,000 instances are used for training, the next 5,000 instances for development and the remaining 10,000 for test.
Architecture and hyperparameters. The model is a VAE with a K-dimensional latent code (with K 10 throughout), both the decoder (which parametrizes the observation model) and the encoder (which parametrizes the inference model) are based on forward neural networks. The decoder maps from a sampled latent code z to a collection of D 28 x 28 Bernoulli distributions. We use a feed-forward decoder with two Rel.U-activated hidden layers, each with 500 units. The encoder maps from D-dimensional data point x to the parameters of variational approximation to the model's posterior distribution. For Gaussian models we predict K locations and K scales (with softplus activation). For Dirichlet models we predict K concentrations using softplus clamped to [10-3 3 10 For Mixed Dirichlet models we predict K log-potentials clamped to [-10, 10). and K concentrations using softplus clamped [10 3, 10 For regularization we employ dropout and L2 penalty. We use two Adam optimizers, one for the parameters of the decoder and another for the parameters of the encoder, each with its own learning rate. We search for the best configuration of hyperparameters using sepoortance-samplll estimates of negative log- ligliiilihood of the model given development set. The parameters we consider are: learning rates (in {10 5 10 10 33, and their halves), regularization strength (in {0, 10 10 10 $1), and dropout rate (in {0, 0.1, 0.2, 0.3)).
All models are trained for 500 epochs with stochastic mini batches of size 100. During training, for gradient estimation, we sample each latent variable exactly once. For importance sampling, 100 samples for model selection, and 1000 samples for evaluation on the test set.
Gradient estimation. We use single-sample reparametrized gradient estimates for most models: Gaussian (unbiased), Dirichlet (unbiased). Gumbel-Softmant ST (biased due to straight-trouge and Mixed Dirichlet (unbiased). A gradient estimate for Mixed Dirichlet models, particular, has two sources of stochasticity, one from the implicit reparametrization for Dirichlet samples, another from score function estimation for Gibbs samples. To reduce variance due to score function estimation, the Mixed Dirichlet model we report employs baseline. We have also looked into a simpler variant, which dispenses with this self-critic baseline, and instead employs a simple running average baseline, this model performed very close to that with a self-critic. The purely discrete latent variable model (Categorical) uses the exact gradient of the ELBO, given mini batch, which we obtain by exactly marginalizing the K 10 assignments of the latent variable.
additional sample, used to compute self-critic
I at https:///iihhb..cc//aaaaaao master/I.CCFS:
on on-mn- mamist/blll
24

---

Published as a conference paper at ICLR 2022
Figure 5: tSNE plots of posterior samples, from left-to-right: Gaussian, Dirichlet, Mixed Dirichlet, Categorical, Gumbel-Softmax ST.
O 3
7
6
3
8
7
9
8
3
7
4
C
6
2
8
9
4
5
+
/
6
9
0
6
3
-
7
Figure 6: Pixel-wise average of 100 model samples generated by conditioning on each of the ten vertices of the simplex. From left-to-right: Mixed Dirichlet, Categorical, Gumbel-Softmal ST, and Dirichlet (as the Dirichlet does not support the vertices of the simplex, we add uniform noise (between 0 and 0.1) to each coordinate of a vertex and renormalize).
Prior, posterior, and KL term in the ELBO. The Gaussian model employs standard Gaussian prior and a parametrized Gaussian posterior approximation, the KL term in the ELBO is computed in closed-form. The Dirichlet model employs uniform Dirichlet prior (concentration 1) and parametrized Dirichlet posterior approximation, the KI. term in the ELBO is also computed in closed-form. Both the Categorical and the Gumbel-Softmax ST models employ uniform Categorical prior and a Categorical approximate posterior (albeit parametrized via Gumbel-Softmax ST for the latter model), the KI. term in the ELBO is between two Categorical distributions and thus computed in closed form. The Mixed Dirichlet model employs maximum entropy prior with bit-precision parameter N = 0 and a parametrized Mixed Dirichlet approximate posterior, the KL term in the ELBO is partly exact and partly estimated, in particular, the contribution of the Gibbs distributions over faces is computed in closed form.
Additional plots. Figure shows (SNE plots where each validation digit is encoded by a sample from the approximate posterior obtained by conditioning that digit. The continuous (Gaussian and Dirichlet) and mixed (Mixed Dirichlet) models learn to represent the digit rather well, with some plausible confusions by the Mixed Dirichlet model (e.g., 4, 7. and 9). The models that can only encode data using the vertices the simplex (Categorical and Gumbel-Softmax ST) typically map multiple digits to the same vertex of the simplex. For models whose priors support the vertices of the simplex (Mixed Dirichlet, Categorical, and Gumbel-Softmax ST), we can inspect what the vertices of the simplex typically map (in data space). Figure 6 shows 100 such samples per vertex. We also include samples from the Dirichlet model, though note that the Dirichlet prior does not support the vertices of the simplex, thus we sample from points very close the vertices (but in the relative interior of the simplex). We also report conditional and unconditional generations from each model in Figures 7 and 8
G.4 SIMPLEX- VALUED REGRESSION
Simplex-valued data are observations in the form of probability vectors, they appear in statistics in contexts such as time series (e.g., modeling polling data) and in machine learning in contexts such as knowledge distillation (e.g., teaching a compact network to predict the categorical distributions predicted by a larger network).
Available under the terms of the Creative Commons AttributionSharr Alike 3.0 license httpp::/ creat vecommens. org/1 cen /by- sa/3.07)
25

---

Published as a conference paper at ICLR 2022
2 3
0
2
3
0
/
2
3
0
2
3
0
/
3
5
6 7
8
9
5
6 7 8
9
5
6
7
8
9
8
6
9
8
6
9
g
Figure 7: Conditional generation. For each instance of each class in the validation set (note that none of the models has access to the label), we sample a latent code conditioned on the digit, and re-sample digit from the model. The illustration displays the pixel-wise average across all validation instances of the same class. From left-to-right: Gaussian, Dirichlet, Mixed Dirichlet, Categorical, Gumbel-Softman ST.
2
3
0
2
3
0
3
0
3
9
5
8
8
5
8
5
6
g
8
Figure 8: Unconditional generation. For each model, we sample a digit via ancestral sampling (i.e., is sampled from the prior, then, given is sampled from the observation model). We gather 5000 such samples and group them by class as predicted by a 5-nearest neighbour classifier trained on the MNIST training set (we use kd tree from scikit-learn), the classifier achieves 95% Fl on the MNIST test set. Each image in the illustration is a mexel-wise average of the samples in the cluster, also report the percentage of digits in each cluster. From left-to-right: Gaussian, Dirichlet, Mixed Dirichlet, Categorical, Gumbel-Softmn ST.
Data and task. We experiment with the UK election data setup by Gordon CaadonnRddriguez et al. (2020) Section 5.2) [ The UK electorate is partitioned into 650 constituencies, each electing member of parliament in a winner-takes-all vote. Hence the data are 650 observed vectors of proportions over the four major parties plus a 'remainder' category (i.e., each observation is a point in the A5-1 simplex, including its faces). Modeling simplex-valued data with the Dirichlet distribution is tricky for the Dirichlet does not support sparse outcomes. While pre-processing the data into the relative interior of the simplex is a simple strategy (e.g., add small positive noise to coordinates and renormalize), it is ineffective for the Dirichlet pdf either diverges or vanishes extrema (neighbourhoods of the liver-dimesiionll faces of the simplex). GGordnnRRodriunn et al.] (2020) document this and other difficulties in modeling with the Dirichlet likelihood function.
address these limitations they develop the continuous categoricul (CC) distribution, exponential family that supports the entire simplex (i.e., it assigns non nonz density to any point in the simplex), and does not diverge at the extrema. The CC enjoys various analytical properties, but it still cannot assign non-zero mass to the lower-dimensional faces of the simplex, thus while it is better choice of likelihood function than the Dirichlet, CC samples never truly sparse (thus test-time predictions always dense).
Architecture and hyperparameters. For Dirichlet and CC, we use a linear layer to map from the input predictors to 5 log-concentration parameters. For Mixed Dirichlet we use 2 linear layers: one maps from the input predictors to scores (clamped to [-10, 10)) which parametrize PF, the other maps to 5 strictly positive concentrations (we use softplus activations, with pre activations constrained to [-10, 10)) which parametrize /y F. We train all models using Adam with learning rate 0.1 and no weight decay for exactly 400 steps without mini-batching with 20% of the available data used for training. Following Gordon-RRdriguezz et al. (2020), we pre-process the data (add 10-3 to each coordinate and renormalize) for Dirichlet and CC, but this is not done for Mixed Dirichlet.
Results. Figure 9 compares the three choices of likelihood function. We report two prediction rules for our Mixed Dirichlet model. Sample mean: predict stochastically by drawing 100 samples and outputting the sample mean. Most probable mean: we predict deterministically by outputting the mean of the Dirichlet on the face that is assigned highest probability by the model (finding this most probable face is an operation that takes time O(K), and recall K = 5 in this task). We can see that Mixed Dirichlet does not suffer from the pathologies of the Dirichlet, due face stratification, and lowers test error bit more than CC (see Table 9). likely due to the ability to sample actual zeros.
httpp:://// hitpss:/ccmmonte aebrarrrr.. wawaaryyparllaa ament wtt.uk///eeeee.. www.erchhhbbbeeeigggeeeee wwt.bbbrreeeinnneeeeeeeeee 8749/
26

---

Published as a conference paper at ICLR 2022
Dir
CC
Mixed Dir (sample mean]
Dir probable mean)
MAE
0.1
50
100
152
250
500
150
100
163
200
253
765
600
Training
Training
Figure 9: Test error noot-mean-squar error on the left, mean absolute error on the right) of generalized linear regression towards 5 3-dimensional vote proportions (UK election data). We compare likelihood functions: Dirichlet, Continuous Categorical (CC), and Mixed Dirichlet.
The Mixed Dirichlet uses twice more parameters (we need to parametrize two components), but training time is barely affected (sampling and density assessments all linear in K), the training loss and its gradients are stable, and the algorithm converges just early CC's. As the Mixed Dirichlet produces sparse samples, it is interesting inspect how often it succeeds to predict whether an output coordinate is zero or not (i.e., whether yk 0, which true for 77.3% of the targets in the text set). The sample mean predicts whether >> O with macro F1 0.92, whereas the most probable mean achieves macro F1 0.94.
Model
RMSE
MAE
CC
0.1124
0.0847
Mixed Dirichlet sample mean
0,1030
0,0774
most probable mean
0.0987
0.0740
Table 4: Test root-mean-seee mean-square error and mean absolute error function of choice of likelihood function and prediction rule.
G.5 COMPUTING INFRASTRUCTURE
Our infrastructure consists of machines with the specifications shown in Table 5 The machines were used interchangeably. and all experiments were executed in a single GPU. Despite having machines with different specifications, we did not observe large differences in the execution time of our models across different machines.
Table 5: Computing infrastructure.
# GPU
CPU
4 Titan Xp 12GB
16 AMD Ryzen 1950X @ 3.40GHz 128GB
4 x GTX 1080 Ti
12GB
x Intel 17-9800X @ 3.80GHz- 128GB
3
2080 Ti- 12GB
12 AMD Ryzen 2920X 3.50GHz 128GB
4. 3 x RTX 2080 Ti- 12GB
12 x AMD Ryzen 2920X @ 3.50GHz 128GB
5. 2 x GTX Titan X 12GB
12 x Intel Xeon E-5 v3 @ 3.50GHz. 64 GB
27


Reviews:
[{"review_id": "warE_0QhIXg", "reviewer": "Reviewer_B5yQ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper was a pleasure to read. The idea is simple and intuitive, and it addresses a recurring issue with commonly-used simplex-valued distributions, allowing to better model sparcity while avoiding diverging likelihoods in the presence of 0s. The paper is clearly written, and the mathematical exposition is formal and well presented. I also believe the idea presented in this paper will be easily used to propose new simplex-valued distributions and be valuable to the community: while the authors propose several instances of mixed distributions, one can easily think of potential alternatives.\n\nMy only complaint about the paper is that the experiments focus mostly on sparcity, and not on avoiding ill-defined log-likelihoods, which I actually believe is another benefit of the proposed distributions. For example, the distribution proposed in [1], which is discussed in the paper, addresses learning $p(y|x)$ when $y$ is simplex-valued in the presence of 0s in the data; which can be handled in a more principled way through mixed distributions. Finally, the experiments are carried out against the Gumbel-Softmax, but not more recent improvements upon it, e.g. [2,3,4]. Nevertheless, I believe the contributions of the paper are enough to warrant publication.\n\nMinor things:\n\n-Section 3.1, when defining a face, I believe \"A face P is any intersection of P with a halfspace such that none of the interior points of P lie on the boundary of the halfspace\" should be replaced by \"A face P is any intersection of P with a closed halfspace such that none of the interior points of P lie on the boundary of the halfspace\" for added clarity.\n\n\n[1] The continuous categorial: A novel simplex-valued exponential family, Gordon-Rodriguez et al., ICML 2020\n\n[2] Estimating Gradients for Discrete Random Variables by Sampling Without Replacement, Kool et al., ICLR 2020\n\n[3] Invertible Gaussian Reparameterization: Revisiting the Gumbel-Softmax, Potapczyski et al., NeurIPS 2020\n\n[4] Rao-Blackwellizing the Straight-Through Gumbel-Softmax Gradient Estimator, Paulus et al., ICLR 2021\n\n========================================================================================================\n\nUPDATE 1 AFTER REBUTTAL\n\n========================================================================================================\n\nI have read the author's rebuttal as well as the other reviews, and my opinion on the paper remains that it should be accepted. I particularly appreciate the authors adding the suggested additional experiment to an already strong paper.", "summary_of_the_paper": "This paper proposes mixed distributions over convex polytopes such as the probability simplex. The proposed distributions are a discrete mixture over the faces of the polytope of \"continuous\" distributions on the corresponding face (formally, absolutely continuous wrt the Lebesgue measure on the face). For example, for the 2-simplex there is a distribution over the triangle, over each edge, and over each vertex. The authors formalize the dominating measure of these distributions, which they call the direct sum measure (given by the mixture over the counting measure over faces and Lebesgue measures on the faces); and derive formulas for entropy and KL divergences between such distributions, as well as characterizing maximum-entropy distributions.", "summary_of_the_review": "This paper proposes a well-motivated and mathematically elegant family of distributions on convex polytopes, allowing to place positive probability not only on the interior of the polytope, but on its faces as well. While there is room for improving the experiments, I believe the presented experiments and theoretical developments -- which can easily enable future work -- should be accepted.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ikK3ibntNvJ", "reviewer": "Reviewer_ePyA", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. The technique in this paper is very solid. I believe the theoretical foundations built in this paper will be useful in the future research on mixed random variables. \n\n2. The idea of this paper makes sense, and this paper is well written and easy to read. \n\n3. The experimental results demonstrate the usefulness of the proposed framework. \n\n\nWeaknesses:\n1. My only concern is that whether ICLR is a proper conference to publish this paper, since it involves some concepts about measure theory and probability theory.  \n", "summary_of_the_paper": "In this paper, the authors build rigorous theoretical foundations for mixed random variables. They first define a natural measure for mixed random variables, which looks like a direct sum. Then they define the entropy and KL divergence based on the proposed measure. They also give two strategies for representing and sampling mixed random variables. At last, they conduct some experiments to illustrate the usefulness of their framework.", "summary_of_the_review": "This paper is very solid. It builds some important theoretical foundations for the mixed random variables. It seems that this will be useful for the future research on this topic. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ZA359BgIhGR", "reviewer": "Reviewer_ASbt", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors are encouraged for their good work.\nThough they do try to introduce \"mixed variables\" (ie random variables consisted by both categorical and continuous data) there are a few unclear points: for example, they consider a (ie one) categorical variable with alphabet K. What is the case when there are more than one categorical points, with different alphabet (eg ordinal data with alphabet X) ?\nAlso, what is the algorithmic complexity with respect to the number of observations, the number/rate of continuous and discrete outcomes and the number of variables (in the case of multivariate mixed variable?\nFinally, Table 1 is quite unclear to me.", "summary_of_the_paper": "The authors wish to introduce a new kind of random variables that are consisted of both continuous and categorical data.\nThey also provide the corresponding theory and take into account information theoretical aspects.", "summary_of_the_review": "My main point is that the authors\n1) do not provide information regarding the case of having more than one discrete outcomes with different alphabets; \n2) what is the algorithmic complexity with respect to the number of observations, the rate of continuous vs discrete outcomes;\n3) they still work on the code and they do not provide it unless the paper is accepted", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "RDMBi-Fvfjh", "reviewer": "Reviewer_F3KA", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Reasons to accept:**\n- The paper is well-written and appears relevant to recent developments. \n- The motivation of the paper in relevance to bridging the continuous representations computed by neural networks and other machine learning models with the discrete representations that characterize humans is interesting. Table 1 nicely summarizes the contributions of this work.\n- The mathematical coverage also seems reasonable. \n- Extensive experiments on three different tasks, including an emergent communication benchmark. Supplementary material also provides qualitative examples.\n\n**Suggested improvements:**\n- Mixed random variables are a standard topic in probability theory. Definitions 1,2,3 appear to be natural in this theoretical context and the same applies to Propositions 1,2. More references in the mathematical and probability literature would be helpful due to the coverage of mixed random variables and associated theory in this literature. \n\n**Update after rebuttal:** I have read the reviews and the author's responses. This is solid well-written work, my review remains unchanged. ", "summary_of_the_paper": "This paper presents multidimensional extensions for mixed random variables originating from discrete-continuous hybrids based on truncation and rectification, which have been proposed for univariate distributions. The proposed extension replaces truncation by sparse projections to the simplex. The authors also propose a direct sum base measure definition on the face lattice of the probability simplex and intrinsic sampling strategies motivated by “manifold stratification”. Based on these introductions, new entropy and Kullback-Leibler divergence functions that subsume the discrete and differential cases and have interpretations in terms of code optimality, are presented. ", "summary_of_the_review": "This is a paper with mathematical rigor and connections to emergent communication, which is a current topic of interest for specific machine learning communities. Albeit the submission of this paper would better fit a probability and statistics venue, to the best of my knowledge, the definitions 1-3 and propositions 1,2 appear to be novel. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes mixed distributions over convex polytopes, and provides theory for mixed distributions that is relevant to the machine learning community. All of the reviewers were positive, and agree that this is a solid contribution. I agree, and I believe that this paper stands a chance of being a foundational paper for future work in probabilistic ML and structured learning.
Decision: Accept (Oral)