An Information Theoretic Approach to Distributed Representation Learning
Authors: Abdellatif Zaidi
Inaki Estella Aguerri
Venue:  (2020)
Keywords: Information Bottleneck
Distributed Learning

Abstract: The problem of distributed representation learning is one in which multiple sources of information X1,...,XK are processed separately so as to extract useful information about some statistically correlated ground truth Y. We investigate this problem from information-theoretic grounds. For both discrete memoryless (DM) and memoryless vector Gaussian models, we establish fundamental limits of learning in terms of optimal tradeoffs between accuracy and complexity. We also develop a variational bound on the optimal tradeoff that generalizes the evidence lower bound (ELBO) to the distributed setting. Furthermore, we provide a variational inference type algorithm that allows to compute this bound and in which the mappings are parametrized by neural networks and the bound approximated by Markov sampling and optimized with stochastic gradient descent. Experimental results on synthetic and real datasets are provided to support the efficiency of the approaches and algorithms which we develop in this paper.

Under review as a conference paper at ICLR 2020
AN INFORMATION THEORETIC APPROACH TO DIS- TRIBUTED REPRESENTATION LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
The problem of distributed representation learning is one in which multiple sources of information XI,..., .x are processed separately so as to extract useful information about some statistically correlated ground truth Y. We investigate this problem from information- theoretic grounds. For both discrete memoryless (DM) and memoryless vector Gaussian models, establish fundamental limits of learning in terms of optimal tradeoff's between relevance and complexity. We also develop a variational bound on the optimal tradeoff that generalizes the evidence lower bound (ELBO) to the distributed setting. Furthermore, we provide a variational inference type algorithm that allows to compute this bound and in which the mappings are parametrized by neural networks and the bound approximated by Markov sampling and optimized with stochastic gradient descent. Experimental results on synthetic and real datasets are provided support the efficiency the approaches and algorithms which we develop in this paper.
INTRODUCTION
Let a measurable variable X E X' and a target variable Y E y with unknown joint distribution Px.y be given. the classic problem of statistical learning, one wishes to infer an accurate predictor of the target variable Y ) based on observed realizations of X E X. That is, for a given class F of admissible predictors : - y and an additive loss function É y - y that measures discrepancies between true values their estimated fits, one aims at finding the mapping s* E F that minimizes the expected risk
Crx,y (6. 2) Epx, [<(Y, ((X))
Because the joint distribution Px,y unknown, in practice the risk equation (also called population risk) cannot be computed directly; and, in the standard approach, one usually resorts choosing the predictor with minimal risk on a training dataset consisting of n labeled samples (Z,, y+)}== that are drawn independently from the unknown joint distribution Px.y. Also, it is important restrict the set J of admissible predictors to low-complexity class to prevent overfitting. This leads to the abstract inference problem shown in Figure |
In this paper, study generalization of this problem in which the prediction is to be performed in distributed manner. The model is shown in Figure 2 Here, the prediction of the target variable Y E y is to be performed on the basis of samples of statistically correlated random variables (Xi,..., Xx) that observed each distinct predictor. We investigate this prohlem in the case in which the loss function E(-) is the logarithmic-loss fidelity measure, given by
(x 9) = log (VV)
(2)
where j(-) designates a probability distribution on )' and ((y) is the value of this distribution evaluated for the outcome y E y. The choice a 'good" loss function often controversial in statistical learning theory. and although complete and rigorous justification of the usage of logarithmic loss as fidelity measure in learning theory still awaited, partial explanations appeared in Jiao et al. (2015) and, especially in Painsky and Wornell (208)) where it is shown that, for binary classification problems, by minimizing the logarithmic-loss one actually minimizes an upper bound to any choice of loss function that smooth, proper (i.c., unbiased and Fisher consistent) and convex. Also, constrain the complexity of the predictors by using mutual information as a regularizer term. This is inline with recent works Xu and Raginsky (2017): y Russo and Zou (0155) that show that the generalization errrr can be upper-boundee using the mutual information between the input dataset and the output of the predictor - see also Bousquet and Elisseeff (2002); Shhlev-Shwartzz Shwartz et al. (2010) where the stabiliry of an algorithm is controlled by constraining the mutual information between its input and output.

---

Under review as a conference paper at ICLR 2020
Yey
Pri
X Ex
U o(X)
8
Figure 1: An abstract inference model for learning.
U
@1
Y Tey
Px....X
15
Pey
Xx
Ux
4%
Figure 2: A model for distributed, e.g., multi-view, learning.
1.1 AN EXAMPLE: MULTI- VIEW LEARNING
In many data analyties problems, data is collected from various sources of information feature extractors; and is intrinsically heterogeneous. For example, an image can be identified by its color or texture features; and document may contain text and images. Conventional machine learning approaches concatenate all available data into one big row vector (or matrix) on which a suitable algorithm is then applied. Treating different observations a single source might cause overfitting and not physically meaningful because each group of data may have different statistical properties. Alternatively, one may partition the data into groups according to samples homogeneity, and each group of data regarded as a separate view. This paradigm, termed multi-view view learning Xu et al. (2013), has received growing interest; and various algorithms exist, sometimes under references such as co-training Blum and Mitchell (1998); Dhillon et al. [2011)] Kumar and Daumé (2011): Gönen and Alpaydmn (201T), multiple kernel learning Gönen and Alpaydın (2011 and subspace learning Via et al. 2010).. By using distinct encoder mappings to represent distinct groups data, and jointly optimizing over all mappings to remove redundancy, multiview learning offers degree of flexibility that is not only desirable in practice but is likely to result in better learning capability. Actually, as shown in /apnik algorithmss produce less errors than global ones. Viewing the problem as that of function approximation, the intuition is that it is usually non-easy to find unique function that holds good predictability properties in the entire data space.
local learning
1.2 INFORMAL SUMMARY OF RESULTS
In this paper, first we characterize the optimal tradeoff between relevance and complexity for the distributed learning model of Figure 2 for both discrete memoryless (DM) and memoryless vector Gaussian models. While the result for the discrete data model (Theorem 1) is not difficult to establish using connections with Courtade and Weissman 2014 Appendix B) which we explicit here, the result for the multivariate Gaussian data model (Theorem 2), which provides sharp analytic characterization of optimal tradeoffss, is new and non-trivial (the proof the converse part not straightforward and was missing before this work in both thcoretic learning and information theoretic communities including in the scalar case). Second, we develop a variational bound on the optimal tradeoff that can be seen as a generalization of the ELBO and the 3- VAE criteria Higgins et al. (2016) to the distributed setting. Furthermore, for both DM and Gaussian models, also provide variational inference type algorithm which is parametrized by neural networks and allows to compute the developed variational bound when the data distribution is not known. Specifically, the main contributions of this paper are:
In Scction 3.2
we find an explicit analytic characterization of optimal tradcoff's between relevance and complexity for the memoryless vector Gaussian model. The result generalizes the Gaussian Information Bottleneck method of Globerson and Tishby (0044) Chechik al. (Feb. 2005) to the
distributed learning scenario.
we study the problem of maximizing relevance under a constraint on the sum complexity for which we establish variational bound which generalizes the ELBO and the 3- VAE criteria to the distributed setting.
In Section 3K
Section 3.4 is algorithmic-oriented. We develop variational inference type algorithm which enables to compute the bound. This algorithm is obtained by parametrizing the encoders, the decoder, and the prior distributions via DNNs and using Monte-Carlo sampling. Also, makes usage of Kingma
2

---

Under review as a conference paper at ICLR 2020
al.'s re-paramettization trick Kingma and Welling 2013 and can be generalization of the variational information bottleneck algorithm in Alemi et al. (0117) to the distributed setting.
Section contains some experimental results on real datasets which show the efficiency the approaches and algorithms that we develop in this paper.
Most relevant to this paper is the single-encoder Information Bottleneck (IB) method of TTishy et al. (1999) which readily and elegantly captures the above mentioned viewpoint seeking the right balance between data fit and generalization by using the mutual information both as a cost function and as regularizer term. Thus, the results of this paper can he seen as generalization of those Tishby et al. (1999) for DM model and Globerson and Tishby (2004); [Cechhik et al. (Feb. 2005) for the Gaussian model to the distributed learning setting.
Remark: Due to space constraints, the proofs of the results of this paper are deferred to the appendices section, which also contains additional experimental results.
1.3 NOTATION
Throughout, upper case letters denote random variables, e.g., X; lower case letters denote realizations of random variables, c.g., x; and calligraphic letters denote sets, c.g., X. The cardinality a set is denoted by [. For random variable X with probability mass function (pmf) Px. for short. Boldface upper case letters denote vectors or matrices, c.g., X, where context should make the distinction clear. For random variables (Xi, (2..... and a set of integers X. C N. Xx denotes the set of random variables with indices in the set K, i... Xx. (Xx l E K.). If = 0, Xx: = 0. For k E IC we let Xx/a (Xi...., XR-1, XA+1y...,y Xx), and assume that Xo Xx+1 0. Also, for zero-mean random vectors X and Y, the quantities Ex. Σx.y and Σx denote respectively the covariance matrix of the vector X. the covariance matric of vector (X, Y) and the conditional covariance matrix of X, conditionally on Y. Finally, for two probability measures Px and Qx on the random variable X E X. the relative entropy or Kullback-Leibler divergence denoted as DKL (Pxl(()
Px (x) p(z),
X
FORMAL PROBLEM FORMULATION
Let K 2 2 and (N....., Y) be a tuple of random variables with given joint probability mass function (pmf) Px.,...Xxx.1
(Z1,..., (2,,,.................. K y) (zi...., .xx) E Xr x... X Xx and y V. where X' designates the alphabet of Xx and y that of Y. Throughout, assume that the Markov chain
Xu Y Xx//
(3)
holds for all & E K. That is, the joint pmf factorizes as
K
Px.....g y (II,..., 2. Py (1) II Pxalr f=1l
The variable Y is target variable; and we seek to characterize how accurate it can be predicted from measurable random vector (X Xx) when the components of this vector are processed separately, each by a distinct encoder. More specifically, let ((, Xx... Y.) be collection of independent copies of (Xi...., Xx, Y). Encoder & E IC only observes the sequence XP; and generates a description Js = du (XF) according some mapping
de : XE M
where M(")) is an arbitrary set of descriptions. The range of allowable description sets will be specified below. decoder (b) collects all deseriptions Jx (J1,..., Jx) and returns an estimate y.n of y
+ M((a))) .... M 5"
(6)
The relevance of the estimation n is measured in terms of the relevance, detined here as the information that the descriptions di (XI),.... information
..,......, &x (X&) collectively preserve about Y". as measured by Shannon mutual
(m) (
Σ
P(w") II P(=El=") log Ply" (o) (xx)))
Ti 2
P(6() (xx))
IPx (Y"; 8'))
'Alternatively, the relevance could defined in a more operational manner by the average logarithmic loss distortion error Rexg. (Y" y²")] ]] H(""
3

---

Under review as a conference paper at ICLR 2020
where Yn = (db computed under the joint distribution Pxg.Y-
(Xx)) and the subscript Pxg.Y indicates that the mutual information is
There various ways control the complexity of the encoding functions [ In this paper. do by restricting their ranges. This is known as minimum description length complexity measure measure|Hinton and van Camp (193)). Specifically, the mapping (x)() at Encoder & K. needs to satisfy
Rx log (XE)l for all X X
(8)
Definition 1 A uulle (A, Ri....,RKy RR) is said to be achievable if there exists an integer n, a family of encoding mappings {6x ) and decoder mapping dp such that
A < T
(V" (X)))))))) ................................ (xxxxxx)))
(9)
R₁ 2 log (b (XE) for all E.K.
(10)
The relelence-ommplexity region IRDIB is given by the closure of all achievable tuples (A, Ri...., Rk)
In some cases, for given Rx (R1,-.., Rx), ) for the of the exposition we will be content with the relevance-complextty function ARRx, Pxx,Y) defined as
A Pxx,Y)= (6
max
(m) (Px,,))
(11)
where the maximization is subjected equation
MAIN RESULTS
3.1
DISCRETE MEMORYLESS DATA MODEL
The following theorem (the proof of which can be found in the appendices section) provides a computable characterization the relevance-complexity region IRDIB. The result can be seen as generalization of Tishby al. Tishby al. (1999) single encoder IB to the distributed learning model with K encoders.
Theorem 1 The eelevance-complesi region IRDIS of the distributed learning problem with Pxg.y for which the Markov chain equation 3 holds is given by the union of all tuples (A, R,...., Rx) ) RK+1 that satisfy for all S S. K.
A < Σ Ru 100-I(Xx 1.Y;
(12)
KES
for some set of pmfs P :== (PuLIxe
PULIXE Pr} with joint distribution of the form
K
K
Pr(t)Pv()) II (xxl( II PULIX (Mexxxt t)-
(13)
Remark 1 Theorem 7 the random variable T stands for a convexification of the region, i.e., convex combination of achievable relevance-complexigy tuples is itself achievable. For given T t, the result of Theoren comprises the optimization over K. conditional distributions (Pug) For k E KC, the conditional distribution Púx Xx represents stochastic encoding of the feature Xx. into latent variable Us. Intuitively, the latent variable Ux should capture all relevant information about Y that is contained in X₃ and non redundant with those carried by (Us)sęx. The requirement of renrredundancy is mandated by the need to operate the minimum possible complexity at which desired relevance level is achievable (recall that minimum complexity, as expressed by algorithm's input-output mutual information, translates directly a better generalization capability). Collectively, however, the set of all latent variables (U1.... "i1...,Uxs, Ukk should be expressive enough to reproduce target variable Y within the desired relevance level.
Like for the single-encoder 1B problem of Tishby et al. (1999) and increasing number of works that followed, including Courtade and Weissman (2014) Section IIIFF), our approach here asymptotic. In addition that it leads an exact characterization. the result also readily provides lower bound the performance in the non-asyne asyptsyptooicc (e.g.. one shot) seying. For the latter setting known approaches (e.g.. the functional representation lemma of and El Gamal (2018) would lead to only non-matching inner and outer bounds on the region of optimal tradeoff pairs, as this is the case even for the single encoder case Li et al. (018))
Remark 2

---

Under review as a conference paper at ICLR 2020
3.2 MEMORYLESS VECTOR GAUSSIAN DATA MODEL
We now turn to a continuous-alphabet setting. Here, ,I....,,.. XK, Y) is zero-mean Gaussian random vector such that
X HLY+N for all & E,K,
(14)
where Hy C* models the linear model connecting the target variable Y Ca to the observation at encoder k, and N& E C k = I...., K, is the noise vector at encoder k, assumed to be Gaussian with zero-mean and covariance matrix Σt, and independent from all other noises and the target variable Y. We denote by Σy the covariance matrix of of the target vector Y E C"v,
For this model, we find an explicit analytic characterization of optimal tradcoff's hetween relevance and complex- ity. The proof relies on deriving an outer bound on the region described by equation 12 and showing that it is achievable with Gaussian distribution, with time-sharing. In doing so, use techniques that rely on the de Bruijn identity and the properties of Fisher information and minimum mean square error (MMSE).
Theorem 2 The relevance complexxyy region IR for the vector Gaussian model is given by the union of all tuples (A, ......, R1) that satisfy for all S S K
A S
Pk + log I Σx
1/
S2 2
1/2
+ log Σ /
Ey H.
+I
kEge
for 0 3 Zz 2
The proof of the direct part follows by evaluating the region of Theorem I which can he extended to the case continuous alphabets using standard discretization (quantization) arguments, with choices T 0 and p(ue) Xx. f) C,, / (St I) ). The main contribution in the proof is that of the converse part. This proof is technical and rather lengthy and, for this reason, is deferred to the appendices section.
Proof:
In the special case in which K 1, the result of Theorem 2 recovers that by Gloherson and Tishby (2004 (see ChechCh ct al. (Feb. 2005)) which establishes the optimal relevance-complexity tradcoff of the single-encoder Gaussian IB problem.
3.3 A VARIATIONAL BOUND
In this section, we consider the problem of learning encoders- and decoder mappings that maximize the relevance level given (fixed) complexity level, i.e., those that perform at the vicinity of the boundary the region IRDB. First, we derive parametrization of the relevance-complexity region; and, then, we develop variational bound which expresses the optimal encoders' and decoder mappings as the solution to an optimization prohlem (an algorithm for solving this problem in the case of unknown distributions is given in the next section).
Roum = Ek_ Rg. Also, let IR denote the region of achievable (relevance, sum-complexity) pairs,
IRDIB sum
= (AA,Runs E RR: 3(B....., E R s.l.
(A, Ri...., Rx) E IRDIB and Rs Roum
Proposition The relevance-con compleexty region under cum-complexity constraint RIDIB given by the comve- hull of all tuples (A, Reum) E R², satisfying A S A(Ram, Pxg.Y) where
A A(Reum PXx,)) max min I(Y;Uk)), Roum A
(15)
and where the maximization is over the set of pmfss P = (Pu, ......., Puxixx3 such that the joint pmf factorizes as pr (y))) 11 PX,IY (xx (y) 11 PURIX2 x.).
The next proposition provides characterization of the pairs (A, Rewm) that lie on the boundary of R.Zuum in terms of nonnegative parameter 2 O.
For every pair (A, Reum) R2 that lies on the boundary of the relevance-compledi region RIzuu there exist 2 0 such that (A, Roum) = (A., Ro), where
Proposition 2
A. (1+
(1 sK)H(Y) + sR. max L. LP()
(16)
5

---

Under review as a conference paper at ICLR 2020
K R. 1(YUU) + - UY
(17)
and P is the set of conditional pmfss P that maximize the cost function
C. = -<(UUUk)) (ire))-sEH(HTOAA (IH(TIU))+ I(Xx;
(18)
Using Proposition 2 is clear that the encoders x heek that achieve the colplext pair (A., R.) can be computed by maximizing the regularized cost equation 18 for the corresponding value of # 2 0. The corresponding optimal decoder PYiux for these encoders can be found as in equation ??. Different
relevance-complexity pairs (A., R.) the boundary of IRDIE and encoders- and decoder mappings that achieve it can found by solving equation 18 for different values of and then evaluating equation [6] and equation 17 for the obtained solution.
The optimization of equation 18 generally requires to compute marginal distributions involving the descriptions Ui...., UK. aspect which can be non-casy computationally costly. To overcome this limitation, in the following we derive tight variational bound on C, (P) which lower bounds the DIB cost function with respect some arbitrary distributions. Let consider the arbitrary decoder QU..uu (y/ (y/ui.... ... ux) for E V un .4...., uK Uk. the K decoders Qyju, (y (uy. for E KC for E V, us Uu. and latent variable priors QU (ux), & E K., us Ux. For short, we denote
Q = {Qy1U Un QYIU,..... QYIUK Qukk
define the variational DIB cost function L (P.(Q
as
K
VB (P.C Q) == Blllog Qr (YIUR)
+
Ellog Qriu, (Y ) - Dxu (PoIIxxxl1)) (19)
Iogarithmii loss
regulasizer
The following lemma states that [P (P. Q) is lower bound to (P) for all distributions Q.
Lemma For fixed pmffs P, we have
C. 2 L. (P,
(P,Q).
for all pmfs Q-
(20)
In addition, there exists a unique Q that achieves the maximum maxQ C. (P, CP(P), and is given by
Qu Piee Qrii PriEe ==
(21)
QYiu...., = PPI
(22)
where Püx. PYIU, and Pyju,....Ux computed from the pmfss P.
Using the above, the optimization in equation 16 can be written in terms of the variational DIB cost function as
max (P) max max L. ( (P,)) Q).
(23)
We close this section by noting that the cost function equation 19 can be seen as generalization of the evidence lower bound (ELBO) as given Rezende al. (2014); Kingma and Welling (2013) for single-encoder learning to the distributed setting. Also, in the specific case in which Y = (X1, , Xx) ) the bound generalizes the ELBO used for VAEs to the case an arbitrary number encoders.
3.4 CASE UNKNOWN DISTRIBUTIONS: VARIATIONAL DISTRIBUTED IB ALGORITHM
In practicc only set of training samples ((Xi... Xx.,, H available. In this section, provide a method to optimize equation 23 in this case by parametrizing the encoding and decoding distributions that optimize using family of distributions whose parameters determined by Deep Neural networks (DNNs). This allows us to formulate equation 23 in terms of the DNN parameters and optimize by using the reparametrization trick Kingma and Welling (203). Monte Carlo sampling, well stochastic gradient descent (SGD) type algorithms.
Let F&N, denote the parametric family of encoding probability distributions Peyix, over Ud for each element Xt.. Each memher of this collection, Pot (xergg described by parameter vector of E Ti C RR where

---

Under review as a conference paper at ICLR 2020
Tž S IR' denotes the set of allowable parameter vectors. The parameter vector 74 is the output of DNN J0p : d' + It,, with network parameters 02 E O S R e.g., the weights of the network at all layers. The DNN fag takes Xx imput and outputs the parameter vector 7x- determining one of the probability members
Pue Xxmt- We have
FNNNA = (oe (ux (x)) for E U42 vx E dx rh = for (x*), Os E O
(24)
For example, the family of multivariate Gaussian distributions is parametrized by the mean fa and covariance matrix M ie.. Y = (mg, > Therefore, given an observation Xx. T = (uf. xX) determined by the output of the DNN fo, and FRN, & is given by PuglX (ux] } (u+s p E.).
Similarly, for decoders Qy over V. define the family of distributions parametrized by a vector 1 S R determined by the output of a DNN for Uu + Th with paramcters Φx 4b C R as
N (Qr (vluz), for y us E Eu: 7h fox (ux): de E Dz).
(25)
and for the distribution QYiue over y for each element in x Ux. define family of distributions parameterized by the output of the DNN før .11 x ... x Ux + T with ox E tx C R and r S R
FNN, -QQ1UU (y (y/lu,...., ux), y us Us : 7k Søx (wi..... ux) ox x0x) (26)
Finally, for the distributions Qu (ua) we define the family of distributions with parameter pp: Vx C R
FNNA = {Quesee (us), for us E Ue : E Vu)
the following, for brevity we use P'. (ux (ax). ). Qin (w/wx), Que (vlux) and Qua (ux) to denote the distribu- tions parametrized by the DNNs feg. fwer fex and yo. respectively.
By restricting the optimization of the variational DIB cost equation within the families of distributions FNN FNN FNNAA
23 to the encoder, decoder and priors
F
NN,A
get
max max L. VB (P.
max L. (0, v) 0.0.0
(27)
where the notation 9
[0 -0..... 0x).
(o), ....... ox ox] and 40 [p1,..., spx] to denote the
DNN and prior parameters and, the cost in equation
is given by
NN C (0, cb sp) == Ery.x Pep ((x1x2))
log Qox (YUUk)
log Qu. (Ku, (U* (U) (U.)))
(28)
Next, we train the DNNs to maximize a Monte Carlo approximation of equation 27 over 0, d. s using SGD. We use the reparameterization trick Kingma and Welling (2013), to sample from 1%,, (U [Xt). In particular, we consider N to consist of a parametric family of distributions that can be sampled by first sampling random variable Z& with distribution Pz, (=k), Zi Za and then transforming the samples using some function go, X x Zx - Lie parameterized by 0g. such that U = go, (Ix, Zx) ~ Po. (U& (xx). The reparametrization trick reduces the original optimization to estimating 6% the deterministic function go, and allows to compute estimates of the gradient using backpropagation Kingma and Welling (2013). The variational DIB cost in equation can be approximated, by sampling ma independent samples {u,.,,]}= Po. (ua (x) for each training sample (.n,,...., LK,,, 3s). = I....,n. Sampling is performed by using Wk.6.j = gos (5R,,, #k.)) with (=r,) 1 i.i.d. sampled from Pz.- . We then have
C (0. (.))))) = 77
log Qox (Dvl Millu.id....... (.,K,)..,
K
(lo Qop (uilumee)) DKL (Pa.. (Un.)))
(29)
1m km
4 EXPERIMENTS, RESILIENCE TO NOISE, ROTATION AND OCCLUSION
In this experiment, we test the robustness of our method against noise, rotation and random occlusion on the MNIST dataset. Specifically, combine two types of random occlusions: the first encoder observes digit from the MNIST that is occluded by square which is rotated randomly (rotation angle uniformly distributed over [-45", 45%)); and the second encoder observes noisy version of the same digit cormupted by additive noise
7

---

Under review as a conference paper at ICLR 2020
(noise level uniform between 0 and 3). The noisy pixels clipped between 0 and 1, with than 60% of the pixels occluded. These occlusions make the problem significantly more involved than the standard MNIST (for which application of our algorithm leads relevance of about 99.9%).
We considered a CNN deterministic networks with dropout which achieves a 99.8% for test data on the clean MNIST data. Then, have trained the same CNN architecture for each of the noisy inputs the encoders, resulting in a relevance of 92.1% % from the input to encoder (randomly rotated occlusion) and 79.68% from the input to encoder (noisy clipped image).
CNN Layers
Original y
Olsarvation X
Observation X,
Encoder k:
conv. ker. [5,5,32]-Red)
maxpool [2,2,2]
3
conv. kcr. [5,5, 15.5,64]RRel...
maxpool [2,2,2]
dense [1024]-ReLu
dropout 0.4
dense [2.56]-relu
Latent space k Decoder
dense [256]-ReLL
dense [256]-Rela
Decoder k
dense --56l
Figure 3: View I: occluded. View 2: noisy.
Table I: Used CNN architecture.
relevance (%) shot
Bolownro
avg.
D-VIB
96.16
97.24
0.-
D-VIB-noRee
96.04
96.72
C-VIB
96.01
96.68
Deterministic CNN
93.18
93.18
Comppeiity
Independent CNNs
92.1 79.68
93.1 82.01
Figure 4: relevance v.s. complexity for = 50.000 and E [10-10,1] .1..
Table 2: Achieved relevance levels.
We applied our VIB algorithm of Section 3.4 to this model with the CNN architecture of Table in which Encoder = 1, 2 is parametrized by an nug 256 dimensional multivariate Gaussian distribution N(ug >j) determined by the ouiput of DNN fe, consisting of the concatenation of convolution, dense and maxpool layers with ReLu activations and dropout. The output of the last layer is followed by dense layer without activation that generate f4 and Mg. The prior chosen Q (u) A/ I). Each decoder takes the samples from Po., (U& X&) and processes its inputs with a dense layer DNN (fex and fe.) each with 256 neurons and ReLu activation, which outputs a vector ý, of size |}]| = 10 normalized with softmax, corresponding distribution over the one- hot encoding of the digit labels (O...., , from the K observations.
Qun (yx (ux) Softmax(Sor, (Ux)), k == 1,2, and Qøc (y jux) = Softmax(fag (U1, U2))),
(30)
(31)
where Softmax(p) (p) for p E IR is vector with i-th entry as [Softmax(p)) = exp(p:) exp(ps).
Figure shows the dievance-complexx
tradcoffs obtained using our D-VIB algorithm Section [.4 with
50.000 and 15 distinct s-values randomly chosen in the range [10 ,1]. For comparison, we also present the performance obtained using three methods among ofate-of the-art multiview learning approaches: (i) applying deterministic CNN on the two views concatenated (deterministic CNN), (ii) applying the single- encoder variational IB method Alemi al. on the two views concatenated (C- (C-VTB), and (iii) learning one function for each view via a distinct CNNs and optimize all CNNs independently (independent CNNs). The For other experimental results, see the appendices section.
achieved relevance is reported in Table 2
We also mention that at high level algorithm D-VIB can be considered performing some form of co- regularization (for instance its Gaussian version is similar to the CCA of Hardonn et al. (2004). Comparatively, the single-vier view algorithm C-VIB can be viewed belonging the family of co-training style algorithms (such as the co-EM Nigam and Ghani (000)) which, as mentioned in the recent survey Zhao et al. (2017). override on single-view algorithms. The performance of D-VIB dominates that of C-VIB, which itself dominates co-EM.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
A. Alemi, Fischer. J. Dillon, and K. Murphy. Deep variational information bottleneck. In ICLR, 2017. URL
https:://arxiv.rrg/aaat wwwpaa/s/.l.......// 00410
R. Blahut. Computation channel capacity and rate-distortion functions. IEEE Transactions on Information Theory, 18(4):460-473, Jul 1972. ISSN 0018-9444)
Blum and T. Mitchell. Combining labeled and unlabeled data with co-training. Proceedings of eleventh annual conference on Computational learning theory, pages 92-100. ACM, 1998.
O. Bousquet and Elisseeff. Stability and generalization. Journal of Machine Learning Research, 2(Mar): 499-526, 2002.
G. Chechik, A. Globerson, N. Tishby, and Y. Weiss. Information bottleneck for Gaussian variables. Journal of Machine Learning Research, 6:165-188, Feb, 2005.
T. A. Courtade and T. Weissman. Multiterminal source coding under logarithmic loss. IEEE Transactions on Information Theory, 60(1):740-761, Jan. 2014. ISSN 0018-9448. doi: 10.1 2212197
T. M. Cover and J. A. Thomas. Elements of Information Theory. Iiley--nterscciene,, 1991.
A. Dembo, T. M. Cover, and A. Thomas. Information theoretic inequalities. IEEE Trans, Inf. Theory, (6):1501-1518, Nov 1991. ISSN 0018-9448 doi: 10. 1009918/ 104312.
A. P. Dempster, N. M. I Laird, and D. B. Rubin. Maximum likelihond from incomplete data via the algorithm. Journal of the Royal Statistical Society, Series B, 39(1):1-38, 1977.
P. Dhillon, D. P. Foster, and L. H. Ungar. Multi-view learning of word embeddings via CCA. In Advances in neural information processing systems, pages 199-207, 2011.
R. Dobrushin and B. Tsybakov. Information transmission with additional noise. IRE Transactions on Information Theory, 8(5):293-304, Sep. 1962. ISSN 0096-1000 doi: 10. 10.1109/IT.1962.1555777
Ekrem and S. Ulukus. An outer bound for the vector Gaussian CEO problem. JEEE Transactions on Information Theory, 60(1)), Nov. 2014.
Gamal and Y.-H. Kim. Network information theory. Cambridge University Press, 2011.
A. Globerson and N. Tishby. On the optimality of the Gaussian information bottleneck curve. Hebrew University Tech. Report, 2004.
M. Gönen and Alpaydin. Multiple kernel learning algorithms. Journal of Machine Learning Research, 12 (lul):221122266 2011.
Hardoon, S. Szedmak, and Taylor. Canonical correlation analysis: overview with application to learning methods. Neur. Comput., 16:2639-266 2004.
Harremoes and N. Tishby. The information bottleneck revisited how choose good distortion measure. In Proc. IEFE Int. Symp. Information Theory, pages 566-570, Jun. 2007.
I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Lerchner. --vac: Learning basic visual concepts with constrained variational framework. 2016.
G. E. Hinton and D. van Camp. Keeping the neural networks simple by minimizing the description length of weights. In Proceedings of the Sixth Anmual Conference on Computational Learning Theory, COLT '93, pages 5-13, New York, NY, USA, 1993. ACM. ISBN 0-89791-611 1-5. doi: 00.11455/68334.166888 URL. http::/dde acm. org 10. 11411144511 683044 168306
Y. Jia, M. Salzmann, and T. Darrell. Factorized latent spaces with structured sparsity. In Advances in Neural Information Processing Systems, pages 982-990, 2010.
J. Jiao, T. A. Courtade, K. Venkat, and T. Weissman. Justification of logarithmic loss via the benefit of side information. IEEF Transactions Information Theory, 61(10):335-53365 2015.

---

Under review as a conference paper at ICLR 2020
D. P. Kingma and M. Welling, Auto-encoding variational bayes. CoRR, abs/1312.6114, 2013. URL http:
// dblp. uni-trier /ttpd///////// htt.aaaas//////oooaaaaas/ss////s htt.://aaa.aarrooooo3........aa.
D. P. Kingma and M. Welling, Auto-Encoding Variational Bayes, ArXiv e-prints, Dec. 2013.
A. Kumar and H. Daumé. A co-training approach for multi-view spectral clustering. In Proceedings of the 28th International Conference on Machine Learning (ICML-I1), pages 393-400, 2011.
C.-T. A. El Gamal. Strong functional representation lemma and applications to coding theorems. IEEE Transactions Information Theory, 64( 1):6967-669777 2018.
C.-T. Li. X. Wu, A. Ozgur, and A. El Gamal. Minimax learning for remote prediction. In
http:://arziv.rr/bbs///be wttp:///aaas///.00000000001 May. 2018.
K. Nigam and R. Ghani. Analyzing the effectiveness and applicability of co-training. In Proc. of the 9th International Conference of Information and Knowledge Management, pages 86-93, 2000.
Painsky and W. Wornell, the universality the logistic function. arXiv preprint arXiv: 1805.03804, 2018.
M. Razaviyayn, M. Hong, and Z.-Q. Luo. A unified convergence analysis of block successive minimization methods for nonsmooth optimization. SIAM J. on Opt., 23:1126-1153, Jun. 2013.
D. J. Rezende, S. Mohamed, and D. Wierstra. Stochastic backpropagation and approximate inference in deep Xing and Jebara, editors, Proceedings of the 31st International Conference on Machine Learning, volume 32 of Proceedings of Machine Learning Research, pages 1278-1281 11886, Bejing, China, 22-24 Jun 2014. PMI.R.
generative models.
D. Russo and J. Zou. How much does your data exploration overfit? controlling bias via information usage. arXiv preprint arXiv:1511.05219 2015.
Shalev-Shwartz, O. Shamir, N. Srebro, and K. Sridharan. Learnability, stability and uniform convergence. Journal of Machine Learning Research, 11(0ct):2635 2670, 2010.
N. Tishby, F. C. Pereira, and W. Bialek. The information bottleneck method. In Proc. 37th Annual Allerton Conf. on Comm., Control, and Computing, pages 368-377, 1999.
Vapnik. The nature of statistical learning theory. Springer science business media, 2013.
A. Xu and M. Raginsky. Information--heerrtiicc analysis of generalization capability of learning algorithms. In Advances in Neural Information Processing Systems, pages 2521-2530 2017.
C. Xu, D. Tao, and C. Xu. survey on multi-viet lcarning. arXiv preprint vrXiv:1304.500 2013.
Zhao, Xie, Xu, and Sun. Multi- learning overview: recent progress new challenges. Information Fusion, 38:43-54, 2017.
10

---

Under review as a conference paper at ICLR 2020
APPENDICES
PROOFS OF MAIN THEOREMS, PROPOSITIONS AND LEMMAS. ADDITIONAL EXPERIMENTAL RESULTS
5 PROOFS OF MAIN THEOREMS, PROPOSITIONS AND LEMMAS
5.1 AUXILIARY LEMMAS
Lemma 2 Dembo et al. 1991
Ekrem and Ulukus 2014 Let (X, Y) be pair of random vectors with pmf
p(x,y) We have
log [TT)JJ (x| & ( S log ((re)
where the conditional Fischer information matrix defined
J(X Y) E log Vogg)))lll)
and the minimun mean squared error (MMSE) matrix is
misse Y = E[(x (X -EEI(XXEEXXNN E|X
Lemma Ekrem and Ulukus 2014 Let (V1, V2) be a random vector with finite second moments and (V1, 1, V2). Then
N~CN((, Ex) independent
mmse 22 V1, V2 + N)
EN +
5.2 PROOF OF THEOREM
the distributed learning problem that we study boils down to known Information Bottleneck (IB) problem of Tishby al. (999). The single-eneoder TB problem is essentially remote point-to-point source coding problem Dobrushin Tsybakov (1962) which distortion measured under logarithm loss fidelity criterion Harremoes and Tishby (2007). In accordance with this analogy, for K > 2 consider the multiterminal source coding problem under logarithmic loss in which the sequence Y models a remote source that observed by K spatially distributed agents; the agents observe noisy versions of the remote source and communicate independently with decoder or Chief Executive Officer (CEO) over rate-constrained noise-free links. For instance, agent k, K. observes X and uses R2 bits per sample describe it the decoder. The decoder wants to reconstruct the remote source Y to within prescribed fidelity level, where incurred distortion is measured using the logarithmic loss criterion, i.c.,
(1o2 (²))
log Py ox (x
(32)
where J = (2I( ox (XR)).
Here, , XR, Y is assumed to be distributed i.i.d. according to the n-product of the pmf Px Xx.Y.
i.e. the Markov chain qquationnBoodd
Definition rate-distortion code (of blocklength 11) for the CEO problem consists of K encoding functions
Φk X {1,..., M(")), for k 1,...,
(33)
11

---

Under review as a conference paper at ICLR 2020
and decoding function
{L, BL...,,M"1. x x (L. M.. - 5mm
(34)
A distortion-rate tuple (D, Ri...., Rk) is achievable for the DM CEO source coding problem with side information if there exist blocklength n, encoding functions { k= and decoding function & such that
Rs log M for k 1....., K
D E fiog (Y", ( (XT),.... .x (XR)))]
The distortion-rate region DReEo of the CEO model is defined as the closure of all non-negative tuples (D, Ri...., Rg) ) that are achievable.
n
Key to the proof of Theorem the following proposition which states that IRDIB and DREBO can be inferred from each other.
Proposition 3 (A, R1, Ri..... Rx) E IRDIS if and only if (H(Y) A, RI. RI)) E DRcBO.
Proof: Let, for I,..., K, Ju = (x(x)) and J = (J1...., Jx). Then,
E[Liog(" "=" E
P(y"li) log
(35)
P(y
3"Ey"
P(w"
E Ply Piy" log y"Ey"
+ +H(Y"JJ
(36)
P(y
(Ru((Pyy()))) (()) ) + H(Y")J
(37)
H(Y"JJ= =),
(38)
where equation 38 is due to the non-negativity of the Baalbbcck-Liibbee divergence and the oquality holds if and only if (y b)
Pjy"lj) where 1)) Pr(Y 1 = j} all j and y" y
Let an achievable tuple (A, Ri..... ..... Rx) E IRDI be given. Then, there must exist functions { J
such that equation and equation 10 hold. Using equation 38 that by letting decoding function (Jx) [Pynjuic [J]]]] we have E[Giog (Y"" Jx] (I" Jxc, which implies (II(Y) R1, Rx) DRcEO-
The result of Theorem | follows easily by combining (Courtade and Weissman 2014 Theorem 10). which provides single-lctter characterization of the rate distortion region DR the CHO problem, and Proposi-
tion 3
5.3 PROOF OF THEOREM
The proof of the direct part Thcorem 2 follows by evaluating the region Theorem I with the choice T (St 2
and p(u Xx, t) = C. (xx,, E
!/3)
The proof of the converse part is as follows. Fix t E T.S C K. and family of distributions (p(us) (okuu//e,20 k
such that the joint distribution factorizes as equation 13 Also, let St I E." and
mmse Xx|Y, Uk., En
(39)
Such au always exists since
mmse X Y, Uk.,, t) Ex
(40)
Then, have
I(X,; Y, 2 log Σx| log jamnsee Xx Y. Uk,.. !)
log E Fh
/ -., 1/2/.
(41)
where the inequality is due to Lemma 2 and equation 41 is due to equation
39
Also, we have
T(Y: (t) log Eyl log J J (Y
(42)
12

---

Under review as a conference paper at ICLR 2020
log
E1/" ,
(43)
where equation 42 follows by using Lemma
and equation 43 holds by using the following equality
s = E
H. Zy
(44)
kege
the proof of which uses a connection between MMSE and Fisher information as shown next.
For the proof of equation
44
first that from the MMSE estimation of Gaussian random vectors El Gamal
and Kim 2011 we have
Y = E Zg=
Σ
GaXx Zse,
(45)
KESt
where Ga Eylege
H Eg and Zg ~ CN((0, Zylxge with
Zyige =yy
Σ H
(46)
KESc
Note that Zge is independent of y= due to the orthogonality principle of the MMSE and its Gaussian distribution. Hence, it also independent of Us We have
mmse
Σ G. Y, Use.nt) Use.t AE
Σ
Gummse (Xul Y, Us G
(47)
kES
Dy xgo
Σ keSc
H (E - -22) H Dyicse
(48)
where equation 47 follows since the terms due the Markov chain (Uk,1Xx)) - (UK/k, Xx/k): and equation 48 follows due to equation 39 and Gu. Finally,
s( Eylkge yixxe mmse
Σ
G.Xe Use..))) Eylsse
(49)
BES
-Zylkge
H [E
su...) H
(50)
KESE
Ey + Σ
(51)
k
where equation 49 due to Lemma equation 46
equation 50 is due to equation 48 and equation
51 follows due to
Now, let Ωr EET (() The rest of the converse proof follows by averaging over the time sharing random variable to gel
1Xx: UxX,112)))22 - Σ p(1) log 11 x 1//2 22,,
2 log I > 22
(52)
where equation 52 follows from concavity of the log-det function Jensen's inequality. Similarly to equation 52 from equation 43 and Jensen's Inequality we have
I(Y; Us: < log Σ M H. +1 keSc
(53)
Finally, using equation 52 and equation in equation ??, noting that Sr Eier Pht) 3 Σ since A Zu,, I Σx1 and taking the union over Sr satisfying 0 A Sr I Ex completes the proof of the converse part; and, hence, that of Theorem
53
13

---

Under review as a conference paper at ICLR 2020
5.4 PROOF OF PROPOSITION
1
For simplicity of exposition, the proof is given for the case K = 2 encoders. The proof for K > 2 follows similarly. By the definition of IIB the accuracy complexity tuple (A, Raum) R4 achievable for random variables Y, XI, U1, U2 with joint pmf satisfying equation 13 if holds that
A S U(Y;U1,Us)
(54)
A Rg - 1Xx;
(55)
A S R2 - -
(56)
A R₂ + - T(X2;
(57)
R1 + R2 S Raum-
(58)
The application of the Fourier-Moizkin elimination to project out Rí and R2 reduces the system on inequalities
equation equation 58 the following system inequalities
A T(Y; U1,
(59)
A Raum
(60)
2A < Asum - 1(x1; -[X:1 +
((Y;U))))))
(61)
follows due to the Markov chain U1 + X1 ee++++ + Xx -- U2 that we have [YYUU, U2) - -1)
Therefore, inequality equation 61 is redundant as it is implied by equation and equation 60 This completes the proof of Proposition
5.5 PROOF OF PROPOSITION 2
Suppose that P yields the maximum in equation 16 Then.
(1 a)A, (1 + *K)H(Y)) RR, + LP(P")
(62)
(1++K)H(()))) + sRs + (HH(YUU)) . (HHIUU + I(X:; UE)
(63)
(1+ (L+) sR. + (-H(Y - sSR, 1(Y:U)) + KH({)))
(64)
(++))[(YUU))
(65)
(1 + s) Px..)),
(66)
where equation 63 is due the definition of C. (P) in equation 18: equation 64 follows since have Ex (T(X;; UN) H( = R₂ 7(Y; U2) + K (YY) from the definition of R, in equation 17 and
equation 66 follows from the definition equation ??.
Conversely, P is the solution to the maximization in the function A ((Rumm Px in equation ?? such that A(Raum: PX A.. then A, < ((Y;UR) and A, < Raum Etm 1(X2s UZ Y) and we have, for any $ 2 0, that
A A(Ream, Px A.
$ A, 1A,-(4. I((;UR)) - A. Roum T(xx:
((Y;UZ) - -sA, + sRam
1(X6:U71)
H(Y) sA. sHamm H(YYU
U H(M sKH(Y)
k=
(67)
H(Y) =A, + &Reum + C: + +KH((
(68)
H(Y) =A, + sRaum + *KH((Y) - ((1 +K)H()))+ ())
(69)
A. + s R(Rum Rs),
(70)
where in equation 67
have >K D I(xx; Y) KHH(Y ((,;UX)) H(Y Us) due to the Markov chain Us X1 Y (Xxre: URAA): equation follows since L. is the maximum over all possible
distributions P (not necessarily P maximizing A (Rrum: Pxx. D) and equation 69 is due equation 16
14

---

Under review as a conference paper at ICLR 2020
Finally, equation 70 is valid for any Raum 0 and Given 8, and hence (A., R.). choosing R.
yields ARR, v I A. Together with equation 66 this completes the proof of Proposition 2
5.6 PROOF OF LEMMA
The proof follows by deriving the following bounds. For any conditional pmf Qriz(ulz)) y y and E 2, e.g. Ux or Mx. proceeding similarly to equation 38 and averaging over Z, we have
H(Y Z) = E(- log Qriz
(71)
Similarly, we have
U()::Uh) = H(U)) - H(Uu(Xb)
(72)
=Il QU (U+)] - (Ru ( (X.(UR)
(73)
(DI,(Py|U, DRu (1Qua)
(74)
Thus, we get
C.(P) = VB (P, Q) DKL (PI1(Pr1ee)00010) + Σ (DKu (Qr1o + DELL (Qu,))
2 L. (P,))
(75)
where equation 75 holds by the non-nigativity of relative entropy: and the equality is met if and only if Q* is as given by equation Iaadd equation 22
OTHER EXPERIMENTAL RESULTS (REGRESSION FOR UNKNOWN GAUSSIAN MODEL)
6.1 D-VIB ALGORITHM FOR VECTOR GAUSSIAN MODEL
the vector Gaussian data model equation 14 the optimal distributions P and Q equation 23 within
the family of multivariate Gaussian distributions. Motivated by this observation, we consider the following parameterization for E IC:
Po. (ux xx) = N(ux ph, E?)
(76)
Qux (flux) NiŠ: pt. Ex)
(77)
Qo (())))))))))) NiS: pii E
(78)
Qu (ua) 1).
(79)
where ph, EE are the output DNN fo, with input X, that encodes observations a Tay -iieensional Gaussian distribution, 4ic, ΣS the outputs of DNN fee with inputs UI...., Ux. sampled from Po. (un X, and f Σj are the output of a DNN for with input Ux, 1,..., K.
With the above choice of parametric encoders and decoders, and using single sample 1, the empirical DIB cost in equation 29 is given for the sample (Xi,s..... XK.i, y:) by
pemp (0, c. sp) = (v: mi,,)) " (2 m12,e) + log det det( (x12.)
K (y - ph Ed (vi - (...)) + log det (2
8
km
(Hi. - 1)² (#4E, - 1) + log Eg--1 nug + tr E2.
ng (1 sK) log(2rr).
2
where (142,5 E12,g) denote output the DNN foe for the i-th sample (xi,,,.... ,,,,,, ys). and similarly for the other mean and covariance terms; and where we have used that each term in the empirical DIB cost equation 29 can be computed noting that d-dimensional Gaussian pmfs N( f4, Σ) we have
log (y:: E) =- (I "T (xy 14) + dllog(2i)) log del del(E))
15

---

Under review as a conference paper at ICLR 2020
and the KL divergence between two multivariate Gaussian pií's P1 N(,,, E1) and P2 NNz,, En) in R is
(RI(P) = (wm) u22 "e2"""" (mn p2a) + log Σn 1r -
(80)
The multivariate Gaussian parametrization of the encoders, decoders and prior distribution given by equa- tion equation [7 can be used for other data models that are not necessary Gaussian, For example, it is particularly suitable for regression problems in which Y lies on continuous space. Also, it is very often used in conjunction with VAE generative problems Rezende et al. (2014): Kingma and Welling 2013).
6.2 REGRESSION FOR VECTOR GAUSSIAN DATA MODEL
Consider a distributed learning model with = encoders, each observing a noisy version an my di-diimnnioonal Gaussian vector Y 0, I), as X H2Y Ng, where H2 E R x and the noises are distributed as NN ~ 1), for & = 1,2.
For this model, the optimal accuracy-complest region can be computed using Theorem 2 In what follows, we evaluate the performance of our D-VIB of the previous section for regression. The algorithm is trained using dataset of i..dd. samples [(X1.,, X2.1, Y.)) form the described vector Gaussian data model. We train the DNNs for various values of the parameter s. We use the multivariate Gaussian parameterization in equation equutiion 79 for the DNNs architecture shown in Table 6.2 Specifically, Encoder k, & 1,2, consists of three dense layers of 512 neurons each followed hy rectified linear unit (Rel.u) activations. The output of encoder & is processed by a dense layer without nonlinear activation to generate ya and Ej, of size 512 and 512 x 512, respectively. Each decoder consists of two dense layers of 512 neurons with ReLu activations. The output of decoder 1, and processed, cach, by a fully connected layer without activation to generate fag and Σg and 1412 and Eig of size 2 and 2 x 2.
It
C.O
ECB
CBB loral
D-13 Ja0t
Figure 5: Accuracy vs. sum-complexity tradcoffs for the Gaussian data model. Pa- rameters: K encoders, fy = 1, v22 and 30.000.
Figure 6: Mean square eтror VS. sum- complexity tradcoffs for the Gaussian data model. Parameters: K = 2 encoders, fl
1, mb n2 3 and 30.000.
region of tuples (A, R obtained from Theorem for a vector Gaussian model with K- encoders, target variable dimension nt, 1, and observations dimension 11 = 122 = 3. A set of 40.000 samples split among training (30.000 samples) and test (10.00H samples). The figure depicts all accuracy complexxty pairs obtained by application of our algorithm D-VIB to this setting. The results are compared to the case of inference with known joint distribution (referred to as D-IB, see next section) the
Figure 5 shows the optimal relevance-comple
as well as the case of centralized inference (C-IB). For the D-VIB algorithm, the the DNN architecture Oshows the evolution of the associated mean squared error (MSE) in the
coders is shown in Table estimation of the label Y using our D-VIB algorithm. As can bee seen from both figures the performance of our VIB algorithm (which does not require knowledge of the joint label- distribution) is very close to that predicted by the theory, i.e., our Theorem
Figure shows similar curves for ny 2, nh = 3 dimensions, for various sizes of the training datset. As expected large training sets allow more accurate prediction. Noteworthy, that the performance during the training phase might be better than that of the centralized learning scenario is indicator can be caused by overfitting. Related to this aspect, recall that although the D- VIB algorithm does not estimate the underlying distribution explicitly, intuitively it does for the computation of the cost function. This is related to that universal compressors also learn the actual distribution of the data that is being compressed. Recall that since the plug-in estimator of entropy is biased downward, estimations of the mutual information terms that involved in the cost function are then biased upward, which is an alternate explanation to the observed overfitting during the training phase.
16

---

Under review as a conference paper at ICLR 2020
DNN Layers
Encoder k
dense [512] ReLu
dense [512]-RelLu
C-B atss Bow
dense [512]-ReLLu
01
Lat. space it
dense (256]-RcLu
D1VE
Decoder 12
dense [256]-Rela)
180
28 Compiesit
Decoder I
dense [256]-ReLu
Figure 7: Effect of varying training set size [5.000, 10.000, 50.000} Gaussian inference for D-VIB.
Table 3: Used DNN architecture.
DISTRIBUTED BLAHUT-ARIMOTO TYPE ALGORIHTMM
7.1 DISCRETE-ALHHAN SETTING
this section, we derive an iterative method to optimize the variational DIB cost function in equation 23 when the data model is discrete and the joint distribution Px..Y is either known, good estimation of it can be obtained from the training samples. these cases, the maximizing distributions P, Q of the variational DIB alternating optimization procedure over P and Q similar to the expectation maximization (EM) algorithm Dempster et al. (1977) and the standard Blahut-Ariman (BA) method 1972]. An extension to the vector Gaussian data model, which involves random variable with continuous alphabets, is also provided. The main idea of the algorithm is that at iteration f, the optimal distributions (()) that maximize the variational D-IB bound LVP (P. ((1)) for fixed (t) be optimized in closed form and, next, the maximizing pmfs Q(") for given p(t) can be also found analytically. So, starting from an initialization p(0) and Q (0) the algorithms performs the following computations successively and this order, until convergence.
cost in equation 23 can be efficiently found
((O)
(0) P
(()
(81)
We refer to such algorithm as "Blahut-Arimote Distributed Information Bottleneck Algorithu (BA-DIB)". Algorithm desccrihes the steps taken by BA-DIB to successively maximize C. (P, Q) by solving concave optimization problem over P and over Q at each iteration. We have the following lemma whose proof follows essentially by using the log-sum inequality Cover and Thomas (1991) and the convexity of the mapping log
Lemma The function L (P, Q) is concave in P and in Q.
For fixed p(() the optimal maximizing the variational D-IB bound in equation 19 follows from Lemma For fixed ((9), the optimal ((t) can be found using the following lemma.
1
as given by equation 21 cquation 22
CVB (P, Q), where PULIxe given Lemma For fixed Q there exists that achieves the maximum maxp C. by
exp p" (uclxx) = g(ux), Eu, Em ((ux) exp(-x's (i,,
(82)
for uh E Us and xx Nx. & E K, and where we define
(x( = (KL (xL(PrieollQy + Eueea PD
(83)
Proof: Due its concavity. to maximize LY (P, Q) with respect to for given Q, we add Lagrange multipliers Aue 2 0 for each constraint Σ., et, uu (rL) = 1 with an E Nr. For each Aoς 2 0 and p(ul(zz) can be explicitly found by solving the KKT conditions. e.g..
p(unleee) -1) MLEM
+)+
Σ Arz FEE
Σ
3p(ueloa
This completes the proof.
17

---

Under review as a conference paper at ICLR 2020
Algorithm 1 BA-DIB training algorithm for discrete data
1: inputs:
discrete pmf X..,. parameter 2
2: output: optimal P Ux|Xa,' pair Rt).
Rt).
3: initialization
Set 0 and set ( with p(ux(r)) = [al for un E Uk Bh E Xx, k = 1,.... K.
4: repeat
5: Compute Q ((+11) using equation 21
and equation 22
6: Compute (t-1) using equation 82
7: t ++++1
8: until convergence.
7.1.1 CONVERGENCE
Algorithm D essentially falls into the class of the Successive Upper-Boune Bound Minimization (SUM) algorithms
Razaviyayn et al. (2013) in which L Q) acts as globally tight lower bound on (PP). Algorithm
I provides sequence p(t) for each iteration t, which converges to a stationary point of the optimization problem equation
23
Proposition 4 Every limit point of the sequence p()) generated by Algorithm 7 converges stationary point
of equation 23
Proof: Let QP(P) maxQ C. (P (P.Q). Using Lemma
for every P * P. it holds that
VB Q' S S. (C.
= CP)).
(84)
Since C, (P) and (P Q" (P')) satisfy the assumptions of Raaaviyyayn al. 2013 Proposition 1). then CYB (P, (P')) satisfics A1-A4 Razaviyayn et al. (2013). Convergence to a stationary point of equation 23 follows from (Razaviyayn al. 2013 Theorem 1).
The self consistent equations equation 21 equation and equation 83 satisfied by any stationary point of the D-IB problem extend those of the standard point-to- point IB problem Globerson and Tishby (2004) to the distributed IB problem with K 2 2 encoders. In particular, note the additional divergence term equation 83
7.2 GAUSSIAN SETTING
Recall Algorithm For finite alphabet sources the updating rules of Q(++1) and ((++1) in Algorithm 1 Mare relatively easy, but they become unfeasible for continuous alphabet sources. We leverage on the optimality of Gaussian test channels, shown Theorem
restrict the optimization of P to Gaussian distributions, which are casily represented by finite set of parameters, namely mean and covariance. We show that if p(() are Gaussian distributions, then p(++1) are also Gaussian distrihutions, which be computed with efficient update algorithm of its representing parameters. In particular, if at time the k-th distributions p!()) by
Ux given
U AiX: Zi
(85)
where Zi ~ CN(O, Egg) ), we show that at L 1, for ((++)) updated as in equation 82 the encoder ((++1)
U |X,
corresponds to UL+1 A A Xx + Z where Z CO, E,112) and Σ E,11, A updated as
E Eett = (1+1) E Eugii
1
( )
(86)
A
(1+1) A AL Exily Ex!)
1
of
A
Dx!)
(87)
The detailed update procedure is given in Algorithm (see the following section for the details of the derivations).
18

---

Under review as a conference paper at ICLR 2020
Algorithm 2 BA-DIB algorithm for the Gaussin Vector D-IB
1: inputs:
covariance Σ "y.x........ parameter & 2
2: output: optimal pairs (A Ez; ), k I..... K.
3: initialization
Set randomly AC and Ege = 0, k E K.
4: repeat
5: Compute Xx, Iuku and update for k E K.
Euily A Ax Ezi
(88)
D 1u = Exulutt A Des
(89)
6: Compute Σz--1 as in equation 86 for E K.
7: Compute A as equation 87 I.
8: t t ++++, 1.
9: until convergence.
7.2.1 DERIVATION OF ALGORITHM
We derive the update rules of Algorithm and show that the Gaussian distribution is invariant to the update rules in Algorithm I
in line with Theorem 2 First, recall that if (X1, X2) are jointly Gaussian, then
Px
CNN C.N Exp(x:)
(90)
where Pxgg |x = Kxil X1, with K Kxglx = Exy,x, Exj
Then, for Q (0+1)) computed as equation 21 and equation 22 from P(O,, which set of Gaussian distributions, have
@ = CN Dylut )
Q CN Eylnk)
Next, look the update p(++) in equation 82 from given Q Q(+11) First, have that p(uf) is the marginal of U given by U ~ CO Eug)) where Enf Ex A + Ezj
Then, to compute via (uf, first, we note that
Eug (Dki 11 11 DRI. DKI.(Py 1Qr:Ueylee) DKL. (PU (ee 1QQu (mn)
(91)
and that for two generic multivariate Gaussian distributions Pı ~ CN(M,, En) and P3 CN E2) in CN the KL divergence computed as equation 80 below.
Applying equation 91 and equation 80 in equation 83 and noting that all involved distributions Gaussian, follows that u. (u Xa) is a quadratic form. Then, since p(u{) is Gaussian, the product log(p(u{) exp(-ut d- (u), xx))) also quadratic form, and identifying constant, first and second order terms, we can write
log (++1) (++l) (uy (xx) = (xx+) + (ux Mugt pap
"E-1 2 (u: Mutt"
(92)
where (xxx) normalization term independent us.
Σ Σ11
2ng + K Zwing Kylut
+ Kange of Poafwiig Kynkealtt KH of Diz affr lug K Kerii
(93)
and
Mut-l/xe Exat1 (K Dylug Mylxa
19

---

Under review as a conference paper at ICLR 2020
Ky Kynkyelu Jug Priaeer uug Myiukre Kukyalu lut "kev lug Mu Mukraala (e
(94)
This shows that p (1x+)(ux/xx) is multivariate Gaussian distribution and that IX xx) is also multivariate Gaussian distributed C. (P "|xx" L (11))
Next, we simplity equation 93 equation 94 to obtain the update rules equation [6 and equation From the matrix inversion lemma, similarly Chechik et al. [Feb. 2005), for (X1, X2) jointly Gaussian we have
Exa + K Dxiixe Kxi 1022
(95)
x1
Applying equation 95 in equation 93 we have
1
Σ
E uflr
+
8 uflyuke
D
s uf ufose"
(96)
(1
E u
1 =
(97)
where equation 97 is due to the Markov chain Us - Y + UKAA.
Then, also from the matrix inversion lemma, have for jointly Gaussian (X1, X2).
E Exix
Exj Exy Ex x,/xa"
(98)
Xgixx
Applying equation 98 to equation 94 for the first term in equation 94 we have
K ylu yyltt Mylxe = 2 yu Dy My
2
(99)
= -Eu A Exy Ey Ey Exi Xx
E A Ex, Exy Josk
(100)
where yy.ug A Ex, ys and equation 100 is due the definition of Exply
Similarly, for second term in equation we have
Ky Kynkye lng Poa lug My "eu Σ bu
Ai E Exglonees )x))
(101)
= - ufly AL(I Exaly Ex. )xi.
(102)
where we use Eug smbr
E Suk and equation 102 is due the Markov chain U
Uk
For the third term in equation 94
K Kukse un "feva lut Mukcr x
E-1 ufee
AL Exuluken Ex )xk-
(103)
Equation equation 87 follows by noting that M4 |xg
A (+1 "k, and that from equation 94 A can be
identified as in equation 87
Finally, we note that due to equation 85 Eajly and Df juku, are given as in equation and equation
where Ex, = E. and E Mes can be computed from its definition. This completes the proof.
20

---

Under review as a conference paper at ICLR 2020
REFERENCES
A. Alemi, Fischer. J. Dillon, and K. Murphy. Deep variational information bottleneck. In ICLR, 2017. URL
httpa:://arxiv.rrg/aaat wwwpaa/s/......... 00410
R. Blahut. Computation channel capacity and rate-distortion functions. IEEE Transactions on Information Theory, 18(4):460-473, Jul 1972. ISSN 0018-9444)
Blum and T. Mitchell. Combining labeled and unlabeled data with co-training. Proceedings of eleventh annual conference on Computational learning theory, pages 92-100. ACM, 1998.
O. Bousquet and Elisseeff. Stability and generalization. Journal of Machine Learning Research, 2(Mar): 499-526, 2002.
G. Chechik, A. Globerson, N. Tishby, and Y. Weiss. Information bottleneck for Gaussian variables. Journal of Machine Learning Research, 6:165-188, Feb, 2005.
T. A. Courtade and T. Weissman. Multiterminal source coding under logarithmic loss. IEEE Transactions on Information Theory, 60(1):740-761, Jan. 2014. ISSN 0018-9448. doi: 10.1 221.097
T. M. Cover and J. A. Thomas. Elements of Information Theory. Iiley--nterscciene,, 1991.
A. Dembo, T. M. Cover, and A. Thomas. Information theoretic inequalities. IEEE Trans. Inf. Theory, (6):1501-1518, Nov 1991. ISSN 0018-9448 doi: 10. 1009918/ 104312.
A. P. Dempster, N. M. I Laird, and D. B. Rubin. Maximum likelihond from incomplete data via the algorithm. Journal of the Royal Statistical Society, Series B, 39(1):1-38, 1977.
P. Dhillon, D. P. Foster, and L. H. Ungar. Multi-view learning of word embeddings via CCA. In Advances in neural information processing systems, pages 199-207, 2011.
R. Dobrushin and B. Tsybakov. Information transmission with additional noise. IRE Transactions on Information Theory, 8(5):293-304, Sep. 1962. ISSN 0096-1000 doi: 10. 10.1109/IT.1962.1555777
Ekrem and S. Ulukus. An outer bound for the vector Gaussian CEO problem. JEEE Transactions on Information Theory, 60(1)), Nov. 2014.
Gamal and Y.-H. Kim. Network information theory. Cambridge University Press, 2011.
A. Globerson and N. Tishby. On the optimality of the Gaussian information bottleneck curve. Hebrew University Tech. Report, 2004.
M. Gönen and Alpaydin. Multiple kernel learning algorithms. Journal of Machine Learning Research, 12 (lul):221122266 2011.
Hardoon, S. Szedmak, and Taylor. Canonical correlation analysis: overview with application to learning methods. Neur. Comput., 16:2639-266 2004.
Harremoes and N. Tishby. The information bottleneck revisited how choose good distortion measure. In Proc. IEFE Int. Symp. Information Theory, pages 566-570, Jun. 2007.
I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Lerchner. --vac: Learning basic visual concepts with constrained variational framework. 2016.
G. E. Hinton and D. van Camp. Keeping the neural networks simple by minimizing the description length of weights. In Proceedings of the Sixth Anmual Conference on Computational Learning Theory, COLT '93, pages 5-13, New York, NY, USA, 1993. ACM. ISBN 0-89791-611 1-5. doi: 00.11455/68334.166888 URL. http::/dde acm. org 10.. 11411144511 683044 168306
Y. Jia, M. Salzmann, and T. Darrell. Factorized latent spaces with structured sparsity. In Advances in Neural Information Processing Systems, pages 982-990, 2010.
J. Jiao, T. A. Courtade, K. Venkat, and T. Weissman. Justification of logarithmic loss via the benefit of side information. IEEF Transactions Information Theory, 61(10):335--5365, 2015.
21

---

Under review as a conference paper at ICLR 2020
D. P. Kingma and M. Welling, Auto-encoding variational bayes. CoRR, abs/1312.6114, 2013. URL http:
// dblp. uni-trier dwtpd////////// htt.aaaas///////ooaaasas//s////s hat.a//aaaaaarrooooo3..2aaaaaaa1
D. P. Kingma and M. Welling, Auto-Encoding Variational Bayes, ArXiv e-prints, Dec. 2013.
A. Kumar and H. Daumé. A co-training approach for multi-view spectral clustering. In Proceedings of the 28th International Conference on Machine Learning (ICML-I1), pages 393-400, 2011.
C.-T. A. El Gamal. Strong functional representation lemma and applications to coding theorems. IEEE Transactions Information Theory, 64( 1):6967-669777 2018.
C.-T. Li. X. Wu, A. Ozgur, and A. El Gamal. Minimax learning for remote prediction. In
http:://arziv.rr/ab////be http///bb//880000707// May. 2018.
K. Nigam and R. Ghani. Analyzing the effectiveness and applicability of co-training. In Proc. of the 9th International Conference of Information and Knowledge Management, pages 86-93, 2000.
A. Painsky and W. Wornell, the universality the logistic function. arXiv preprint arXiv: 1805.03804, 2018.
M. Razaviyayn, M. Hong, and Z.-Q. Luo. A unified convergence analysis of block successive minimization methods for nonsmooth optimization. SIAM J. on Opt., 23:1126-1153, Jun. 2013.
D. J. Rezende, S. Mohamed, and D. Wierstra. Stochastic backpropagation and approximate inference in deep Xing and Jebara, editors, Proceedings of the 31st International Conference on Machine Learning, volume 32 of Proceedings of Machine Learning Research, pages 1278-122 11886, Bejing, China, 22-24 Jun 2014, PMI.R.
generative models.
D. Russo and J. Zou. How much does your data exploration overfit? controlling bias via information usage. arXiv preprint arXiv:1511.05219 2015.
Shalev-Shwartz, O. Shamir, N. Srebro, and K. Sridharan. Learnability, stability and uniform convergence. Journal of Machine Learning Research, 11(0ct):2635 2670, 2010.
N. Tishby, F. C. Pereira, and W. Bialek. The information bottleneck method. In Proc. 37th Annual Allerton Conf. on Comm., Control, and Computing, pages 368-377, 1999.
Vapnik. The nature of statistical learning theory. Springer science business media, 2013.
A. Xu and M. Raginsky. Information--heerrtiicc analysis of generalization capability of learning algorithms. In Advances in Neural Information Processing Systems, pages 2521-2530 2017.
C. Xu, D. Tao, and C. Xu. survey on multi-vict lcarning. arXiv preprint vrXiv:1304.500 2013.
Zhao, Xie, Xu, and Sun. Multi-vier learning overview: recent progress new challenges. Information Fusion, 38:43-54, 2017.
22


Reviews:
[{"review_id": "rygizoc0qr", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies a distributed representation problem where multiple features X_1,...,X_K are processed (or encoded) separately to estimate (or decode) some quantity of interest Y.\nThe log loss is considered throughout, which amounts to measuring the mutual information between Y and \\hat Y, defined as the \"accuracy\" of the estimation method. The average rate (measured in number of bits per sample) of the encoded feature is defined as the \"complexity\" of the representation method.\nThe author derived the fundamental trade-off between the accuracy and the complexity for any representation-estimation (or encoding-decoding) method.\nThe author also derived a variational representation of the optimal accuracy-complexity region, which also expresses the optimal encoder and decoder map as the solution of the optimization problem.\nFinally, the author considered the case where the joint distribution of P_{X_1,,,,X_K,Y} is unknown, and encoder and decoder are parameterized by neural networks, parameters of which are tuned using data.\n\nI incline to reject the paper, for the following reasons.\n1. The accuracy-complexity trade-off studied in the paper is more of a rate-distortion type of information-theoretic problem, where the joint distribution P_{X_1,,,,X_K,Y} is assumed to be known. Its connection to the learning problem, where the joint distribution P_{X_1,,,,X_K,Y} is unknown, is unclear. Even if the precise accuracy-complexity region is obtained, it says little about the sample complexity needed by a learning algorithm to achieve this region.\n2. Deriving the optimal encoder-decoder mapping from the variational representation of the accuracy-complexity region also requires the joint distribution, which violates the basic assumption of the learning problem.\n3. The author did consider the case where the joint distribution is unknown, and the encoder-decoder pair is learned from data. However, this learning problem is somewhat artificial: each encoder only encodes one of the features, but in order to encode optimally, it has to know the entire joint distribution, hence need to access all the features during training. This discrepancy of seeing different components of the data set during training and inference is not well-motivated.\nThe author mentioned \"multi-view learning\" at the beginning of the paper. It would be good if the author can elaborate more on this problem in Sec 4 of Experiment Results, and discuss with more detail on how the proposed method solves this problem and how it is different from the existing results, both in terms of the algorithm and the performance.\n\n\n\n=================================================\nFeedback to authors' reply\n\nI got a better understanding on how the proposed learning algorithms works after reading the authors' reply.\nI guess the idea for the case where the joint distribution is unknown is that, for encoding, different nodes uses its own training data (without accessing other nodes' data) to optimize the encoder separately; while for decoding, the master node trains the decoder uses data available to all nodes to estimate the joint distribution.\nIn this way, the encoders and the decoder jointly optimizes a variational lower bound of the optimal rate region.\n\nIf this is the case, I think the proposed method may have some value in practice. \nBut now the question is how good the variational lower bound is compared to the optimal region, and how well can this variational lower bound be approximated by neural networks and how efficient can the training be done. Without theoretical analysis on these questions, one may only use experiments to assess the performance. From Table 2, it looks like the improvement of the proposed method on the existing method is quite marginal.\n\nIn summary, I would like to thank the authors' valuable reply. I encourage the authors to study the gap between the variational lower bound and the optimal region, and maybe do more experiments to find a good use case of the proposed method.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "rklfW0_6Fr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper extended the Gaussian Information Bottleneck method to the case of multi-view learning and provided a variation bound for the accuracy optimization with constrain on the sum of complexity. It also proposed an algorithm to learn the distributed representation without any prior knowledge of the data distribution.\n\nThe multi-view learning problem has been quite well studied in the literature. The paper reformulated the multi-view learning problem as a Bayesian inference problem and provided solid analysis for it.\n\nThe writing of the paper was pretty hard to follow for me, with a lot of notations that are not defined clearly. \n* For example, I can roughly guess that U in theorem 1 represent the learned descriptors, but what’s the variable T in theorem 1?\n* What is \\Omega in Theorem 2?\n\nThe experimental result doesn’t look very comprehensive at all as it was mostly compared with variations of the proposed algorithm and it doesn’t include any other multi-view learning algorithms.\n\nThe algorithms in the experimental result are not very clearly defined. I don’t see much explanation of what is exactly D-VIB and C-VIB. There’s some formulation of the algorithm in Section 3.4, but it only gives a loss function briefly. I’m not sure if many practitioners will be able to implement this algorithm from the description here.\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1lFhyDE5B", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I am not an expert in this area and the paper involves a lot of derivations and proofs, but I did not check the correctness of those derivations. In summary, this paper proposed a framework for integrating multiple data sources for representing data. In the framework, each data source was mapped to a latent data variable by using a nonlinear function which is called an encoder; then the mapped latent variables were jointly mapped to the target data by using another nonlinear function which is called a decoder. To make this idea to work, the paper used mutual information as the objective function to control the accuracy if the model, and at the same time to avoid overfitting the paper proposed to use MDL as a measure to control the complexity of the model. If I was right, this was the whole picture of the proposed model. My questions are the following:\n1) I am not very clear how the model complexity was automatically incorporated with the objective function. It seems to me that the objective function was finally the equation (29) and then the neural networks for encoder and decoder were optimized. If this was the case, how the model complexity was incorporated, that is, how the R_k was used in the model? Was the values R_k constant in the model - I mean they are fixed constant values? How these values,i.e.,R_k, were chosen?\n2) I am a mathematician, but to be honest, I feel that the Maths in the paper is huge and heavy and I thought it could not be that complex for the model. The consequence is that it make the paper to be hard to read. This is a personal feeling, you could just ignore this point.\n3) Experiments: there are a lot of papers describing to integrate data sources for at least the MNIST example. It would be interesting to compare the proposed method to the literature. The experiment in 4.1 obviously is a toy data problem - I mean although the data is real, but the data generated was using noisy and rotations. It would be more interesting to apply the method to a real-world problem.\n4) I think it would be more friendly to explicitly define the concepts of Discrete Memoryless and Memoryless Vector Gaussian Models. \n5) The Markov chain represented in equation (3) is not well defined. I do not understand these notations. \n6) Before the equation (4), is it equivalent X_k^n and X_{k,n}? I am confused by these notations\n7) In equation (6), it is more readable to explicitly define the Shannon Mutual Information.\n8) The second paragraph on Page 5: you use Gaussian pmfs here, but pmf denotes discrete variable. But Gaussian I assume is continuous.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJlpd_F6tr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors studied the distributed representation learning problem, where multiple sources of data are processed to provide information about Y. They studied this problem from information-theoretic point of view. Their main contribution can be summarized as follows.\n    1. The optimal trade-off between the accuracy and complexity were studied for discrete memoryless data model as well as memoryless vector Gaussian model.\n    2. A variational bound were constructed in order to connect the optimal encoder and decoder mappings with the solution of an optimization algorithm.\n    3. If only samples from an unknown distribution are available, an algorithm were proposed to find the optimal encode and decoder. Moreover, some experiment were conducted to support the approach.\n\nIn general, I think the paper is well-organized. The definition of the problem and the motivation of the approach are clear. The theorems, algorithms and experiments are solid enough to support the whole story of this paper. Generally I wish to see  this paper being accepted.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors study generalization in distributed representation learning by describing limits in accuracy and complexity which stem from information theory. 

The paper has been controversial, but ultimately the reviewers who provided higher scores presented weaker and fewer arguments. By recruiting an additional reviewer it became clearer that, overall the paper needs a little more work to reach ICLR standards. The main suggestions for improvements have to do with improving clarity in a way that makes the motivation convincing and the practicality more obvious. Boosting the experimental results is a complemental way of increasing convincingness, as argued by reviewers.

Decision: Reject