The Dual Information Bottleneck

1 INTRODUCTION . 1.1 THE INFORMATION BOTTLENECK METHOD . The Information Bottleneck ( IB ) method ( Tishby et al. , 1999 ) , is an information-theoretic framework for describing efficient representations of a large scale “ input ” random variable X ( input patterns ) , for predicting an “ output ” variable Y ( desired label ) . In this setting the joint distribution of X and Y , p ( x , y ) defines the problem , or rule , and the training data are a finite sample from this distribution . In general , we assume that p ( y | x ) is strictly stochastic , and hence bounded away from { 0 , 1 } 1 . The representation variable X̂ is in general a stochastic function of X which forms a Markov chain Y → X → X̂ , and only depends on Y through the input X . We call the map p ( x̂ | x ) the encoder of the representation and denote by p ( y | x̂ ) the Bayes optimal decoder for this representation ; i.e. , the best possible prediction of the desired label Y from the representation X̂ . The IB has direct successful applications for representation learning in various domains , from vision and speech processing ( Ma et al. , 2019 ) , to neuroscience ( Schneidman et al. , 2001 ) , and Natural Language Processing ( Li & Eisner , 2019 ) . Due to the notorious difficulty in estimating mutual information in high dimension , variational approximations to the IB have been suggested and applied also to Deep Learning ( Alemi et al. , 2016 ; Parbhoo et al. , 2018 ) . For large scale learning , when almost all input patterns can be considered typical in the information theoretic sense ( Cover & Thomas , 2006 ) , the cardinality of the typical inputs is characterized by the pattern ’ s entropy , H ( X ) = −∑x p ( x ) log p ( x ) . Under similar typicality assumption , the mutual 1This may seem like a limitation of the method , but it can be shown that in the deterministic limit the IB is well defined . This stochastic assumption simplifies the analysis but does not pose any real limitation . information ( MI ) of the encoder , I ( X ; X̂ ) = ∑ x , x̂ p ( x , x̂ ) log p ( x|x̂ ) p ( x ) characterizes the cardinality of the typical representations , X̂ . On the other hand , the mutual information of the Bayes optimal decoder , I ( Y ; X̂ ) = ∑ y , x̂ p ( y , x̂ ) log p ( y|x̂ ) p ( y ) , is equivalent to the cross-entropy generalization error . The IB trade off between the encoder and decoder mutual information values is defined by the minimization of the Lagrangian : F [ pβ ( x̂ | x ) ; pβ ( x̂ ) ; pβ ( y | x̂ ) ] = I ( X ; X̂ ) − βI ( Y ; X̂ ) , ( 1 ) independently over the convex sets of the normalized distributions , { pβ ( x̂ | x ) } , { pβ ( x̂ ) } and { pβ ( y | x̂ ) } , given a positive Lagrange multiplier β . As shown in ( Tishby et al. , 1999 ; Shamir et al. , 2010 ) , this is a natural generalization of the classical concept of Minimal Sufficient Statistics ( Cover & Thomas , 2006 ) , where the estimated parameter is replaced by the output variable Y and exact statistical sufficiency is characterized by the mutual information equality : I ( X̂ ; Y ) = I ( X ; Y ) . The minimality of the statistics is captured by the minimization of I ( X ; X̂ ) , due to the Data Processing Inequality ( DPI ) . However , non-trivial minimal sufficient statistics only exist for very special parametric distributions known as exponential families ( Brown , 1986 ) . Thus in general , the IB relaxes the minimal sufficiency problem to a continuous family of representations X̂ which are characterized by the trade off between compression , I ( X ; X̂ ) ≡ IX , and accuracy , I ( Y ; X̂ ) ≡ IY , along a convex line in the Information-Plane ( IY vs. IX ) . When the rule p ( x , y ) is strictly stochastic , the convex optimal line is smooth and each point along the line is uniquely characterized by the value of β . We can then consider the optimal representations x̂ = x̂ ( β ) as encoder-decoder pairs : ( pβ ( x|x̂ ) , pβ ( y|x̂ ) ) 2 - a point in the continuous manifold defined by the Cartesian product of these distribution simplexes . We also consider a small variation of these representations , δx̂ , as an infinitesimal change in this ( encoder-decoder ) continuous manifold ( not necessarily on the optimal line ( s ) ) . 1.2 IB AND RATE-DISTORTION THEORY . The IB optimization trade off can be considered as a generalized rate-distortion problem ( Cover & Thomas , 2006 ) with the distortion function between a data point , x and a representation point x̂ taken as the KL-divergence between their predictions of the desired label y : dIB ( x , x̂ ) = D [ p ( y | x ) ||pβ ( y|x̂ ) ] = ∑ y p ( y | x ) log p ( y | x ) pβ ( y | x̂ ) . ( 2 ) The expected distortion Epβ ( x , x̂ ) [ dIB ( x , x̂ ) ] for the optimal decoder is simply the label-information loss : I ( X ; Y ) − I ( X̂ ; Y ) , using the Markov chain condition . Thus minimizing the expected IB distortion is equivalent to maximizing I ( X̂ ; Y ) , or minimizing equation 1 . Minimizing this distortion is equivalent to minimizing the log-loss or the cross-entropy loss , as done in most deep learning applications , and it upper-bounds other loss functions such as the L1-loss ( due to the Pinsker inequality , or ( Painsky & Wornell , 2018 ) ) . The Pinsker inequality shows that both orders of the cross-entropy upper bound to the L1-loss , or total-variation distance , min { D [ q||p ] , D [ p||q ] } ≥ 12 log 2‖p− q‖21 . 1.3 THE IB EQUATIONS . For discrete X and Y , a necessary condition for the IB ( local ) minimization is given by the three self-consistent equations for the optimal encoder-decoder pairs , known as the IB equations : ( i ) pβ ( x̂ | x ) = pβ ( x̂ ) Z ( x ; β ) e−βD [ p ( y|x ) ‖pβ ( y|x̂ ) ] ( ii ) pβ ( x̂ ) = ∑ x pβ ( x̂ | x ) p ( x ) ( iii ) pβ ( y | x̂ ) = ∑ x p ( y | x ) pβ ( x | x̂ ) , ( 3 ) where Z ( x ; β ) is the normalization function . Iterating these equations is a generalized , BlahutArimoto , alternating projection algorithm ( Tusnady & Csiszar , 1984 ; Cover & Thomas , 2006 ) and it converges to a stationary point of the Lagrangian , equation 1 ( Tishby et al. , 1999 ) . Notice that the minimizing decoder , ( equation 3- ( iii ) ) , is precisely the Bayes optimal decoder for the representation x̂ ( β ) , given the Markov chain conditions . 2Here we use the inverse encoder , which is in the fixed dimension simplex of distributions over X . 1.4 CRITICAL POINTS AND CRITICAL SLOWING DOWN . One of the most interesting aspects of the IB equations is the existence of critical points along the optimal line of solutions in the Information-Plane ( i.e . the information curve ) . At these points the representations change topology and cardinality ( number of clusters ) ( Zaslavsky & Tishby , 2019 ; Parker et al. , 2003 ) and they form the skeleton of the information curve and representation space . Under the strict stochastic assumption , the information-curve is a smooth function of the Lagrange multiplier β , but its derivative may not be smooth . Critical points are bifurcations of the solutions , which are values of β for which two different solutions ( representations ) co-exist . To identify such points we perform a perturbation analysis of the IB equations , as in ( Zaslavsky & Tishby , 2019 ) . Taking a small perturbation of the representation , denoted for brevity by δx̂ , the changes in the log encoder and log decoder that satisfy equation 3 for a given β can be determined through the nonlinear eigenvalues problems3 : [ I − βCIBxx′ ( x̂ , β ) ] ∂ log pβ ( x′ | x̂ ) ∂x̂ = 0 , [ I − βCIByy′ ( x̂ , β ) ] ∂ log pβ ( y′ | x̂ ) ∂x̂ = 0 , ( 4 ) with the two square matrices C defined by : CIBxx′ ( x̂ , β ) = ∑ y p ( y | x ) pβ ( x ′ | x̂ ) pβ ( y | x̂ ) p ( y | x′ ) , CIByy′ ( x̂ , β ) = ∑ x p ( y | x ) pβ ( x | x̂ ) pβ ( y | x̂ ) p ( y′ | x ) . ( 5 ) As shown in ( Zaslavsky & Tishby , 2019 ) , these two matrices have the same eigenvalues and have non-trivial eigenvectors ( i.e. , different co-existing optimal representations ) precisely at the critical values of β , the bifurcation points of the IB solution . At these points the cardinality of the representation X̂ ( the number of “ IB-clusters ” ) changes due to splits of clusters , resulting in topological phase transitions in the encoder . These critical points form the “ skeleton ” of the topology of the optimal representations . Between critical points the optimal representations change continuously ( with β ) . The important computational consequence of critical points is known as critical slowing down ( Tredicce et al. , 2004 ) . For binary Y , near a critical point the convergence time , τβ , of the iterations of equation 3 scales like : τβ ∼ 1/ ( 1− βλ2 ) , where λ2 is the second eigenvalue of either CIByy′ or C IB xx′ . At criticality , λ2 ( x̂ ) = β −1 and the number of iterations diverges . This phenomenon dominates any local minimization of equation 3 which is based on alternate encoder-decoder optimization . To illustrate the importance of the critical points and the critical slowing-down we consider a lowdimensional problem that is easy to visualize . Taking the joint distribution , p ( y , x ) with a binary Y and only 5 possible inputs . For a given β , the Blahut-Arimoto iterations converge to a stationary solutions of the IB equations , which amounts to discrete points in the interval [ 0 , 1 ] - the simplex of the p ( y ) distributions - for each value of the Lagrange multiplier , β . The evolution of the optimal decoder , pβ ( y = 0 | x̂ ) , ∀x̂ ∈ X̂ , as a function of β , is a bifurcation diagram ( Figure 1 ( a ) ) . The critical points of the IB equations correspond to bifurcations in the decoders ’ graph . At these critical points the non-trivial eigenvalues of CIByy′ ( β ) cross the β −1 value ( Figure 1 ( b ) ) , and the number of iterations diverges ( critical slowing down ) ( Figure 1 ( c ) ) .

Summary: The information bottleneck (IB) is an information theoretic principle for optimizing a mapping (encoding, e.g. clustering) of an input, to trade off two kinds of mutual information: minimize mutual information between the original input and the mapped version (to compress the input), and maximize mutual information between the mapped input and an output variable. It is related to a minimization of an (expected) Kullback-Leibler divergence betwen conditional distributions of an output variable.