Perturbation Theory for the Information Bottleneck

1 Information Bottleneck . Extracting relevant information from data is crucial for all forms of learning . Animals are very adept at isolating biologically useful information from complicated real-world sensory stimuli : for example , we instinctively ignore pixel-level noise when looking for a face in a photo . A failure to disregard irrelevant bits could lead to suboptimal generalization performance especially when the data contains spurious correlations . For instance , an image classifier that relies on background texture to identify objects is likely to fail when presented with a new image showing an object in an ‘ unusual ’ background ( see , e.g. , Refs [ 7 , 30 ] ) . Understanding the principles behind the identification and extraction of relevant bits is therefore of fundamental and practical importance . Formalizing this aspect of learning , the information bottleneck ( IB ) method provides a precise notion of relevance with respect to a prediction target : the relevant information in a source ( - ) is the bits that carry information about the target ( . ) [ 26 ] . The relevant bits in - are summarized in a representation ( / ) via a stochastic map defined by an encoder @ ( I |G ) , obeying the Markov constraint /↔-↔ . .1 In general a trade-off exists between the amount of discarded information ( compression ) and the remaining relevant information in / ( prediction ) , thus motivating the IB cost function,2 ! [ @ ( I |G ) ] = ( / ; - ) − V ( / ; . ) , ( 1 ) where V > 0 denotes the trade-off parameter and ( ; ) the mutual information . The first term favors succinct representations whereas the second encourages predictive ones . The IB loss is minimized by the representations that are most predictive of . at fixed compression , parametrized by the Lagrange multiplier V ( see , Fig 1a ) . The IB method offers a highly versatile framework with wide-ranging applications , including neural coding [ 16 ] , evolutionary population dynamics [ 22 ] , statistical physics [ 9 ] , clustering [ 25 ] , deep 1This Markov chain implies % . |- , / = % . |- and % / |- , . = % / |- ( see , e.g. , Ref [ 6 ] ) . 2The optimization problem involving Eq ( 1 ) first appeared in a different context ( see , e.g. , Ref [ 27 ] ) . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . learning [ 1–3 ] and reinforcement learning [ 10 ] . However the nonlinearity of the IB problem makes it computationally expensive and difficult to analyze , barring a few special cases [ 5 ] . This necessitates an investigation of tractable methods for solving the IB problem . The use of variational approximations to reduce the computational cost has paved the way for a massive scale-up of the IB method [ 3 ] . Complementing this approach , we report a new analytical result for the IB problem in the tractable limiting case of learning onset . 2 Learning Onset . Although the IB loss in Eq ( 1 ) favors a representation that encodes every relevant bit in - when V → ∞,3 the optimal representation needs not contain any relevant information at finite V. To see this , we note that the loss vanishes for any uninformative representation ( / ; - ) = ( / ; . ) = 0 , and thus an informative representation yields a lower loss only when the relevant information in / is adequately large : a negative IB loss requires ( / ; . ) > V−1 ( / ; - ) . But the relevant information is also bounded from above by the data processing inequality ( DPI ) , ( / ; . ) ≤ ( / ; - ) , resulting from the the Markov constraint /↔-↔ . [ 6 ] ( see , Fig 1a ) . Combining these inequalities yields V−1 ( / ; - ) < ( / ; . ) ≤ ( / ; - ) , ( 2 ) which can not be met when V−1 > 1 . Hence the existence of an informative IB minimizer requires V−1 ≤ 1 . Indeed for any % - , . with ( - ; . ) > 0 , there exists a critical trade-off parameter V2 ( - → . ) ≥ 1 that marks the learning onset , separating two qualitatively distinct regimes : uninformative regime at V < V2 and informative regime at V > V2 . The learning onset is the first in a series of transitions that emerges from the hierarchy of relevant information in the data [ 26 ] . Galvanized in part by the recent applications of the IB principle in deep learning , several works have attempted to characterized the IB transitions [ 8 , 17 , 28 , 29 ] . However the IB problem remains intractable even in limiting cases and a complete characterization of the IB transitions remains elusive . In fact the only exception is the special case of Gaussian variables for which an exact solution exists [ 5 ] . In this work we derive a perturbation theory for the IB problem and offer the first complete description of the learning onset . We elaborate on the subtle differences between our theory and the previous works in Sec 6 . The learning onset is not only a special limit in the IB problem but also physically and practically relevant . It corresponds to the region where the relevant information per encoded bit is greatest and 3The compression term , while infinitesimally small in this limit , still penalizes irrelevant information and prefers a representation / that is the minimal sufficient statistics of - for . . thus places a tight bound on the thermodynamic efficiency of predictive systems [ 23 , 24 ] . An analysis the IB learning onset has recently found applications in statistical physics [ 9 ] . The ( inverse ) critical trade-off parameter is also a useful measure of correlation between two random variables [ 13 ] ; indeed its square root satisfies all but the symmetry property of Rényi ’ s axioms for statistical dependence measures [ 21 ] . Finally estimating the upper bound of V2 might help weed out non-viable values of hyperparameters in deep learning techniques such as the variational information bottleneck [ 28 , 29 ] . 2.1 Strong data processing inequality . We can improve the bound on V2 with the tight version of the DPI , the strong data processing inequality ( SDPI ) [ 4 , 18 , 20 ] ( see , Fig 1a ) ( / ; . ) ≤ [ KL ( - → . ) ( / ; - ) ( 3 ) where [ KL ( -→ . ) denotes the contraction coefficient for the Kullback-Leibler divergence , defined via [ KL ( - → . ) ≡ sup '-≠ % - DKL ( ' . ‖ % . ) DKL ( '- ‖ % - ) . ( 4 ) Here % - and % . denote the probability distributions of - and . . The supremum is over all allowed distributions given the space of - , and ' . is related to '- via the channel % . |- . Replacing the DPI with the SDPI in Eq ( 2 ) , we obtain V2 ( - → . ) ≥ [ KL ( - → . ) −1 . ( 5 ) In the following section we show that the equality holds , as expected ( since the SDPI is tight ) . Note that [ KL ( -→ . ) and V2 ( -→ . ) are generally asymmetric under -↔ . . 3 Perturbation Theory . We investigate the learning onset through the lens of perturbation theory . This method constructs the solution for a problem as a power series in a small parameter Y , when the solution for the limiting case Y=0 , the unperturbed solution , is accessible . For small Y , the higher order terms in this series represent ever smaller corrections to the unperturbed solution . To obtain these corrections , we insert the series solution into the initial problem and expand the resulting expressions as power series in Y , truncated at appropriate order . For example , the first-order theory drops all quadratic and higher terms ( those proportional to Y2 , Y3 , . . . ) , resulting in a consistency condition for the linear correction ( i.e. , the term proportional to Y ) . Requiring consistency up to Y= leads to the nth-order perturbation theory . In practice the first few corrections suffice for a characterization of the problem in the vicinity of Y=0 . Our theory is based on a controlled expansion around the critical trade-off parameter V2 and some uninformative encoder @ 0 ( I |G ) = @ 0 ( I ) , @ ( I |G ) = @ 0 ( I |G ) + Y @ 1 ( I |G ) + Y2 @ 2 ( I |G ) + . . . ( 6 ) ( / ; - ) = Y ( 1 ) / ; - [ @ 1 ] + Y2 ( 2 ) / ; - [ @ 1 , @ 2 ] + . . . , ( 7 ) where Y ≡ V − V2 → 0+ and ∑I @ = ( I |G ) = 0 for = ≥ 1 to ensure normalization . Note that ( 0 ) / ; - vanishes for uninformative @ 0 . The first and second-order informations capture the first and secondorder growths of information as V rises above V2 and are given by ( see Appendix for derivation ) ( 1 ) / ; - [ @ 1 ] = ∑ G ? ( G ) ∑ I∈Z1 @ 1 ( I |G ) ln @ 1 ( I |G ) @ 1 ( I ) ( 8 ) ( 2 ) / ; - [ @ 1 , @ 2 ] = ∑ G ? ( G ) ( ∑ I∈Z0 @ 1 ( I |G ) 2 − @ 1 ( I ) 2 2 @ 0 ( I ) + ∑ I∈Z1 @ 2 ( I |G ) ln @ 1 ( I |G ) @ 1 ( I ) + ∑ I∈Z2 @ 2 ( I |G ) ln @ 2 ( I |G ) @ 2 ( I ) ) , ( 9 ) where Z0 = supp ( @ 0 ) and Z= = supp ( @ = ) \ ⋃=−18=0 Z8 ( i.e. , Z= contains representation classes or space that first appear in the support of the =th-order encoder ) .4 The expansions for @ ( I ) and @ ( I |H ) 4Our theory generalizes the expansions in Refs [ 28 , 29 ] which considered the caseZ1=Z2=∅ . take the same form as Eq ( 6 ) , and the expressions for ( / ; . ) are identical to Eqs ( 7 ) - ( 9 ) but with . replacing - everywhere . Finally we write down the loss function as a power series in Y , ! [ @ ( I |G ) ] = Y ! ( 1 ) [ @ 1 ] + Y2 ! ( 2 ) [ @ 1 , @ 2 ] + . . . , ( 10 ) where ! ( 1 ) [ @ 1 ] = ( 1 ) / ; - [ @ 1 ] − V2 ( 1 ) / ; . [ @ 1 ] ( 11 ) ! ( 2 ) [ @ 1 , @ 2 ] = ( 2 ) / ; - [ @ 1 , @ 2 ] − V2 ( 2 ) / ; . [ @ 1 , @ 2 ] − ( 1 ) / ; . [ @ 1 ] . ( 12 )

Summary: The authors investigate the learning onset for the information bottleneck method, i.e., the behavior of the problem at \beta being close to the critical parameter \beta_c. More specifically, the IB curve is studied via a perturbation around the optimal compression map. In contrast to previous work, the authors allow this perturbation also outside of the alphabet of the optimal compression.