Perturbation Theory for the Information Bottleneck

1 Information Bottleneck . Extracting relevant information from data is crucial for all forms of learning . Animals are very adept at isolating biologically useful information from complicated real-world sensory stimuli : for example , we instinctively ignore pixel-level noise when looking for a face in a photo . A failure to disregard irrelevant bits could lead to suboptimal generalization performance especially when the data contains spurious correlations . For instance , an image classifier that relies on background texture to identify objects is likely to fail when presented with a new image showing an object in an ‘ unusual ’ background ( see , e.g. , Refs [ 7 , 30 ] ) . Understanding the principles behind the identification and extraction of relevant bits is therefore of fundamental and practical importance . Formalizing this aspect of learning , the information bottleneck ( IB ) method provides a precise notion of relevance with respect to a prediction target : the relevant information in a source ( - ) is the bits that carry information about the target ( . ) [ 26 ] . The relevant bits in - are summarized in a representation ( / ) via a stochastic map defined by an encoder @ ( I |G ) , obeying the Markov constraint /↔-↔ . .1 In general a trade-off exists between the amount of discarded information ( compression ) and the remaining relevant information in / ( prediction ) , thus motivating the IB cost function,2 ! [ @ ( I |G ) ] = ( / ; - ) − V ( / ; . ) , ( 1 ) where V > 0 denotes the trade-off parameter and ( ; ) the mutual information . The first term favors succinct representations whereas the second encourages predictive ones . The IB loss is minimized by the representations that are most predictive of . at fixed compression , parametrized by the Lagrange multiplier V ( see , Fig 1a ) . The IB method offers a highly versatile framework with wide-ranging applications , including neural coding [ 16 ] , evolutionary population dynamics [ 22 ] , statistical physics [ 9 ] , clustering [ 25 ] , deep 1This Markov chain implies % . |- , / = % . |- and % / |- , . = % / |- ( see , e.g. , Ref [ 6 ] ) . 2The optimization problem involving Eq ( 1 ) first appeared in a different context ( see , e.g. , Ref [ 27 ] ) . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . learning [ 1–3 ] and reinforcement learning [ 10 ] . However the nonlinearity of the IB problem makes it computationally expensive and difficult to analyze , barring a few special cases [ 5 ] . This necessitates an investigation of tractable methods for solving the IB problem . The use of variational approximations to reduce the computational cost has paved the way for a massive scale-up of the IB method [ 3 ] . Complementing this approach , we report a new analytical result for the IB problem in the tractable limiting case of learning onset . 2 Learning Onset . Although the IB loss in Eq ( 1 ) favors a representation that encodes every relevant bit in - when V → ∞,3 the optimal representation needs not contain any relevant information at finite V. To see this , we note that the loss vanishes for any uninformative representation ( / ; - ) = ( / ; . ) = 0 , and thus an informative representation yields a lower loss only when the relevant information in / is adequately large : a negative IB loss requires ( / ; . ) > V−1 ( / ; - ) . But the relevant information is also bounded from above by the data processing inequality ( DPI ) , ( / ; . ) ≤ ( / ; - ) , resulting from the the Markov constraint /↔-↔ . [ 6 ] ( see , Fig 1a ) . Combining these inequalities yields V−1 ( / ; - ) < ( / ; . ) ≤ ( / ; - ) , ( 2 ) which can not be met when V−1 > 1 . Hence the existence of an informative IB minimizer requires V−1 ≤ 1 . Indeed for any % - , . with ( - ; . ) > 0 , there exists a critical trade-off parameter V2 ( - → . ) ≥ 1 that marks the learning onset , separating two qualitatively distinct regimes : uninformative regime at V < V2 and informative regime at V > V2 . The learning onset is the first in a series of transitions that emerges from the hierarchy of relevant information in the data [ 26 ] . Galvanized in part by the recent applications of the IB principle in deep learning , several works have attempted to characterized the IB transitions [ 8 , 17 , 28 , 29 ] . However the IB problem remains intractable even in limiting cases and a complete characterization of the IB transitions remains elusive . In fact the only exception is the special case of Gaussian variables for which an exact solution exists [ 5 ] . In this work we derive a perturbation theory for the IB problem and offer the first complete description of the learning onset . We elaborate on the subtle differences between our theory and the previous works in Sec 6 . The learning onset is not only a special limit in the IB problem but also physically and practically relevant . It corresponds to the region where the relevant information per encoded bit is greatest and 3The compression term , while infinitesimally small in this limit , still penalizes irrelevant information and prefers a representation / that is the minimal sufficient statistics of - for . . thus places a tight bound on the thermodynamic efficiency of predictive systems [ 23 , 24 ] . An analysis the IB learning onset has recently found applications in statistical physics [ 9 ] . The ( inverse ) critical trade-off parameter is also a useful measure of correlation between two random variables [ 13 ] ; indeed its square root satisfies all but the symmetry property of Rényi ’ s axioms for statistical dependence measures [ 21 ] . Finally estimating the upper bound of V2 might help weed out non-viable values of hyperparameters in deep learning techniques such as the variational information bottleneck [ 28 , 29 ] . 2.1 Strong data processing inequality . We can improve the bound on V2 with the tight version of the DPI , the strong data processing inequality ( SDPI ) [ 4 , 18 , 20 ] ( see , Fig 1a ) ( / ; . ) ≤ [ KL ( - → . ) ( / ; - ) ( 3 ) where [ KL ( -→ . ) denotes the contraction coefficient for the Kullback-Leibler divergence , defined via [ KL ( - → . ) ≡ sup '-≠ % - DKL ( ' . ‖ % . ) DKL ( '- ‖ % - ) . ( 4 ) Here % - and % . denote the probability distributions of - and . . The supremum is over all allowed distributions given the space of - , and ' . is related to '- via the channel % . |- . Replacing the DPI with the SDPI in Eq ( 2 ) , we obtain V2 ( - → . ) ≥ [ KL ( - → . ) −1 . ( 5 ) In the following section we show that the equality holds , as expected ( since the SDPI is tight ) . Note that [ KL ( -→ . ) and V2 ( -→ . ) are generally asymmetric under -↔ . . 3 Perturbation Theory . We investigate the learning onset through the lens of perturbation theory . This method constructs the solution for a problem as a power series in a small parameter Y , when the solution for the limiting case Y=0 , the unperturbed solution , is accessible . For small Y , the higher order terms in this series represent ever smaller corrections to the unperturbed solution . To obtain these corrections , we insert the series solution into the initial problem and expand the resulting expressions as power series in Y , truncated at appropriate order . For example , the first-order theory drops all quadratic and higher terms ( those proportional to Y2 , Y3 , . . . ) , resulting in a consistency condition for the linear correction ( i.e. , the term proportional to Y ) . Requiring consistency up to Y= leads to the nth-order perturbation theory . In practice the first few corrections suffice for a characterization of the problem in the vicinity of Y=0 . Our theory is based on a controlled expansion around the critical trade-off parameter V2 and some uninformative encoder @ 0 ( I |G ) = @ 0 ( I ) , @ ( I |G ) = @ 0 ( I |G ) + Y @ 1 ( I |G ) + Y2 @ 2 ( I |G ) + . . . ( 6 ) ( / ; - ) = Y ( 1 ) / ; - [ @ 1 ] + Y2 ( 2 ) / ; - [ @ 1 , @ 2 ] + . . . , ( 7 ) where Y ≡ V − V2 → 0+ and ∑I @ = ( I |G ) = 0 for = ≥ 1 to ensure normalization . Note that ( 0 ) / ; - vanishes for uninformative @ 0 . The first and second-order informations capture the first and secondorder growths of information as V rises above V2 and are given by ( see Appendix for derivation ) ( 1 ) / ; - [ @ 1 ] = ∑ G ? ( G ) ∑ I∈Z1 @ 1 ( I |G ) ln @ 1 ( I |G ) @ 1 ( I ) ( 8 ) ( 2 ) / ; - [ @ 1 , @ 2 ] = ∑ G ? ( G ) ( ∑ I∈Z0 @ 1 ( I |G ) 2 − @ 1 ( I ) 2 2 @ 0 ( I ) + ∑ I∈Z1 @ 2 ( I |G ) ln @ 1 ( I |G ) @ 1 ( I ) + ∑ I∈Z2 @ 2 ( I |G ) ln @ 2 ( I |G ) @ 2 ( I ) ) , ( 9 ) where Z0 = supp ( @ 0 ) and Z= = supp ( @ = ) \ ⋃=−18=0 Z8 ( i.e. , Z= contains representation classes or space that first appear in the support of the =th-order encoder ) .4 The expansions for @ ( I ) and @ ( I |H ) 4Our theory generalizes the expansions in Refs [ 28 , 29 ] which considered the caseZ1=Z2=∅ . take the same form as Eq ( 6 ) , and the expressions for ( / ; . ) are identical to Eqs ( 7 ) - ( 9 ) but with . replacing - everywhere . Finally we write down the loss function as a power series in Y , ! [ @ ( I |G ) ] = Y ! ( 1 ) [ @ 1 ] + Y2 ! ( 2 ) [ @ 1 , @ 2 ] + . . . , ( 10 ) where ! ( 1 ) [ @ 1 ] = ( 1 ) / ; - [ @ 1 ] − V2 ( 1 ) / ; . [ @ 1 ] ( 11 ) ! ( 2 ) [ @ 1 , @ 2 ] = ( 2 ) / ; - [ @ 1 , @ 2 ] − V2 ( 2 ) / ; . [ @ 1 , @ 2 ] − ( 1 ) / ; . [ @ 1 ] . ( 12 )

Summary: The paper studies the learning onset of Information Bottleneck (IB). The authors derive perturbation theory for the IB method and characterize the learning onset, and verifies with experiments. The authors also compare the difference and subtleties in their derivation and previous attempts at deriving a perturbation theory for the learning onset.