A Theory of Usable Information under Computational Constraints

We propose a new framework for reasoning about information in complex systems . Our foundation is based on a variational extension of Shannon ’ s information theory that takes into account the modeling power and computational constraints of the observer . The resulting predictive V-information encompasses mutual information and other notions of informativeness such as the coefficient of determination . Unlike Shannon ’ s mutual information and in violation of the data processing inequality , V-information can be created through computation . This is consistent with deep neural networks extracting hierarchies of progressively more informative features in representation learning . Additionally , we show that by incorporating computational constraints , V-information can be reliably estimated from data even in high dimensions with PAC-style guarantees . Empirically , we demonstrate predictive V-information is more effective than mutual information for structure learning and fair representation learning . 1 INTRODUCTION . Extracting actionable information from noisy , possibly redundant , and high-dimensional data sources is a key computational and statistical challenge at the core of AI and machine learning . Information theory , which lies at the foundation of AI and machine learning , provides a conceptual framework to characterize information in a mathematically rigorous sense ( Shannon & Weaver , 1948 ; Cover & Thomas , 1991 ) . However , important computational aspects are not considered in information theory . To illustrate this , consider a dataset of encrypted messages intercepted from an opponent . According to information theory , these encrypted messages have high mutual information with the opponent ’ s plans . Indeed , with infinite computation , the messages can be decrypted and the plans revealed . Modern cryptography originated from this observation by Shannon that perfect secrecy is ( essentially ) impossible if the adversary is computationally unbounded ( Shannon & Weaver , 1948 ) . This motivated cryptographers to consider restricted classes of adversaries that have access to limited computational resources ( Pass & Shelat , 2010 ) . More generally , it is known that information theoretic quantities can be expressed in terms of betting games ( Cover & Thomas , 1991 ) . For example , the ( conditional ) entropy of a random variable X is directly related to how predictable X is in a certain betting game , where an agent is rewarded for correct guesses . Yet , the standard definition unrealistically assumes agents are computationally unbounded , i.e. , they can employ arbitrarily complex prediction schemes . Leveraging modern ideas from variational inference and learning ( Ranganath et al. , 2013 ; Kingma & Welling , 2013 ; LeCun et al. , 2015 ) , we propose an alternative formulation based on realistic computational constraints that is in many ways closer to our intuitive notion of information , which we term predictive V-information . Without constraints , predictive V-information specializes to classic mutual information . Under natural restrictions , V-information specializes to other well-known notions of predictiveness , such as the coefficient of determination ( R2 ) . A consequence of this new formulation is that computation can “ create usable information ” ( e.g. , by decrypting the intercepted messages ) , invalidating the famous data processing inequality . This generalizes the idea that clever feature extraction enables prediction with extremely simple ( e.g. , linear ) classifiers , a key notion in modern representation and deep learning ( LeCun et al. , 2015 ) . As an additional benefit , we show that predictive V-information can be estimated with statistical guarantees using the Probably Approximately Correct framework ( Valiant , 1984 ) . This is in sharp contrast with Shannon information , which is well known to be difficult to estimate for high dimensional or continuous random variables ( Battiti , 1994 ) . Theoretically we show that the statistical guarantees of estimating V information translate to statistical guarantees for a variant of the Chow-Liu algorithm for structure learning . In practice , when the observer employs deep neural networks as a prediction scheme , V-information outperforms methods that approximate Shannon information in various applications , including Chow-Liu tree contruction in high dimension and gene regulatory network inference . 2 DEFINITIONS AND NOTATIONS . To formally define the predictive V-information , we begin with a formal model of a computationally bounded agent trying to predict the outcome of a real-valued random variable Y ; the agent is either provided another real-valued random variable X as side information , or provided no side information ∅ . We use X and Y to denote the samples spaces of X and Y respectively ( while assuming they are separable ) , and use P ( X ) to denote the set of all probability measures over the Borel algebra on X ( P ( Y ) similarly defined for Y ) . Definition 1 ( Predictive Family ) . 1 Let Ω = { f : X ∪ { ∅ } → P ( Y ) } . We say that V ⊆ Ω is a predictive family if it satisfies ∀f ∈ V , ∀P ∈ range ( f ) , ∃f ′ ∈ V , s.t . ∀x ∈ X , f ′ [ x ] = P , f ′ [ ∅ ] = P ( 1 ) A predictive family is a set of predictive models the agent is allowed to use , e.g. , due to computational or statistical constraints . We refer to the additional condition in Eq . ( 1 ) as optional ignorance . Intuitively , it means that the agent can , in the context of the prediction game we define next , ignore the side information if she chooses to . Definition 2 ( Predictive conditional V-entropy ) . Let X , Y be two random variables taking values in X × Y , and V be a predictive family . Then the predictive conditional V-entropy is defined as HV ( Y |X ) = inf f∈V Ex , y∼X , Y [ − log f [ x ] ( y ) ] HV ( Y |∅ ) = inf f∈V Ey∼Y [ − log f [ ∅ ] ( y ) ] We additionally call HV ( Y |∅ ) the V-entropy , and also denote it as HV ( Y ) In our notation f is a function X ∪ { ∅ } → P ( Y ) , so f [ x ] ∈ P ( Y ) is a probability measure on Y chosen based on the received side information x ( we use f [ · ] instead of the more conventional f ( · ) ) ; and f [ x ] ( y ) ∈ R is the value of the density evaluated at y ∈ Y . Intuitively , V ( conditional ) entropy is the smallest expected negative log-likelihood that can be achieved predicting Y given observation ( side information ) X ( or no side information ∅ ) , using models from V . Eq . ( 1 ) means that whenever the agent can use P to predict Y ’ s outcomes , it has the option to ignore the input , and use P no matter whether X is observed or not . Definition 2 generalizes several known definitions of uncertainty . For example , as shown in proposition 2 , if the V is the largest possible predictive family that includes all possible models , i.e . V = Ω , then Definition 2 reduces to Shannon entropy : HΩ ( Y |X ) = H ( Y |X ) and HV ( Y |∅ ) = HΩ ( Y ) = H ( Y ) . By choosing more restrictive families V , we recover several other notions of uncertainty such as trace of covariance , as will be shown in Proposition 1 . Shannon mutual information is a measure of changes in entropy when conditioning on new variables : I ( X ; Y ) = H ( Y ) −H ( Y |X ) = HΩ ( Y ) −HΩ ( Y |X ) ( 2 ) Here , we will use predictive V-entropy to define an analogous quantity , IV ( X → Y ) , to represent the change in predictability of an output variable Y when given side information X . 1Regularity Conditions : To minimize technical overhead we restrict out discussion only to distributions with probability density functions ( PDF ) or probability mass functions ( PMF ) with respect to the underlying measure . Also ∅ 6∈ X . Definition 3 ( Predictive V-information ) . Let X , Y be two random variables taking values in X × Y , and V be a predictive family . The predictive V-information from X to Y is defined as IV ( X → Y ) = HV ( Y |∅ ) −HV ( Y |X ) ( 3 ) 2.1 IMPORTANT SPECIAL CASES . Several important notions of uncertainty and predictiveness are special cases of our definition . Note that when we are defining V-entropy of a random variable Y in sample space Y ∈ Rd ( without side information ) , out of convenience we can assume X is empty X = ∅ ( this does not violate our requirement that ∅ 6∈ X . ) Proposition 1 . For V-entropy and V-information , we have 1 . Let Ω be as in Def . 1 . Then HΩ ( Y ) is the Shannon entropy , HΩ ( Y | X ) is the Shannon conditional entropy , and IΩ ( Y → X ) is the Shannon mutual information . 2 . Let Y = Rd and V = { f : { ∅ } → Pµ | µ ∈ Rd } , where Pµ is the distribution with density y 7→ 1Z e −‖y−µ‖2 where Z = ∫ e−‖y−µ‖2dy , then the V-entropy of a random variable Y equals its mean absolute deviation , up to an additive constant . 3 . Let Y = Rd and V = { f : { ∅ } → N ( µ , Σ ) | µ ∈ Rd , Σ = 1/2Id×d } , then the V-entropy of a random variable Y equals the trace of its covariance tr ( Cov ( Y ) ) , up to an additive constant . 4 . Let V = { f : { ∅ } → Qt , θ , θ ∈ Θ } , where Qt , θ is a distribution in a minimal exponential family with sufficient statistics t : Y → Rd and set of natural parameters Θ . For a random variable Y with expected sufficient statistics µY = E [ t ( Y ) ] , the V-entropy of Y is the maximum Shannon entropy over all random variables Ŷ with identical expected sufficient statistics , i.e . E [ t ( Ŷ ) ] = µY . 5 . Let Y = Rd , X be any vector space , and V = { f : x 7→ N ( φ ( x ) , Σ ) , x ∈ X ; ∅ 7→ N ( µ , Σ ) |µ ∈ Rd ; Σ = 1/2Id×d , φ ∈ Φ } , where Φ is the set of linear functions { φ : X → Rd } , then V-information IV ( X → Y ) equals the ( unnormalized ) maximum coefficient of determination R2 · tr ( Cov ( Y ) ) for linear regression . The trace of covariance represents a natural notion of uncertainty – for example , a random variable with zero variance ( when d = 1 , tr ( Cov ( Y ) ) = Var ( Y ) ) ) is trivial to predict . Proposition 1.3 shows that the trace of covariance corresponds to a notion of surprise ( in the Shannon sense ) for an agent restricted to make predictions using certain Gaussian models . More broadly , a similar analogy can be drawn for other exponential families of distributions . In the same spirit , the coefficient of determination , also known as the fraction of variance explained , represents a natural notion of informativeness for computationally bounded agents . Also note that in the case of Proposition 1.4 , the V-entropy is invariant if the expected sufficient statistics remain the same .

Summary: The paper presents a generalization of classical definitions of entropy and mutual information that can capture computational constraints. Intuitively, information theoretic results assume infinite computational resources, so they may not correspond to how we treat "information" in practice. One example is public-key encryption. An adversary that has infinite time will eventually break the code so the decrypted message conveys the same amount of information (in a classical sense) as the plaintext message. In practice, this depends on computational time.