Quantifying Task Complexity Through Generalized Information Measures

1 INTRODUCTION . Deep networks have shown remarkable progress in both simple and complex machine learning tasks . But how does one measure the “ complexity ” of a learning task ? Is it possible to ascertain in a principled manner which tasks are “ harder ” to solve than others ? How “ close ” is one task to another ? Answers to these questions would have implications in many fields of machine learning such as transfer learning , multi-task learning , un/semi/self-supervised learning , and domain adaptation . In classical information theory , the entropy of a random variable X is a useful measure of complexity for tasks such as compression and transmission , which essentially require reconstructing X . However , the entropy ofX is insufficient for measuring the complexity of a supervised learning task TX , Y , where the goal is to predict an output Y from an input X , i.e. , to estimate the conditional pY |X ( y | x ) from a finite set of samples from pXY ( x , y ) , which we refer to as solving the learning task . Complexity measures provided by statistical learning theory like VC-dimension or covering numbers are also inadequate for this purpose because they ignore the dependence between X and Y for the particular task at hand . Information-theoretic measures such as mutual information , information bottleneck ( Tishby et al. , 2000 ) and its variants ( Strouse & Schwab , 2017 ) have been used to study the trade-off between model complexity and accuracy , but have not been developed to focus on assessing task complexity and can provide unsatisfactory results when comparing different tasks ( see Section 5 for details ) . Measures based on Kolmogorov complexity ( Li , 2006 ; Vereshchagin & Vitányi , 2004 ) could in principle be used to compare different tasks , but they are dataset permutation sensitive and not easily computable . The work of ( Achille et al. , 2019a ) proposes to quantify task complexity by measuring the information stored on the network weights , but the approach depends on the specific neural network architecture used for training . The work of ( Tran et al. , 2019 ) does not require or assume trained models , but makes strict assumptions that limit its broad applicability . In this work , we introduce a novel perspective on task complexity which generalizes classical measures from information theory . Specifically , one well-known interpretation of classical Shannon entropy is , given a random variable X , find the minimum number of bits that are needed on average to encode instances of X so that the instances can be perfectly recovered from the binary code . Stated differently , if one lets Q be defined as the set of all possible binary functions , on the domain of X , then Shannon entropy essentially asks what is the optimal sequence of queries to compute { q1 ( X ) , q2 ( X ) , . . . : qi ∈ Q } ( i.e. , how to encode X as a binary string ) so that X can be perfectly recovered from the shortest ( on average ) sequence of binary answers to the queries ( see Section 2 for more discussion of this interpretation ) . As discussed above , however , in most learning tasks we are not interested in simply compressing X but rather making a prediction about some other variable Y . Further , notions of complexity can potentially be made more relevant to a specific task by not having Q to be the set of all possible binary functions on X but rather a smaller set of queries specific to a measure of interest . From this intuition , we define the complexity of a learning task as the minimum expected number of queries , selected from a fixed set Q , one needs to ask to predict Y ( to some user-defined level of confidence ) from the respective answers to the queries . As a few specific examples of potential query sets : • Decision boundary complexity : Here , Q is the the set of all possible half-spaces in Rd ( assuming X ∈ Rd ) and q ( x ) is a binary function response indicating whether x lies in a particular half-space ( q ∈ Q ) . Then task complexity formalizes the intuition of “ level of non-linearity ” of the decision boundary . For example , the complexity of any linearly-separable binary classification task is 1 , whereas , for a non-linearly separable task , this value depends on the curvature of the decision boundary . • Task-specific input feature complexity : Here , Q is the set of projection functions of X and q is of the form q ( x ) = xq , where xq is the value observed at the qth entry of x . Then task complexity formalizes the intuition “ the greater the redundancy between the input features the easier it is to solve the task ” . For example , Y being a constant function of X would be the simplest task with complexity 0 , since no feature needs to be queried to predict it . This notion of complexity would help in answering questions such as “ which input features are most important for solving a given task ? ” and could in turn help in developing more “ interpretable ” learning algorithms . • Visual semantic complexity : Given a vocabulary V of different possible entities , their attributes and relations in a visual scene , Q could be the set of all binary functions indicating the presence or absence of an entity , its attribute or its relation with other entities ( supplied by V ) in a designated region of the image X . For example , a particular q could be the function implementing the query “ Is there a person in the top left corner of the image ? ” . This notion of complexity would allow one to gauge the semantic complexity of a visual task . For instance , tasks which ask complex questions like “ Is there a person playing with his dog , next to a river in the image ? ” would inherently be more complex than simple object detection tasks , “ Where is the dog in this image ? ” and could be quantified by semantically relevant queries . While our proposed formal definition of task complexity will be applicable to all such choices of query functions { q } q∈Q and enjoys several nice theoretical properties that we discuss in section 2 , its computation will generally be intractable . As a result , in section 3 we propose to reduce the complexity of selecting a minimal set of questions by using the Information Pursuit ( IP ) algorithm , which selects questions sequentially , depending on previous questions and answers , in order of information gain . While IP is generally applicable to any task and query set , its implementation is still intractable depending on the complexity of the model p ( X , Y ) and of the set Q . To address this issue , we posit a tractable graphical model for p ( X , Y ) and learn the required distributions using variational autoencoders and normalizing flows . In section 4 we evaluate our approach on various binary image classification tasks ( MNIST , KMNIST , FashionMNIST , Caltech Silhouettes ) that can be tackled using a common set of queries ( the set of image patches ) . Our results show that complexity computed using patch queries aligns with the intuition that the complexity of a classification task increases as signal-to-noise ratio decreases , and that classification of the KMNIST dataset is more complex than classification of the FashionMNIST dataset , something that isn ’ t obvious a priori . While these experiments are restricted to simple tasks and queries , the proposed framework is generally applicable provided that tractable models , inference and learning methods can be developed , which is the subject of ongoing and future work . Finally , we note that to the best of our knowledge , this is the first time that a subjective notion of task complexity has been proposed in literature , where the user can incorporate in Q the perception of complexity he/she wishes to measure . 2 QUANTIFYING TASK COMPLEXITY . Let the input data be represented by random variable X and the corresponding output/hypothesis by random variable Y with sample spaces X and Y respectively . A supervised learning task TX ; Y is defined as the task of estimating the conditional distribution pY |X ( y | x ) from a finite set of samples from the joint distribution pXY ( x , y ) .1 We propose to quantify the complexity of a task as the minimum expected number of queries needed to solve the task . Queries , q ∈ Q , are functions of the input data , whose answers for a given x are denoted as { q ( x ) } q∈Q . Naturally , the query set needs to be sufficiently rich so that the task is solvable based on answers to the queries . More formally , we say the set Q is sufficient for task TX ; Y if ∀ ( x , y ) ∈ X × Y , p ( y | x ) = p ( y | { x′ ∈ X : q ( x′ ) = q ( x ) ∀q ∈ Q } ) . ( 1 ) In other words , Q is sufficient for task TX ; Y if whenever two points x and x′ have identical answers for all queries in Q , then their true posteriors must be equal , p ( y | x ) = p ( y | x′ ) . Given a fixed query set Q , we now formally define an encoding function EQ , which we refer to as a Q-Coder , along with our proposed complexity measure CQ ( X , Y ) for task TX , Y . Defining the Encoder function . Given a query set Q , an Encoder is a function , E : S∗ → Q where S∗ is the set of all finite-length sequences generated using elements from the set S = { ( q , q ( x ) ) | q ∈ Q , x ∈ X } . Additionally , we require that Q contains a special query qSTOP which signals the encoder to stop asking queries and outputs the code for x as the set of query-answer pairs that have been asked . The process can be described as follows , given E and input sample x . 1. q1 = E ( ∅ ) . The first question is independent of x . 2. qk+1 = E ( { qi , qi ( x ) } 1 : k ) . All subsequent queries depend on the query-answer pairs ob- served so far for x . 3 . If qL+1 = qSTOP terminate and return CodeEQ ( x ) : = ( qi , qi ( x ) ) 1 : L as the code for x . Note that each qi depends on x , but we drop this in the notation for brevity . Note also that the number of queries L = |CodeEQ ( x ) | for a particular x generalizes coding length in information theory . The query qSTOP constrains the code CodeEQ ( x ) to be prefix-free . Defining task complexity . Given a joint distribution pXY ( x , y ) and a sufficient query set Q , we define task complexity , CQ ( X ; Y ) , as the minimum over all encoders E ( which are mappings from X to a subset of queries in Q ) of the mean coding length : CQ ( X ; Y ) : = min E EX [ |CodeEQ ( X ) | ] s.t . p ( y | x ) = p ( y | CodeEQ ( x ) ) ∀x ∈ X , y ∈ Y ( Sufficiency ) ( 2 ) Figure 1 : Schematic view of the overall framework for quantifying complexity of a task TX ; Y . The first constraint ensures sufficiency of the code ∀x ∈ X . By this we mean that there exists at least one Encoder E for which the first constraint of “ sufficiency ” is satisfied , where p ( y | CodeEQ ( x ) ) should be interpreted as the conditional probability of y given the event { x′ ∈ X | CodeEQ ( x ) = CodeEQ ( x′ ) } . The solution to ( 2 ) provides the optimal encoder for task TX ; Y , and Fig . 1 illustrates the overall framework in detail . Connection with Shannon entropy H . It can be shown that when Q is taken as the set for all possible binary questions on X and if Y is a function of X ( which it usually is for supervised classification problems ( Kolchinsky et al. , 2018 ) ) , then the solution , E∗ , to ( 2 ) gives a coding length within one bit of the entropy of Y , denoted as H ( Y ) . More formally , one can show that H ( Y ) ≤ CQ ( X ; Y ) = EX [ |CodeE ∗ Q ( X ) | ] < H ( Y ) + 1 . ( 3 ) 1As commonly used , we denote random variables by capital letters and their realizations with small letters . Note that one example of such an optimal encoder , E∗ ( x ) , is given by the Huffman code for Y . Connection with task complexity & equivalence classes . Notions of complexity of an entity/object , often relate to the level of structural regularity present in the entity ( Gell-Mann , 2002 ) . CQ ( X ; Y ) measures the degree of regularity present in the structure of TX ; Y . This structure is implicitly defined by the conditional pY |X ( y | x ) and the query set Q . Notice that any E partitions X into equivalence classes . [ x ] = { x′ ∈ X | CodeEQ ( x ) = CodeEQ ( x′ ) } . ( 4 ) The prefix-free constraint on the codes generated by E ( due to ) ensures that ∀x′ ∈ [ x ] , ∀y ∈ Y , pY |X=x′ ( y ) = pY |X=x ( y ) . It is then natural to relate task complexity with the number of equivalence classes induced by the optimal E∗ . The greater the number of equivalence classes , the higher the complexity of the task . This is because knowing the distribution pY |X ( y | x′ ) for any one element x′ in each equivalence class is sufficient to predict Y . An extreme case of this would be the constant function which arguably has the simplest possible structure of any task TX , Y . The equivalence class in this case is just X since pY |X ( y | x ) is same for all x ∈ X . Thus , the number of equivalence classes for the constant function is 1 ( the minimum possible ) . This is expected since for a constant function there is no structure to learn from data whatsoever ! The following lemma ( see Appendix A.1 for a proof ) relates CQ ( X ; Y ) and the number of equivalence classes , the latter being not easy to compute from finite samples . Proposition 1 . Given a finite query set Q , b-valued query-answers { q ( X ) } q∈Q and any δ > 0 , the number of equivalence classes induced by the minimizer of ( 2 ) can be upper bounded by bCQ ( X ; Y ) +|Q| √ 2 log ( 1δ ) ) with probability of misclassifying X at most δ . Proposition 1 indicates that for the sameQ , tasks with greaterCQ ( X ; Y ) will have larger complexity ( by comparing the upper bound on the number of equivalence classes ) . This also illustrates the role different bases play . For example , in visual recognition tasks if one queries intensities of all the pixels in the image at once then CQ ( X ; Y ) = 1 ( Observing intensities at all the pixels is sufficient information to predict Y from X ) . However b in that case would be large ( exponential in the size of the image ) . Instead if one queries individual pixels , b would be the number of intensity values each pixel can take but CQ ( X ; Y ) ≥ 1 . Properties of CQ ( X ; Y ) . The following proposition , whose proof can be found in Appendix A.2 , establishes some key properties of our proposed measure of complexity . Proposition 2 . For any query setQ that is sufficient for task TX , Y , CQ ( X ; Y ) satisfies the following properties : 1 . CQ ( X ; Y ) ≥ 0 . ( non-negativity ) 2 . CQ ( X ; Y ) = 0 iff X ⊥⊥ Y . ( trivial structure ) 3 . If ∀x , x′ ∈ X , x 6= x′ , ∃y ∈ Y , such that pY |X=x ( y ) 6= pY |X=x′ ( y ) , then CQ ( X ; Y ) ≥ CQ ( X ; Ỹ ) for all tasks TX , Ỹ provided Q is sufficient for TX , Ỹ . ( total structure ) 4 . CQ ( X ; Y1 , Y2 ) ≤ CQ ( X ; Y1 ) + CQ ( X ; Y2 ) for any two tasks with X ∼ pX ( x ) and Y1 ⊥⊥ Y2 | X . ( sub-additivity under union ) The property “ trivial structure ” captures the fact that if Y is independent of X , the learning task is trivial ( i.e. , there is nothing to be learned about Y from observing X or functions of X ) . The property “ total structure ” captures the intuition that if for a given task TX ; Y , the conditional distribution functioini pY |X=x ( y ) is different ∀x ∈ X , then learning is “ impossible ” ( assume Y is a categorical variable , which it is for most classification tasks ) as no inductive bias would help in generalization to unseen inputs for such a task . For example , Y = f ( X ) where f : X → Y is injective . The last property “ sub-additivity under union ” is especially interesting in transfer learning settings where source task TX ; Y1 and target task TX ; Y2 have the same marginal for X ∼ pX ( x ) but different conditionals pY1|X ( y1 | x ) and pY2|X ( y2 | x ) . CQ ( X ; Y1 , Y2 ) refers to the complexity of task TX ; Y1 , Y2 , defined by pXY1Y2 ( x , y1 , y2 ) , where the corresponding sufficiency constraint in ( 2 ) becomes p ( y1 , y2|x ) = p ( y1 , y2|CodeEQ ( x ) ) ∀x ∈ X , y1 ∈ Y1 , y2 ∈ Y2 . ( 5 ) This property could potentially be exploited to predict the success of transfer learning for different choices of source and target tasks . Further , the assumption Y1 ⊥⊥ Y2 | X is not particularly strict as it simply implies given input X , knowledge of Y1 is not required to predict Y2 and vice-versa . -approximate task complexity . In practice , we are often interested in solving a task “ approximately ” rather than “ exactly ” . To accommodate this , we extend the definition of Sufficiency in ( 2 ) to -Approximate Sufficiency by relaxing it to d ( p ( y | x ) , p ( y | CodeEQ ( x ) ) ) ≤ ∀x ∈ X . Here , d is any distance-like metric on distributions such as the KL-divergence , total variation , Wasserstein distance , etc . ( Refer Appendix A.3 ) . Having established the above properties of our complexity measure , we note that unfortunately for any general query set Q , the problem in ( 2 ) is known to be NP-Complete and hence generally intractable ( Laurent & Rivest , 1976 ) . As a result , we instead consider a greedy approximation to CQ ( X ; Y ) via an algorithm called Information Pursuit ( IP ) introduced by Geman & Jedynak ( 1996 ) , which we describe in detail next .

Summary: This paper proposes a method to quantify the complexity of a learning task. The paper is motivated from the “20 questions“ game where an agent computes the answer (label) via a sequence of questions asked on the input data with answers given by an Oracle (simple functions of the data, in this case). The authors formalize this process and define the complexity of a learning task as the smallest number of questions from a given set Q necessary to predict the labels accurately averaged across the dataset. Information Pursuit (IP) of Geman & Jedynak 1996 is used to instantiate this definition using variational and normalizing-flow based models to learn the conditional distributions. Experimental results are shown for MNIST, Fashion-MNIST, KMNIST and Caltech Silhouettes datasets.