Quantifying Task Complexity Through Generalized Information Measures

1 INTRODUCTION . Deep networks have shown remarkable progress in both simple and complex machine learning tasks . But how does one measure the “ complexity ” of a learning task ? Is it possible to ascertain in a principled manner which tasks are “ harder ” to solve than others ? How “ close ” is one task to another ? Answers to these questions would have implications in many fields of machine learning such as transfer learning , multi-task learning , un/semi/self-supervised learning , and domain adaptation . In classical information theory , the entropy of a random variable X is a useful measure of complexity for tasks such as compression and transmission , which essentially require reconstructing X . However , the entropy ofX is insufficient for measuring the complexity of a supervised learning task TX , Y , where the goal is to predict an output Y from an input X , i.e. , to estimate the conditional pY |X ( y | x ) from a finite set of samples from pXY ( x , y ) , which we refer to as solving the learning task . Complexity measures provided by statistical learning theory like VC-dimension or covering numbers are also inadequate for this purpose because they ignore the dependence between X and Y for the particular task at hand . Information-theoretic measures such as mutual information , information bottleneck ( Tishby et al. , 2000 ) and its variants ( Strouse & Schwab , 2017 ) have been used to study the trade-off between model complexity and accuracy , but have not been developed to focus on assessing task complexity and can provide unsatisfactory results when comparing different tasks ( see Section 5 for details ) . Measures based on Kolmogorov complexity ( Li , 2006 ; Vereshchagin & Vitányi , 2004 ) could in principle be used to compare different tasks , but they are dataset permutation sensitive and not easily computable . The work of ( Achille et al. , 2019a ) proposes to quantify task complexity by measuring the information stored on the network weights , but the approach depends on the specific neural network architecture used for training . The work of ( Tran et al. , 2019 ) does not require or assume trained models , but makes strict assumptions that limit its broad applicability . In this work , we introduce a novel perspective on task complexity which generalizes classical measures from information theory . Specifically , one well-known interpretation of classical Shannon entropy is , given a random variable X , find the minimum number of bits that are needed on average to encode instances of X so that the instances can be perfectly recovered from the binary code . Stated differently , if one lets Q be defined as the set of all possible binary functions , on the domain of X , then Shannon entropy essentially asks what is the optimal sequence of queries to compute { q1 ( X ) , q2 ( X ) , . . . : qi ∈ Q } ( i.e. , how to encode X as a binary string ) so that X can be perfectly recovered from the shortest ( on average ) sequence of binary answers to the queries ( see Section 2 for more discussion of this interpretation ) . As discussed above , however , in most learning tasks we are not interested in simply compressing X but rather making a prediction about some other variable Y . Further , notions of complexity can potentially be made more relevant to a specific task by not having Q to be the set of all possible binary functions on X but rather a smaller set of queries specific to a measure of interest . From this intuition , we define the complexity of a learning task as the minimum expected number of queries , selected from a fixed set Q , one needs to ask to predict Y ( to some user-defined level of confidence ) from the respective answers to the queries . As a few specific examples of potential query sets : • Decision boundary complexity : Here , Q is the the set of all possible half-spaces in Rd ( assuming X ∈ Rd ) and q ( x ) is a binary function response indicating whether x lies in a particular half-space ( q ∈ Q ) . Then task complexity formalizes the intuition of “ level of non-linearity ” of the decision boundary . For example , the complexity of any linearly-separable binary classification task is 1 , whereas , for a non-linearly separable task , this value depends on the curvature of the decision boundary . • Task-specific input feature complexity : Here , Q is the set of projection functions of X and q is of the form q ( x ) = xq , where xq is the value observed at the qth entry of x . Then task complexity formalizes the intuition “ the greater the redundancy between the input features the easier it is to solve the task ” . For example , Y being a constant function of X would be the simplest task with complexity 0 , since no feature needs to be queried to predict it . This notion of complexity would help in answering questions such as “ which input features are most important for solving a given task ? ” and could in turn help in developing more “ interpretable ” learning algorithms . • Visual semantic complexity : Given a vocabulary V of different possible entities , their attributes and relations in a visual scene , Q could be the set of all binary functions indicating the presence or absence of an entity , its attribute or its relation with other entities ( supplied by V ) in a designated region of the image X . For example , a particular q could be the function implementing the query “ Is there a person in the top left corner of the image ? ” . This notion of complexity would allow one to gauge the semantic complexity of a visual task . For instance , tasks which ask complex questions like “ Is there a person playing with his dog , next to a river in the image ? ” would inherently be more complex than simple object detection tasks , “ Where is the dog in this image ? ” and could be quantified by semantically relevant queries . While our proposed formal definition of task complexity will be applicable to all such choices of query functions { q } q∈Q and enjoys several nice theoretical properties that we discuss in section 2 , its computation will generally be intractable . As a result , in section 3 we propose to reduce the complexity of selecting a minimal set of questions by using the Information Pursuit ( IP ) algorithm , which selects questions sequentially , depending on previous questions and answers , in order of information gain . While IP is generally applicable to any task and query set , its implementation is still intractable depending on the complexity of the model p ( X , Y ) and of the set Q . To address this issue , we posit a tractable graphical model for p ( X , Y ) and learn the required distributions using variational autoencoders and normalizing flows . In section 4 we evaluate our approach on various binary image classification tasks ( MNIST , KMNIST , FashionMNIST , Caltech Silhouettes ) that can be tackled using a common set of queries ( the set of image patches ) . Our results show that complexity computed using patch queries aligns with the intuition that the complexity of a classification task increases as signal-to-noise ratio decreases , and that classification of the KMNIST dataset is more complex than classification of the FashionMNIST dataset , something that isn ’ t obvious a priori . While these experiments are restricted to simple tasks and queries , the proposed framework is generally applicable provided that tractable models , inference and learning methods can be developed , which is the subject of ongoing and future work . Finally , we note that to the best of our knowledge , this is the first time that a subjective notion of task complexity has been proposed in literature , where the user can incorporate in Q the perception of complexity he/she wishes to measure . 2 QUANTIFYING TASK COMPLEXITY . Let the input data be represented by random variable X and the corresponding output/hypothesis by random variable Y with sample spaces X and Y respectively . A supervised learning task TX ; Y is defined as the task of estimating the conditional distribution pY |X ( y | x ) from a finite set of samples from the joint distribution pXY ( x , y ) .1 We propose to quantify the complexity of a task as the minimum expected number of queries needed to solve the task . Queries , q ∈ Q , are functions of the input data , whose answers for a given x are denoted as { q ( x ) } q∈Q . Naturally , the query set needs to be sufficiently rich so that the task is solvable based on answers to the queries . More formally , we say the set Q is sufficient for task TX ; Y if ∀ ( x , y ) ∈ X × Y , p ( y | x ) = p ( y | { x′ ∈ X : q ( x′ ) = q ( x ) ∀q ∈ Q } ) . ( 1 ) In other words , Q is sufficient for task TX ; Y if whenever two points x and x′ have identical answers for all queries in Q , then their true posteriors must be equal , p ( y | x ) = p ( y | x′ ) . Given a fixed query set Q , we now formally define an encoding function EQ , which we refer to as a Q-Coder , along with our proposed complexity measure CQ ( X , Y ) for task TX , Y . Defining the Encoder function . Given a query set Q , an Encoder is a function , E : S∗ → Q where S∗ is the set of all finite-length sequences generated using elements from the set S = { ( q , q ( x ) ) | q ∈ Q , x ∈ X } . Additionally , we require that Q contains a special query qSTOP which signals the encoder to stop asking queries and outputs the code for x as the set of query-answer pairs that have been asked . The process can be described as follows , given E and input sample x . 1. q1 = E ( ∅ ) . The first question is independent of x . 2. qk+1 = E ( { qi , qi ( x ) } 1 : k ) . All subsequent queries depend on the query-answer pairs ob- served so far for x . 3 . If qL+1 = qSTOP terminate and return CodeEQ ( x ) : = ( qi , qi ( x ) ) 1 : L as the code for x . Note that each qi depends on x , but we drop this in the notation for brevity . Note also that the number of queries L = |CodeEQ ( x ) | for a particular x generalizes coding length in information theory . The query qSTOP constrains the code CodeEQ ( x ) to be prefix-free . Defining task complexity . Given a joint distribution pXY ( x , y ) and a sufficient query set Q , we define task complexity , CQ ( X ; Y ) , as the minimum over all encoders E ( which are mappings from X to a subset of queries in Q ) of the mean coding length : CQ ( X ; Y ) : = min E EX [ |CodeEQ ( X ) | ] s.t . p ( y | x ) = p ( y | CodeEQ ( x ) ) ∀x ∈ X , y ∈ Y ( Sufficiency ) ( 2 ) Figure 1 : Schematic view of the overall framework for quantifying complexity of a task TX ; Y . The first constraint ensures sufficiency of the code ∀x ∈ X . By this we mean that there exists at least one Encoder E for which the first constraint of “ sufficiency ” is satisfied , where p ( y | CodeEQ ( x ) ) should be interpreted as the conditional probability of y given the event { x′ ∈ X | CodeEQ ( x ) = CodeEQ ( x′ ) } . The solution to ( 2 ) provides the optimal encoder for task TX ; Y , and Fig . 1 illustrates the overall framework in detail . Connection with Shannon entropy H . It can be shown that when Q is taken as the set for all possible binary questions on X and if Y is a function of X ( which it usually is for supervised classification problems ( Kolchinsky et al. , 2018 ) ) , then the solution , E∗ , to ( 2 ) gives a coding length within one bit of the entropy of Y , denoted as H ( Y ) . More formally , one can show that H ( Y ) ≤ CQ ( X ; Y ) = EX [ |CodeE ∗ Q ( X ) | ] < H ( Y ) + 1 . ( 3 ) 1As commonly used , we denote random variables by capital letters and their realizations with small letters . Note that one example of such an optimal encoder , E∗ ( x ) , is given by the Huffman code for Y . Connection with task complexity & equivalence classes . Notions of complexity of an entity/object , often relate to the level of structural regularity present in the entity ( Gell-Mann , 2002 ) . CQ ( X ; Y ) measures the degree of regularity present in the structure of TX ; Y . This structure is implicitly defined by the conditional pY |X ( y | x ) and the query set Q . Notice that any E partitions X into equivalence classes . [ x ] = { x′ ∈ X | CodeEQ ( x ) = CodeEQ ( x′ ) } . ( 4 ) The prefix-free constraint on the codes generated by E ( due to ) ensures that ∀x′ ∈ [ x ] , ∀y ∈ Y , pY |X=x′ ( y ) = pY |X=x ( y ) . It is then natural to relate task complexity with the number of equivalence classes induced by the optimal E∗ . The greater the number of equivalence classes , the higher the complexity of the task . This is because knowing the distribution pY |X ( y | x′ ) for any one element x′ in each equivalence class is sufficient to predict Y . An extreme case of this would be the constant function which arguably has the simplest possible structure of any task TX , Y . The equivalence class in this case is just X since pY |X ( y | x ) is same for all x ∈ X . Thus , the number of equivalence classes for the constant function is 1 ( the minimum possible ) . This is expected since for a constant function there is no structure to learn from data whatsoever ! The following lemma ( see Appendix A.1 for a proof ) relates CQ ( X ; Y ) and the number of equivalence classes , the latter being not easy to compute from finite samples . Proposition 1 . Given a finite query set Q , b-valued query-answers { q ( X ) } q∈Q and any δ > 0 , the number of equivalence classes induced by the minimizer of ( 2 ) can be upper bounded by bCQ ( X ; Y ) +|Q| √ 2 log ( 1δ ) ) with probability of misclassifying X at most δ . Proposition 1 indicates that for the sameQ , tasks with greaterCQ ( X ; Y ) will have larger complexity ( by comparing the upper bound on the number of equivalence classes ) . This also illustrates the role different bases play . For example , in visual recognition tasks if one queries intensities of all the pixels in the image at once then CQ ( X ; Y ) = 1 ( Observing intensities at all the pixels is sufficient information to predict Y from X ) . However b in that case would be large ( exponential in the size of the image ) . Instead if one queries individual pixels , b would be the number of intensity values each pixel can take but CQ ( X ; Y ) ≥ 1 . Properties of CQ ( X ; Y ) . The following proposition , whose proof can be found in Appendix A.2 , establishes some key properties of our proposed measure of complexity . Proposition 2 . For any query setQ that is sufficient for task TX , Y , CQ ( X ; Y ) satisfies the following properties : 1 . CQ ( X ; Y ) ≥ 0 . ( non-negativity ) 2 . CQ ( X ; Y ) = 0 iff X ⊥⊥ Y . ( trivial structure ) 3 . If ∀x , x′ ∈ X , x 6= x′ , ∃y ∈ Y , such that pY |X=x ( y ) 6= pY |X=x′ ( y ) , then CQ ( X ; Y ) ≥ CQ ( X ; Ỹ ) for all tasks TX , Ỹ provided Q is sufficient for TX , Ỹ . ( total structure ) 4 . CQ ( X ; Y1 , Y2 ) ≤ CQ ( X ; Y1 ) + CQ ( X ; Y2 ) for any two tasks with X ∼ pX ( x ) and Y1 ⊥⊥ Y2 | X . ( sub-additivity under union ) The property “ trivial structure ” captures the fact that if Y is independent of X , the learning task is trivial ( i.e. , there is nothing to be learned about Y from observing X or functions of X ) . The property “ total structure ” captures the intuition that if for a given task TX ; Y , the conditional distribution functioini pY |X=x ( y ) is different ∀x ∈ X , then learning is “ impossible ” ( assume Y is a categorical variable , which it is for most classification tasks ) as no inductive bias would help in generalization to unseen inputs for such a task . For example , Y = f ( X ) where f : X → Y is injective . The last property “ sub-additivity under union ” is especially interesting in transfer learning settings where source task TX ; Y1 and target task TX ; Y2 have the same marginal for X ∼ pX ( x ) but different conditionals pY1|X ( y1 | x ) and pY2|X ( y2 | x ) . CQ ( X ; Y1 , Y2 ) refers to the complexity of task TX ; Y1 , Y2 , defined by pXY1Y2 ( x , y1 , y2 ) , where the corresponding sufficiency constraint in ( 2 ) becomes p ( y1 , y2|x ) = p ( y1 , y2|CodeEQ ( x ) ) ∀x ∈ X , y1 ∈ Y1 , y2 ∈ Y2 . ( 5 ) This property could potentially be exploited to predict the success of transfer learning for different choices of source and target tasks . Further , the assumption Y1 ⊥⊥ Y2 | X is not particularly strict as it simply implies given input X , knowledge of Y1 is not required to predict Y2 and vice-versa . -approximate task complexity . In practice , we are often interested in solving a task “ approximately ” rather than “ exactly ” . To accommodate this , we extend the definition of Sufficiency in ( 2 ) to -Approximate Sufficiency by relaxing it to d ( p ( y | x ) , p ( y | CodeEQ ( x ) ) ) ≤ ∀x ∈ X . Here , d is any distance-like metric on distributions such as the KL-divergence , total variation , Wasserstein distance , etc . ( Refer Appendix A.3 ) . Having established the above properties of our complexity measure , we note that unfortunately for any general query set Q , the problem in ( 2 ) is known to be NP-Complete and hence generally intractable ( Laurent & Rivest , 1976 ) . As a result , we instead consider a greedy approximation to CQ ( X ; Y ) via an algorithm called Information Pursuit ( IP ) introduced by Geman & Jedynak ( 1996 ) , which we describe in detail next .

Summary: The paper claims that existing measures of complexity such as entropy are not suitable for measuring task complexity since they focus on the complexity of X rather than the predictive relationship from X to Y. The paper argues that mutual information is not useful for comparing different learning tasks, as two tasks (MNIST vs Fashion-MNIST) can have similar MI but intuitively different complexity (second-to-last paragraph in related work). The paper proposes a measure for task complexity based on the number of queries required to predict a label of an input. The form of the queries is not specified, and the provided examples include half-space queries, single feature (decision-tree-like) queries, or high level semantic queries.  The proposed method instead considers a query generator E, then encoding X as the answers to the sequence of queries generated by E, and predicting Y from the answers. The complexity of a task is related to the number of equivalence classes induced by the input X and the query generator E.