More Side Information, Better Pruning: Shared-Label Classification as a Case Study

1 INTRODUCTION . Pruning Neural networks , the task of compressing a network by removing parameters , has been an important subject both for practical deployment and theoretical research . Some pruning algorithms have focused on manipulating pre-trained models , ( Mozer & Smolensky , 1989 ; LeCun et al. , 1990 ; Reed , 1993 ; Han et al. , 2015 ) while recent work have identified that there exist sparse subnetwork ( also called winning tickets ) in randomly-initialized neural networks that , when trained in isolation , can match and often even surpass the test accuracy of the original network ( Frankle & Carbin , 2019 ; Frankle et al. , 2020 ) . There is a vast literature on network pruning , and we refer the reader to Blalock et al . ( 2020 ) ; Sze et al . ( 2017 ) ; Reed ( 1993 ) for an excellent survey . In this work , we adopt the pruning methods of Tanaka et al . ( 2020 ) ; Lee et al . ( 2019 ) ; Wang et al . ( 2020 ) ; Han et al . ( 2015 ) which have been influential in our experiments . More crucially , most literature on pruning has been focused on designing a machine that converts a fixed deep learning solution to a prediction problem , to a more efficient version thereof . The pruning machine has a compression knob which trades off the level of pruning with accuracy of the prediction . The more resources we are willing to expend in prediction ( measured here using floating-point operations ( FLOPs ) ) , the more information we can obtain , where information here is measured as prediction accuracy , or as reduction of uncertainty ( defined below ) . We now ask what happens when we want to prune a network , but also possess information on the prediction coming from another source . Intuitively , given some form of additional side information , we should be able to prune our network with a higher compression ratio to reach the same level of accuracy for the prediction task , compared with a scenario with no additional side information . But how can we take the side information into account when pruning ? 1.1 MOTIVATION . This question was motivated by an actual real-life scenario . We describe the scenario in detail , although the actual problem we thoroughly study in what follows is much simpler . Imagine a database retrieval system with a static space of objects X . Given a query object q , the goal is to return an object x from X that maximizes a ground-truth retrieval value function fq ( x ) . We have access to a function f̃q ( x ) expressed as a deep network , which approximates fq , and was trained using samples thereof . The function f̃q is very expensive to compute . ( Note that we keep q fixed here , as part of the definition of fq ( · ) , although in an actual setting both q and x would be input to a bivariate retrieval function f̃ . ) Computing f̃q ( x ) for all x ∈ X is infeasible . One way to circumvent this is by computing a less accurate , but efficient function ˜̃fq ( · ) , defined by the network resulting in a pruning of the network defining f̃q . Then compute ˜̃ fq ( · ) on all x ∈ X to obtain a shortlist of candidates X ′ , and then compute f̃q ( x ) on x ∈ X ′ only . This idea can also be bootstrapped , using rougher , more aggresively pruned estimates ˜̃̃ fq , f̃ ( 4 ) q , f̃ ( 5 ) q ... and increasingly shorter shortlist . However , an important point is ignored in this approach : The space X is structured , and we expect there to be prior connections between its elements . This is the side information . Such connections can be encoded , for example , as a similarity graph over X where it is expected that fq ( x1 ) is close to fq ( x2 ) whenever there is an edge between x1 , x2 . There is much work on deep networks over graphs ( Zhou et al. , 2018 ; Kipf & Welling , 2017 ; Wu et al. , 2020 ) . But how can the extra information , encoded as a graph , be used in conjunction with the pruning process ? Let us simplify the information retrieval scenario . First , assume that we are in a classification and not in a regression scenario , so that fq ( x ) can take a finite set of discrete values , and f̃q ( x ) returns a vector of logits , one coordinate per class . Second , assume the side information on X is a partitioning of X into cliques , or clusters X1 ... Xk where on each clique the value of fq ( · ) is fixed , and written as fq ( Xi ) , i = 1 .. k. Now the problem becomes that of estimating the fq ( Xi ) ’ s using n random samples xi1 ... xin ∈ Xi , i = 1 .. k. 1 Fixing the cluster Xi , one obvious thing to do in order to estimate fq ( Xi ) is to take an average of the logit vectors f̃q ( xi1 ) ... f̃q ( xin ) , where f̃q is some fixed ( possibly pruned ) network , and use the argmax coordinate as prediction . Assuming each pruned network f̃q outputs a prediction vector with a certain level of uncertainty , the averaged vector should have lower uncertainty , and this can be quantified using simple probabilistic arguments . This will henceforth be called the baseline method . Intuitively the baseline method , though easy to do using out-of-the-box pruning libraries , can not possibly be optimal given the side information of same label across Xi . Indeed , the baseline method feeds all the examples xi1 ... xin independently through separate instantiations of f̃q , and nothing 1Continuing the retrieval story , the practitioner would now find the Xi that maximizes fq , and then further focus the search in that cluster . prevents the different instantiations to learn overlapping pieces of information . Hence it makes sense to somehow interconnect these networks as a meta-network , and possibly do the pruning on the meta-network . In this work , we experiment with several methods for performing this task , and compare our results with the baseline . 1.2 THE SHARED-LABEL PREDICTION PROBLEM . We depart from the original motivating information retrieval scenario , and henceforth consider a simpler , toy problem which we call the shared-label prediction problem . We are given an underlying space of instances X and an unknown ground truth labelling function f : X 7→ Y for some discrete set Y of labels . The goal is to train a classifier that , given a random n-tuple of instances x1 ... xn ∈ Xn sharing the same unknown label y ( so that f ( x1 ) = · · · = f ( xm ) = y ) , outputs a prediction of y . This is the shared prediction problem . Our work is empirical , and the goal is to develop general methods for the shared prediction problem , given a base network , designed for the standard ( non-shared ) prediction problem , and a base pruning method , we ask : How do we reuse and rewire these readily available tools to effectively solve the shared-label prediction problem on tuples of n-instances ? 2 OUR CONTRIBUTION . Below in Section 2.1 we present four methods . Each method uses a baseline CNN model , together with a pruning method with a compression ratio knob ρ , and creates a meta-network that is parameterized by the information size n and by ρ , designed to solve the shared classification problem . To measure our success , we will both use a measure of accuracy as well as a measure of relative information which we define below . We will compute these measures extensively over a grid of possible pairs ( n , ρ ) , for each method . Visualization of the results highlights an interesting invariant that is worth studying . Intuitively , the measure of relative information tells us how efficiently each method uses its computational resource , without wasting time on computing the same pieces of information over and over on the n-tuple of instances . Therefore , it allows us to obtain a quantitative comparison between the methods . To define the measure , we first recall some information theory . Given a random variable Y over a discrete space , the Shannon entropy , or uncertainty of Y is H [ Y ] = − ∑ Pr [ Y = y ] log Pr [ Y = y ] , where the sum ranges over possible values of Y . In our case , we will use H ( Y ) to measure the uncertainty in the label of a randomly drawn instance , which is also the uncertainty in the label of a randomly drawn n-tuple in the shared label setting . Given a random variable Ỹ ( an estimate of Y ) , the information gain measures the difference between the entropy of Y and the expectation with respect to Ỹ of H ( Y |Ỹ ) . More precisely , IG ( Y ; Ỹ ) = H ( Y ) − EỸ [ − ∑ y Pr [ Y = y|Ỹ ] log Pr [ Y = y|Ỹ ] ] . Note that information gain is symmetrical , that is IG ( Ỹ ; Y ) = IG ( Y ; Ỹ ) . Therefore it is also called mutual information and denoted I ( Ỹ ; Y ) . In our setting , Ỹ will be an estimator of Y obtained using the output of the network on an n-tuple of instances in the shared label setting , and I ( Ỹ ; Y ) will measure the expected amount of information we learn about Y using the network output on that tuple . For a given network , we will be computing IG ( Ỹ ; Y ) empirically in what follows , by taking Ỹ to be the prediction obtained by selecting the argmax coordinate ( logit ) of the output of a network . Given a method for the shared-label scenario , we define the relative information ( RI ) to be RI ( Ỹ , Y , n , ρ ) = IG ( Ỹ ; Y ) n/ρ . In words , this is a measure of information that the network learns , per computational cost . The denominator n/ρ is a reasonable measure of computational cost for the methods we study , because for these methods , the amount of computational effort we expend for shared label instance x1 ... xn is proportional to n , and inverse proportional to the compression ratio ρ . We believe it is also a reasonable measure of computational cost for other natural methods . For all methods we study , fixing the information size n , our experiments suggest that there exists a sweet spot phenomenon , or a `` compression threshold '' in the sense that RI , as a function of ρ , has a global maximum ρ∗ . If the compression ratio ρ is smaller than ρ∗ , than we are at the undercompressed regime , where we can still save computational resources without relatively deteriorating the results , or the information , to a large extent . On the other side , if the compression ratio ρ is bigger than ρ∗ , than we are at the over-compressed regime , where we can gain a lot more information , by using a relatively mere amount of computational resources . We believe that a better understanding of this phenomenon can shed light on the interaction between different compression ratios , information sizes , and the information gains achieved by the methods ( which is equivalent to test performances , as our experiments show ) . We show that the above is a robust phenomenon that occurs in a variety of settings .

Summary: The authors study how to improve the prediction and pruning performance with additional information generated by labels in the shared-label classification problem. As a starting point, the authors consider a simple scenario where side information can be extracted from the same labeled batch. To train the neural network, the authors use a balanced loss consisting of a weighted sum of general cross-entropy and cross-entropy of average batch prediction. The authors also suggest a new CNN-LSTM architecture to improve predictive performance to exploit the side information. The experiments section shows the proposed method performs well and achieves a high compression rate.