Enhancing Visual Representations for Efficient Object Recognition during Online Distillation

1 INTRODUCTION . Deep CNNs that are widely used for image classification ( Huang et al . ( 2017 ) ) often require large computing resources and process each image with high computational complexity ( Livni et al . ( 2014 ) ) . In real-world scenarios , the prior probability of occurrence of individual classes in an image stream is often unknown and varies with the deployed environment . For example , in a zoo , the image stream input to the deep CNN will mostly consist of animals , while that of vehicles would be rare . Other object classes such as furniture and aircraft would be absent . Therefore , only a subset of the many classes known to a deep CNN may be presented to it for classification during its deployment . To adapt to the varying prior class probability in the deployed scenario with high efficiency , we propose an online distillation framework - ENVISE . Here , we employ a high capacity general purpose image classifier as the teacher network ( TN ) , while the student network ( SN ) is a low capacity network . For greater efficiency and faster convergence , we require the coefficients of the SN to be binary and refer to it as the binary student network ( BSN ) . When the BSN is first deployed , it is trained on the unlabeled image stream using the predicted labels of the TN as pseudo-labels . Once the BSN converges to the performance of the TN , it is used as the primary classifier to rapidly classify the frequent classes faster than the TN . However , if a rare class appears ( i.e . a class absent during online training ) in the image stream , the BSN must accurately detect it as a class it has not yet encountered , which is then processed by the TN . Since the BSN is trained only on the frequent classes , we refer to these classes as inlier ( IL ) and the rare classes as outlier ( OL ) . It is important to note , that the OL classes are outliers with respect to the BSN only , but are known to the TN . Detecting extremely rare classes which are unknown to both the BSN and TN ( global unknowns ) is beyond the scope of this paper . Thus , assuming that the TN knows all possible classes from the deployed environment , we aim to increase the overall efficiency of the system ( without sacrificing performance ) by exploiting the higher probability of occurrence of frequent classes in a given scenario . Our approach for detecting OL classes is motivated from the observation by Ren et al . ( 2019 ) , where networks incorrectly learn to emphasize the background rather than the semantically important regions of the image leading to poor understanding of the IL classes . We know that attention maps highlight the regions of the image responsible for the classifier ’ s prediction ( Selvaraju et al . ( 2017 ) ) . We empirically observe that , the attention map of the BSN may focus on the background even when the attention map of the TN emphasizes the semantically meaningful regions of the image . In doing so , the BSN memorizes the labels of the TN , making it difficult to differentiate between the representations of IL and OL classes . To mitigate these issues , we propose an attention triplet loss that achieves two key objectives - a ) guide the attention map of the correct prediction of the BSN to focus on the semantically meaningful regions , and b ) simultaneously ensure that the attention map from the correct and incorrect predictions of the BSN are dissimilar . We show that by focusing on the semantically relevant regions of the image , the BSN will learn to distinguish between the representations of IL and OL classes , thereby improving its ability to detect OL classes . To assess the overall gain in efficiency of ENVISE , we propose a new evaluation metric - GiE , based on the number of times the BSN and TN are used to process the image stream . Since the deployed scene is comprised mostly of IL classes with few OL classes , we expect the BSN to be employed most of the time for classifying IL classes . The TN is used rarely i.e . only when the BSN detects an OL class . We refer to efficiency as the overall reduction in FLOPs in the online distillation framework to process the varying prior probability of classes in the image stream . This term differs from conventional model compression techniques ( Frankle & Carbin ( 2019 ) ; Chen et al . ( 2020 ) ) which process the image stream comprising of classes with equal probability using a single compressed model . To the best of our knowledge , we are the first to propose supervision on attention maps for OL detection , and a new evaluation metric that measures the gain in computational efficiency of an online distillation framework . A summary of our main contributions is : • Faster convergence of BSN : We theoretically justify and empirically illustrate that the BSN adapts to the performance of the TN faster than the real-valued SN ( RvSN ) . We also demonstrate the faster convergence of the BSN for different BSN architectures over its corresponding RvSN . • Attention triplet loss ( Lat ) , which guides the BSN to focus on the semantically meaningful regions of the image , thereby improving OL detection . • New evaluation metric - GiE to measure the overall gain in computational efficiency of the online distillation framework , and • We benchmark CIFAR-100 and tiny-imagenet datasets with SOTA OL detection meth- ods by creating meaningful IL and OL class pairs . ENVISE outperforms these baseline methods for OL detection , improves separation of IL and OL classes in the feature space , and yields the highest gain in computational efficiency . 2 RELATED WORK . Online distillation : Distilling knowledge to train a low-capacity student network from a highcapacity teacher network has been proposed as part of model compression ( Hinton et al . ( 2015 ) ) . Wang & Yoon ( 2020 ) provide a detailed review of different knowledge distillation methods . Mullapudi et al . ( 2019 ) propose an online distillation framework for semantic segmentation in videos , while Abolghasemi et al . ( 2019 ) use knowledge distillation to augment a visuomotor policy from visual attention . Lin et al . ( 2019 ) propose an ensemble student network that recursively learns from the teacher network in closed loop manner while Kim et al . ( 2019 ) use a feature fusion module to distill knowledge . Gao et al . ( 2019 ) propose online mutual learning and Cioppa et al . ( 2019 ) propose to periodically update weights to train an ensemble of student networks . These ensemble based methods require large computational resources and are expensive to train . However , ENVISE involves training a compact model that mimics the performance of the TN with less computation . Outlier detection : Outlier detection or out-of-distribution detection ( OOD ) refers to detecting a sample from an unknown class ( Hendrycks & Gimpel ( 2016 ) ) . Existing SOTA OOD methods use outlier samples during training or validation . Yu & Aizawa ( 2019 ) increase the distance between IL and OL samples during training while Vyas et al . ( 2018 ) ; Lee et al . ( 2018 ) add perturbations onto test images and train their network to differentiate between IL and OL samples . However in ENVISE , the BSN is adaptively trained only on IL class images without any knowledge of OL class images , which is a more realistic scenario . Without using outlier classes during training , Hendrycks & Gimpel ( 2016 ) ; Papadopoulos et al . ( 2019 ) ; Bendale & Boult ( 2016 ) ; Hendrycks et al . ( 2019 ) employ class probabilities or measure the similarity between IL and OL features . Yoshihashi et al . ( 2019 ) generate discriminative feature space using reconstruction based approaches . However , the likelihood of correct classification by these networks is based on background regions as described in Ren et al . ( 2019 ) . Hence , these methods fail when applied to real-world IL and OL class pairs since the confidence of prediction is based on background information . We overcome this drawback by proposing supervision on attention maps where we guide the BSN to focus on the important semantic regions using the attention map of the teacher network as a reference . This enables ENVISE to classify IL images with high confidence while also improving OL detection . 3 ONLINE DISTILLATION FOR EFFICIENT OBJECT RECOGNITION . The framework of ENVISE is shown in Figure 1 ( a ) . Initially , the TN ( indicated by “ 1 ” in the figure ) classifies the images from the input stream with high accuracy , albeit at a slower rate . The BSN ( red dotted box ) comprises of an adaptation module ( “ 2 ” ) and a binary CNN ( “ 3 ” ) . Given an image stream , the adaptation module learns the optimum binary weights to mimic the performance of the TN . Once its accuracy converges to that of the TN , the adaption of the real-valued weights stops , but the inference with binary weights continues . The OL detector ( “ 4 ” ) uses the softmax output of the BSN as confidence to either generate the final prediction ( if the confidence is high ) , or treats the image as an OL class and redirects it to the TN for classification . Binarization algorithm for student network : During adaptive training , the BSN does not have access to the labeled data and mimics the behavior of the TN on the IL classes as shown in Figure 1 ( b ) . For an input image x , let f ( g ( wi , j ) , x ) represent the output of the BSN where wi , j are real-valued weights of ith kernel in the jth layer , and g ( . ) is a binarizing function for these weights . During adaptive training , the error between the output of the BSN and TN is minimized by optimizing w∗i , j = arg ming ( wi , j ) ‖L‖ , where L is the overall loss function used for training the BSN . While the intermediate weight wi , j is computed during adaptive learning , the binary versions of these weights w∗i , j = g ( wi , j ) are used for fast computation . After convergence , the real valued weights are discarded , and only the binary weights are used . The question becomes what is a good choice for the binarization function g ( . ) ? Following Rastegari et al . ( 2016 ) , we find a binary vector bi , j ( whose elements are either +1 or −1 ) , and a non-negative scalar αi , j such that w∗i , j = αi , j .bi , j is the best approximation of wi , j in a minimum squared error sense , where αi , j = 1N |wi , j |1 , bi , j = sign ( wi , j ) . However , unlike Rastegari et al . ( 2016 ) where a standalone network is binarized , our BSN learns the w∗i , j during adaptive training in an online distillation framework . In doing so , the BSN compensates for the effects of binarization on the overall loss function . Our motivation for using the BSN is that it classifies the image stream with high accuracy and utilizes fewer FLOPs as compared to the RvSN . This results in an increase in efficiency of the overall framework . Furthermore , the BSN converges to the performance of the TN faster than the RvSN , which we theoretically emphasize in the Lemma below and experimentally validate in Section 4 . We provide the sketch proof here and the complete proof in Appendix A.1 . Lemma 3.1 . Let RvSN and BSN represent the real-valued student network and binary student network respectively with the same network architecture . Let R ( . ) denote the rate of convergence in terms of accuracy when the student network is adaptively trained using the pseudo-labels from the teacher network . Then , R ( BSN ) > R ( RvSN ) for the same image stream and number of iterations . Proof : We assume our image stream as x ( n ) where n = 1 , 2 , 3 ... N , comprises of N samples from the deployed scenario . Since the weights of the BSN are derived from the RvSN , we prove this lemma first for RvSN and then extend it to the BSN . Let w∗ be the optimal weight value which represents network convergence i.e . misclassification error = 0 . The weight update rule using back propagation is given by : w ( n+ 1 ) = w ( n ) + ηx ( n ) [ Since η = 1 ] w ( n+ 1 ) = w ( n ) + x ( n ) ( 1 ) Computing the bounds of the weights of the RvSN , we have n2α2 ‖w∗‖2 ≤ ‖w ( n+ 1 ) ‖2 ≤ nβ ( 2 ) where α = argminx ( n ) ∈C w ? x ( n ) and β = argmaxx ( n ) ∈C ‖x ( k ) ‖ 2 . From eq . 2 , we can say that for this inequality to be satisfied , there exists n ? r which denotes the optimal number of samples for which the RvSN converges i.e . we obtain w ? at n = n ? r . This is given as follows : n ? 2 r α 2 ‖w ? ‖2 = n ? rβ n ? r = β ‖w ? ‖2 α2 ( 3 ) Thus from eq . 3 , we can say that the RvSN achieves convergence after being trained on β‖w ∗‖2 α2 number of samples . Since , the weights of the BSN are derived from the RvSN , we substitute the value of binary weight ŵ∗ from Rastegari et al . ( 2016 ) to obtain n ? b = n ? r N ( 4 ) Here , n ? b is the optimal number of samples for which the BSN converges . Comparing eq . 3 and eq . 4 , we observe that number of n∗b < n ? r i.e . the BSN takes fewer samples to converge to the performance of the TN than RvSN given the same network architecture . Knowledge transfer from teacher to student network : Given an unlabeled image stream , the BSN is trained from the predictions of the TN as hard pseudo-labels using cross-entropy loss , as Ld = − 1N ∑ i yilog ( si ) . Here , y and s are the pseudo-labels generated by the TN and the softmax output of the BSN respectively , N is the total images in the image stream . Once the BSN converges to the performance of the TN , we employ the BSN as the primary classifier during inference . When the deployed scenario does not change for a long duration , the BSN classifies the IL classes faster than the TN without relying on the latter . This improves the overall efficiency of the framework as the TN is now used to classify only the OL classes ( number of OL images < < number of IL images ) . However , to maintain overall accuracy , the BSN must also accurately determine when the image from the input stream belongs to an OL class so that the TN can be used in such cases . We know that attention maps highlight the regions of the image responsible for the classifier ’ s prediction ( Li et al . ( 2018 ) ) . Some examples of the attention maps from the correct predictions of the BSN and the TN are shown in Figure 2 ( a ) . The first three columns in Figure 2 ( a ) show that although the BSN ( adaptively trained using Ld ) and the TN both correctly classify the images , their attention maps are significantly different . The attention map from the correct prediction of the BSN focuses on the background while that of the TN lies on the semantically meaningful regions of the image . Furthermore , the attention maps of the BSN ’ s correct and incorrect predictions are visually similar which causes the BSN to learn incorrect representations of the IL images while correctly classifying them . To better differentiate between IL and OL classes , the BSN should not memorize the predictions of the TN , but learn proper representations of the IL images . To achieve this , we propose the attention triplet loss given in Equation 5 that causes the BSN ’ s attention map to be similar to that of the TN while forcing the attention map from the BSN ’ s correct and incorrect prediction to be different . We use Grad-CAM ( Selvaraju et al . ( 2017 ) ) to compute the attention map A from the one-hot vector of the predicted class . To generate the attention map triplets , we use the attention map from the TN ’ s prediction At as the anchor . The hard positive attention map Asp is obtained from correct prediction of the BSN . When the BSN misclassifies an image ( prediction of BSN and TN are different ) , we use the label from the TN to compute Asp . Motivated by the findings in Wang et al . ( 2019 ) , we observe that the attention maps generated from the correct and incorrect predictions of the BSN are visually similar . Hence , we use the attention map from an incorrect prediction as the hard negative attention map Asn to enforce its separability from Asp . To avoid bad local minima and a collapsed model early on during training , mining suitable hard negatives are important ( Schroff et al . ( 2015 ) ) . Hence , we use the attention map from the second most probable class ( incorrect prediction ) as our hard negative due to its similarity with the hard positive ( third and fourth column in Figure 2 ( a ) ) . Thus , we formulate the attention triplet loss Lat as : Lat = 1 N 1 K N∑ n ( K∑ k ∥∥∥Atk −Aspk∥∥∥2 − K∑ k ∥∥∥Atk −Asnk∥∥∥2 + δ ) ( 5 ) Initially , the hard negative lies within the margin since its squared distance with the hard positive is small . Hence , Lat enforces separation between hard positive and hard negative by a distance greater than the value of margin δ , which we empirically set as 1.5 . N is the total number of samples and K is the total number of pixels in A . Using Lat and Ld , we formulate our final objective function as : Lf = λ1Ld + λ2Lat ( 6 ) where λ1 and λ2 are the weights of Ld and Lat which we empirically set as 1 and 0.2 respectively . The effect of Lat is shown in the last column of Figure 2 ( a ) where the BSN learns proper representations of the IL image due to its attention map being very similar to that of the TN ( second column ) . Furthermore , the improvement in feature space separation between the IL and OL classes in Figure 2 ( b ) shows that the BSN not only forms clusters of the different IL classes ( in red ) , but is also well separated from the OL classes ( in purple ) .

Summary: A knowledge distillation framework is proposed for efficient object recognition. In this framework, the teacher network (TN) performs high accuracy prediction while two student networks (SN) mimic the prediction from TN. The first SN learns from TN while the second SN is a binarized form (BSN) of the first SN.  The design is made for online inference is that the BSN first recognizes the image, leaving the rare category objects to be recognized by the TN. Furthermore, an attention supervision scheme is proposed to enhance the CNN prediction by focusing on meaningful image content.  The proposed method has been validated on CIFAR-100 and Tiny-Imagenet.