PCNN: Environment Adaptive Model Without Finetuning

1 INTRODUCTION . Convolutional Neural Networks ( CNNs ) have achieved tremendous success for many computer vision tasks , such as image classification , object detection , and image segmentation . The success on these vision tasks has fueled the desire to deploy CNNs on mobile platforms , due to the pervasive use of mobile and wearable devices , and the rise of emerging applications such as self-driving car . While the deployment of CNNs on mobile platforms provides a fascinating perspective , two significant but conflicting challenges stand in the way of the future deployment : accuracy and resourceefficiency . The strict resource requirements on mobile platforms generally forces the deployment of a simpler model with lower accuracy , as trading accuracy for resource-efficiency . Therefore , how to improve the accuracy given the limited resource budget has become an important research topic in the filed of deep learning , and also is the focus of this paper . An emerging opportunity to resolve the conflict between accuracy and resource-efficiency on mobile platforms is the class skew , i.e. , the strong temporal and spatial locality of the appearance of classes . For example , only a few people may appear in our lab , even if we may meet thousands of people through the whole year . While a complex model with thousands of classes is required to classify thousands of people , a small model with less than 10 classes can be sufficient in the lab . Less number of classes indicates a higher accuracy and less resource consumption . For example , if we randomly guess from 1000 classes , the accuracy is 0.1 % , while the accuracy would increase to 10 % for randomly guessing from 10 classes . This increase in accuracy provides more space on pruning the architecture , since the main constraint of pruning methods is the decrease in accuracy . The fascinating perspective of both increasing accuracy and decreasing resource consumption motivates the utilization of class skew on mobile platforms . While utilizing class skews provides a fascinating perspective , it is challenging to adapt the model efficiently towards the class skew during runtime , especially when we consider the frequent switches of class skews , e.g. , every 10 minutes . It is also infeasible to pre-train a sequence of models for each class skew , since there is a mind-bogglingly huge number of class skews . For example , if we take 10 out of 100 classes , there would be 1.731014 class skews . Existing works choose to fine-tune the model towards the class skew during runtime when class skew switches , based on the technique from transfer learning . With transfer learning , the number of nodes in the last layer will be reduced according to class skew and last few layers will be fine-tuned by several epochs , which introduce lots of computation overhead and latency . A 14-second or even minutes latency will occur every time the class skew switches and the model is adapted . To efficiently adapt the model during runtime towards the class skew , we bring up probability layer , a novel add-on module with contributions as the following . • No overhead is introduced while probability layer can still adapt the model towards the class skew , achieving an equivalent or better accuracy than transfer learning . • An effective Class Skew Detector ( CSD ) is proposed to efficiently detect the class skew , serving as the base for model adaption . • Both increasing accuracy and decreasing resource consumption can be achieved by probability layer when combined with pruning techniques , which is critical to the successful deployment of CNNs on mobile platforms . 2 RELATED WORKS . Environment information Environment information is a promising approach for energy-efficiency in the deployment of deep learning techniques . ( Kang et al . ( 2017 ) ) skips redundant images from surveillance cameras fixed at a crosswalk , by comparing the input stream with the fixed background . ( Jiang et al . ( 2018 ) ) utilizes the temporal redundancy of input stream to reduce the processing rate while keeping a similar accuracy . In this paper , we utilize another environment information , class skew , which is complimentary to existing approaches . Our approach shares some similarities with the previous work ( Han et al . ( 2016 ) ; Shen et al . ( 2017 ) ) in utilizing class skew during runtime . Compared to their approach that uses transfer learning to adapt the model towards class skew , we can adapt the model without any overhead and achieving an equivalent or better performance , which suits better with the nature of limited resource budget on mobile platforms . Transfer learning Transfer learning is currently the dominant method for domain adaption problems . In transfer learning , we will have a source domain and a target domain . A model will be pretrained on the source domain and the last few layers will be retrained on the target domain such that the model can handle testing data from the target domain ( Doersch et al . ( 2015 ) ; Han et al . ( 2016 ) ; Oquab et al . ( 2014 ) ; Shen et al . ( 2017 ) ; Yosinski et al . ( 2014 ) ) . Benefits of transfer learning have been shown in various areas , including utilizing unlabeled data ( Doersch et al . ( 2015 ) ; Noroozi & Favaro ( 2016 ) ) and reconciling the lack of data in the target domain ( Oquab et al . ( 2014 ) ; Yosinski et al . ( 2014 ) ) . While the transfer learning shows merit in domain adaption and utilized in ( Han et al . ( 2016 ) ; Shen et al . ( 2017 ) ) for runtime model adaption towards class skews , it does not fit well with the nature of limited resource budget and desired property of low latency on mobile platforms . It is also hard to conduct transfer learning without human guidance on mobile devices , due to the complex decision on how many layers we should fine-tuning , as reported by ( Yosinski et al . ( 2014 ) ) , and the hyper-parameters utilized in the training process , e.g. , number of epoches and learning rate , which has important influence over the fine-tuning process . Comparing to transfer learning , our approach fits better with the limited resource budget on mobile platforms by requiring no fine-tuning for model adaption while still provides an equivalent or better accuracy . Pruning methods Pruning methods utilize various model redundancy to decrease resource consumption with the cost of accuracy . Quantization replace convolutional filters with quantilized vectors ( Bagherinezhad et al . ( 2017 ) ) to reduce computation and replace 32-bit weights with 8-bit weights ( Han et al . ( 2015 ) ) to use less memory . Filter pruning ( Lin et al . ( 2018 ) ) reduces computation by decreasing number of filters that has little influence over the accuracy . Fabrics ( Saxena & Verbeek ( 2016 ) ; Huang et al . ( 2017a ) ; Tom & Ludovic ( 2018 ) ) targets for discovering the optimal architecture by explore an exponentially large number of architectures . While pruning methods reduce resource consumption , accuracy is also decrease . Instead , both decreasing resource consumption and increasing accuracy can be achieved during runtime without any overhead , through combining pruning methods with probability layer . 3 PROBABILITY LAYER . In this section , we introduce the probability layer and the PCNN . Before proceeding further , we introduce the notation that will be used in the rest of the paper . Notation . CNN can be viewed as a feed-forward multi-layer architecture that maps the input images X to a vector of estimated probability for each class ~p = ( p1 , p2 , ... , pn ) , where n is the number of classes and pi = P ( i|X ) is the estimated probability pi for the label i given the input image X . Key Assumption . The main difference between the proposed layer and the original CNNs is that we take the environment information into consideration . In the original CNNs , the prediction for each image will be made individually , assuming a sequence of images is independent and identically distributed ( i.i.d ) . However , in real life , this assumption does not hold and strong temporal and spatial locality may exist . Instead , we assume that class skew exists , which means the number of classes and class types are fixed during a time period . Meanwhile , it is still possible that the class skew switches to another class skew after a few minutes , which can be detected and handled by the profiler , as detailed in section 4 . Further , we assume the existence of a model classifying all possible classes , called the general model . This assumption is reasonable because modern CNNs can be trained to classify thousands of classes ( Krizhevsky et al . ( 2012 ) ; Simonyan & Zisserman ( 2014 ) ; Szegedy et al . ( 2015 ) ; He et al . ( 2016 ) ; Huang et al . ( 2017b ) ) . Intuition . Probability layer helps by using environment information that original CNNs does not use . When human recognizes an object , both vision and environment information will be used , i.e. , what we have seen recently and which objects may appear here . However , CNNs can only make use of visual information while discarding environment information , which makes it extremely difficult to distinguish classes with similar shapes . For example , Figure 1a and 1b shows images from CIFAR-100 for bottle and rocket respectively . It is hard to distinguish these two classes only from images while environment information can easily rule out rocket in most scenarios . Figure 1 gives intuition on how probability layer utilize environment information . In Figure 1c , the lower row represents the outputs from softmax layer and the upper row represents the probability layer . The orange nodes stand for the classes with high predicted probability in softmax layer and the red nodes stand for the suggestion from the environment . The prediction from probability layer will be selected from the intersection of the set of red nodes and orange nodes , which rules out confusing classes for CNNs . Approach . Probability layer is an extra layer after the CNN model , rescaling the output of softmax layer . Rescaling is a topic in statistics ( Saerens et al . ( 2002 ) ) . To the best of our knowledge , we are the first to discuss rescaling in CNN context . The outputs of original CNNs predict the probability for each class and the probability layer will adjust this prediction based on the difference of class distribution in training and testing dataset . In particular , for classes with different distributions in training and testing dataset , the probability layer will rescale the corresponding outputs from softmax layer according to the difference in distribution . For other classes with the same distribution in both training and testing dataset , the outputs of the proposed layer are equal to the outputs of the softmax layer . The probability layer will take as input the originally predicted probability , class distribution in training dataset , as well as the distribution in testing dataset , and output a vector for the rescaled prediction . The first input is the prediction vector P ( ·|X ) from the softmax layer , which represents the originally predicted probability for each class from the original CNNs . The second input is a vector of class distribution P ( · ) in training dataset and the third one is a vector of class distribution Pt ( · ) in testing dataset . The probability layer will rescale the predicted probability in P ( ·|X ) element wisely and produce as output a vector Pt ( ·|X ) for the rescaled prediction of each class . Formally , let the outputs of CNNs with and without rescaling are Pt ( i|X ) = Pt ( X|i ) · Pt ( i ) Pt ( X ) ( 1 ) and P ( i|X ) = P ( X|i ) · P ( i ) P ( X ) ( 2 ) respectively . Here Pt ( i ) means the class distribution in testing dataset and Pt ( i|X ) represents the predicted probability for class i after the probability layer . We assume that Pt ( X|i ) equals P ( X|i ) approximately , where P ( X|i ) is the distribution of image data for class i . This assumption makes sense since , for a class i , the selection of input x is random . Through transforming equation 4 and equation 5 as well as utilizing Pt ( X|i ) = P ( X|i ) , we can derive that Pt ( i|X ) = Pt ( i ) P ( i ) · P ( X ) Pt ( X ) · P ( i|X ) . ( 3 ) Considering ∑n i=1 Pt ( i|X ) = 1 , we can get the rescaling formular as Pt ( i|X ) = Pt ( i ) P ( i ) · P ( i|X ) ∑n j=1 Pt ( i ) P ( j ) · P ( j|X ) ( 4 ) To give probability layer the ability to detect new classes , we choose not to rescale the outputs from softmax layer when the original model has strong confidence in its prediction and set the formula of probability layer as Pt ( i|X ) = Pt ( i ) P ( i ) · P ( i|X ) ∑n j=1 Pt ( i ) P ( j ) · P ( j|X ) · I { P ( i|X ) < ω } + P ( i|X ) · I { P ( i|X ) > =ω } , ( 5 ) where ω is the threshold above which we should trust the original prediction and IX is the indicator function such that IX ( x ) = ∆ if x ∈ X , return 1 , otherwise return 0 . If a model has a strong confidence in its prediction , the accuracy would be much higher than the model ’ s average accuracy . Our experiments show that CNNs will give most of the images high predicted probability and the accuracy of these images will exceed average accuracy a lot . Probability layer helps when the original model is confused on the prediction and will not interfere with the decision when the original model has confidence in its prediction .

Summary: This paper proposed a way to detect a skew in the distribution of classes in a stream of images and reweight the class priors accordingly, to estimate the final posterior probabilities of present classes. This probability re-calibration is referred to as the probability layer. A simple algorithm is proposed to detect the class distribution skew. The proposed benefit of this method is that they do not require fine-tuning any network parameters using newly skewed data.