Mixture of Pre-processing Experts Model for Noise Robust Deep Learning on Resource Constrained Platforms

1 INTRODUCTION . Internet-of-Things ( IoT ) platforms with distributed edge devices promise tremendous enhancements in many application domains including smart-cities , traffic management , surveillance , security , energy , health-care , and defense , to name a few . In a smart IoT platform , an edge device captures the data from sensors and processes it to extract meaningful information for pre-defined tasks . Deep learning promises major advancements in the quality of information that can be automatically extracted from streaming sensor data . The deep neural networks ( DNN ) can be deployed at the edge device and/or at the cloud to extract information . Data captured by real-time sensors can be distorted during acquisition , in particular , when hardware operates under power constraints . For example , the spatial resolution of an image sensor can be reduced on-demand to save power , but low-resolution distorts the captured image . Likewise , lowpower operation can enhance random noises introduced by a sensor . Therefore , a critical challenge for deploying DNN in smart IoT platforms is to meet the target output quality for a given task even with distorted and noisy input . Data augmentation and denoising techniques can be used to make a DNN robust against input perturbations , however , it lowers detection accuracy for the clean images due to the regularization effect for the noisy images ( Na et al. , 2018 ) . Mixture of experts can be used with the gating network which discriminates input distributions and multiple object detection networks trained with images from different distributions ( Jacobs et al. , 1991 ; Jordan & Jacobs , 1994 ) . The model requires large memory for each expert , thus , it is impractical to use on the edge devices . On the other hand , lightweight average filtering , commonly used for image noise reduction , introduces undesired spatial distortions that can negatively impact accuracy for ’ good ’ images , specially , for complex tasks such as object detection or temporal activity classifications . This paper presents a lightweight mixture of pre-processing experts ( MoPE ) model for noise robust deep learning . We use the pre-processing to minimize the variance of the input images for the object detection network . Adversarially trained auto-encoder with skip connection is used as a pre-processing unit for the noisy images and identity unit is used as a pre-processing for the clean and low resolution images . Gating network is trained to learn input distribution and used to select proper pre-processing units given input distributions . The proposed approach is applied to DNNs designed for different tasks including object detection , multiple object tracking , and human activity classification . Experimental results demonstrate following key contributions : • For distorted images/videos , our approach achieves better detection accuracy than a baseline network trained with only data augmentation . • For clean images/videos , our approach demonstrates accuracy comparable to a vanilla network , and better than a network attached just with pre-processing modules ( without MoPE ) . • The proposed MoPE approach is effective for different classes of DNN including convolutional and recurrent networks . • The complexity analysis shows the proposed approach adds negligible overheads ( 0.67 % and 0.17 % in terms of memory and computation ) to the baseline DNNs . 2 RELATED WORK . Denoising techniques : Several image denoising algorithms exist in the literature ( Buades et al. , 2005 ; Dabov et al. , 2007 ; Tomasi & Manduchi , 1998 ) . Those algorithms assumed data distribution as a known prior , and proposed fine-tuned algorithms for the specific noise distributions . Milani et al . ( 2012 ) proposed a much simpler adaptive filtering strategy for Histogram of Oriented Gradients ( HOG ) based object detection in noisy images . Recently , convolutional neural network ( CNN ) based object detection with noisy images is proposed in Milyaev & Laptev ( 2017 ) . They used an adaptive bilateral filter that considers local texture properties to decide the amount of intensity smoothing required . They also showed the ability of CNN to adapt with image noise by re-training it with noisy images . However , they have not reported how this re-trained network performs with clean images with or without pre-processing . We observed that applying pre-processing on clean images degrades the accuracy of re-trained network . Training techniques : Training with data augmentation can always be used to increase the robustness of the network . Applying similarity loss ( Na et al. , 2018 ) on top of that has shown to increase noise robustness for image classification task . However , we show that it is not beneficial for the complex task like object detection which we would like to solve . To compare the effect of similarity loss for the object detection network , two faster R-CNN networks ( Ren et al. , 2015 ) have been trained with and without similarity loss . We have applied similarity loss to the output of the first feature extractor in the Faster R-CNN . Table 1 shows that the difference between mAPs for those networks is negligible . The results differ from the results for the image classification task as in Na et al . ( 2018 ) . Possible explanation would be that the similarity loss is effective only for task which deals with continuous variables . Even though region proposal network uses the same features with the image classifier , generation of bounding boxes relates with many discrete components including anchors and number of region proposals , etc . And the object detection accuracy is determined based on the results from both region proposal network and image classifier at the second stage . 3 MIXTURE OF PRE-PROCESSING EXPERTS . Reasonable amount of image distortion is useful as a regularization when the test data distribution is expected to be in the same distribution with the training data . In our scenario , the input distribution is diverse including various resolution and noisy images . Thus , data augmentation increases the overall detection accuracies , however , it reduces individual accuracy compared to the network trained only on the specific data distribution . Table 2 shows that the network trained with clean , low resolution and noisy images improves detection accuracy for low resolution and noisy images compared to the baseline at the expense of decreased accuracy . To tackle this problem , we propose simple MoPE model to incorporate various input distributions as shown in Figure 1 . Each pre-processing expert is trained on the target input distribution like low resolution or noisy images and used as a pre-processing for the object detection network . Preprocessing unit trained on each data distribution minimizes the variance of the input distribution seen by the object detection network . A gating network is used to discriminate the input distribution whether the image is clean or noisy as shown in Figure 1 . The gating network selects an input image for the object detection network among the output candidate images from pre-processing units . 3.1 GATING NETWORK . We use modified PatchGAN ( Isola et al. , 2017 ) for the gating network . We use 31x31 receptive filed , which is smaller than that from the original paper as Gaussian noise in nature is easy to identify in a relatively small region . As shown in Figure 2 , we use 3x3 kernels for convolution layers , instance normalization ( Ulyanov et al. , 2016 ) and use shallower network than those in the original paper . It is also well aligned with our objective which is to make minimal changes for the edge devices , and it will be discussed in Section 6.4 . 3.2 PRE-PROCESSING UNITS . Pre-processing for the clean and low-resolution images : We use no pre-processing for the highresolution images and up-sampled version of low resolution images . As the objective of using the mixture of the pre-processing units was to develop a computationally efficient architecture , we found data augmentation was sufficient to enhance the robustness against high-resolution and low-resolution images . Pre-processing for the noisy images : We consider average filter and denoise neural network as a candidate pre-processing for the noisy images . Average filter was very powerful pre-processor for the noisy images considering its simpleness , thus , included in our experiments . We also consider U-Net like denoise neural network ( Ronneberger et al. , 2015 ) as shown in Figure 3 . We intentionally use the simple architecture to reduce the computational complexity . All the convolution and transposed convolution have 3x3 kernels , and the number of filters increases/decreases by 2 when the features are down/up sampled with stride 2 . As our denoise network is pixel to pixel transformation between similar images , skip connection helps preserve the color information in original images , thus , eventually produce better quality . Average filter at front reduces the random noise in images at the expense of loss the edge information . Loss of the edge information is recovered by training denoise network with adversarial loss ( Goodfellow et al. , 2014 ) which will be discussed in the following section . 4 TRAINING OBJECTIVES . 4.1 DENOISE NETWORK . We use adversarial loss ( Goodfellow et al. , 2014 ) on top of the similarity loss ( L2 loss ) as shown in Figure 4 for realistic denoising . For the noise function F : X → Y and denoise function G : Y → X and its discriminator D , we express the objective as : LGAN ( G , D|F ) = Ex∼pdata ( x ) [ logD ( x ) ] + Ey∼pdata ( y ) [ log ( 1−D ( G ( y ) ) ) ] , ( 1 ) where G aims to generate images G ( y ) that look similar to images from clean data distribution X , while D tries to distinguish between denoised images G ( y ) and the original images x for a given noise function y = F ( x ) . We add similarity loss that encourages the denoised images G ( F ( x ) ) to be similar to the original images x : Lsim ( G|F ) = ||G ( F ( x ) ) − x||22 . ( 2 ) The total objective is : L ( G , D|F ) = LGAN ( G , D|F ) + λLsim ( G|F ) , ( 3 ) where λ controls the relative importance of the two objectives . We use λ = 1 in the following experiments . Finally , we solve : G∗ = argmin G max D L ( G , D|F ) . ( 4 ) 4.2 GATING NETWORK . We use softmax gating network . As we know the expert that is responsible for X , we consider the gating function as a classification problem . For the gating function H , the original images x and the noisy images F ( x ) , the training objective is : LGate ( H ) = − log ( H ( x ) ) − log ( 1−H ( F ( x ) ) ) , ( 5 ) where the output ofH is the result from the sigmoid function . We directly use the output ofH to select the input images for the object detection network among the candidate images from pre-processing experts . 5 IMPLEMENTATION . We use faster R-CNN ( Ren et al. , 2015 ) for object detection network and inception v2 ( Szegedy et al. , 2016 ) as a backbone feature extractor . Tensorflow object detection API ( Huang et al. , 2017 ) is used and modified to integrate MoPE model . We use MS-COCO 2014 dataset ( Lin et al. , 2014 ) for training and follow the default configuration except the learning rate schedule in Huang et al . ( 2017 ) . We use the up-sampled version of the 2x and 4x low-resolution images and noisy images along with the clean images for data augmentation . Gaussian noise N ( µ = 0 , σ2 ) is used where σ is selected randomly in the interval [ 0 , max_σ ] per each image . For every iteration , we randomly select one image between the clean and the low-resolution image . The selected image is injected with the corresponding Gaussian noise added image . Faster R-CNN is trained with stochastic gradient descent ( SGD ) optimizer with momentum of 0.9 for 600k iterations . We start training with a learning rate of 2e-4 and divide it by 10 at 200k , 400k iterations . Denoise network and gating network is trained with Adam optimizer for 20k iterations . We also consider fine-tuning of faster R-CNN and MoPE model for extra 200k iterations with a learning rate of 2e-6 .

Summary: This paper introduced a parameterized image processing technique to improve a robustness of visual recognition systems against noisy input data. The proposed method is composed of two components; a denoising network that suppresses the noise signals in an image, and gating network that predicts whether to use the original input image or the one produced by the denoising network. The proposed idea is evaluated on three tasks of object detection, tracking and action recognition.