Transferable Recognition-Aware Image Processing

1 INTRODUCTION Unlike in image recognition where a neural network maps an image to a semantic label , a neural network used for image processing maps an input image to an output image with some desired properties . Examples include image super-resolution ( Dong et al. , 2014 ) , denoising ( Xie et al. , 2012 ) , deblurring ( Eigen et al. , 2013 ) , colorization ( Zhang et al. , 2016 ) and style transfer ( Gatys et al. , 2015 ) . The goal of such systems is to produce images of high perceptual quality to a human observer . For example , in image denoising , we aim to remove noise in the signal that is not useful to an observer and restore the image to its original “ clean ” form . Metrics like PSNR and SSIM ( Wang et al. , 2004 ) are often used ( Dong et al. , 2014 ; Tong et al. , 2017 ) to approximate human-perceived similarity between the processed images with the original images , and direct human assessment on the fidelity of the output is often considered the “ gold-standard ” assessment ( Ledig et al. , 2017 ; Zhang et al. , 2018b ) . Therefore , many techniques ( Johnson et al. , 2016 ; Ledig et al. , 2017 ; Isola et al. , 2017 ) have been proposed for making the output images look perceptually pleasing to human . However , image processing outputs may not be accurately recognized by image recognition systems . As shown in Fig . 1 , the output image of an denoising model could easily be recognized by a human as a bird , but a recognition model classifies it as a kite . One could specifically train a recognition model only on these output images produced by the denoising model to achieve better performance on such images , or could leverage some domain adaptation approaches to adapt the recognition model to this domain , but the performance on natural images can be harmed . This retraining/adaptation scheme might also be impractical considering the significant overhead induced by catering to various image processing tasks and models . With the fast-growing size of image data , many images are often “ viewed ” and analyzed more by machines than by humans . Nowadays , any image uploaded to the Internet is likely to be analyzed by certain vision systems . For example , Facebook uses a system called Rosetta to extract texts from over 1 billion user-uploaded images every day ( Maria , 2018 ) . It is of great importance that the processed images be recognizable by not only humans , but also by machines . In other words , recognition systems ( e.g. , image classifier or object detector ) , should be able to accurately explain the underlying semantic meaning of the image content . In this way , we make them potentially easier to search , recommended to more interested audience , and so on , as these procedures are mostly executed by machines based on their understanding of the images . Therefore , we argue that image processing systems should also aim at better machine recognizability . We call this problem “ Recognition-Aware Image Processing ” . It is also important that the enhanced recognizability is not specific to any concrete neural networkbased recognition model , i.e. , the improvement on recognition performance is only achieved when the output images are evaluated on that particular model . Instead , the improvement should ideally be transferable when evaluated on different models , to support its usage without access to possible future recognition systems , since we may not decide what model will be used for recognizing the processed image , for example if we upload it to the Internet or share it on social media . We may not know what network architectures ( e.g . ResNet or VGG ) will be used for inference , what object categories the downstream model recognizes ( e.g . animals or scenes ) , or even what task will be performed on the processed image ( e.g . classification or detection ) . Without these specifications , it might be hard to enhance image ’ s machine semantics . In this work , we propose simple and highly effective approaches to make image processing outputs more accurately recognized by downstream recognition systems , transferable among different recognition architectures , categories and tasks . The approaches we investigate add a recognition loss optimized jointly with the image processing loss . The recognition loss is computed using a fixed recognition model that is pretrained on natural images , and can be done in an unsupervised manner , e.g. , without semantic labels of the image . It can be optimized either directly by the original image processing network , or through an intermediate transforming network . We conduct extensive experiments , on multiple image enhancement/restoration ( super-resolution , denoising , and JPEG-deblocking ) and recognition ( classification and detection ) tasks , and demonstrate that our approaches can substantially boost the recognition accuracy on the downstream systems , with minimal or no loss in the image processing quality measured by conventional metrics . Also , the accuracy improvement transfers favorably among different recognition model architectures , object categories , and recognition tasks , which renders our simple solution effective even when we do not have access to the downstream recognition models . Our contributions can be summarized as follows : • We propose to study the problem of enhancing the machine interpretability of image processing outputs , a desired property considering the amount of images analyzed by machines nowadays . • We propose simple and effective methods towards this goal , suitable for different use cases , e.g. , without ground truth semantic labels . Extensive experiments are conducted on multiple image processing and recognition tasks , demonstrating the wide applicability of the proposed methods . • We show that using our simple approaches , the recognition accuracy improvement could transfer among recognition architectures , categories and tasks , a desirable behavior making the proposed methods applicable without access to the downstream recognition model . 2 RELATED WORK . Image processing/enhancement problems such as super-resolution and denoising have a long history ( Tsai , 1984 ; Park et al. , 2003 ; Rudin et al. , 1992 ; Candès et al. , 2006 ) . Since the initial success of deep neural networks on these problems ( Dong et al. , 2014 ; Xie et al. , 2012 ; Wang et al. , 2016b ) , a large body of works try to investigate better model architecture design and training techniques ( Dong et al. , 2016 ; Kim et al. , 2016b ; Shi et al. , 2016 ; Kim et al. , 2016a ; Mao et al. , 2016 ; Lai et al. , 2017 ; Tai et al. , 2017a ; Tong et al. , 2017 ; Tai et al. , 2017b ; Lim et al. , 2017 ; Zhang et al. , 2018d ; Ahn et al. , 2018 ; Lefkimmiatis , 2018 ; Chen et al. , 2018 ; Haris et al. , 2018b ) , mostly on the image superresolution task . These works focus on generating high visual quality images under conventional metrics or human evaluation , without considering recognition performance on the output . There are also a number of works that relate image recognition with processing . Some works ( Zhang et al. , 2016 ; Larsson et al. , 2016 ; Zhang et al. , 2018c ; Sajjadi et al. , 2017 ) use image classification accuracy as an evaluation metric for image colorization/super-resolution , but without optimizing for it during training . Wang et al . ( 2016a ) incorporates super-resolution and domain adaptation techniques for better recognition on very low resolution images . Bai et al . ( 2018 ) train a superresolution and refinement network simultaneously to better detect faces in the wild . Zhang et al . ( 2018a ) train networks for face hallucination and recognition jointly to achieve better recover the face identity from low-resolution images . Liu et al . ( 2018 ) considers 3D face reconstruction and trains the recognition model jointly with the reconstructor . Sharma et al . ( 2018 ) trains a classification model together with an enhancement module . Our problem setting is different from these works , in that we assume we do not have the control on the recognition model , as it might be on the cloud or decided in the future , thus we advocate adapting the image processing model only . This also ensures the recognition model is not harmed on natural images . Haris et al . ( 2018a ) investigate how super-resolution could help object detection in low-resolution images . VidalMata et al . ( 2019 ) and Banerjee et al . ( 2019 ) also aims to enhance machine accuracy on poor-conditioned images but mostly focus on better image processing techniques without using recognition models . Wang et al . ( 2019 ) propose a method to make denoised images more accurately segmented , also presenting some interesting findings in transferability . Most existing works only consider one image processing task or image domain , and develop specific techniques , while our simpler approach is task-agnostic and potentially more widely applicable . Our work is also related but different from those which aims for robustness of the recognition model ( Hendrycks & Dietterich , 2019 ; Li et al. , 2019 ; Shankar et al. , 2018 ) , since we focus on the training of the processing models and assume the recognition model is given . 3 METHOD . In this section we first introduce the problem setting of “ recognition-aware ” image processing , and then we develop various approaches to address it , each suited for different use cases . 3.1 PROBLEM SETTING . In a typical image processing problem , given a set of training input images { Ikin } and corresponding target images { Iktarget } ( k = 1 , · · ·N ) , we aim to train a neural network that maps an input image to its corresponding target . For example , in image denoising , Ikin is a noisy image and I k target is the corresponding clean image . Denoting this mapping network as P ( for processing ) , parameterized by WP , during training our optimization objective is : min WP Lproc = 1 N N∑ k=1 lproc ( P ( Ikin ) , Iktarget ) , ( 1 ) where P ( Ikin ) is simply the output of the processing model Iout , and lproc is the loss function for each sample . The pixel-wise mean-squared-error ( MSE , or L2 ) loss is one of the most popular choices . During evaluation , the performance is typically measured by average similarity ( e.g. , PSNR , SSIM ) between Iktarget and I k out = P ( Ikin ) , or through human assessment . In our problem setting of recognition-aware processing , we are interested in a recognition task , with a trained recognition model R ( R for recognition ) , parameterized by WR . We assume each input/target image pair Ikin/I k target is associated with a ground truth semantic label S k for the recognition task . Our goal is to train a image processing model P such that the recognition performance on the output images { Ikout = P ( Ikin ) } is high , when evaluated using R with the semantic labels { Sk } . In practice , the recognition model R might not be available ( e.g. , on the cloud ) , in which case we could resort to other models if the performance improvement transfers among models .

Summary: This paper proposes a setting called "recognition-aware image processing." The key idea is to make the images output by image processing methods still be readily recognized by image recognition methods. Realizing this will help to better meet the requirement from both human observers and machines. Formally, this is formulated as a combined optimization in which the losses from image processing and recognition tasks are jointly considered. This framework is further extended to unsupervised case and the case of intermediate transformer to make it more flexible. The transferability issue is discussed and it is observed that the model trained by the proposed method can generally help even when other recognition models or tasks are used. Experimental study is conducted to demonstrate the performance of the proposed method.