Quantifying the Cost of Reliable Photo Authentication via High-Performance Learned Lossy Representations

1 INTRODUCTION . Increasing adoption of machine learning in computer graphics has rapidly decreased the time-frame and skill set needed for convincing photo manipulation . Point-and-click solutions are readily available for plausible object insertion ( Portenier et al. , 2019 ) , removal ( Xiong et al. , 2019 ) , sky replacement ( Tsai et al. , 2016 ) , face editing ( Portenier et al. , 2018 ) and many other popular operations . While often performed with humorous or artistic intent , they can wreak havoc by altering medical records ( Mirsky et al. , 2019 ) , concealing scientific misconduct ( Gilbert , 2009 ; Bik et al. , 2016 ; Bucci , 2018 ) or even interfering with democratic elections ( Chesney & Citron , 2019 ) . Reasoning about photo integrity and origin relies on subtle statistical traces , e.g. , fingerprints of imaging sensors ( Chen et al. , 2008 ) , color interpolation artifacts ( Popescu & Farid , 2005 ) , or pixel co-occurrence patterns ( Marra et al. , 2019b ; Mayer & Stamm , 2019 ) . Unfortunately , such traces are commonly destroyed during online dissemination , since social networks are forced to aggressively compress digital media to optimize storage and bandwidth expenditures - especially on mobile devices ( Cabral & Kandrot , 2015 ) . As a result , detection of photo manipulations online is notoriously unreliable . Some platforms perform forensic photo analysis at the ingress ( Truepic , 2019 ) , but it may already be too late . Existing photo compression standards , like JPEG , optimize for human perception alone and aggressively remove weak micro-signals already at the device . We demonstrate that huge gains in photo manipulation detection accuracy are possible at low cost by carefully optimizing lossy compression . Thanks to explicit optimization , fractional increase in bitrate is sufficient to significantly increase the detection accuracy . We build upon the work of Korus & Memon ( 2019 ) and use their toolbox for end-to-end modeling of photo dissemination channels . We design a lightweight and high-performance lossy image codec , and optimize for reliable manipulation detection - a backbone of modern forensic analysis ( Wu et al. , 2019 ; Mayer & Stamm , 2019 ) . Interestingly , the model learns complex frequency attenuation patterns as simple inclusion of high-frequency information turns out to be insufficient . This suggests new directions in ongoing efforts to revisit the standard rate-distortion paradigm ( Blau & Michaeli , 2019 ) . We believe such solutions could be useful for social media platforms , photo attestation services , or insurance companies , which may exploit asymmetric compression setups and acquire photos from smart-phones in analysis-friendly formats . In terms of rate-distortion , our model is comparable with modern hand-engineered codecs , like BPG ( Bellard , 2014 ) which delivers only slightly better results . On GPU-enabled platforms , our codec can be faster , even without low-level optimization . 2 RELATED WORK . Learned Compression : Rapid progress in deep learning has rekindled interest in lossy image compression . While some studies consider fully end-to-end solutions dispensing with conventional entropy coding ( Toderici et al. , 2017 ) , the most successful solutions tend to be variations of autoencoders combined with context-adaptive arithmetic coding . Such codecs have recently surpassed state-of-the-art hand-crafted solutions ( Rippel & Bourdev , 2017 ; Mentzer et al. , 2018 ) . Adoption of generative models allows to hallucinate unimportant details , and reach extreme compression rates while maintaining good perceptual quality ( Agustsson et al. , 2018 ) . This research direction makes explicit provenance objectives increasingly pressing . Compression vs High-level Vision : JPEG compression is commonly used for data augmentation to retain high machine vision performance on compressed images . Despite this , severe compression is known to degrade accuracy ( Dodge & Karam , 2016 ) , and restoration techniques are often used to mitigate the problem ( Wang et al. , 2016 ) . Some studies optimize JPEG compression to encode semantically salient regions with better quality in a format-compliant way ( Prakash et al. , 2017 ) . Researchers also explore trainable variations of the JPEG codec optimized for minimal performance degradation and low power use in IoT devices ( Liu et al. , 2018 ) . In high-volume applications , computational footprint can be reduced by running high-level vision directly on the DCT coefficients ( Gueguen et al. , 2018 ) . Adoption of trainable latent representations gives more flexibility and allows for end-to-end training ( Torfason et al. , 2018 ) . Optimization of Photo Dissemination Channels : Large volume of photos shared online spawned the need to aggressively optimize all steps of photo dissemination ( uplink , downlink and storage ) . Social media platforms already rely on in-house solutions ( Facebook , 2018 ) , and employ extreme measures , like header transplantation , to minimize overhead and improve user experience ( Cabral & Kandrot , 2015 ) . The platforms actively engage in research and development of image compression , including optimization of the standard JPEG codec ( Google , 2016 ) , development of new backwardcompatible standards like JPEG-XL ( Rhatushnyak et al. , 2019 ) , and development of entirely new codecs - both hand-engineered ( e.g. , WebP ) and end-to-end trained ( Toderici et al. , 2017 ) . 3 END-TO-END TRAINABLE PHOTO DISSEMINATION MODEL . We build upon a recently published end-to-end trainable model of photo acquisition and dissemination ( Korus & Memon , 2019 ) . The model uses a forensic analysis network ( FAN ) for photo manipulation detection , and allows for joint optimization of the FAN and the camera ISP , leading to distinct imaging artifacts that facilitate authentication . The published toolbox included only stan- dard JPEG compression , and we extended it to support trainable codecs . We show a generic version of the updated model in Fig . 1 with highlighted potentially trainable elements . In this study , we fixed the camera model , and jointly optimize the FAN and a deep compression network ( DCN ) . We describe the design of our DCN codec , and its pre-training protocol below . 3.1 BASELINE DCN ARCHITECTURE . Our DCN model follows the general auto-encoder architecture proposed by Theis et al . ( 2017 ) , but uses different quantization , entropy estimation and entropy coding schemes ( Section 3.2 ) . The model is fully convolutional , and consists of 3 sub-sampling ( stride-2 ) convolutional layers , and 3 residual blocks in between ( Fig . 2 ) . We do not use any normalization layers ( such as GDN ) , and rely solely on a single trainable scaling factor . Distribution shaping occurs organically thanks to entropy regularization ( see Fig . A.3b in the appendix ) . The decoder mirrors the encoder , and implements up-sampling using sub-pixel convolutions ( combination of convolutional and depth-to-space layers ) . We experimented with different variants of latent representation quantization , eventually converging on soft-quantization with a fixed codebook of integers with a given maximal number of bits per feature ( bpf ) . We used a 5-bpf uniform codebook ( M = 32 values from -15 to 16 ) . We show the impact of codebook size in the appendix ( Fig . A.3a ) . The model is trained to minimize distortion between the input and reconstructed images regularized by entropy of the latent representation : Ldcn = E X [ d ( X , D ◦ Q ◦ E ( X ) ) + λHH ( Q ◦ E ( X ) ) ] , ( 1 ) where X is the input image , and E , Q , and D denote the encoder , quantization , and decoder , respectively . We used a simpleL2 loss in the RGB domain as the distortion measure d ( · , · ) , a differentiable soft estimate of entropy H ( Section 3.2 ) , and SSIM as the validation metric . 3.2 SOFT QUANTIZATION AND ENTROPY ESTIMATION . We developed our own quantization and entropy estimation mechanism , because we found existing approaches unnecessarily complicated and/or lacking in accuracy . Some of the most recent solutions include : ( 1 ) addition of uniform random noise to quantized samples and non-parametric entropy modeling by a fitted piece-wise linear model ( Ballé et al. , 2016 ) ; ( 2 ) differentiable entropy upper bound with a uniform random noise component ( Theis et al. , 2017 ) ; ( 3 ) regularization by penalizing norm of quantized coefficients and differences between spatial neighbors ( Rippel & Bourdev , 2017 ) ; ( 4 ) PixelCNN for entropy estimation and context modeling ( Mentzer et al. , 2018 ) . Our approach builds upon the soft quantization used by Mentzer et al . ( 2018 ) , but is extended to address numerical stability problems , and allow for accurate entropy estimation . Let z be a vectorized latent representation Z of N images , i.e . : zk = zn , i , j , c where n , i , j , c advance sequentially along an arbitrary memory layout ( here image , width , height , channel ) . Let c denote a quantization codebook withM centers [ c1 , . . . , cM ] ( code words ) . Then , given a weight matrix W ∈ [ 0 , 1 ] N , M : ∀m ∑ n wn , m = 1 , we can define : hard quantization as ẑ = [ c argmaxmw : ,m ] ; and soft quantization as z̃ = Wc . Hard quantization replaces an input value with the closest available codeword , and corresponds to a rounding operation performed by the image codec . Soft quantization is a differentiable relaxation , which uses a linear combination of all code-words - as specified by the weight matrix . A detailed comparison of both quantization modes , along with an illustration of potential numerical pitfalls , can be observed in the top row of Fig . A.1 in the appendix . The hard and soft quantization are used in the forward and backward passes , respectively . In Tensorflow , this can be implemented as z = tf.stop gradient ( ẑ - z̃ ) + z̃ . The weights for individual code-words in the mixture are computed by applying a kernel κ to the distances between the values and the code-words , which can be organized into a distance matrix D : D = z− cᵀ = [ dn , m = zn − cm ] , ( 2 ) W = κ ( D ) = [ wn , m = κ ( dn , m ) ] . ( 3 ) The most commonly used implementations use a Gaussian kernel : κγ = e −γd2n , m , ( 4 ) which suffers from numerical problems for edge cases overflowing the codebook range ( see Fig . A.1 top row in the 4-th and 5-th columns ) . To alleviate these problems , we adopt a t-Student kernel : κγ , v = ( 1 + γd2n , m v ) − ( v+1 ) /2 , ( 5 ) which behaves much better in practice . We do not normalize the kernels , and ensure correct proportions of the weights by numerically normalizing rows of the weight matrix . We estimate entropy of the quantized values by summing the weight matrix along the sample dimension , which yields an estimate of the histogram w.r.t . codebook entries ( comparison with an actual histogram is shown in Fig . A.3 ) : h̃ = [ h̃m = ∑ n wn , m ] . ( 6 ) This allows to estimate the entropy of the latent representation by simply : Ĥ = − ∑ m h̃mlog2h̃m . ( 7 ) We assess the quality of the estimate both for synthetic random numbers ( 1,000 numbers sampled from Laplace distributions of various scales ) and an actual latent representation of 128 × 128 px RGB image patches sampled from the clic test set ( see Section 3.5 and examples in Fig . 4a ) . For the random sample , we fixed the quantization codebook to integers from−5 to 5 , and performed the experiment numerically . For the real patches , we fed the images through a pre-trained DCN model ( a medium-quality model with 32 feature channels ; 32-C ) and used the codebook embedded in the model ( integers from −15 to 16 ) . Fig . 3 shows the entropy estimation error ( both absolute and relative ) and scatter plots of real entropies vs. their soft estimates using the Gaussian and t-Student kernels . It can be observed that the t-Student kernel consistently outperforms the commonly used Gaussian . The impact of the kernels ’ hyperparameters on the relative estimation error is shown in Fig . A.2 . The best combination of kernel parameters ( v = 50 , γ = 25 ) is highlighted in red and used in all subsequent experiments .

Summary: This paper presents a learned image compression method that is able to be robust under a variety of tasks. The results aren't state of the art in terms of rate-distortion performance, but this paper has a very good analysis of the results, and has produced a very fast codec. In that sense, this is a very interesting paper that may lead to other fast methods (the other fast method they compared the runtime against - WaveOne never published a complete description).