Title: QUANTIFYING THE COST OF RELIABLE PHOTO AU-THENTICATION VIA HIGH-PERFORMANCE LEARNED LOSSY REPRESENTATIONS

Abstract: Detection of photo manipulation relies on subtle statistical traces, notoriously removed by aggressive lossy compression employed online. We demonstrate that end-to-end modeling of complex photo dissemination channels allows for codec optimization with explicit provenance objectives. We design a lightweight trainable lossy image codec, that delivers good rate-distortion performance, comparable with the popular hand-crafted BPG, and has low computational footprint on modern GPU-enabled platforms. Our results show that significant improvements in manipulation detection accuracy are possible at fractional costs in bandwidth/storage. Our codec improved the accuracy from 37% to 86% even at very low bit-rates, well below the practicality of JPEG (QF 20).
We build upon a recently published end-to-end trainable model of photo acquisition and dissemination (Korus & Memon, 2019) . The model uses a forensic analysis network (FAN) for photo manipulation detection, and allows for joint optimization of the FAN and the camera ISP, leading to distinct imaging artifacts that facilitate authentication. The published toolbox included only stan-Kernel: Gaussian, γ=1 → 23.88 Kernel: t-Student(5), γ=1 → 40.45 Kernel: t-Student(10), γ=1 → 36.98 Kernel: t-Student(25), γ=1 → 32.86 Kernel: t-Student(50), γ=1 → 31.02

1. INTRODUCTION:

Increasing adoption of machine learning in computer graphics has rapidly decreased the time-frame and skill set needed for convincing photo manipulation. Point-and-click solutions are readily available for plausible object insertion (Portenier et al., 2019) , removal (Xiong et al., 2019) , sky replacement (Tsai et al., 2016) , face editing (Portenier et al., 2018) and many other popular operations. While often performed with humorous or artistic intent, they can wreak havoc by altering medical records (Mirsky et al., 2019) , concealing scientific misconduct (Gilbert, 2009; Bik et al., 2016; Bucci, 2018) or even interfering with democratic elections (Chesney & Citron, 2019) .

Reasoning about photo integrity and origin relies on subtle statistical traces, e.g., fingerprints of imaging sensors (Chen et al., 2008) , color interpolation artifacts (Popescu & Farid, 2005) , or pixel co-occurrence patterns (Marra et al., 2019b; Mayer & Stamm, 2019) . Unfortunately, such traces are commonly destroyed during online dissemination, since social networks are forced to aggressively compress digital media to optimize storage and bandwidth expenditures -especially on mobile devices (Cabral & Kandrot, 2015) . As a result, detection of photo manipulations online is notoriously unreliable. Some platforms perform forensic photo analysis at the ingress (Truepic, 2019) , but it may already be too late. Existing photo compression standards, like JPEG, optimize for human perception alone and aggressively remove weak micro-signals already at the device.

We demonstrate that huge gains in photo manipulation detection accuracy are possible at low cost by carefully optimizing lossy compression. Thanks to explicit optimization, fractional increase in bitrate is sufficient to significantly increase the detection accuracy. We build upon the work of Korus & Memon (2019) and use their toolbox for end-to-end modeling of photo dissemination channels. We design a lightweight and high-performance lossy image codec, and optimize for reliable manipulation detection -a backbone of modern forensic analysis (Wu et al., 2019; Mayer & Stamm, 2019) . Interestingly, the model learns complex frequency attenuation patterns as simple inclusion of high-frequency information turns out to be insufficient. This suggests new directions in ongoing efforts to revisit the standard rate-distortion paradigm (Blau & Michaeli, 2019) .

We believe such solutions could be useful for social media platforms, photo attestation services, or insurance companies, which may exploit asymmetric compression setups and acquire photos from smart-phones in analysis-friendly formats. In terms of rate-distortion, our model is comparable with modern hand-engineered codecs, like BPG (Bellard, 2014) which delivers only slightly better results. On GPU-enabled platforms, our codec can be faster, even without low-level optimization.

2. RELATED WORK:

Learned Compression: Rapid progress in deep learning has rekindled interest in lossy image compression. While some studies consider fully end-to-end solutions dispensing with conventional entropy coding (Toderici et al., 2017) , the most successful solutions tend to be variations of autoencoders combined with context-adaptive arithmetic coding. Such codecs have recently surpassed state-of-the-art hand-crafted solutions (Rippel & Bourdev, 2017; Mentzer et al., 2018) . Adoption of generative models allows to hallucinate unimportant details, and reach extreme compression rates while maintaining good perceptual quality (Agustsson et al., 2018) . This research direction makes explicit provenance objectives increasingly pressing.

Compression vs High-level Vision: JPEG compression is commonly used for data augmentation to retain high machine vision performance on compressed images. Despite this, severe compression is known to degrade accuracy (Dodge & Karam, 2016) , and restoration techniques are often used to mitigate the problem (Wang et al., 2016) . Some studies optimize JPEG compression to encode semantically salient regions with better quality in a format-compliant way (Prakash et al., 2017) .

Researchers also explore trainable variations of the JPEG codec optimized for minimal performance degradation and low power use in IoT devices (Liu et al., 2018) . In high-volume applications, computational footprint can be reduced by running high-level vision directly on the DCT coefficients (Gueguen et al., 2018) . Adoption of trainable latent representations gives more flexibility and allows for end-to-end training (Torfason et al., 2018) .

Optimization of Photo Dissemination Channels: Large volume of photos shared online spawned the need to aggressively optimize all steps of photo dissemination (uplink, downlink and storage). Social media platforms already rely on in-house solutions (Facebook, 2018) , and employ extreme measures, like header transplantation, to minimize overhead and improve user experience (Cabral & Kandrot, 2015) . The platforms actively engage in research and development of image compression, including optimization of the standard JPEG codec (Google, 2016) , development of new backwardcompatible standards like JPEG-XL (Rhatushnyak et al., 2019) , and development of entirely new codecs -both hand-engineered (e.g., WebP) and end-to-end trained (Toderici et al., 2017) . dard JPEG compression, and we extended it to support trainable codecs. We show a generic version of the updated model in Fig. 1 with highlighted potentially trainable elements. In this study, we fixed the camera model, and jointly optimize the FAN and a deep compression network (DCN). We describe the design of our DCN codec, and its pre-training protocol below.

3.1. BASELINE DCN ARCHITECTURE:

Our DCN model follows the general auto-encoder architecture proposed by Theis et al. (2017) , but uses different quantization, entropy estimation and entropy coding schemes (Section 3.2). The model is fully convolutional, and consists of 3 sub-sampling (stride-2) convolutional layers, and 3 residual blocks in between (Fig. 2 ). We do not use any normalization layers (such as GDN), and rely solely on a single trainable scaling factor. Distribution shaping occurs organically thanks to entropy regularization (see Fig. A.3b in the appendix). The decoder mirrors the encoder, and implements up-sampling using sub-pixel convolutions (combination of convolutional and depth-to-space layers).

We experimented with different variants of latent representation quantization, eventually converging on soft-quantization with a fixed codebook of integers with a given maximal number of bits per feature (bpf). We used a 5-bpf uniform codebook (M = 32 values from -15 to 16). We show the impact of codebook size in the appendix (Fig. A.3a ).

The model is trained to minimize distortion between the input and reconstructed images regularized by entropy of the latent representation:

EQUATION

where X is the input image, and E, Q, and D denote the encoder, quantization, and decoder, respectively. We used a simple L 2 loss in the RGB domain as the distortion measure d(• , • ), a differentiable soft estimate of entropy H (Section 3.2), and SSIM as the validation metric.

3.2. SOFT QUANTIZATION AND ENTROPY ESTIMATION:

We developed our own quantization and entropy estimation mechanism, because we found existing approaches unnecessarily complicated and/or lacking in accuracy. Some of the most recent solutions include: (1) addition of uniform random noise to quantized samples and non-parametric entropy modeling by a fitted piece-wise linear model (Ballé et al., 2016) ; (2) differentiable entropy upper bound with a uniform random noise component (Theis et al., 2017) ; (3) regularization by penalizing norm of quantized coefficients and differences between spatial neighbors (Rippel & Bourdev, 2017) ; (4) PixelCNN for entropy estimation and context modeling (Mentzer et al., 2018) . Our approach builds upon the soft quantization used by Mentzer et al. (2018) , but is extended to address numerical stability problems, and allow for accurate entropy estimation.

Let z be a vectorized latent representation Z of N images, i.e.: z k = z n,i,j,c where n, i, j, c advance sequentially along an arbitrary memory layout (here image, width, height, channel Laplace distribution scale The weights for individual code-words in the mixture are computed by applying a kernel κ to the distances between the values and the code-words, which can be organized into a distance matrix D:

EQUATION

W = κ(D) = w n,m = κ(d n,m ) .

(3) The most commonly used implementations use a Gaussian kernel:

EQUATION

) which suffers from numerical problems for edge cases overflowing the codebook range (see Fig . A .1 top row in the 4-th and 5-th columns). To alleviate these problems, we adopt a t-Student kernel:

EQUATION

)

which behaves much better in practice. We do not normalize the kernels, and ensure correct proportions of the weights by numerically normalizing rows of the weight matrix.

We estimate entropy of the quantized values by summing the weight matrix along the sample dimension, which yields an estimate of the histogram w.r.t. codebook entries (comparison with an actual histogram is shown in Fig. A .3):

EQUATION

This allows to estimate the entropy of the latent representation by simply:

EQUATION

)

We assess the quality of the estimate both for synthetic random numbers (1,000 numbers sampled from Laplace distributions of various scales) and an actual latent representation of 128 × 128 px RGB image patches sampled from the clic test set (see Section 3.5 and examples in Fig. 4a ). For the random sample, we fixed the quantization codebook to integers from -5 to 5, and performed the experiment numerically. For the real patches, we fed the images through a pre-trained DCN model (a medium-quality model with 32 feature channels; 32-C) and used the codebook embedded in the model (integers from -15 to 16).

Fig. 3 shows the entropy estimation error (both absolute and relative) and scatter plots of real entropies vs. their soft estimates using the Gaussian and t-Student kernels. It can be observed that the t-Student kernel consistently outperforms the commonly used Gaussian. The impact of the kernels' hyperparameters on the relative estimation error is shown in 

3.3. ENTROPY CODING AND BIT-STREAM STRUCTURE:

We used a state-of-the-art entropy coder based on asymmetric numeral systems (Duda, 2013; Duda et al., 2015) and its finite state entropy (FSE) implementation (Collet, 2013) . For simplicity and computational efficiency, we did not employ a context model1 and instead encode individual feature channels (channel EC). Bitrate savings w.r.t. global entropy coding (global EC) vary based on the model, image size and content. For 512 × 512 px images, we observed average savings of ≈ 12%, but for very small patches (e.g., 128 px), it may actually result in overhead (Tab. A.2). This can be easily addressed with a flag that switches between different compression modes, but we leave practical design of the format container for future work. We use a simple structure of the bit-stream, which enables variable-length, per-channel entropy coding with random channel access (Tab. A.1). Such an approach offers flexibility and scalability benefits, e.g.: (1) it allows for rapid analysis of selected feature channels (Torfason et al., 2018) ; (2) it enables trivial parallel processing of the channels to speed up encoding/decoding on modern multi-core platforms.

3.4. TRAINING PROTOCOL AND DATA:

We pre-trained the DCN model in isolation and minimize the entropy-regularized L 2 loss (equation 1) on mixed natural images (MNI) from 6 sources: (1) native camera output from the RAISE and MIT-5k datasets (Dang-Nguyen et al., 2015; Bychkovsky et al., 2011) ; (2) photos from the Waterloo exploration database (Ma et al., 2016) ; (3) HDR images (Hasinoff et al., 2016) ; (4) computer game footage (Richter et al., 2016) ; (5) city scapes (Cordts et al., 2016) ; and (6) the training sub-set of the CLIC professional dataset (CLIC, 2019) . In total, we collected 32,000 square crops ranging from 512 × 512 to 1024 × 1024 px, which were subsequently down-sampled to 256 × 256 px and randomly split into training and validation subsets.

We used three augmentation strategies:

(1) we trained on 128 × 128 px patches randomly sampled in each step;

(2) we flip the patches vertically and/or horizontally with probability 0.5; and (3) we apply random gamma correction with probability 0.5. This allowed for reduction of the training set size, to ≈10k images where the performance saturates. We used batches of 50 images, and learning rate starting at 10 -4 and decaying by a factor of 0.5 every 1,000 epochs. The optimization algorithm was Adam with default settings (as of Tensorflow 1.12). We train by minimizing MSE (L 2 loss) until convergence of SSIM on a validation set with 1,000 images.

3.5. BASELINE MODELS AND EVALUATION:

We control image quality by changing the number of feature channels. We consider three configurations for low, medium, and high quality with 16, 32, and 64 channels, respectively.

Standard Codecs: As hand-crafted baselines, we consider three codecs: JPEG from the libJPEG library via the imageio interface, JPEG2000 from the OpenJPEG library via the Glymur interface, and BPG from its reference implementation (Bellard, 2014) . Since our model uses full-resolution RGB channels as input, we also use full-resolution chrominance channels whenever possible (JPEG and BPG). To make the comparison as fair as possible, we measure effective payload of the codecs.

For the JPEG codec, we manually seek byte markers and include only the Huffman tables and Huffman-coded image data. For JPEG2000, we add up lengths of tile-parts, as reported by jpylyzer.

For BPG, we seek the picture data length marker.

Rate-distortion Trade-off: We used 3 datasets for the final evaluation (Fig. 4 We measured PSNR and SSIM using the scikit-image package and MS-SSIM using sewar. We used the default settings and only set the data range to [0, 1]. The values are computed as the average over RGB channels. The bit-rate for hand-crafted codecs was computed using the effective payload, as explained above. For the DCN codec, we completely encoded and decoded the images (Section A.1).

Fig. 5 shows rate-distortion curves (SSIM vs. effective bpp) for the clic and raw datasets (see appendix for additional results). We show 4 individual images (Fig. 4 ) and averages over the respective datasets. Since standard quality control (e.g., quality level in JPEG, or number of channels in DCN) leads to a wide variety of bpps, we fit individual images to a parametric curve f (x) = (1 + e -αx β +γ ) -1δ and show the averaged fits. It can be observed that our DCN model delivers significantly better results than JPEG and JPEG2000, and approaches BPG.

Processing Time: We collected DCN processing times for various platforms (Table 1 ), including desktops, servers, and low-power edge AI. We report network inference and complete encoding/decoding times on 512 × 512 px and 1920 × 1080 px images, averaged over the clic dataset (separate runs with batch size 1) and obtained using an unoptimized Python 3 implementation2 . On GPUenabled platforms, the inference time becomes negligible (over 100 fps for 512 × 512 px images, and over 20 fps for 1920 × 1080 px images on GeForce 1080 with a 2.6 GHz Xeon CPU), and entropy coding starts to become the bottleneck (down to 13 and 5 fps, respectively). We emphasize that the adopted FSE codec is one of the fastest available, and significantly outperforms commonly used arithmetic coding (Duda, 2013) . If needed, channel EC can be easily parallelized, and the ANS codec could be re-implemented to run on GPU as well (Weißenberger & Schmidt, 2019) .

As a reference, we measured the processing times of 1920 × 1080 px images for the standard codecs on the i7-7700 CPU @ 3.60GHz processor. JPEG coding with 1 thread takes between 0.061 s (Q=30) and 0.075 s (Q=90) (inclusive of writing time to RAM disk; using the pillow library). JPEG 2000 with 1 thread takes 0.61 s regardless of the quality level (inclusive of writing time to RAM disk; glymur library). BPG with 4 parallel threads takes 2.4 s (Q=1), 1.25 s (Q=20) and 0.72 s (Q=30) (inclusive of PNG reading time from RAM disk; bpgenc command line tool). While not directly comparable and also not optimized, some state-of-the-art deep learned codecs require minutes to process even small images, e.g., 5-6 min for 768 × 512 px images from the Kodak dataset reported by Mentzer et al. (2018) . The fastest state-of-the-art learned codec is reported to run at ≈100 fps on images of that size on a GPU-enabled desktop computer (Rippel & Bourdev, 2017) . 

4. OPTIMIZATION FOR MANIPULATION DETECTION:

We consider the standard photo manipulation detection setup where an adversary uses contentpreserving post-processing, and a forensic analysis network (FAN) needs to identify the applied operation or confirm that the image is unprocessed. We use a challenging real-world setup, where the FAN can analyze images only after transmission through a lossy dissemination channel (Fig. 1 ).

In such conditions, forensic analysis is known to fail (Korus & Memon, 2019) . We consider several versions of the channel, including: standard JPEG compression, pre-trained DCN codecs, and trainable DCN codecs jointly optimized along with the FAN. We analyze 128 × 128 px patches, and don't use down-sampling to isolate the impact of the codec.

4.1. PHOTO MANIPULATION AND DETECTION STRATEGY:

We consider 6 benign post-processing operations which preserve image content, but change lowlevel traces that can reveal a forgery. Such operations are commonly used either during photo manipulation or as a masking step afterwards. We include: (a) sharpening -implemented as an unsharp mask operator applied to the luminance channel in the HSV color space; (b) resampling involving successive down-sampling and up-sampling using bilinear interpolation and scaling factors 1:2 and 2:1; (c) Gaussian filtering with a 5 × 5 filter and standard deviation 0.83; (d) JPEG compression using a differentiable dJPEG model with quality level 80; (e) AWGN noise with standard deviation 0.02; and (f) median filtering with a 3 × 3 kernel. The operations are difficult to distinguish visually from native camera output -even without lossy compression (Fig. 7 ).

The FAN is a state-of-the-art image forensics CNN with a constrained residual layer (Bayar & Stamm, 2018) . We used the model provided in the toolbox (Korus & Memon, 2019) , and optimize for classification (native camera output + 6 post-processing classes) of RGB image patches. In total, the model has 1.3 million parameters. 

EQUATION

and the DCN to minimize its combination with the original fidelity/entropy loss (equation 1):

EQUATION

where λ c is used to control the balance between the objectives (we consider values from 10 -3 to 1).

We start from pre-trained DCN models (Section 3.4). The FAN model is trained from scratch.

When JPEG compression was used in the channel, we used the differentiable dJPEG model from the original study (Korus & Memon, 2019) , but modified it to use hard-quantization in the forward pass to ensure results equivalent to libJPEG. We used quality levels from 10 to 95 with step 5.

We followed the same training protocol as Korus & Memon (2019) , and trained on native camera output (NCO). We used the DNet pipeline for Nikon D90, and randomly sampled 128 × 128 px RGB patches from 120 full-resolution images. The remaining 30 images were used for validation (we sampled 4 patches per image to increase diversity). We used batches of 20 images, and trained for 2,500 epochs with learning rate starting at 10 -4 and decaying by 10% every 100 epochs. For each training configuration, we repeated the experiment 3-5 times to validate training stability.

4.3. QUANTITATIVE ANALYSIS:

We summarize the obtained results in Fig. 8 which shows the trade-off between effective bpp (rate), SSIM (distortion), and manipulation detection accuracy. The figure compares standard JPEG compression (diamond markers), pre-trained basic DCN models (connected circles with black border), and fine-tuned DCN models for various regularization strenghts λ c (loose circles with gray border). Fine-tuned models are labeled with a delta in the auxiliary metric (also encoded as marker size and color), and the text is colored in red/green to indicate deterioration or improvement.

Fig. 8a shows how the manipulation detection capability changes with effective bitrate of the codec. We can make the following observations. Firstly, JPEG delivers the worst trade-off and exhibits irregular behavior. The performance gap may be attributed to: (a) better image fidelity for the DCN codec, which retains more information at any bitrate budget; (b) presence of JPEG compression as one of the manipulations. The latter factor also explains irregular drops in accuracy, which coincide with the quality level of the manipulation (80) and unfavorable multiples of the quantization tables (see also Fig. B.1) . Secondly, we observe that fine-tuning the DCN model leads to a sudden increase in payload requirements, minor improvement in quality, and gradual increase in manipulation detection accuracy (as λ c → 0). We obtain significant improvements in accuracy even for the lowest quality level (37% → 85%; at such low bitrates JPEG stays below 30%). Interestingly, we don't see major differences in payload between the fine-tuned models, which suggests that qualitative differences in encoding may be expected beyond simple inclusion of more information.

Fig. 8b shows the same results from a different perspective, and depicts the standard rate-distortion trade-off supplemented with auxiliary accuracy information. We can observe that DCN fine-tuning moves the model to a different point (greater payload, better quality), but doesn't seem to visibly deteriorate the rate-distortion trade-off (with the exception of the smallest regularization λ c = 0.001 which consistently shows better accuracy and worse fidelity).

4.4. QUALITATIVE ANALYSIS:

To explain the behavior of the models, we examine frequency attenuation patterns. We compute FFT spectra of the compressed images, and divide them by the corresponding spectra of uncompressed images. We repeat this procedure for all images in our raw test set, and average them to show consistent trends. The results are shown in Fig. 9 for the pre-trained DCN model (1st column) and fine-tuned models with decreasing λ c (increasing emphasis on accuracy). The plots are calibrated to show unaffected frequencies as gray, and attenuated/emphasized frequencies as dark/bright areas.

The pre-trained models reveal clear and gradual attenuation of high frequencies. Once the models are plugged in to the dissemination workflow, high frequencies start to be retained, but it does not suffice to improve manipulation detection. Increasing importance of the cross-entropy loss gradually changes the attenuation patterns. Selection of frequencies becomes more irregular, and some bands are actually emphasized by the codec. The right-most column shows the most extreme configuration where the trend is clearly visible (the outlier identified in quantitative analysis in Section 4.3).

The changes in codec behavior generally do not introduce visible differences in compressed images (as long as the data distribution is similar, see Section 5). We show an example image (raw test set), its compressed variants (low-quality, 16-C), and their corresponding spectra in Fig. 10 . The spectra follow the identified attenuation pattern (Fig. 9 ). The compressed images do not reveal any obious artifacts, and the most visible change seems to be the jump in entropy, as predicted in Section 4.3. We ran additional experiments by skipping photo acquisition and fine-tuning directly on MNI from the original training set (subset of 2,500 RGB images). We observed the same behavior (see Appendix C), and the optimized codec was artifact-free on all test sets. (Although, due to a smaller training set, the model loses some of its performance; cf. MNI results in Fig. A.6.) However, the FANs generalized well only to clic and kodak images. The originally trained FANs generalized reasonably well to different NCO images (including images from other 3 cameras) but not to clic or kodak. This confirms that existing forensics models are sensitive to data distribution, and that further work will be needed to establish more universal training protocols (see detailed discussion in Appendix D). Short fine-tuning is known to help (Cozzolino et al., 2018) , and we leave this aspect for future work. We are also planning to explore new transfer learning protocols (Li & Hoiem, 2017) .

Generalization should also consider other forensic tasks. We optimized for manipulation detection, which serves as a building block for more complex problems, like processing history analysis or tampering localization (Korus, 2017; Mayer & Stamm, 2019; Wu et al., 2019; Marra et al., 2019a) . However, additional pre-screening may also be needed, e.g., analysis of sensor fingerprints (Chen et al., 2008) , or identification of computer graphics or synthetic content (Marra et al., 2019b) .

6. CONCLUSIONS:

Our study shows that lossy image codecs can be explicitly optimized to retain subtle low-level traces that are useful for photo manipulation detection. Interestingly, simple inclusion of high frequencies in the signal is insufficient, and the models learns more complex frequency attenuation/amplification patterns. This allows for reliable authentication even at very low bit-rates, where standard JPEG compression is no longer practical, e.g., at bit-rates around 0.4 bpp where our DCN codec with lowquality settings improved manipulation detection accuracy from 37% to 86%. We believe the proposed approach is particularly valuable for online media platforms (e.g., Truepic, or Facebook), who need to pre-screen content upon reception, but need to aggressively optimize bandwidth/storage.

Source Code: github.com/pkorus/neural-imaging. The standard soft quantization with a Gaussian kernel (Mentzer et al., 2018) works well for rounding to arbitrary integers, but leads to numerical issues for smaller codebooks. Values significantly exceeding codebook endpoints have zero affinity to any of the entries, and collapse to the mean (i.e., ≈ 0 in our case; Fig. A.1a) . Such issues can be addressed by increasing numerical precision, sacrificing accuracy (due to larger kernel bandwidth), or adding explicit conditional statements in the code. The latter approach is inelegant and cumbersome in graph-based machine learning frameworks like Tensorflow. We used a t-Student kernel instead and increased precision of the computation to 64bits. This doesn't solve the problem entirely, but successfully eliminated all issues that we came across in our experiments, and further improved our entropy estimation accuracy. Fig. A .2 shows entropy estimation error for Laplace-distributed random values, and different hyper-parameters of the kernels. We observed the best results for a t-Student kernel with 50 degrees of freedom and bandwidth γ = 25 (marked in red). This kernel is used in all subsequent experiments.

We experimented with different codebooks and entropy regularization strengths. Fig . A .3a shows how the quantized latent representation (QLR) changes with the size of the codebook. The figures also compare the actual histogram with its soft estimate (equation 6). We observed that the binary codebook is sub-optimal and significantly limits the achievable image quality, especially as the number of feature channels grows. Adding more entries steadily improved quality and the codebook with M = 32 entires (values from -15 to 16) seemed to be the point of diminishing returns.

Our entropy-based regularization turned out to be very effective at shaping the QLR (Fig. A.3b) and dispensed with the need to use other normalization techniques (e.g., GDN). We used only a single scalar multiplication factor responsible for scaling the distribution. All baseline and finetuned models use λ H = 250 (last column). 

A.2 ENTROPY CODING AND BIT-STREAM STRUCTURE:

We used a rudimentary bit-stream structure with the essential meta-data and markers that allow for successful decoding (Tab. A.1). Feature channels are entropy-coded independently (we refer to this approach as channel EC), and can be accessed randomly after decoding a lookup table of their sizes. This simple approach can yield considerable savings w.r.t. global entropy coding (global EC), especially as the image size increases (Tab. A.2). - 

C FINE-TUNING ON MIXED NATURAL IMAGES:

As discussed in Section 5, we ran additional experiments by skipping photo acquisition and finetuning directly on mixed natural images (MNI) -a subset of the original DCN training set (2,500 images). Images in this dataset tend to have more details and depict objects at a coarser scale, since they were all down-sampled to 256 × 256 px (from various original sizes). This required adjusting manipulation strength to maintain visual similarity between photo variations. In particular, we used weaker sharpening, Gaussian filtering with a smaller standard deviation (0.5), down&up-sampling via 75% of the image size (instead of 50%), Gaussian noise with standard deviation 0.012, and JPEG quality level 90. We fine-tuned for 600 epochs.

We summarize the obtained results in Fig C .1 using images from the clic test set. In this experiment, the gap in manipulation detection accuracy between JPEG and baseline DCN has disappeared, except for remaining sudden drops at selected JPEG quality levels (corresponding to the manipulation quality factor 90). We still observe significant improvement for fine-tuned DCN models, but here it tends to saturate around 86%, which might explain negligible improvement of the high-quality 64-C model. By inspecting confusion matrices, we see most of the confusion between native, sharpen and awgn classes where the differences are extremely subtle.

The fine-tuned DCN models remain close to the baseline rate-distortion behavior. Interestingly, except for the weakest regularization (λ c = 0.001), all fine-tuned models seem to be equivalent (w.r.t. the trade-off). We did not observe any obvious artifacts, even in the most aggressive models. The only image with deteriorated SSIM is the alejandro-escamilla-6 image from clic, which was consistently the most affected outlier in nearly all fine tuned models for both NCO and MNI (1st row in Fig. C.2) . In some replications it actually managed to improve, e.g., for the shown cases with λ c = 0.005 and 0.001. However, we don't see any major differences between these variations.

Visualization of frequency attenuation patterns (Fig. C .3) shows analogous behavior, but the changes are more subtle on MNI. We included additional difference plots w.r.t. baseline and weakly finetuned models, where the changes are better visible. On NCO, due to less intrinsic high-frequency content, the behavior is still very clear (cf. bottom part of Fig. C .3).

:

Previous studies report ≈10% performance gains due to context modeling(Agustsson et al.,

2018).

We used low-level Cython wrappers for the FSE entropy coder(Collet, 2013).

Figures and Tables:

Figure 1: A generic end-to-end trainable model of photo acquisition and dissemination: camera ISP is modeled by a neural imaging pipeline (NIP); manipulation detection is performed by a forensic analysis network (FAN); the channel may use either JPEG or a trainable deep compression network (DCN). Potentially trainable elements are shown in yellow.

Figure 2: Architecture of our deep compression network: an auto-encoder with 3 sub-sampling stages and residual units in between. (Empty arrows: no activation; filled arrows: leaky ReLU.)

Figure 3: Entropy estimation error for a Laplacian distribution with varying scale and for the latent space of 128 × 128 px images. The t-Student kernel is significantly more accurate -especially for wide distributions overflowing the codebook range.

Fig.3shows the entropy estimation error (both absolute and relative) and scatter plots of real entropies vs. their soft estimates using the Gaussian and t-Student kernels. It can be observed that the t-Student kernel consistently outperforms the commonly used Gaussian. The impact of the kernels' hyperparameters on the relative estimation error is shown in Fig. A.2. The best combination of kernel parameters (v = 50, γ = 25) is highlighted in red and used in all subsequent experiments.

Figure 4: Example images from the considered clic, kodak and raw test sets (512×512 px).

Figure 5: Rate-distortion trade-offs on the clic, kodak and raw test sets. (See Fig. A.5 for MS-SSIM).

Figure 6: Comparison of our DCN codec with low-quality settings (16-C) against JPEG with matching SSIM and matching bpp. Samples from clic, kodak, and raw datasets.

Figure 7: Subtle photo manipulation: (left) 128×128 px native patch; (rest) various post-processing.

Figure 8: The rate-distortion-accuracy trade-off (raw test set) reveals significant improvements in manipulation detection accuracy while maintaining similar rate-distortion performance. basic → acc: 0.37 λc = 1.000 → acc: 0.41 λc = 0.050 → acc: 0.48 λc = 0.010 → acc: 0.62 λc = 0.005 → acc: 0.71 λc = 0.001 → acc: 0.85

Figure 10: Compression results for various versions of the low-quality DCN: (1st column) original image; (2nd) pre-trained model; (3rd-6th) fine-tuned models with decreasing λ c .

Fig. A.4 visually compares the QLRs of our baseline low-quality codec (16 feature channels) with weak and strong regularization.

Figure A.4: Comparison of the latent representations (16-C model) of an example 512 × 512 px image (thong-vo-428) learned with weak and strong entropy regularization (5-bpf codebook).

Figure A.5: Rate-distortion trade-offs of the baseline DCNs on the clic, kodak and raw test sets: (left) average over all images; (2nd-5th columns) sample images from Fig. 4; (top section) linearscale SSIM; (bottom section) MS-SSIM in dB scale f (x) = -10 • log 10 (1x).

Figure B.1: Impact of compression quality and fine-tuning regularization on the achievable detection accuracy and its variation. Sudden drops for JPEG are caused by inclusion of this compression as one of the manipulations, and correspond to the manipulation quality level (80).

Figure B.3: Difference in image fidelity (SSIM) after fine-tuning the low-quality DCN model within the dissemination workflow (weak CE objective with little to no improvement in detection accuracy): raw, kodak and clic datasets.

Figure C.1: Visualization of the rate-distortion-accuracy trade-off on the clic dataset after fine-tuning on mixed natural images.

Average compression/decompression time on different platforms (in seconds) with breakdown into NN inference and complete processing; in-memory processing using the 32-C model.

Table content: <table><tr><td/><td/><td/><td colspan="2">512×512 px images</td><td/><td/><td colspan="2">1920×1080 px images</td><td/></tr><tr><td>GPU</td><td>CPU / Platform</td><td colspan="2">inference</td><td colspan="2">whole codec</td><td colspan="2">inference</td><td colspan="2">whole codec</td></tr><tr><td/><td/><td colspan="8">Encode Decode Encode Decode Encode Decode Encode Decode</td></tr><tr><td>Maxwell</td><td colspan="2">ARM 57 (nVidia Jetson Nano) 0.2076</td><td>0.5333</td><td>0.6507</td><td>0.6721</td><td>1.6348</td><td>3.6057</td><td>4.4978</td><td>4.9722</td></tr><tr><td>-</td><td>i7-7700 @ 3.60GHz</td><td>0.2165</td><td>0.3330</td><td>0.2272</td><td>0.3317</td><td>1.8052</td><td>2.7678</td><td>1.8753</td><td>2.7901</td></tr><tr><td>-</td><td>i7-9770 @ 3.60Ghz</td><td>0.0648</td><td>0.1396</td><td>0.0762</td><td>0.1397</td><td>0.5197</td><td>1.1685</td><td>0.6080</td><td>1.1728</td></tr><tr><td>GF 1080</td><td>Xeon E5-2690 @ 2.60GHz</td><td>0.0083</td><td>0.0173</td><td>0.0742</td><td>0.0498</td><td>0.0597</td><td>0.1244</td><td>0.1805</td><td>0.1714</td></tr><tr><td>P40</td><td>Xeon E5-2680 @ 2.40GHz</td><td>0.0093</td><td>0.0160</td><td>0.0720</td><td>0.0375</td><td>0.0558</td><td>0.1123</td><td>0.1895</td><td>0.1684</td></tr><tr><td>V100</td><td>Xeon E5-2680 @ 2.40GHz</td><td>0.0065</td><td>0.0071</td><td>0.0604</td><td>0.0209</td><td>0.0416</td><td>0.0489</td><td>0.1735</td><td>0.0979</td></tr><tr><td colspan="2">GF 2080S i7-9770 @ 3.60Ghz</td><td>0.0059</td><td>0.0132</td><td>0.0421</td><td>0.0244</td><td>0.0399</td><td>0.0953</td><td>0.1343</td><td>0.1320</td></tr></table>

Table A.1: Structure of the bit-stream describing a DCN-compressed image -inclusive of both ANS probability tables and entropy-coded data Table A.2: Bit-stream length of channel entropy coding (EC) relative to global EC for different quality levels and image patches of various size.

Table content: <table><tr><td>Section</td><td colspan="2">Content</td><td/><td/><td/><td>Data Type</td><td>Bytes</td></tr><tr><td>Basic meta-data:</td><td colspan="5">Latent shape H x W x N Length of coded channel sizes = 2 bytes (uint16)</td><td>uint8 uint16</td><td>3 2</td></tr><tr><td>Channel sizes (shorter of a/b)</td><td colspan="4">(a) FSE-encoded channel sizes 1 (b) raw bytes</td><td/><td>uint16 uint16</td><td>var 2N</td></tr><tr><td>Image data (N × shorter of a/b)</td><td colspan="6">(a) FSE-encoded latent channel 1 (b) RLE-encoded latent channel (#repetitions + byte) uint16 + uint8 3 uint8 var</td></tr><tr><td>1 DCN model</td><td colspan="3">Avg. bit-stream size</td><td/><td colspan="2">Bit-stream size range</td></tr><tr><td/><td>128</td><td>256</td><td>512</td><td>128</td><td>256</td><td>512</td></tr><tr><td>low quality (16-C)</td><td colspan="6">1.03 0.934 0.882 0.917 -1.098 0.829 -1.005 0.755 -0.980</td></tr><tr><td colspan="7">medium quality (32-C) 1.05 0.933 0.874 0.961 -1.108 0.821 -0.998 0.742 -0.968</td></tr><tr><td>high quality (64-C)</td><td colspan="6">1.07 0.948 0.887 0.977 -1.119 0.833 -0.998 0.773 -0.964</td></tr><tr><td colspan="2">A DCN CODEC DETAILS</td><td/><td/><td/><td/></tr></table>

Figure A.1: Comparison of soft quantization with Gaussian and t-Student kernels for a Laplace distribution of increasing scale: the t-Student kernel is more accurate and robust to outliers. Entropy estimation error for a Laplacian distribution with varying scale and various hyper-parameters of the kernels: the t-Student kernel (2nd-6th column) is more accurate than the Gaussian (1st column) -especially for wide distributions overflowing the codebook range.

Table content: <table><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td colspan="6">Random sample: Laplace dist. λ=2;</td><td/><td colspan="6">Random sample: Laplace dist. λ=4;</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td colspan="3">real quantization</td><td/><td/><td/><td/><td colspan="3">real quantization</td><td/><td/><td/><td/><td colspan="3">real quantization</td><td/></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td colspan="3">soft est. (Gaussian)</td><td/><td/><td>4</td><td/><td colspan="3">soft est. (Gaussian)</td><td/><td/><td/><td>4</td><td colspan="3">soft est. (Gaussian)</td><td/></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td colspan="3">soft est. (t-Student)</td><td/><td/><td/><td/><td colspan="3">soft est. (t-Student)</td><td/><td/><td/><td/><td colspan="3">soft est. (t-Student)</td><td/></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>2</td><td/><td/><td/><td/><td/><td/><td/><td>2</td><td/><td/><td/><td/></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>0</td><td/><td/><td/><td/><td/><td/><td/><td>0</td><td/><td/><td/><td/></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>-2</td><td/><td/><td/><td/><td/><td/><td colspan="2">-2</td><td/><td/><td/><td/></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>-4</td><td/><td/><td/><td/><td/><td/><td colspan="2">-4</td><td/><td/><td/><td/></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>-20</td><td/><td>-10</td><td>0</td><td>10</td><td/><td/><td>-40</td><td>-20</td><td/><td>0</td><td>20</td></tr><tr><td/><td/><td>0.8 1.0</td><td/><td/><td/><td/><td/><td/><td>0.5 0.6</td><td/><td/><td/><td/><td/><td/><td>0.4 0.3</td><td/><td/><td/><td colspan="3">real (H=2.45) soft/Gaussian (H=2.47) soft/t-Student (H=2.46)</td><td>0.20</td><td/><td/><td/><td colspan="3">real (H=3.19) soft/Gaussian (H=3.19) soft/t-Student (H=3.19)</td><td colspan="2">0.16 0.14</td><td colspan="5">real (H=3.33) soft/Gaussian (H=3.38) soft/t-Student (H=3.33)</td></tr><tr><td colspan="2">Histograms</td><td>0.2 0.4 0.6</td><td/><td/><td colspan="3">soft/Gaussian (H=0.45) soft/t-Student (H=0.28) real (H=0.26)</td><td/><td>0.1 0.2 0.3 0.4</td><td/><td/><td colspan="4">soft/Gaussian (H=1.61) soft/t-Student (H=1.56) real (H=1.55)</td><td>0.1 0.2</td><td/><td/><td/><td/><td/><td/><td>0.05 0.10 0.15</td><td/><td/><td/><td/><td/><td/><td colspan="2">0.06 0.08 0.10 0.12</td><td/><td/><td/><td/></tr><tr><td/><td/><td>0.0</td><td/><td/><td/><td/><td/><td/><td>0.0</td><td/><td/><td/><td/><td/><td/><td>0.0</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td colspan="2">0.04</td><td/><td/><td/><td/></tr><tr><td/><td/><td/><td>-4</td><td>-2</td><td>0</td><td>2</td><td>4</td><td/><td/><td>-4</td><td>-2</td><td>0</td><td>2</td><td>4</td><td/><td/><td>-4</td><td colspan="2">-2</td><td>0</td><td>2</td><td>4</td><td/><td colspan="2">-4</td><td>-2</td><td>0</td><td>2</td><td>4</td><td/><td>-4</td><td>-2</td><td/><td>0</td><td>2</td><td>4</td></tr><tr><td/><td colspan="2">100</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>Relative entropy error [%]</td><td colspan="2">20 40 60 80</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td/><td/><td>0</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td/><td colspan="2">100</td><td colspan="4">Kernel: Gaussian, γ=50 → 10.32</td><td/><td/><td colspan="4">Kernel: t-Student(5), γ=50 → 7.80</td><td/><td colspan="5">Kernel: t-Student(10), γ=50 → 5.85</td><td/><td/><td colspan="4">Kernel: t-Student(25), γ=50 → 3.07</td><td/><td/><td colspan="4">Kernel: t-Student(50), γ=50 → 2.43</td><td/><td colspan="5">Kernel: t-Student(100), γ=50 → 7.46</td></tr><tr><td>Relative entropy error [%]</td><td colspan="2">20 40 60 80</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td/><td/><td>0</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td/><td/><td>0</td><td>2</td><td>4</td><td>6</td><td>8</td><td>10</td><td>0</td><td>2</td><td>4</td><td>6</td><td>8</td><td>10</td><td>0</td><td>2</td><td>4</td><td>6</td><td>8</td><td>10</td><td>0</td><td>2</td><td>4</td><td>6</td><td>8</td><td>10</td><td>0</td><td>2</td><td>4</td><td>6</td><td>8</td><td>10</td><td>0</td><td>2</td><td>4</td><td>6</td><td>8</td><td>10</td></tr><tr><td/><td/><td/><td colspan="4">Laplace distribution scale</td><td/><td/><td colspan="4">Laplace distribution scale</td><td/><td/><td colspan="4">Laplace distribution scale</td><td/><td/><td colspan="4">Laplace distribution scale</td><td/><td/><td colspan="4">Laplace distribution scale</td><td/><td/><td colspan="4">Laplace distribution scale</td></tr><tr><td colspan="6">Figure A.2:</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr></table>

1: Example confusion matrices for the baseline and fine-tuned low-quality DCN models.

Table content: <table><tr><td>True</td><td>Predicted</td><td>(a) pre-trained 16-C DCN → 36.9% native sharpen resample gaussian jpeg awgn</td><td>median</td><td>(b) fine-tuned w. λ c = 0.0010 → 87.0% native sharpen resample gaussian jpeg awgn median</td></tr><tr><td colspan="5">native 92 0.0010 0.0050 0.0100 0.0500 0.1000 0.5000 1.0000 8 10 10 24 7 18 23 basic</td></tr></table>

Peer review (iclr2020):
The paper describes a pipeline for image compression which allows to reliably detect specific manipulation patterns in compressed images. The results show that it is possible to learn image compression that performs similarly to a modern image compression algorithm while in the same time is optimized to reveal specific kinds of manipulations. The authors build upon (Korus & Memon, 2019), but use a learnable codec instead of differentiable JPEG. The idea to regularize entropy of the latent representation of images is interesting. A method to train a well-performing image compression system which can also follow additional constraints (such as ability to reveal certain manipulations) is very valuable for practice. Unfortunately, there are already available trainable compression methods and the authors do not compare to these methods. However, in my opinion to detect manipulation in the image one should prove that visual content in some area of the image was significantly changed with respect to some original, while in the other parts of the image it was not changed. Otherwise it becomes impossible to distinguish in-camera filtering and secondary postprocessing. Basically, the authors present a method to detect whether a very particular configuration of some basic image processing filters (Gaussian blur, median filter, resampling) was applied to the image. Therefore the particular problem formulation looks very artificial. With regards to the experiments in the paper, I was somewhat lost. Compare the Fig. 5 and the Fig. 8. In the Fig. 8, we see a big set of possible system configurations having different manipulation detection accuracy, image quality and compression performance. In the Fig. 5, we see a compression efficiency-image quality dependency. However, it remains unclear how do the systems represented at these two graphs relate to each other, or, in the other words, what is he mapping between points of these graphs. Next, poor performance of JPEG manipulation detection by the proposed network does not prove that JPEG manipulation cannot be detected, it just shows that the proposed architecture does not perform well in this problem. A comparative study which relates a new system to a current state of the art is required to claim that a proposed approach is better. Finally, SSIM is not a standard way to compute image quality. MS-SSIM and PSNR are also popular, and a user study is usually recommended to claim that some method generates images of better visual quality. Summarizing, the authors do not provide a new best-performing image compression algorithm, and neither solve a problem of image manipulation detection, but show that it is possible to learn an image compression system with some additional constraints. I believe it is an interesting contribution, and I hope the authors can improve presentation of the experiments.