Quantifying the Cost of Reliable Photo Authentication via High-Performance Learned Lossy Representations
Authors: Pawel Korus
Nasir Memon
Venue:  (2020)
Keywords: image forensics
photo manipulation detection
learned compression
lossy compression
image compression
entropy estimation

Abstract: Detection of photo manipulation relies on subtle statistical traces, notoriously removed by aggressive lossy compression employed online. We demonstrate that end-to-end modeling of complex photo dissemination channels allows for codec optimization with explicit provenance objectives. We design a lightweight trainable lossy image codec, that delivers competitive rate-distortion performance, on par with best hand-engineered alternatives, but has lower computational footprint on modern GPU-enabled platforms. Our results show that significant improvements in manipulation detection accuracy are possible at fractional costs in bandwidth/storage. Our codec improved the accuracy from 37% to 86% even at very low bit-rates, well below the practicality of JPEG (QF 20). 

Published as a conference paper at ICLR 2020
QUANTIFYING THE COST OF RELIABLE PHOTO AU- THENTICATION VIA HIGH-PERFORMANCE LEARNED
LOSSY REPRESENTATIONS
Pawel Korusi.2 Nasir Memon
TTandon School of Engineering, New York University, USA Department of Telecommunications, AGH University of Science and Technology, Poland (pkorus, memon) @nyu. edu
ABSTRACT
Detection of photo manipulation relies subtle statistical traces, notoriously re- moved by aggressive lossy compression employed online. We demonstrate that end-to-end modeling of complex photo dissemination channels allows for codec optimization with explicit provenance objectives. We design lightweight train- able lossy image codec, that delivers good rate-distortion performance, compara- ble with the popular hand-crafted BPG, and has low computational footprint on modern GPU-enabled platforms. Our results show that significant improvements in manipulation detection accuracy are possible at fractional costs in bandwidth/s- torage. Our codec improved the accuracy from 37% to 86% even at very low bit-rates, well below the practicality of JPEG (QF 20).
INTRODUCTION
Increasing adoption of machine learning in computer graphics has rapidly decreased the time-frame and skill set needed for convincing photo manipulation. Point-and-click solutions are readily avail- able for plausible object insertion (Portenier et al.] 2019), removal (Xiong et al., 2019), sky replace- ment (Tsai et al., 2016). face editing (Portenier al. 2018) and many other popular operations. While often performed with humorous or artistic intent, they can wreak havoc by altering medi- cal records (Mirsky et al. 2019), concealing scientific misconduct (Gilbert 2009:; (Bik et al. 2016 Bucci 2018 even interfering with democratic elections (Chesney & Citron 2019)).
Reasoning about photo integrity and origin relies on subtle statistical traces, c.g., fingerprints of imaging sensors (Chen et al. 2008), color interpolation artifacts (Popescu & Farid (005). or pixel co-occurrence patterns (Marra et al. 2019b Mayer & Stamm 2019). Unfortunately, such traces are commonly destroyed during online dissemination. since social networks are forced to aggressively compress digital media to optimize storage and bandwidth expenditures especially on mobile de- vices (Cabral & Kandrot 2015). As result, detection of photo manipulations online is notoriously unreliable. Some platforms perform forensic photo analysis at the ingress (Truepic) 2019). but may already be too late. Existing photo compression standards, like JPEG, optimize for human perception alone and aggressively remove weak micro-signals already at the device.
We demonstrate that huge gains in photo manipulation detection accuracy are possible low cost by carefully optimizing lossy compression. Thanks to explicit optimization, fractional increase in bitrate sufficient to significantly increase the detection accuracy. We build upon the work of Korus & Memon (2019) and use their toolbox for end-to-end modeling of photo dissemination channels. We design a lightweight and high-performance lossy image codee, and optimize for reliable ma- nipulation detection backbone of modern forensic analysis (Wu et al. 2019 Mayer & Stamm 2019). Interestingly, the model learns complex frequency attenuation patterns as simple inclusion of high-frequency information turns out to be insufficient. This suggests directions in ongoing efforts to revisit the standard rate-distortion paradigm (Blau & Michaeli 2019).
We believe such solutions could be useful for social media platforms, photo attestation services, or insurance companies, which may exploit asymmetric compression setups and acquire photos from smart-phones in friiedly formats. In terms of rate-distortion, model is comparable

---

Published as a conference paper at ICLR 2020
Clociiestin bhcle prasine, pasegeac A
Pastytoce
RAW mmage
NI
troge
RGB
MUX
Dowascale
Coppesion (JFG DCN)
FAN
Closs enoopy los
imago
Acqualiin
Post Poot-poocssing 'olling
Transnisiio
Analysia
Figure I: A generic end-to-end trainable model of photo acquisition and dissemination: camera ISP modeled by neural imaging pipeline (NIP); manipulation detection is performed by forensic analysis network (FAN); the channel may use either JPEG or trainable deep compression network (DCN). Potentially trainable elements are shown in yellow.
with modern hand-engineered codecs, like BPG (Bellard, 2014) which delivers only slightly better results. On GPU-enabled platforms, our codec can be faster, even without low-level optimization.
2 RELATED WORK
Learned Compression:
Rapid progress in deep learning has rekindled interest in lossy image compression. While some studies consider fully end-to-end solutions dispensing with conventional entropy coding (Toderici et al. 2017). the most successful solutions tend to be variations of auto- encoders combined with context- -adaptive arithmetic coding. Such codecs have recently surpassed state-of-the-arr hand-crafted solutions (Rippel & Bourdev [017] Mentzer et al., 20188. Adoption of generative models allows to hallucinate unimportant details, and reach extreme compression rates while maintaining good perceptual quality (Agustsson et al. 2018). This research direction makes explicit provenance objectives increasingly pressing.
Compression vs High-level Vision: JPEG compression is commonly used for data augmentation to retain high machine vision performance compressed images. Despite this, severe compression known to degrade accuracy (Dodge & Karam 2016), and restoration techniques are often used mitigate the problem (Wang al.) 2016). Some studies optimize JPEG compression to encode semantically salient regions with better quality in format-compliant way (Prakash et al. 2017). Researchers also explore trainable variations of the JPEG codec optimized for minimal performance degradation and low power use in IoT devices (Liu et al. 2018). In high-volume applications, computational footprint can be reduced by running high-level vision directly on the DCT coeff- cients (Gueguen et al.) (018). Adoption of trainable latent representations gives more flexibility and allows for end-to-end training (Torfason et al. 2018).
Optimization of Photo Dissemination Channels: Large volume of photos shared online spawned the need to aggressively optimize all steps of photo dissemination (uplink, downlink and storage). Social media platforms already rely on in-house solutions (Facebook 2018), and employ extreme measures, like header transplantation, to minimize overhead and improve user experience (Cabral & Kandrot 2015). The platforms actively engage in research and development image compression, including optimization of the standard JPEG codec (Google) 2016), development of new backward- compatible standards like JPEG-XL (Rhatushnyak et al. 2019), and development of entirely new codecs both hand-engineered (e.g., WebP) and end-to-end trained (Toeeriii et al. 2017).
END-TO-END TRAINABLE PHOTO DISSEMINATION MODEL.
We build upon a recently published end-to-end trainable model of photo acquisition and dissem- ination (Korus& & Memon 2019). The model uses a forensic analysis network (FAN) for photo manipulation detection, and allows for joint optimization the FAN and the camera ISP, leading to distinct imaging artifacts that facilitate authentication. The published toolbox included only stan-
2

---

Published as a conference paper at ICLR 2020
Geaee dacaeds 0&R
Insader
PM 1231
10
ON
Fartage
Quatteation A
Temmyy
Decoder
U2 C 133
Sayt SER 022 1L.-11
SAA Te ON 123
SH0 0M 77p
XC DIL 12
GEN-OE DNL 129
Architecture of our deep compression network: an auto-encoder with 3 sub-sampling Figure 2:
stages and residual units in between. (Empty arrows: no activation; filled arrows: leaky ReLU.)
dard JPEG compression, and extended it to support trainable codecs. We show generic version of the updated model in Fig. [ with highlighted potentially trainable elements. In this study, we fixed the camera model, and jointly optimize the FAN and a deep compression network (DCN). We describe the design of our DCN codec, and its pre-training protocol below.
3.1 BASELINE DCN ARCHITECTURE
Our DCN model follows the general auto-encoder architecture proposed by Theis et al. (2017) 7), but uses different quantization, entropy estimation and entropy coding schemes (Section 3.2). The model fully convolutional, and consists of 3 sub-sampling (stride-2) convolutional layers, and residual blocks in between (Fig.2). We do not use any normalization layers (such as GDN), and rely solely on a single trainable scaling factor. Distribution shaping occurs organically thanks to entropy regularization (see Fig. A.3p in the appendix). The decoder mirrors the encoder, and implements up-sampling using sub-pixel convolutions (combination of convolutional and depth- tepth-to-sppp to-s-ec layers).
We experimented with different variants of latent representation quantization, eventually converging on soft- suft-quantization with a fixed codebook of integers with a given maximal number of bits per feature (bpf). We used 5-bpf uniform codebook (M 32 values from -15 to 16). We show the impact of codebook size in the appendix (Fig. A.3p).
The model is trained to minimize distortion between the input and reconstructed images regularized by entropy of the latent representation:
Leen [X,
+ Ay (Qo
where is the input image, and E, º, and 'D denote the encoder, quantization, and decoder, respec- tively. We used simple L2 loss in the RGB domain as the distortion measure d(-,-). a differentiable soft estimate of entropy И (Section 3.2). 3.2) and SSIM as the validation metric.
3.2 SOFT QUANTIZATION AND ENTROPY ESTIMATION
We developed our own quantization and entropy estimation mechanism, because we found existing approaches unnecessarily complicated and/or lacking in accuracy. Some of the most recent solutions include: (1) addition uniform random noise to quantized samples and non-parametric entropy modeling by fitted piece-wise linear model (Ballé al. 2016); (2) differentiable entropy upper
bound with uniform random noise component (Theis et al. 2017 (3) regularization by penalizing
norm of quantized coefficients and differences between spatial neighbors (Rippel & Bourdey 2017); (4) PixelCNN for entropy estimation and context modeling (Mentzer et al. 2018). Our approach builds upon the soft quantization used by Mentzer et al. (2018). but is extended to address numerical stability problems, and allow for accurate entropy estimation.
Let be vectorized latent representation Z of N images, i.e.: Z4 = En.ijc where n,, J, advance sequentially along an arbitrary memory layout (here image, width, height, channel). Let e denote a quantization codebook with M centers (ci...., CMrl (code words). Then, given a weight matrix W define: hard quantization as à (rggmax,, W., and soft
[0, 1] M Vm En Wn.m
3

---

Published as a conference paper at ICLR 2020
Syatheels dota (Taaparion die)
steet space
tmagh
Taplice distharkn
13 eerrog
retrops
Figure 3: Entropy estimation error for a Laplacian distribution with varying scale and for the latent space of 128 x 128 px images. The t-Student kernel is significantly more accurate -sspccially for wide distributions overflowing the codebook range.
quantization ă We. Hard quantization replaces an input value with the closest available code- word, and corresponds to rounding operation performed by the image codec. Soft quantization differentiable relaxation, which uses linear combination of all code-words the weight matrix. A detailed comparison of both quantization modes, along with an illustration of potential numerical pitfalls, be observed in the top row of Fig. [A. in the appendix. The hard and soft quantization are used in the forward and backward passes, respectively. In Tensorflow, this can be implemented
specified by
tf. stop-grossereee gradient
2) z.
The weights for individual code-words in the mixture are computed by applying kernel x to the distances between the values and the code-words, which can be organized into distance matrix D
D = z - c [dn,m Zn Cm
(2)
W <D)) [wn.m x(dn,m)
(3)
The most commonly used implementations use Gaussian kernel:
Kry y
which suffers from numerical problems for edge cases overflowing the codebook range (see Fig. A. top row in the 4-th and 5-th columns). To alleviate these problems, we adopt t-Student kernel:
(v+1)/2
ydl
Kyyz
1+
(5)
which behaves much better in practice. We do normalize the kernels, and ensure correct propor- tions of the weights by numerically normalizing rows of the weight matrix.
We estimate entropy of the quantized values by summing the weight matrix along the sample dimen- sion, which yields an estimate of the histogram w.r.t. codebook entries (comparison with an actual histogram is shown in Fig. A.3):
h
hm
Σ
Wn.m
(6)
This allows to estimate the entropy of the latent representation by simply:
i = Σ hm
(7)
We assess the quality of the estimate both for synthetic random numbers (1,000 numbers sampled from Laplace distributions of various scales) and an actual latent representation of 128 x 128 px RGB image patches sampled from the clic the random sample, we fixed the quantization codebook to integers from -5 to 5, and performed the experiment numerically. For the real patches, we fed the images through a pre-trained DCN model (a medium-qualid model with 32 feature channels; 32-C) and used the codebook embedded in the model (integers from 15 to 16).
(see Section [.5] and examples in Fig. 44.. For
Fig. shows the entropy estimation error (both absolute and relative) and scatter plots of real en- tropies vs. their soft estimates using the Gaussian and t-Student kernels. It can be observed that the t-Student kernel consistently outperforms the commonly used Gaussian. The impact of the ker- nels' hyperparameters on the relative estimation error is shown in Fig. A.2 The best combination of kernel parameters (u 50, 7 25) is highlighted in red and used in all subsequent experiments.
4

---

Published as a conference paper at ICLR 2020
PUR
Stride di
GRAPPERAA
P
A
Figure 4: Example images from the considered clic, kodak and raw test sets (512x512 px).
3.3 ENTROPY CODING AND BIT-STREAM STRUCTURE
We used a state-of-the-ar entropy coder based on asymmetric numeral systems (Duda 2013 Duda et al. 2015)) and its finite state entropy (FSE) implementation (Collet 2013). For simplicity and computational efficiency. we did not employ a context mode and instead encode individual feature channels (channel EC). Bitrate savings w.r.t global entropy coding (global EC) vary based on the model, image size and content. For 512 x 512 px images, we observed average savings of 12%, but for very small patches (e.g.. 128 px), it may actually result in overhead (Tab. A.2). This can be casily addressed with a flag that switches between different compression modes, but we leave practical design of the format container for future work. We use simple structure of the bit-stream, which enables variable-length, per-channel entropy coding with random channel access (Tab. [A.I). Such approach offers flexibility and scalability benefits, e.g.: (1) it allows for rapid analysis of selected feature channels (Torfason et al. 22018;; (2) it enables trivial parallel processing of the channels to speed up encoding/decoding on modern multi-core platforms.
3.4 TRAINING PROTOCOL AND DATA
We pre- trained the DCN model in isolation and minimize the tatropy-regularizee L2 loss (equa- tion T on mixed natural images (MNI) from sources: (1) native camera output from the RAISE and MIT-5k datasets (Dang-Nguyer
al. 2015 (ychkoosky al. 2011): (2) photos from the Wa-
terloo exploration database (Ma et al. 2016); :): (3) HDR images (Hasinoff et al. 2016); (4) computer game footage (Richter et al. 2016);
city scapes (Cordts et al. | 2016);; and (6) the training sub-set of the CLIC professional dataset (CLIC) 2019). In total, we collected 32,000 square crops ranging
from 512 512 1024 x 1024 px, which were subsequently down-sampled to 256 x 256 px and randomly split into training and validation subsets.
We used three augmentation strategies: (1) we trained on 128 x 128 px patches randomly sampled in each step; (2) we flip the patches vertically and/or horizontally with probability 0.5; and (3) we apply random gamma correction with probability 0.5. This allowed for reduction of the training set size, to s0 10k images where the performance saturates. We used batches of 50 images, and learning rate starting at 10 and decaying by a factor of 0.5 every 1,000 epochs. The optimization algorithm was Adam with default settings (as of Tensorflow 1.12). We train by minimizing MSE (L2 loss) until convergence of SSIM on a validation set with 1,000 images.
3.5 BASELINE MODELS AND EVALUATION
We control image quality by changing the number of feature channels. We consider three configu- rations for low, medium, and high quality with 16, 32, and 64 channels, respectively.
Standard Codecs:
As hand-crafted baselines, we consider three codecs: JPEG from the libJPEG library via the imageio interface, JPEG2000 from the OpenJPEG library via the Glymur interface, and BPG from its reference implementation (Bellard) 2014). Since our model uses full-resolution
Previous studies report s: 100% performance gains due context modeling (Agustsson et al. 2018
5

---

Published as a conference paper at ICLR 2020
clie512 amages
kodak5112 images
39 images
1.00
0.33
SSN
0000
680
Elllettoee hpp
12Boerive bpp
Ellectvve hpp
Figure 5: Rate-distortion trade-offss on the clic, kodak and raw test sets. (See Fig. A.5 for MS-SSIM).
RGB channels as input, we also use full-resolution chrominance channels whenever possible (JPEG and BPG). To make the comparison as fair as possible, we measure effective payload of the codecs. For the JPEG codec, we manually seek byte markers and include only the Huffman tables and Huffman-coded image data. For JPEG20000, we add up lengths of tile-parts, as reported by jpylyzer. For BPG, we seek the picture.data..engg.
marker.
Rate-distortion Trade-off: We used 3 datasets for the final evaluation (Fig. 4 (raw) 39 images with native camera output from 4 different cameras 2011]; (clic) 39 images from the professional validation subset of CLIC (2019); (019)) (kodak) 24 images from the standard Kodak dataset. All test images are of size 512 x 512px, and were obtained either by cropping directly without re-sampling (raw, kodak) or by resizing central square crop (clic).
et al. 2015 Bychkovsky et al.)
We measured PSNR and SSIM using the scikit-image package and MS-SSIM using sewar. We used the default settings and only set the data range to [0. 1]. The values are computed as the average over RGB channels. The bit-rate for hand-crafted codecs was computed using the effective payload, as explained above. For the DCN codec, we completely encoded and decoded the images (Section(A.) A..I)
Fig. 5) shows rate-distortion curves (SSIM vs. effective bpp) for the clic and raw datasets (see appendix for additional results). We show 4 individual images (Fig. 4 and averages over the re- spective datasets. Since standard quality control (e.g.. quality level in JPEG, or number of chan- nels in DCN) leads to a wide variety of bpps, we fit individual images to a parametric curve f(z) = (1 te az
e - - and show the averaged fits. It can be observed that our DCN model delivers significantly better results than JPEG and JPEG2000, and approaches BPG.
Processing Time: We collected DCN processing times for various platforms (Table I including desktops, servers, and low-power edge AL We report network inference and complete encoding/de- coding times on 512 x 512 px and 1920 x 1080 px images, averaged over the clic dataset (separate runs with batch size and obtained using an unoptimized Python 3 implementation On GPU- enabled platforms, the inference time becomes negligible (over 100 fps for 512 x 512 px images, and over 20 fps for 1920 x 1080 px images on GeForce 1080 with a 2.6 GHz Xeon CPU), and entropy coding starts to become the bottleneck (down to 13 and 5 fps, respectively). We emphasize that the adopted FSE codec is one of the fastest available, and significantly outperforms commonly used arithmetic coding (Duda] 2013). If needed, channel EC can be easily parallelized, and the ANS codec could be re-implemented to run on GPU as well (WeiBenberger & Schmid! 2019).
As reference, we measured the processing times of 1920 x 1080 px images for the standard codecs on the i7-7700 CPU @ 3.60GHz processor. JPEG coding with thread takes between 0.061 s (Q=30) and 0.075 (Q=90) (inclusive of writing time to RAM disk; using the pillow library). JPEG 2000 with thread takes 0.61 s regardless of the quality level (inclusive of writing time to RAM disk; glymur library). BPG with 4 parallel threads takes 2.4 (Q=1), 1.25 (Q=20) and 0.72 (Q=30) (inclusive of PNG reading time from RAM disk; bpgenc command line tool). While not directly comparable and also not optimized, some state-of- -f-e---a-h-ar deep learned codecs require minutes to process even small images, e.g., 5-6 min for 768 x 512 px images from the Kodak dataset reported by Mentzer et al. 22018) The fastest state-of-the-ait learned codec reported to run 1000 fps on images of that size on a GPU-enabled desktop computer (Rippel & Bourdey 2017)
We used lewel Cython wrappers for the FSE entropy coder Collet 2013

---

Published as a conference paper at ICLR 2020
Table I: Average comppessin/decomnessin time different platforms (in seconds) with break-
down into NN inference and complete processing; in mn-memory processing using the 32-C model.
512x px images
1920x 1080 px images
GPU
CPU/ Platform
inference
whole codec
inference
whole codec
Encode
Decode
Encode
Decode
Encode
Decode
Encode
Decode
Maxwell
ARM (nVidia Nano)
0.2076
0.5333
0.6507
0.6721
1.6348
3.6057
4,4978
4.9722
17-7700 @ 3.60GHz
0,2165
0.3330
0.2272
0.3317
1.8052
2.7678
1.8753
2.7901
17-9770 3.60Ghz
0.0648
0.1396
0.0762
0.1397
0,5197
1.1685
0,6080
1.1728
GF
Xeon 2690 2.60HHz
0.0173
0.0498
0.0897
0.1244
0.1714
P40
Xeon 55-2680 @ 2,40GHZ
0.0093
0.0160
0.0720
0.0375
0.0558
0.1123
0.1895
0.1684
V100
Xeon E5-2680 @ 2,40GHz
0.0065
0.0071
0.0604
0.0209
0.0416
0.0489
0.1735
0.0979
GF 2080S
17-9770 3.60Ghz
0.0132
0.0244
0.1343
750 tos matt11
Figure 6: Comparison of our DCN codec with low www-qulltt settings (16-C) against JPEG with match- ing SSIM and matching hpp. Samples from clic, kodak, and raw datasets.
OPTIMIZATION FOR MANIPULATION DETECTION
We consider the standard photo manipulation detection setup where adversary uses content- preserving post-processing, and a forensic analysis network (FAN) needs to identify the applied operation confirm that the image is unprocessed. We a challenging real-world setup. where the FAN can analyze images only after transmission through lossy dissemination channel (Fig. [ In such conditions, forensic analysis is known to fail (Korus & Memon 2019). We consider sev- eral versions of the channel, including: standard JPEG compression, pre-trained DCN codecs, and trainable DCN codecs jointly optimized along with the FAN. We analyze 128 x 128 px patches, and don't use down-sampling to isolate the impact of the codec.
4.1 PHOTO MANIPULATION AND DETECTION STRATEGY
We consider benign post-processing operations which preserve image content, but change low- level traces that can reveal forgery. Such operations are commonly used either during photo ma- nipulation or as masking step afterwards. We include: (a) sharpening implemented as an unsharp mask operator applied the luminance channel in the HSV color space; (b) resampling involving successive down-sampling and up- ap-sampling using bilinear interpolation and scaling factors 1:2 and 2:1; (c) Gaussian filtering with a
filter and standard deviation 0.83; (d) JPEG compression using a differentiable dJPEG model with quality level 80; (c) AWGN noise with standard deviation 0.02; and (f) median filtering with 3 3 kernel. The operations are difficult to distinguish visually from native camera output even without lossy compression (Fig.
The FAN is a state-of-the-ar image forensics CNN with a constrained residual layer (Bayar & Stamm 2018). We used the model provided in the toolbox (Korus & Memon 2019), and optimize for classification (native camera output + post-processing classes) of RGB image patches. In total, the model has 1.3 million parameters.

---

Published as a conference paper at ICLR 2020
sharpen
Jing
mediian
Comprosoni Jnsampesee
Figure 7: Subtle photo manipulation: (left) 128x x 128 px native patch; (rest) various post- post-processing.
4.2 TRAINING PROTOCOL
We jointly train the entire workflow and optimize both the FAN and DCN models. Let M. denote the e-th manipulation (including identity for native camera output), and J denote the output of the FAN with Fe being the probability of the corresponding manipulation class c. Let also C denote the adopted lossy compression model, e.g., D o Q o for the DCN. We denote sRGB images rendered by the camera ISP as X. The FAN model is trained to minimize a cross-entropy loss:
Lee
log (Fe oCo Mcx))
(8)
and the DCN minimize its combination with the original fidelity/entropy loss (equation T
- AcLuen
(9)
where A is used to control the balance between the objectives (we consider values from 10-3 to 1). We start from pre-trained DCN models (Section 3.4p. The FAN model trained from scratch.
When JPEG compression was used in the channel, we used the differentiable dJPEG model from the original study (Korus & Memon, 2019), but modified it pass to ensure results equivalent to libJ We used quality levels from 10 to 95 with step 5.
hard-quantization in the forward
We followed the same training protocol as Korus & Memon (2019). and trained on native camera outpur (NCO). We used the DNer pipeline for Nikon D90, and randomly sampled 128 x 128 px RGB patches from 120 full-resolution images. The remaining 30 images were used for validation (we sampled 4 patches per image to increase diversity). We used batches of 20 images, and trained for 2,500 epochs with learning rate starting 10-4 and decaying by 10% every 100 epochs. For each training configuration, we repeated the experiment 3-5 times to validate training stability.
4.3 QUANTITATIVE ANALYSIS
We summarize the obtained results in Fig. & shows the trade-off between effective hpp (rate), SSIM (distortion), and manipulation detection accuracy. The figure compares standard JPEG com- pression (diamond markers), pre-trained basic DCN models (connected circles with black border), and fine-tuned DCN models for various regularization strenghts A (loose circles with gray border). Fine-tuned models are labeled with delta in the auxiliary metric (also encoded as marker size and color), and the text colored in red/green indicate deterioration or improvement.
Fig. Ba shows how the manipulation detection capability changes with effective bitrate of the codec. We can make the following observations. Firstly, JPEG delivers the worst trade-off and exhibits irregular behavior. The performance gap may be attributed to: (a) better image fidelity for the DCN codec, which retains more information at any bitrate budget; (b) presence of JPEG compression as one of the manipulations. The latter factor also explains irregular drops in accuracy, which coincide with the quality level of the manipulation (80) and unfavorable multiples of the quantization tables (see also Fig. B.I.. Secondly, in payload requirements, minor improvement in quality, and gradual increase in manipulation de- tection accuracy (as X - 0). We obtain significant improvements in accuracy even for the lowest
observe that fine-tuning the DCN model leads to sudden increase
8

---

Published as a conference paper at ICLR 2020
rato RODLEaC
disortion
000
077
1.0
TAN
@
Edliction Epp
Fifetivo bpp
Figure 8: The rate- suueddistottoonaccueee trade-off (raw test set) reveals significant improvements in manipulation detection accuracy while maintaining similar rate-distortion performance.
bodr 0-37
Au 1.000 ace 0:41
0,000 acti 0:48
0.000 and 0.62
O 003 aoe 0.71
As 0001 ooe 0.83
Figure 9: Visualization of frequency attemnation//applifccttooo patterns in the FFT domain for the fine-tuned DCN codec (low-quality, 16-C model).
quality level (37% 85%; such low bitrates JPEG stays below 30%). Interestingly, don't t see major differences in payload between the fine-tuned models, which suggests that qualitative differences in encoding may be expected beyond simple inclusion of more information.
Fig. &p shows the same results from a different perspective, and depicts the standard rate-distortion trade-off supplemented with auxiliary accuracy information. We can observe that DCN fine-tuning moves the model to a different point (greater payload, better quality), but doesn't 't seem to visibly deteriorate the rate-distortion trade-off (with the exception of the smallest regularization A, 0.001 which consistently shows better accuracy and worse fidelity).
4.4 QUALITATIVE ANALYSTS
To explain the behavior of the models, we examine frequency attenuation patterns. We compute FFT spectra of the compressed images, and divide them by the corresponding spectra of uncompressed images. We repeat this procedure for all images in our raw test set, and average them to show consistent trends. The results are shown Fig. for the pre-trained DCN model (Ist column) and fine-tuned models with decreasing A₂ (increasing emphasis on accuracy). The plots are calibrated to show unaffected frequencies as gray, and attenuated/emphhasizd frequencies as dark/bright areas.
The pre- pre-trained models reveal clear and gradual attenuation of high frequencies. Once the models are plugged in the dissemination workflow, high frequencies start be retained, but does not suffice to improve manipulation detection. Increasing importance of the cross-entropy loss gradually changes the attenuation patterns. Selection of frequencies becomes more irregular, and some bands actually emphasized by the codec. The right-most column shows the most extreme configuration where the trend is clearly visible (the outlier identified in quantitative analysis in Section 4.3
The changes in codec behavior generally do not introduce visible differences in compressed images (as long as the data distribution is similar, see Section 55. We show an example image (raw test set), its compressed variants (low-quality, 16-C), and their corresponding spectra in Fig. 10 The spectra follow the identified attenuation pattern (Fig. 19). The compressed images do not reveal any obious artifacts, and the most visible change seems to be the jump in entropy, predicted in Section 4.3

---

Published as a conference paper at ICLR 2020
Orignal
Teak Ho 122, seer U9M
100 141 sieer 1ES
2.20 wete EMM A E01 L-4k simer 1E
Figure 10: Compression results for various versions of the low-quality DCN: (Ist column) original
image; (2nd) pre-trained model; (3rd-6th) fine-tuned models with decreasing X
DISCUSSION, LIMITATIONS AND FUTURE WORK
While the proposed approach can successfully facilitate pre-screening of photographs shared on- line, further research is needed to improve model generalization. We observed that the fine-tuning procedure tends bias the DCN/FAN models towards the secondary image dataset, in our case the native camera output (NCO). The baseline DCN was pre-trained on mixed natural images (MNI) with extensive augmentation, leading to competitive results on all test sets. However, fine-tuning was performed on NCO only. Characteristic pixel correlations, e.g., due color interpolation, bias the codec and lead to occasional artifacts in MNIs (mostly in the clic test set; see Appendix [B), and deterioration of the rate-distortion trade-off. The problem is present regardless of λe. which suggests issues with the fine-tuning protocol (data diversity) and not the forensic optimization objective.
We additional experiments by skipping photo acquisition and fine-tuning directly MNI from the original training set (subset of 2,500 RGB images). We observed the same behavior (see Ap- pendix C. and the optimized codec was artifact-free all test sets. (Although, due to smaller training set, the model loses some of its performance; cf. MNI results in Fig. A.6) However, the FANs generalized well only clic and kodak images. The originally trained FANS generalized rea- sonably well to different NCO images (including images from other 3 cameras) but not to clic or kodak. This confirms that existing forensics models are sensitive data distribution, and that further work will be needed to establish more universal training protocols (see detailed discussion in Ap- pendix DD. Short fine-tuning is known to help (Cozzolino et al. 2018), and we leave this aspect for future work. We are also planning to explore new transfer learning protocols (Li & Hoiem 2017).
Generalization should also consider other forensic tasks. We optimized for manipulation detection, which serves building block for more complex problems, like processing history analysis or tampering localization (Korus 2017 Mayer & Stamm 2019, |Wu et al., 2019] Marra et al. (019a). However, additional pre-screening may also be needed, e.g., analysis of sensor fingerprints (Chen et al. 2008), or identification of computer graphics or synthetic content (Marra et al. (019b).
6 CONCLUSIONS
Our study shows that lossy image codecs can be explicitly optimized to retain subtle low-level traces that are useful for photo manipulation detection. Interestingly, simple inclusion of high frequencies in the signal is insufficient, and the models learns more complex frequency tion patterns. This allows for reliable authentication even at very low bit-rates, where standard JPEG compression no longer practical, c.g., at bit-rates around 0.4 bpp where our DCN codec with low- quality settings improved manipulation detection accuracy from 37% 86%. We believe the pro- posed approach is particularly valuable for online media platforms (e.g.. Truepic, or Facebook), who need pre-screen content upon reception, but need to aggressively optimize bandwidth/storage.
cnpttaation/mmpfficcaaa
Source Code: git.ub...boo.. httpub..om//ooooo wtt../oo/ooooouuuuuuaaaaaa
Acknowledgements: This work supported by the NSF award number 1909488.
10

---

Published as a conference paper at ICLR 2020
REFERENCES
E. Agustsson, M. Tschannen, F. Mentzer, R. Timofte, and L. Van Gool. Generative Adversarial Networks for Extreme Learned Image Compression. arXiv: 02958, 2018.
J. Ballé, V. Laparra, and E. Simoncelli. End-to-end Optimized Image Compression.
arXiv:661101104,, 2016.
B. Bayar and M. Stamm. Constrained convolutional neural networks: A new approach towards general purpose image manipulation detection. IEEE Trans. Information Forensics and Security, 13(11), 2018.
F. Bellard. Better portable graphic. hutps://blllrdd.or//b//// 2014.
E. M Bik, A. Casadevall, and F. Fang. The prevalence of inappropriate image duplication in biomed- ical research publications. MBio, 7(3), 2016.
Y. Blau and Michaeli. Rethinking lossy compression: The rale-distortion-perreppiio tradeoff.
190.072,,,,, 2019.
E. Bucci. Automatic detection of image manipulations in the biomedical literature. Cell death & disease, 9(3):000, 2018.
V. Bychkovsky, S. Paris, E. Chan, and F. Durand. Learning photographic global tonal adjustment with database of input/output image pairs. In IEEE Conf. computer vision and pattern recogni- tion, 2011.
B. Cabral and E. Kandrot. http:://gggnieeiii..cooommtttttt behind- photos/ 2015.
bereroo-e--bbiindd-reieeww. preview-
M. Chen, J. Fridrich, M. Goljan, and J. Lukás. Determining image origin and integrity using sensor noise. IEEE Trans. Information Forensics and Security, 3(1), 2008
R. Chesney and D. Citron. Deepfakes and the new disinformation war: The coming age of post-truth geopolitics. Foreign Aff., 98, 2019.
CLIC. Image compression challenge. http://www..oonpeesssooccccaaall 2019.
Y. Collet. FSE Codec. https//iihhbbooommmaa///////iuut 2013.
M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, Benenson, U. Franke, S. Roth, and B. Schiele. The cityscapes dataset for semantic urban scene understanding. In IEEE Conf. computer vision and pattern recognition, 2016.
D. Cozzolino, Thies, A. Rössler, C. Riess, M. Nießner, and L. Verdoliva. Forensictransfer: Weakly-supervised domain adaptation for forgery detection. arXiv: 1812.02/0 2018.
DT
C. Pasquini, V. Conotter, and G. Boato. Raise: A raw images dataset for digital image forensies. In ACM Multimedia Systems Conference, 2015.
Dodge and L. Karam. Understanding how image quality affects deep neural networks. In JEEE Int. Conf. Quality of Multimedia Experience, 2016.
J. Duda. Asymmetric numeral systems: entropy coding combining speed of huffiman coding with compression rate of arithmetic coding. arXiv:1311.2540, 2013.
Duda, K. Tahboub, N. Gadgil, and E. Delp. The use of asymmetric numeral systems as an accurate replacement for Huffman coding. In Picture Coding Symposium, 2015. doi: 1091109/PS..0015. 7170048.
Facebook. Spectrum, image transcoding library. https://lbsspecruum.i// 2018.
N. Gilbert. Science journals crack down on image manipulation. https//wwww.aauure.cmmm/eew wwwp:////w/ll//ee22/.../../. 2009.
Google. Guetzli perceptual jpeg encoder. http://gginubccommggogge///eeell 2016. 2016.

---

Published as a conference paper at ICLR 2020
L. Gueguen, A. Sergeev, B. Kadlec, R. Liu, and J. Yosinski. Faster neural networks straight from jpeg. Advances in Neural Information Processing Systems, 2018.
S. Hasinoff, D. Sharlet, R. Geiss, A. Adams, J. Barron, F. Kainz, J. Chen, and M. Levoy. Burst pho- tography for high dynamic range and low-light imaging on mobile cameras. ACM Transactions Graphics, 35(6), 2016.
P. Korus. Digital image integrity-a survey of protection and verification techniques. Digital Signal Processing, 71, 2017.
P. Korus and N. Memon. Content authentication for neural imaging pipelines: End-to-end optimiza- tion of photo provenance in complex distribution channels. In IEEE Conf. Computer Vision and Pattern Recognition, 2019.
Z. Li and D. Hoiem. Learning without forgetting. IEEE Trans. pattern analysis and machine intelligence, 40(12), 2017.
Z. Liu, T. Liu, W. Wen, I.. Jiang, J. Xu, Y. Wang, and G. Quan. DeepN-JPEG: A deep neural net- work favorable JPEG-based image compression framework. In ACM Annual Design Automation Conference, 2018.
K. Ma, Duanmu, Q Wu, Z. Wang, H. Yong, H. Li, and L. Zhang. Waterloo exploration database: New challenges for image quality assessment models. IEEE Trans. on Image Processing, 26(2), 2016.
F. Marra, D. Gragnaniello, L. Verdoliva, and G. Poggi. A full-image full-resolution end-to-end- trainable enn framework for image forgery detection. arXiv:1909.06755 2019a.
F. Marra, D. Gragnaniello, L. Verdoliva, and G. Poggi. Do GANs leave artificial fingerprints? In IEEE Conf. Multimedia Information Processing and Retrieval, pp. 506-511, 2019b.
O. Mayer and M. Stamm. Forensic similarity for digital images. arXiv:1902.04665. 2019.
F. Mentzer, E. Agustsson, M. Tschannen, R. Timofte, and L. Van Gool. Conditional Probability Models for Deep Image Compression. IEEE Conf. Computer Vision and Pattern Recognition, 2018. doi: 10. 1099CV
Y. Mirsky, Mahler, Shelef, and Y. Elovici. CT-GAN: Malicious Tampering of 3D Medical Imagery using Deep Learning.
2019,
A. Popescu and H. Farid. Exposing digital forgeries in color filter array interpolated images. JEEE Trans. Signal Processing, 53(10), 2005.
T. Portenier, Q. Hu, A. Szabo, S. Bigdeli, Favaro, and M. Zwicker. Faceshop: Deep sketch-based face image editing. ACM Trans. Graph., 37(4):99, 2018.
T. Portenier, Q Hu, P. Favaro, and M. Zwicker. Smart, deep copy-paste. arXiv: 2019.
A. Prakash, N. Moran, S. Garber, A. DiLillo, and J. Storer. Semantic perceptual image compression using deep convolution networks. In Data Compression Conference, 2017.
A. Rhatushnyak, Wassenberg. Sneyers, Alakuijala, L. Vandevenne, Versari, R. Obryk,
Z. Szabadka, A. Deymo, E. Kliuchnikov, et al. Committee Draft of JPEG XL Image Coding System. arXiv: 03565, 2019.
S. Richter, V. Vineet, S. Roth, and V. Koltun. Playing for data: Ground truth from computer games. In European conference on computer vision, 2016.
O. Rippel and L. Bourdev. Real-Time Adaptive Image Compression. arXiv:1705.0588 2017.
L. Theis, W. Shi, A. Cunningham, and F. Huszár, Lossy Image Compression with Compressive Autoencoders. arXiv: 2017.
12

---

Published as a conference paper at ICLR 2020
G. Toderici, D. Vincent, N. Johnston, S. Jin Hwang, D. Minnen, J. Shor, and M. Covell. Full resolution image compression with recurrent neural networks. In IEEE Conf. Computer Vision and Pattern Recognition, 2017.
R. Torfason, F. Mentzer, Agustsson, M. Tschannen, R. Timofie, and L. Van Gool. Towards Image Understanding from Deep Compression without Decoding. arXiv:/803.06133 2018.
Truepic. https///ruepic.com 2019.
YH. Tsai, X. Shen, Z. Lin, K. Sunkavalli. and MH. Yang. Sky is not the limit: semantic-aware sky replacement. ACM Trans. Graph., 35(4): 1,9-1, 2016.
Z. Wang, D. Liu, S. Chang, Q. Ling, Y. Yang, and T. Huang. D3: Deep dual-domain based fast restoration of jpeg-compressed images. In IEEE Conf. Computer Vision and Puttern Recognition, 2016.
A. Weißenberger and B. Schmidt. Massively parallel ans decoding on gpus. In ACM Int. Conf. Parallel Processing, 2019.
Y. Wu, W. AbdAlmageed, and P. Natarajan. Mantra-net: Manipulation tracing network for detection and localization of image forgeries with anomalous features. In IEEE Conf. Computer Vision and Pattern Recognition, 2019.
W. Xiong, J. Yu, Z. Lin, J. Yang, X. Lu, C. Barnes, and J. Luo. Foreground-aware image inpainting. In IEEE Conf. Computer Vision and Pattern Recognition, pp. 5840-5848, 2019.
13

---

Published as a conference paper at ICLR 2020
Table A.1: Structure of the bit-stream describing DCN image
Section
Content
Data Type
Bytes
Latent shape H W x N
uint8
3
Basic meta-data:
Length of coded channel sizes 2 bytes (uint
uint16
2
Channel sizes (shorter of a/b)
(a) FSE-encoded channel sizes'
uint
var
(b) raw bytes
uint16
2N
Image data (Nx shorter a/b)
(a) FSF-encoded latent channel!
uint8
var
(b) RLEE-ncoded latent channel (frepetitions byte)
uint16 uint8 3
T. inclusive of both ANS probability tables and entropy-coded data
Table A.2: Bit-stream length of channel entropy coding (EC) relative to global EC for different quality levels and image patches of various size.
Avg. bit-stream size
Bit-stream size range
DCN model
128
256
512
128
256
512
low quality (16-C)
1.03
0.934
0.882
0.917
1.098
0.829
1.005
0.755
0,980
medium quality (32-C)
1.05
0.933
0.874
0.961
1.108
0.821
0.968 18833-0999
0,998
0.742
high quality (64-C)
1.07
0.948
0.887
0.977
1.119
0.773 0.964
A DCN CODEC DETAILS
A.1 QUANTIZATION AND ENTROPY REGULARIZATION
The standard soft quantization with a Gaussian kernel (Mentzer et al. 2018) works well for round- ing to arbitrary integers, but leads to numerical issues for smaller codebooks. Values significantly exceeding codebook endpoints have zero affinity to any of the entries, and collapse to the mean (i.e., so 0 in case; Fig. a...). Such issues can be addressed by increasing numerical precision, sacrific- ing accuracy (due to larger kernel bandwidth), or adding explicit conditional statements in the code. The latter approach is inelegant and cumbersome in graph-based machine learning frameworks like Tensorflow. We used a t-Student kernel instead and increased precision of the computation to 64- bits. This doesn't solve the problem entirely, but successfully eliminated all issues that we came across in our experiments, and further improved our entropy estimation accuracy. Fig. A.2 shows entropy estimation error for Laplace-distributed random values, and different hyper-parameters of the kernels. We observed the best results for a t-Student kernel with 50 degrees of freedom and bandwidth 7 = 25 (marked in red). This kernel is used in all subsequent experiments.
We experimented with different codebooks and entropy regularization strengths. Fig. A.3 shows how the quantized latent representation (QLR) changes with the size of the codebook. The figures also compare the actual histogram with its soft estimate (equation ( We observed that the binary codebook is sub-optimal and significantly limits the achievable image quality, especially as the number of feature channels grows. Adding more entries steadily improved quality and the codebook with M = 32 entires (values from -15 to 16) seemed to be the point of diminishing returns.
Our mntroy-bbsed regularization turned out to be very effective shaping the QLR (Fig. A.3 and dispensed with the need to use other normalization techniques (e.g., GDN), We used only single scalar multiplication factor responsible for scaling the distribution. All baseline and fine- tuned models use λи = 250 (last column). Fig. A.4 visually compares the QLRs of our baseline low-quality codec (16 feature channels) with weak and strong regularization.
A.2 ENTROPY CODING AND BIT-STREAM STRUCTURE
We used a rudimentary bit-stream structure with the essential meta-data and markers that allow for successful decoding (Tab. I.I.. Feature channels are entropy-coded independently (we refer to this approach as channel EC), and can be accessed randomly after decoding lookup table of their sizes. This simple approach can yield considerable savings w.r.t. global entropy coding (global EC), especially the image size increases (Tab. A.2
14

---

Published as a conference paper at ICLR 2020
<55 -34
40
Mears 31-2.2)
641
Figure A.1: Comparison of soft quantization with Gaussian and t-Student kemnlss for a Laplace distribution of increasing scale: the t-Student kernel is more accurate and robust to outliers.
Rmat Lomee 25 2M
Fand 1i cAa
Sen Livina ml -M
Gma Liweeel De
Hand Secuu vicit tte
Krmel Loobe Tn TM
Fand Ca vidi 11
Gend Livindl 1+2
Lhwwett vid -12
Liaee 220 110
220
Figure A.2: Entropy estimation error for Laplacian distribution with varying scale and various hyper-parameters of the kernels: the t-Student kernel (2nd-6th column) is more accurate than the Gaussian (Ist column) especially for wide distributions overflowing the codebook range.
15

---

Published as a conference paper at ICLR 2020
(Bock
impact of quantization codebook size
(b) impact of entropy regularization given fixed 5-bpf codebook
Figure A.3: Comparison of the distributions of the learned quantized latent representations (QLRs): (a) impact of codebook size; (b) impact of entropy regularization.
Botas
Chacnd&
Chasal
Cloced Hucar
Cheeai
Clarac
Clacsi
Dhanel
(a) weak entropy regularization (Aæ=1)
(b) strong entropy regularization (Az==550)
Figure A.4: Comparison of the latent representations (16-C model) of an example 512 x 512 px image (thong-vo-428) learned with weak and strong entropy regularization (5- codebook).
16

---

Published as a conference paper at ICLR 2020
<l:522: 30
go: abbettorwentaivddd 17937
bete- ore Lineo
050 12s
<tts sttu-ttiita 2015
640
BDoslied
Bladidd hpp
Bladts kpp
Electi App
top e22 ladlinds
kocho011 24 aungpe
*4: ladiuute
#14 kodaa11
#20 lodla121
1.50
Eibetiie
Mastis
Edbcton
ceei12: 31 issago
ell: 2el22
#31 m22654994552
4ot tleckii
begkin
ESoctine
Edectin
Eixecin
<i4532 20 truggen
4155 witsoroootdooe 393001
(333 thoag 425
J:3 sttartiettaa00
345 90111
10
10
31
10
10
tepp
Indu0512 teagge
24: bndtevl5
210: ovttn15
A Indre:
Edeelin
Seage
210
365 0000
1leellntg Tpp
10
Figure A.5: Rate-distortion trade-offs of the baseline DCNs on the clic, kodak and raw test sets: (left) average over all images: (2nd-5th columns) sample images from Fig. [ (top section) linear- scale SSIM; (bottom section) MS-SSIM in dB scale f(x) - 10 log (1 - x).
17

---

Published as a conference paper at ICLR 2020
back
30)
201)
8403
Effective hyp
Effective lop
comparian los onlput (noe) image
oxtourtion un natural images (dic) imsaga
300 1000
dia (LA40
31060
byp
bpp
Figure A.6: Rate-distortion performance for standard codecs and all DCN versions including the pre-trained baselines (b) and fine-tuned models with the the weakest (A, 1) and the strongest emphasis on manipulation detection (A, = 0.005 and 0.001). The latter incure only fractional cost in payload/quality but bring significant benefits for manipulation detection. The top (bottom) rows show results for SSIM (MS-SSIM), respectively, and include DCN models fine tuned on native camera output (raw) and mixed natural images (clic).
18

---

Published as a conference paper at ICLR 2020
Table B.1: Example confusion matrices for the baseline and fine-tuned low-quality DCN models.
(a) pre-trained 16-C DCN + 36.9%
(b) fine-tuned w. A -0.0010 - 87.0%
sharpen
resample
gaussian
median
sharper
resample
gaussian
median
native
jp
awgi
native
jpeg
Predicted
True
native
10
24
23
92
*
sharpen
12
resample
58
25
5
11
gaussian jpeg
65
7
15
*
95
12
22
20
awgn
12
11
41
10
*
median
45
9
29
28
B FINE-TUNING ON NATIVE CAMERA OUTPUT
Controlling Detection Accuracy: Fig. visualizes the trade-offs in image compression and foren- sic analysis performance. Here we show direct impact of image compression and fine-tuning settings on the achievable manipulation detection accuracy and its variations (Fig. B. For the JPEG codec, we observe nearly perfect manipulation detection for the highest quality level (QF=95), and steady decline starting immediately below. The sudden drop accuracy corresponds the quality level of JPEG as one of the manipulations (QF=80). For DCN models, we clearly see steady improvement of fine-tuning w.r.t. the baseline models (on the right). Interestingly, the high-quality model shows slight decline at first.
Qualitative Analysis: The learned frequency attennatiio//ampliiiatton patterns for all of the con- sidered quality levels shown in Fig. B.2 The visualizations computed in the FFT domain and show the relative magnitude of individual frequencies w.r.t. original uncompressed images (av- eraged over all test images). In all cases, we observe complex behavior beyond simple inclusion of freu-rry content. The pattern seem to have a stable trajectory, despite independent runs of the experiment with different regularization strengths Ae. The same patterns will also be visible in individual image spectra (Fig. B.4 Fig. B.6
Generalization and Imaging Artifacts: While our baseline DCN models were pre-trained on large and diverse training set, the fine-tuning procedure relied the complete model of photo acquisition and dissemination. Photo acquisition with digital cameras yields characteristic imaging artifacts, e.g., due to color filtering and interpolation. This leads to characteristic distribution of native camera output (NCO), and ultimately biases the codec. Fig. [B.3] shows the differences in SSIM between the baseline models and the models fine-tuned with very weak cross-entropy objective (leading to no improvement in manipulation detection accuracy). For NCO (raw test set), we observe improvement of image quality (and corresponding increase in bitrate). For the kodak set, the quality remains mostly unaffected (with an increased bitrate). On the clic set, we observe minor quality loss, and occasional artifacts (see examples Fig. B.4).
In Fig. B.4 Fig. B.6 we collected example images from all test sets (elic, kodak, and raw) and compress them with baseline and fine-tuned models. The images are ordered by SSIM deterioration due to weak fine-tuning (quality loss without gains in accuracy; Fig. B.3) the worst cases are shown at the top. (Note that some of artifacts are caused by JPEG encoding of the images embedded in the PDF, and some geometric distortions were introduced by imperfect scaling in malploilib.) In the first clic image (Ist row in Fig. B.4h, we can see color artifacts along the high-contrast wires. In the second image, see distortions in the door blinds, and subtle shift in the hue of the bike. For the remaining two images, SSIM remains the same or better and we do not see any artifacts. In the kodak set, the worst image quality observed for kodim05 (1st row in Fig. B.5) but we don't any artifacts.
19

---

Published as a conference paper at ICLR 2020
Figure B.1: Impact of compression quality and fin-tuning regularization on the achievable detection accuracy and its variation. Sudden drops for JPEG caused by inclusion of this compression as one of the manipulations, and correspond to the manipulation quality level (80).
1.000 0:20
0J00 00000
bust
Au 1,000 aut 0/M
0,000 acc: D.S
0.000 ace (c50
0.00 aod 0:02
A, 0100 aott (340)
Figure B.2: Visualization of frequency attenuation'amplifcatoon patterns in the FFT domain for the fine-tuned DCN codees (on native camera output). From the top: 16-C, 32-C, and 64-C models.
5004 Aondouttina Birned modil
153/ Brofecttic chaica moCi
Figure B.3: Difference in image fidelity (SSIM) after fine-tuning the low-quality DCN model within the dissemination workflow (weak CE objective with little to no improvement in detection accuracy): raw, kodak and clic datasets.
20

---

Published as a conference paper at ICLR 2020
EI0-H 1T%
N-
Figure B.4: Changes in the image compressed with various versions of the medium-quality DCN codec: (Ist column) sample image from the clic dataset; (2nd) pre-trained DCN model; (3rd-6th) fine-tuned models with decreasing Xç. Images are ordered by SSIM fidelity loss of A = 1.0 w.r.t. the pre-trained model.
21

---

Published as a conference paper at ICLR 2020
Figure B.5: Changes in the image compressed with various versions of the medium-quality DCN codec: (Ist column) sample image from the kodak dataset; (2nd) pre-trained DCN model; (3rd-6th) fine-tuned models with decreasing Xç. Images are ordered by SSIM fidelity loss of A 1.0 w.r.t. the pre-trained model.
22

---

Published as a conference paper at ICLR 2020
S MAOK
TUA
TRA
PAKA
PRR
PAR
A-1-0-N
A- LR
A-
Strudel
Strudel mo
Strndel mi
Stridel m
strudel
strudel mo
GRAPPERIA
P
GRAPPERIA
P
GRAPPERIA
P
ORAPPERRA
P
GRAPPERIA SPECK
P
#GRAPPERIL
P
SPECK
SPECK
SPECN
SPEEK
SPECK
Figure B.6: Changes in the image compressed with various versions of the medium-quality DCN
codec: (Ist column) sample image from the raw dataset; (2nd) pre-trained DCN model; (3rd-6th) fine-tuned models with decreasing Xç. Images are ordered by SSIM fidelity loss of X = 1.0 w.r.t. the pre-trained model.
23

---

Published as a conference paper at ICLR 2020
distortinn
648
0.43
6M0
0.72
1:14
30
1.2
Fifuctio hpp
Fifrctiv hpp
Figure C.I: Visualization of the wwwdddisoottioncacccrree trade-off on the clic dataset after fine-tuning on mixed natural images.
C FINE-TUNING ON MIXED NATURAL IMAGES
As discussed in Section ran additional experiments by skipping photo acquisition and fine- tuning directly on mixed natural images (MNT) a subset of the original DCN training set (2,500 images). Images in this dataset tend to have more details and depict objects coarser scale, since they were all down-sampled to 256 x 256 px (from various original sizes). This required adjusting manipulation strength maintain visual similarity between photo variations. In particular, we used weaker sharpening, Gaussian filtering with a smaller standard deviation (0.5), down&up-sampling via 75% of the image size (instead of 50%). Gaussian noise with standard deviation 0.012, and JPEG quality level 90. We fine-tuned for 600 epochs.
We summarize the obtained results in Fig C.J the gap in manipulation detection accuracy between JPEG and baseline DCN has disappeared, ex- cept for remaining sudden drops at selected JPEG quality levels (corresponding to the manipulation quality factor 90). We still observe significant improvement for fine-tuned DCN models, but here it tends to saturate around 86%, which might explain negligible improvement of the high-quality 64-C model. By inspecting confusion matrices, and awgn classes where the differences are extremely subtle.
Fug/C.Juusig using images from the clic test set. In this experiment,
most of the confusion between native, sharpen
The fine-tuned DCN models remain close to the baseline rate-distortion behavior. Interestingly, except for the weakest regularization (A, = 0.001), all fine-tuned models seem to equivalent (w.rt. the trade-off). We did not observe any obvious artifacts, even in the most aggressive models. The only image with deteriorated SSIM is the alejandro-escane escamilla-ō image from clic, which was consistently the most affected outlier in nearly all fine tuned models for both NCO and MNI (Ist row in Fig.
In some replications it actually managed to improve, e.g., for the shown cases with A. 0.005 and 0.001. However, we don't see any major differences between these variations.
Visualization of frequency attenuation patterns (Fig. C.3) shows analogous behavior, but the changes are more subtle on MNI. We included additional difference plots w.r.t. baseline and weakly fine- tuned models, where the changes are better visible. On NCO, due to less intrinsic high- high-frequency content, the behavior still very clear (cf. bottom part of Fig.
24

---

Published as a conference paper at ICLR 2020
II.
0407 (90
3.17
Chipadd Chiiaac
Oceral
SIIC 222 oicc 000
obnc
Figure C.2: Changes in images compressed with various versions of the medium-quality DCN codec fine-tuned on MNI: (1st column) sample image from the clic dataset; (2nd) pre-trained DCN model; (3rd-6th) fine-tuned models with decreasing Ae. The top image corresponds to the most consistent outlier with the worst SSIM degradation.
25

---

Published as a conference paper at ICLR 2020
.000 - ace: 1LB7
0.500 acc: (L:0
00000 are (66)
0,055 acc: (LS7
00021 apc DU78
Tart
bomic zeet DS3
0000 are
8,000
0.000 are
0.05 are
0.001
THt
Figure C.3: Visualization of frequency attennation/amplffcttion patterns for DCN codecs fine-tuned on MNI: (top) low-quality codec tested clic images; (bottom) the same codec tested images. Difference plots show changes w.r.t. the baseline and weakly fine-tuned models.
26

---

Published as a conference paper at ICLR 2020
training dats = rw comgresiio dem+
Inining dala our
compinestoon den
training data raw compession Jpg
1.0
tesk
kodakk
mow
0.6
0.2
training
compression
trainin
comproesion
training
compresson Jpm
1.0
0.5
0.4
0.4
0.5
1.0
0.8
0.6
1.0
0.2
0.5
validation socunacy
Validition aceuzocy
validation securacy
Figure D.1: Transferability of the trained FAN models to different data distributions: (top) models trained on native camera output can generalize to raw test images from different cameras; (bottom) models trained on diverse images generalize well to different except native camera output.
D TRANSFERABILITY OF THE FAN
In this study, considered two broad classes of images: native camera output (NCO) and mixed natural images (MNI) which exhibit significant differences pixel distribution. For DCN pre- training, we relied large MNI dataset with images down-sampled 256 x 256 px (Section 3.4 Fine-tuning was then performed on either NCO from a single camera (Nikon D90; Section 4 or smaller sub-set of the original training MNI (2,500 images; Appendix C). Finally, considered three test sets: raw with NCO from different cameras; clic and kodak with MNI.
We observed that the FAN models tend have limited generalization capabilities images with different pixel distribution. We ran additional experiments to quantify this phenomenon, and show the obtained results in Fig. D.I where we compare test validation accuracy for various sets (we also included a version of the clic set resized to 256 x 256 px). In the top row, we show results for models trained on NCO from a single camera. We can see imperfect, but reasonable generalization to the output of various cameras (red markers). Once the data distribution changes, the performance drops significantly. Analogously, FAN models trained on MNI generalize well to other MNI datasets (clic and kodak), but fail on NCO. We see additional bias towards images down-sampled to the same resolution as the training data (compare clic vs clic-256 images), but here the difference is small and consistent between 5.2 6% based on linear fit to the data.
27


Reviews:
[{"review_id": "H1l-LJJAKS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nSummary of the paper\n- This work proposes a new deep-learning-based method to replace the lossy compression techniques of images., jpg.\n- The work investigates the role of codec and shows that the proposed complex photo dissemination channels optimizes the codec related traits on images.\n- The method achieved much better performance in compressing images compared to practically used JPEG (QF-20)\n\nI think the paper is well written and the experiment seems to support the author's argument. Unfortunately, this field is not overlapped to my research field, and it is hard for me to judge this paper.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Byegr-2J5B", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a learned image compression method that is able to be robust under a variety of tasks. The results aren't state of the art in terms of rate-distortion performance, but this paper has a very good analysis of the results, and has produced a very fast codec. In that sense, this is a very interesting paper that may lead to other fast methods (the other fast method they compared the runtime against - WaveOne never published a complete description).\n\nThis paper should be likely accepted, but the authors should town down the claims a bit. The results presented do NOT show that this method is better than the best hand engineered approach, despite what they claim. Even compared to BPG, which is NOT state of the art, the results are a mixed bag. \n\nWe would like to point out to the authors that the VVC codec has shown much stronger performance than BPG, and similarly the AV1 codec has surpassed the performance of BPG. Moreover, even Pik has also surpassed the performance of BPG, so just showing stronger performance than BPG is not grounds to make the claim that this method is superior to hand engineered approaches.\n\nMoreover, as I stated earlier, this method is not even better all the time, therefore weakening the claim.\n\nOn the positives:\n- the paper fully describes the architecture, unlike WaveOne\n- the runtime numbers are impressive (as far as I know, there is no faster published method)\n- the authors consider applications other than compression performance (such as classification performance in forensic analysis)\n\nOn the negatives, which I highly suggest that the authors fix if this paper is to be taken seriously by the community:\n- please be sure to explain that SSIM is computed in <RGB | grayscale>\n- please be more explicit about which loss is used during training for distortion (i.e., \"we use MSE for the training loss, but stop training when SSIM converges\")\n- please provide PSNR numbers for the method; and ideally MS-SSIM (in decibels) instead of PSNR\n- please add other neural compression methods to the graphs \n- please clarify that you create a file and decode a file for each image used to create the graphs (very important topic), as opposed to using the estimated file size\n- tone down the claims w.r.t. beating classical codecs", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJeC0l9yjS", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper describes a pipeline for image compression which allows to reliably detect specific manipulation patterns in compressed images.  The results show that it is possible to learn image compression that performs similarly to a modern image compression algorithm while in the same time is optimized to reveal specific kinds of manipulations. The authors build upon (Korus & Memon, 2019), but use a learnable codec instead of differentiable JPEG. \n\nThe idea to regularize entropy of the latent representation of images is interesting. A method to train a well-performing image compression system which can also follow additional constraints (such as ability to reveal certain manipulations) is very valuable for practice. Unfortunately, there are already available trainable compression methods and the authors do not compare to these methods. However, in my opinion to detect manipulation in the image one should prove that visual content in some area of the image was significantly changed with respect to some original, while in the other parts of the image it was not changed. Otherwise it becomes impossible to distinguish in-camera filtering and secondary postprocessing. Basically, the authors present a method to detect whether a very particular configuration of some basic image processing filters (Gaussian blur, median filter, resampling)  was applied to the image. Therefore the particular problem formulation looks very artificial. \n\nWith regards to the experiments in the paper, I was somewhat lost. Compare the Fig. 5 and the Fig. 8. In the Fig. 8, we see a big set of possible system configurations having different manipulation detection accuracy, image quality and compression performance.  In the Fig. 5, we see a compression efficiency-image quality dependency. However, it remains unclear how do the systems represented at these two graphs relate to each other, or, in the other words, what is he mapping between points of these graphs. Next, poor performance of JPEG manipulation detection by the proposed  network does not prove that JPEG manipulation cannot be detected, it just shows that the proposed architecture does not perform well in this problem. A comparative study which relates a new system to a current state of the art is required to claim that a proposed approach is better. Finally, SSIM is not a standard way to compute image quality. MS-SSIM and PSNR are also popular, and a user study is usually recommended to claim that some method generates images of better visual quality.\n\nSummarizing, the authors do not provide a new best-performing image compression algorithm, and neither solve a problem of image manipulation detection, but show that it is possible to learn an image compression system with some additional constraints. I believe it is an interesting contribution, and I hope the authors can improve presentation of the experiments.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a new image compression approach that preserves the patterns indicating image manipulation. The reviewers appreciate the idea and the method. Please take into account the suggestions of Reviewer1, when preparing the final version.
Decision: Accept (Poster)