Provenance detection through learning transformation-resilient watermarking
Authors: Jamie Hayes
Krishnamurthy Dvijotham
Yutian Chen
Sander Dieleman
Pushmeet Kohli
Norman Casagrande
Venue:  (2020)
Keywords: watermarking
provenance detection

Abstract: Advancements in deep generative models have made it possible to synthesize images, videos and audio signals that are hard to distinguish from natural signals, creating opportunities for potential abuse of these capabilities. This motivates the problem of tracking the provenance of signals, i.e., being able to determine the original source of a signal. Watermarking the signal at the time of signal creation is a potential solution, but current techniques are brittle and watermark detection mechanisms can easily be bypassed by doing some post-processing (cropping images, shifting pitch in the audio etc.). In this paper, we introduce ReSWAT (Resilient Signal Watermarking via Adversarial Training), a framework for learning transformation-resilient watermark detectors that are able to detect a watermark even after a signal has been through several post-processing transformations. Our detection method can be applied to domains with continuous data representations such as images, videos or sound signals. Experiments on watermarking image and audio signals show that our method can reliably detect the provenance of a synthetic signal, even if the signal has been through several post-processing transformations, and improve upon related work in this setting. Furthermore, we show that for specific kinds of transformations (perturbations bounded in the $\ell_2$ norm), we can even get formal guarantees on the ability of our model to detect the watermark.  We provide qualitative examples of watermarked image and audio samples in the anonymous code submission link.

Under review as a conference paper at ICLR 2020
PROVENANCE DETECTION TRANSOORMATION-RRSIIIENY WATERMARKING
THROUGH LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
Advancements in deep generative models have made it possible to synthesize images, videos and audio signals that are hard distinguish from natural signals, creating opportunities for potential abuse of these capabilities. This motivates the problem of tracking the provenance of signals, i.e., being able determine the original source of signal. Watermarking the signal at the time of signal creation is a potential solution, but current techniques are brittle and watermark detection mechanisms can easily be bypassed by doing some post-processing (cropping images, shifting pitch in the audio etc.). In this paper, we introduce ReSWAT (Resilient Signal Watermarking via Adversarial Training), framework for learning wransforatiion-rssiiene watermark detectors that are able to detect a water- mark even after signal has been through several post-processing transformations. Our detection method can be applied to domains with continuous data represen- tations such as images, videos or sound signals. Experiments on watermarking image and audio signals show that our method can reliably detect the provenance of synthetic signal, even if the signal has been through several post-processing transformations, and improve upon related work in this setting. Furthermore, we show that for specific kinds of transformations (perturbations bounded in the & norm), we can even get formal guarantees the ability of our model to detect the watermark. We provide qualitative examples of watermarked image and audio samples in the anonymous code submission link.
INTRODUCTION
Generative models have contributed to impressive advancements in content generation and repre-
sentation learning in both digital image and audio domains (Brock et al. (2018); Kalchbrenner et al. (2018); Mchri et al. (2016); Zhu et al. (renger et (2019); Donahue & Simonyan (2019); Oord et al. (0016); Goodfellow et al. (2014); Kingma & Welling (2013). However, as generative models learn better match target distribution, the distinction between natural signals and synthetic
signals generated by model has blurred, leading to a raft of concerns over the potential misuse of these models (Chesney& & Citron (218)). For example, synthetic videos that are indistinguishable
from natural videos, sometimes referred to as deep fakes, have the potential to cause widespread distrust in traditional media.
While the problem of detecting synthetic signals is interesting in its own right, it is challenging to do in manner that is independent of the model used to generate the signal. Instead, we consider the problem of provenance detection via watermarking, a technique that involves injection of a carefully chosen (but imperceptible) perturbation (watermark) into the signal the time of creation. The presence of the perturbation in the signal can be later used to detect the provenance (or ultimate source) of this signal. While the simplicity of the technique makes watermarking an appealing technique for detecting synthetic (as well naturally generated) signals, it is susceptible to adversarial actors that can systematically try to remove the watermark by transforming the signal (so as to obfuscate the provenance). In the case of images, this may take the form of cropping, addition of Gaussian noise or blurring, rotating images etc. Many watermarking schemes break down under these types of transformations.
In this paper, we propose a novel transformation--eetten watermarking scheme- Resilient Signal Watermarking via Adversarial Training (ReSWAT) that is able detect the presence of watermark

---

Under review as a conference paper at ICLR 2020
after the signal has been through systematic attempts remove the watermark. We use ideas from the literature on adversarial training (Madry et al.) 2017) to learn to synthesize watermarks (encoding the provenance of signal) that can be detected even after the watermarked signal has been through transformations that seek to remove the watermark. Our detection method can operate in any domain with continuous data representations. Experimental results demonstrate that method is effective and substantially improves over existing methods it can reliably detect the provenance of a synthetic signal even if the signal has been deformed or manipulated.
1.1 CONTRIBUTIONS
We formulate the transformation resilient watermarking problem and show how it can be reduced to an empirical risk minimization problem with minimax loss function.
We develop a transformation resilient watermarking scheme, named ReSWAT (Resilient Signal Watermarking via Adversarial Training) and show that the learning procedure is closely related to adversarial training techniques (Madry et al. 2017 Athalye et al.) (017).
Empirically, show across set of image and audio datasets that scheme can produce imperceptible watermarks that can be detected even after the signal has been through a series of adversarial transformations that preserve fidelity to the original signal. We do this using both standard metrics for signal watermarking and via human evaluation on audio signals. Furthermore, we formal provable guarantees of detection if an adversarial transformation has a bounded &g norm.
1.2 RELATED WORK
The problem of detecting synthetic signals has recently received a significant amount of attention. Marra et al. (2018) investigate how generative adversarial network (GAN) (Goodfellow et al. (014)) leaves an identifiable fingerprint in the image it generates, while Nataraj et al. (2019) detect if an image generated by GAN by training detection model on co-occurence matrices of the RGB channels of images. Similarly, Yu et al. (2018) also train a model that learns to detect if an image was generated by GAN, and attribute a generated image to its source model. Unfortunately, these detection methods suffer from some limitations: (1) they are specifically designed to operate only in the image domain and (2) they are not designed be resilient transformations that an attacker could apply.
Digital watermarking consists of two stage process: An embedding stage, where the original signal is combined with hidden message (also referred to as the watermark), producing a watermarked version of the signal. Secondly, decoding stage, where the hidden message is retrieved from the watermarked signal. Watermarking schemes are typically evaluated on fidelity of the watermarked signal to the original signal, the resilience the watermark to various post-processing transformations and the false positive rates (i.e. whether the decoder detects watermarks in watermarrdrreed signals). Robustness of watermarking techniques is measured against a number practical attacks that aim to remove degrade the watermark signal. For example, adding Gaussian noise, image cropping and image compression represent watermark degrading, watermark removal and watermark geometric (re- positioning) attacks, respectively. In this paper, are primarily interested in zero-bit watermarking, where one is simply interested in detecting the presence of a watermark (rather than decoding hidden message from the watermark). The state-of-the-arr zero-bit watermarking scheme in the image domain is referred to as Broken Arrows (BA) (Furon & Bas 2008) and won the "Break Our Watermarking System" (BOWS) competition (Bennour et al. 2007) ). BÄ has a provable minimum false positive rate under Gaussian noise, however the scheme does not provide robustness against geometric attacks (like rotations and cropping).
FORMULATION OF THE ROBUST WATERMARKING PROBLEM
We study signals that live in a space X and are generated by a probabilistic source P, our framework applies regardless of whether P, is natural source (images of natural scenes) artificial source (samples from a VAE or a GAN). We are interested developing a scheme to watermark signals generated by this source, with the requirement that:
2

---

Under review as a conference paper at ICLR 2020
The detector should detect watermark in any watermarked signal from source P. and not detect a watermark any other signal.
Even if post- processonn transformations (for example, compeessio/coooinngfrququn shift etc) applied to the signal, the detector detects the watermark, or the absence thereof.
Formally, we define the robust watermarking problem as follows:
Definition 2.1 (Transformation resilient watermarking scheme). Consider a watermarking scheme (W, where W X' + is watermarking routine and : X - {0. 1} watermark detector. Let T be space of transformations with T X' + X for each T T. We consider the watermarking scheme resilient with respect to a set of transformations for source P, D ((T(Ws))) = (T())) 0, T with high probability for Ps.
This formulation suggests empirical risk minimization approach for training D. Given fixed W, we can train to minimize the empirical risk
E max e ( er
TET
where l loss function measuring the discrepancy between the prediction of D and the desired label (0 for non-watermarked signals and 1 for watermarked signals).
The inner maximization transformations resembles the objective used in adversarial training (Madry et al. 2017), motivating our approach to learning a transformation resilient watermarking, We develop this idea the following section.
RESWAT: RESILIENT WATERMARKING VIA ADVERSARIAL TRAINING
We parameterize the detector D as a neural network fø. parameterized by If we fix the detector, the watermark embedding process W should embed watermark that is "strongly detected" by the detector, i.e. the watermark embedding mechanism pushes the watermark deep inside the decision boundary of D.. However, watermark does not significantly change the original signal according to some distance measure. While it is challenging to define the space of imperceptible distortions of the input, convenient proxy is to limit the change in terms of the foo norm between the watermarked and the original signal. Thus, we construct the watermark, 8, by solving the following optimization problem:
also want the watermark to be imperceptible
require that the
(()) min E(So(s 1)
|| Sc
This optimization problem can be solved efficiently using a projected gradient descent (PGD) method, and is mathematically very similar to computing an adversarial example for neural network (however, in this case, the adversary friendly and actually pushes the example minimize the loss of the detector). The parameter controls the perceptibility of the watermark. If is too large, the watermark would clearly perceptible while if it too small, the watermark may easily be washed out by post processing steps. Thus, the right choice of achieves the optimal trade-off between perceptibility and transformation resilient detection. Note that we use the lo norm of the perturbation instead of another since this distributes the watermark over the entire signal, and will be resilient to geometric attacks (that black out patches of the input, or obfuscate inputs).
With this watermark embedding method, resilient watermark detection as follows:
train the detector perform transformation
minimize E max e (fo(T((+))), + 81
(el.1))((ffoo((ss) (fo(T())).o)
(I)
P. TET
This is similar to the expectation over transformation attack Athalyy et al. (017)), that aims to construct adversarial examples that persist through of transformations. During training, only
3

---

Under review as a conference paper at ICLR 2020
Table 1: Transformations used evaluate the robustness of ReSWAT in the image space.
Transform
Transform description
Gaussian noise
Add Gaussian noise with standard deviation ø.
Rotation
Rotate an image by a random angle uniformly sampled from range r-r
Randomly crop an image's height by c pixels and width by ew pixels, and project back to the original image size.
Cropping
Horizontal flip
With probability 1/2, flip the image on the horizontal axis.
Randomly adjust the brightness of an image by factor, which ranges from 0 (no increase in brightness) to (maximum whitening).
Brightness
sample from differentiable transformations, this is so we do not have to approximate the gradient, which would be prohibitively slow to training.
In practice, at each iteration in training we approximate finding maXTET with ma, T......... where T; E T E {1....n). n). Theoretically, if we sample more transformations we achieve better approximation for the true gradient over the full transformation distribution, however, empirically, we found it too expensive to trade more parallel transformations for longer training time. We explore how resilience to transformations is affected by the number of sampled transformations each step in training in appendix (C] All following experimental results are taken from a model trained with sample size of one each iteration of training.
3.1 OPTIMIZING FOR NON-TRANSFERABLE WATERMARKS
Traditional watermarking schemes are hased on a secret key that ensures that only parties with access the key can create watermarked content; attacker that can watermark content without access to the key represents an integrity attack on the scheme. We refer to this kind of attack as a specificity attack that attempts introduce false positives. In our case, the classifier detecting/construucing the watermark can be thought of as an approximation of a secret key.
We will now propose method construct watermark perturbations for given watermark classifier, fo, which do not transfer to other watermark classifiers trained on the same data.
Recall that given signal 8, to create a watermark perturbation we use PGD which takes steps in the direction of
V. max TET (((fo(((s)).1)
(2)
To encourage that the generated perturbation doesn't 't transfer to other watermark classifiers trained on the same data, generate the watermark that is robust against ensemble of models. Let {ff X' + {0, 1). E
n}} be an ensemble of watermark classifiers trained on the same problem fo- To limit the transferability of watermarks fo, create watermark perturbations by stepping in the direction of
& max (v
TET ((F((T())))
(((F(C(())))))))
V. max iTET
(3)
Taking steps in the direction of this loss will increase the probability that watermark perturbations only classified as watermarked by fo and do not transfer to other models.
4 EVALUATION
Here, we give experimental results that our scheme is robust to a variety attacks. We first evaluate the scheme image data in section 4.T and section 4.2 and then audio data in section 4.3
4

---

Under review as a conference paper at ICLR 2020
1.0
1.0
10
S5IM Watermark
SSIM Watermark
SSIM Watermark
0.6
0.4
0.4
5.2
5.2
0.
ReSWAT
200
02
044
06
033
%
02
04
08
03
92
2.4
08
1.0
SSIM Transformed
SSIM Transformed
SSIM Transformed
Gaussian noise.
(b) Rotation.
(c) Crop.
Figure 1: Distortion needed for perfect provenance detection (by fe against different signal quality degradation levels suffered by the attacker. We measure the amount of distortion the watermarking scheme must introduce in order to guarantee perfect detection accuracy under transformation, and compare with the Broken Arrows zero-bit watermarking scheme.
(a) Not watermarked
(b) Watermarked (SSIM 0.82) (c) Watermarked + Gaussian noise (a = 0.4, 0.1)
Figure 2: Sample of a transformation attack on ImageNet (famp)) (a) non-watennee watermarked test set image, (b) is a watermarked version with the minimum necessary watermark required to be correctly classified by a detection model when Gaussian noise added at standard deviation depicted in (c).
4.1 EVALUATION PROTOCOL ON IMAGES
During training sample from of transformations, to which the scheme should be robust. In this evaluation we use transformations detailed in table []] We evaluate watermark detection robustness in the image space on both Cifar10 (Kizzhevsky et al.) and ImageNet (Deng et al., 2009).
We defer evaluation of Cifarl0 ( to appendix A and detail only ImageNet experiments here.
ImageNet. We train standard ResNet152 classifier (He al. (2016)) modified for binary prediction. for the watermark detector. We replace batch normalization with instance normalization (Ulyanov al. 2016)), as we noticed that batch statistics were heavily skewed by the transformations applied during training. We train with a mini-batch size of 32 for 60,000 steps, with initial learning rate of 0.1 and decaying this by factor of 10 every 20,000 steps.
All images are normalized into the range [0, and a batch of watermarked images are constructed with five PGD steps at each iteration. With respect to the set transformations. during training, we set to 0.25, to 4/2, C₁. and c, 10, and to 0.I. We trained four classifiers under these hyperparameters, two where at each step we randomly sample a single transformation, and watermark with 5/255 and 10/255. referred to f and fra,, respectively, and two where apply composition of all transformations, and watermark with 5/255 and 10/255, referred to as fcomp and &omp respectively.
5

---

Under review as a conference paper at ICLR 2020
All watermark detection classifiers achieved 100% test set accuracy, where the test set consists of 10,000 -oneaaaeradreed and 10,000 watermarked ImageNet test set images. detailed comparison of the differences between these models is given in appendix A however, trained under a composition of transformations are more robust to all attacks and so remaining experiments will use only Soomp and fommp For evaluation, we measure the success of an attack with respect to the distortion introduced by the attack measured by structural similarity (SSIM) score (Wang et al. 2004)).
found that models
4.2 EVALUATING PERFORMANCE AGAINST ATTACKS
We now describe experiments measuring the quality our watermarking scheme under various transformations seeking induce misclassification in the watermark detector: We study both attacks that seek to remove a watermark from given watermarked signals (thus inducing a false negative for the detector) and attacks that seek make the detector detect watermark in non-watermarked image (thus inducing a false positive).
4.2.1 SIGNAL TRANSFORMATION ATTACKS (FALSE NEGATIVES)
We investigate the trade-off in distortion of the watermarked image when require perfect de- tection under various transformations. Given a watermarked signal s, we sample from a space of transformations to create copies of this input under, referred as s i (1, (1,-.,)) n). We increase the watermark e value until the accuracy of the detection model on 8 n1....n is >99%. We measure the perceptibility of the watermark and the perceptibility of the transformation used in the attack, both in terms of SSIM, and compare with Broken Arrows (BA), where for each transformation we set the number of random samples, n, million.
Figure shows that for various distributions of transformations the amount of distortion introduced by the watermarking scheme is dominated by the amount of distortion introduced by the transformation. For Gaussian noise, the necessary distortion introduced by the watermark (for perfect detection under a transformation) using ReSWAT is approximately equivalent to BA and substantially smaller than the distortion introduced by the transformation. While for cropping and rotation attacks, our watermarks incur negligible levels of distortion while BA fails to watermark content without incurring large distortions. We show qualitative examples in fig. 2 for Gaussian noise attack, and analogous plots of fig. D Jsiig Seak-Siinnllto-- -o-iose ratio as the distortion metric in appendix
4.2.2 SPECIFICITY ATTACKS (FALSE POSITIVES)
Using the ImageNet dataset, we trained model with the same hyperparameters as Seomp- however the model was optimized by constructing watermarks using eq. 3 instead of eq. (2). using five other pre-trained watermark classifiers; we denote this model by feomp- Given 20 pre-trained watermark classifiers only differing from fcamp by random initialization of weights,
compare how well 1,000
watermark's 's constructed using each of these models transfers to fcimp and feimp representing an attack that attempts to introduce false positives. Figure]JJshwss the difference average false positive rate between demp and ftx /somp- At a watermark SSIM score of 0.60, fcomp has false positive rate of below 20% while the false positive rate these inputs is nearly 50% for feomp. Clearly, optimizing eq. (3) improves resilience to specificity attacks.
4.2.3 CERTIFIED ROBUSTNESS
The previous sets of experiments present results on the robustness of the watermarking scheme against "best effort" attacks, i.e, we attempt to compute attacks that break the watermarking system and declare success if we fail to do However, it is possible that attack algorithm failed to find the worst case transformation that would break the watermark detection. Thus, it is desirable to have stronger guarantee against all attacks within certain transformation space.
While provable guarantees are hard to obtain in general, we can leverage recent work on randomized smoothing techniques (Cohen et al. 2019) that are able to obtain certified guarantees against trans- formations constrained in the & norm, i.c., the adversary can transform the signal by any amount within distance in the fa norm. On 200 examples from the ImageNet test set that watermarked

---

Under review as a conference paper at ICLR 2020
1.0
1.0
Robust Non-robust
Robust Non-robust
False Positive Rate
0.8
Certified Accuracy
0.8
0.6
0.6
0.4
0.4
0.2
0.2
00
0.60
0.65
0.70
0.75 0.80 0.85 0.90 0.95 1.00 SSIM
0.0
0.0
0.5
1.0
1.5
2.0 Radius
2.5
3.0
3.5 4.0
Figure 3: Average false positive rate of f (non-robust) and füomp (robust) on watermark's constructed by other water- mark classifiers. Given as a function of the average watermark image SSIM.
Figure 4: Certified accuracy of 200 wa- termarked ImageNet examples. Plot- ted as a function of the & radius of ro- bustness. We compare with robustness guarantees against a non-robust water- mark detection model, that has not been trained on set of transformations.
with 20/255, estimate lower bound, p, the probability of the most-likely class under Gaussian noise parameterized by (O, I), using 10,000 random samples. Given p. the detector is robust to adversarial perturbations, y, if > 222 < o x (p) (as proven in Cohen et al. 2019)), )) up to confidence level of which we set to 0.99. Figure shows the certified accuracy of these examples as a function of the certified radius in the 22-norm. For e = 20/255 nearly 200 watermarked images robust any perturbation with &2-norm smaller than 1.5. We also include results a non-robust watermark detection model a model trained without sampling from a distribution of transformations. This model has comparatively small certified region of robustness, implying that training on a distribution of transformations does improve robustness.
4.2.4 OUT-OF-DISTRIBUTION EVALUATION
The previous section focused on adversarial attacks on signals drawn from the same distribution the model was trained on. A natural question to ask is, would the system fail when signals are drawn from different distribution? To evaluate this, we verified that, given model trained on Cifarl0 images, all SVHN test images are correctly classified to the non-waterne non-watermarked class. Secondly, we watermarked 500 images generated by BigGAN (Brock et al.) 2018) using model trained on ImageNet samples (S3ia We then measured resilience to trans mranffomattinss as detailed in section 4.2.1 we attacked both watermaerrdr and watermarked images with Gaussian noise, rotation and cropping such that the average SSIM of these images is 0.6. In comparison, the average SSIM of watermarked images was 0.82. The false positive rate (attacking won-raatrrrrked images) was 1.2% and the false negative rate (attacking watermarked images) was 0%. BigGAN watermarked images exhibited comparative levels of resilience transformations despite the model being trained on ImageNet samples.
4.3 EVALUATION ON TEXT-TO-SPEECH DATASET
To evaluate ReSWAT on audio data, we train a watermark detection model using a proprietary dataset composed of hours of high quality speech data, where each audio sample is short speech utterance lasting between 1 and 10 seconds. We use a DeepSpeaker architecture (Li al. 2017). modified for binary prediction. The pre- propesscssing stage takes input, variable length waveform, normalizes values between -0.5 and 0.5, and outputs the fixed length mel-spectro which is then used as input to the model.
7

---

Under review as a conference paper at ICLR 2020
Table 2: Transformations used evaluate the robustness of ReSWAT the audio space. Transformation values specify the amount of distortion that can be added to an audio sample while maintaining perfect detection accuracy under trained ReSWAT model. All audio samples are normalized into [-0.5, 0.5] range.
Input
Transform
Gaussian noise
Uniform noise
Pitch shift
Silence
Roll
7
Watermarked (e x 10 10
0.01 0.013
0.0056
0.16
0.78
1.0
Watermarked (c 4.8 10-3)
0.0278
0.5
0.78
1.0
Transform description
Gaussian noise
Add Gaussian noise with standard deviation Add uniform noise sampled from a range [-a, a:. Pitch-shifts audio by randomly sampled scale factor in range [B
Uniform noise
Pitch shift
Silence
Randomly silence a contiguous fraction, 7, of the audio.
Rolls values in audio by fraction of audio length uniformly sampled at random from range [-5, ö].
Roll
We train the detector model for 100,000 steps, with an exponentially decaying learning rate initialized at 0.001 and decayed by factor of 0.9 every 1000 steps. The watermarking value is initialized at 0.04 and decreased by 0.00001 whenever the detection rate is 100% based on a moving average of the previous 100 steps. We five PGD steps at each iteration create watermarked inputs. During training we sample from a set of transformations, described table 2 to which the scheme should be robust.
At test time we achieve perfect detection rate on 200 watermarked audio samples using e 4 10-4 Similarly to section 1.2.1 we measure the robust- ness of watermarked audio samples with respect the amount of distortion introduced by transforma- tions, under the requirement that detection accuracy is >99%. For given transformation and input, we
1200
1000.
Frequency
800
randomly sample the input 100 times under the trans-
6:00
formation, this creates a test set of 20,000 data points on which we measure the accuracy of the detector. Table[[]show> the results of attacking watermarked inputs for different values using various transfor- mations. Watermarks imperceptible, while a faint background noise can be heard for 4.8 x 10-3, The detection model robust to a large number of transformations, for exam- ple, 78% of audio can be obscured without decreasing detection accuracy.
200
4 x 10-4 are almost
Much
Worke
Worn
sautey
Wome
samee
statels
Better
Better
Figure 5: AB experiment. 660 non- watermarked and watermarked audio samples presented to 1320 people in total. Partici- pants were played the original sample and watermarked sample and asked to rate how different the original audio was from the wa- termarked version.
Human evaluation.
Here, we conduct human evaluation of watermark perceptibility using both mean opinion score (MOS) and A/B tests. MOS (Streijl et al. (016) is com- monly used measure for audio quality; is expressed as single rational number, typically in the range 1-5, where is lowest perceived quality, and 5 the highest perceived quality. The MOS is calculated as the arithmetic mean over single ratings performed by human subjects. Due the large number of audio samples used in the study, we use a single rating per audio sample. Our dataset consisted of 2000 watermarked and oon-waaermrrked audio samples, that were all correctly classified by the water-
8

---

Under review as a conference paper at ICLR 2020
mark detector model under transformations listed in table The average rating of non- won-wattrmarked content was 4.595=10.57, and the average rating of watermarked content (at = 4 x 10-4) was 4.530=0.530. Clearly, human participants did not perceive the watermarked audio as significantly worse or degraded.
For A/B tests took subset of 660 watermarked and non- non- watermarked audio samples. We played both the non-watermarked and watermarked audio sample to participants and asked if the watermarked audio sample was worse or better than the original. Results shown in fig. [ over 80% of human participants rated the quality of watermarked audio samples as the same as non-watermarked audio samples.
CONCLUSION
We presented a general solution, that leverages imperceptible watermark, to the problem of detecting the provenance of a signal. In a departure from related work, our watermarking schemes attempts to learn constructions of watermarks that are resilient to adversarial transformations. Our solution can be applied in numerous environments such as in image, audio and video domains. Results presented both on images and audio suggest that is possible to construct watermarks that simultaneously maintain a high level of signal fidelity, and are resilient to adversarial transformations, with minimal false positive rate.
REFERENCES
Anish Athalye, Logan Engstrom, Andrew Ilyas, and Kevin Kwok. Synthesizing robust adversarial examples. arXiv preprint arXiv:1707.07397, 2017.
Jihane Bennour, J-L Dugelay, and Federico Matta. Watermarking attack: Bows contest. In Secu- rity, Steganography, and Watermarking of Multimedia Contents IX, volume 6505, pp. 650518. International Society for Optics and Photonics, 2007.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 1006, 2018.
Robert Chesney and Danielle Keats Citron. Deep fakes: A looming challenge for privacy, democracy, and national security. 2018.
Jeremy M Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. arXiv preprint ArXiv: 1902.02922 2019.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L.. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09, 2009.
Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. arXiv preprint arXiv:1907.02544, 2019.
Teddy Furon and Patrick Bas. Broken arrows. EURASIP Journal on Information Security, 2008(1): 597040, 2008
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Parde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural informa- tion processing systems, pp. 2672-2680, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lockhart, Florian Stimberg, Aaron van den Oord, Sander Dieleman, and Koray Kavukcuoglu. Efficient neural audio synthesis. arXiv preprint arXiv:1882.088435 2018.
Diederik P Kingma and Max Welling. arXiv:1312.611 2013.
Auto-encoding variational bayes. arXiv preprint

---

Under review as a conference paper at ICLR 2020
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced research).
html
Chao Li, Xiaokong Ma, Bing Jiang, Xiangang Li, Xuewei Zhang, Xiao Liu, Ying Cao, Ajay Kannan, and Zhenyao Zhu. Deep speaker: end-to-end neural speaker embedding system. arXiv preprint arXiv:1705.02304 2017.
Aleksander Madry. Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: 770606608, 2017.
Francesco Marra, Diego Gragnaniello, Luisa Verdoliva, and Giovanni Poggi. Do gans leave artificial fingerprint?? arXiv preprint
1842, 2018.
Soroush Mehri, Kundan Kumar, Ishaan Gulrajani, Rithesh Kumar, Shubbam Jain, Jose Sotelo, Aaron Courville, and Yoshua Bengio. Samplemn: An unconditional end-to-end neural audio generation model. arXiv preprint arXiv:1612.0783 2016.
Lakshmanan Nataraj. Tajuddin Manhar Mohammed, BS Manjunath, Shivkumar Chandrasekaran, Arjuna Flenner, Jawadul H Bappy, and Amit K Roy-Chowdhury. Detecting gan generated fake images using co-occurrence matrices. arXiv preprint arXiv:1903.06830 2019.
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv: (600::399999999 2016.
Ryan Prenger, Rafael Valle, and Bryan Catanzaro. Waveglow: flow-based generative network for speech synthesis. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3617-3621. IEFE, 2019.
Robert C Streijl, Stefan Winkler, and David S Hands. Mean opinion score (mos) revisited: methods and applications, limitations and alternatives. Multimedia Systems, 22(2):213-22 227, 2016.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint wrXiv:1607.00022 2016.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, Eero P Simoncelli, et al. Image quality assessment: from error visibility structural similarity. IEEE transactions on image processing, 1304)600--12, 2004.
Ning Yu, Larry Davis, and Mario Fritz. Attributing fake images to gans: Analyzing fingerprints in generated images. arXiv preprint 1.Xi::811.008888 2018.
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 2223-2232, 2017.
10

---

Under review as a conference paper at ICLR 2020
Attack SUcceSS
watermank Rotation
Attack success
Attack success
Attack success
0.2
2.0
Brightress
Crop
0.4
3.4
0.
135
1.
9.
05
02
13
5.0
02
1.0
1.0
1.2
03
1.
SSIM
SSIM
SSIM
SSIM
(a) JCHAR: = 1/255
(b) fCHAR: = 14/255
(c) fCIFAR. c 23/255
(d) SCIAAR 33/255
Attack success
Attack success
Attack success
Attack success
0.6
3.6
0.
22
0.
SSIM
SSIM
SSIM
SSIM
(e) fÊIARR 1/200
(1) fCFAR 12/255
(g) SÂPAR: 23/255
(h) fCIAAR 33/255
Figure 6: Transformation attack results on Cifarl0. Each subligure plotted as function of the distortion introduced by the transformation, and evaluate for watermark perturbation ranging from c = 1/255 to c = 33/255. The solid markers are the attack success against watermarked images, and the faded markers, the attack success on non-watermarked images.
A EVALUATION ON CIFAR10 & FURTHER EVALUATION ON IMAGENET AND TEXT-TO-SPEECH DATA
Cifar10 (Krizhevsky al.). We use a wide ResNet classifier (He al. (2016) for the watermark detector. We replace batch normalization with instance normal (Ulyanov et al. (016)). We trained with a mini-batch size of for 60,000 steps, with an initial learning rate of 0.01 and decaying this by factor of 10 every 20,000 steps. All images are normalized into the range [0,1]. We set the maximum size the watermark perturbation, e, to 20/255, and the watermark perturbation is constructed with five PGD steps at each iteration during training. With respect to the set of transformations, we set a to 0.5, r to 7/2, Cj and cw to 2, and b to 0.25. We trained two classifiers under these hyperparameters, one where at each step randomly sample single transformation. and another classifier where we apply a composition of all transformations, which shall be denoted by SeFAR and Su respectively. Both classifiers achieved 100% test accuracy, where the test set consists of 10,000 non-watermarked images and 10,000 watermarked images.
Given transformation function to X' - X. parameterized by which encodes some randomness, and an input E X' with class E {0, 1}. we create n copies of this input under fg. s', i E {I...., n1.. We then measure both the SSIM of the watermarked images and SSIM of both transformed watermarked and transformed non- watermarked images as a function of the attack success over these inputs. Figure 6 shows the results of transformation attack: for each of 2,000 Cifar10 test set images (1,000 non-watee watermarked and 1, 1,000 watermarked) we create 1000 new images under transformation; each marker in the figure the average attack success over million test images. We evaluate the attack success for number of different watermarking values. Large increase the perceptibility of the watermark, and so correspondingly decrease the SSIM score. Note, attacking non-watermarked content is unaffected by the value chosen for watermarking.
For = 1/25s, the SSIM score watermarked content is z5 0.99, and thus the watermark is highly imperceptible. However, number of attacks on watermark content succeed with high SSIM score, such as for example transformations that modify the brightness of an image. Slightly increasing the perceptibly of the watermark to 12/255, reduces the attack success of nearly all transformations to
11

---

Under review as a conference paper at ICLR 2020
Attack SUccess
Attack success
Attack success
Attack success
Rotation Brightness Crop
0.4
2.4
00
9.
72 02
12
20
0.
02
0D
03
SSIM
SSIM
SSIM
SSIM
(a) plics, c 1/256
f e 12/255
(c) fe10,
1/255
(d) fr1,
12/255
1.0
Attack success
Attack success
Attack success
Attack success
3.3
9.6
3.6
8.
0.2
22
SSIM
SSIM
SSIM
SSIM
(c) Jonips 1/255
(f) fccspp
12/255
(g)
2510 camps e
1/235
(h) J f500
12/255
Figure 7: Transformation attack results on ImageNet. Each subfigure plotted as function of the distortion introduced by the transformation, and we evaluate for watermark perturbation ranging from 1/255 12/255. The solid markers are the attack success against watermarked images, and the faded markers, the attack success on non-warermarked images.
watermark
Gaussian noise
Attack success
0.8
Attack success
0.8
Time roll
Time shift
Silent chunk
Pitch
Uniform noise
0.4
0.4
0.2
0.0
10
40
70
80
100
30
60
100
PSNR
PSNR
(a) Watermarked with = 4 x 10-4
(b) Watermarked with e 4.8 x 10
Figure 8: Transformation attack results on audio data. Each marker represents the average attack success versus average PSNR of 10,000 inputs sampled under transformation. The solid markers are the attack success against watermarked images, and the faded markers, the attack success on non-watermarked images.
Overall, the classifier trained composition of transformations is robust attacks non-watermarked content.
Similar effects can be observed ImageNet shown in fig. 7 and the propriety audio dataset in fig, g Interestingly, watermark classifiers on ImageNet seem to be more robust at smaller values than on 10, conjecture this is because there is larger space in which to distribute the watermark.
12

---

Under review as a conference paper at ICLR 2020
B
FURTHER COMPARISON WITH BROKEN ARROWS
50
50
ReSWAT
PSNR Watermark
PSNR Watermark
PSNR Watermark
20
20
15
10
20
30
50
30
20
40
13
20 30
PSNR Transformed
PSNR Transformed
PSNR Transformed
(a) Gaussian noise.
(b) Rotation.
(c) Crop.
Figure 9: Distortion needed for perfect provenance detection (by f against different signal quality degradation levels suffered by the attacker. We measure the amount of distortion the watermarking scheme must introduce in order to guarantee perfect detection accuracy under transformation, and compare with the Broken Arrows zero-bit watermarking scheme.
1.0
0.8
Accuracy
0.6
0.4
0.2
25
71
0.0
0.1
0.3
0.5
Figure 10: Comparison of ReSWAT trained using a single sample from the transformation distribution each training step, and with 25 samples each training step, for Cifar10, against Gaussian noise attack.
Figure gives analogous plots fig. [ when the measure of distortion introduced by both the water- mark and transformation is Feak-Signal-tor -o-oos- ratio (PSNR). As one may expect, results exhibited here mirror those of fig. [. ReSWAT and BA are comparable under Gaussian noise transformations, while ReSWAT is substantially better than BA under rotation and cropping transformations.
C MEASURING HOW THE NUMBER OF SAMPLES FROM THE TRANSFORMATION DISTRIBUTION USED DURING TRAINING AFFECTS RESILIENCE TO ATTACKS
Here, we evaluate the increase in resilience transformations when we optimize ReSWAT using eq. (I) with 25 samples from the transfonmation distribution each step of training, as opposed to a single sample. Figure I0 shows the resilience improvements ReSWAY against a Gaussian noise transformation the Cifar10 dataset. We evaluate test accuracy 50,000 watermarked and
13

---

Under review as a conference paper at ICLR 2020
non-watermarked images with various levels of Gaussian noise applied. However, the improvement in resilience was at the expense of approximately a 5x increase in training time.
14


Reviews:
[{"review_id": "ryeRqiZaKH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a watermark design algorithm that is based on an adversarial training paradigm where a watermark detector and a watermark generator are jointly trained to maximize the likelihood of watermark detection while bounding the manipulation of individual pixels. \n\nI have several concerns about the clarity of the paper, its applicability to the motivating problem, and the completeness of the results that motivate my \"reject\" recommendation.\n\nThe description does not make it clear how the detector and watermark mechanism are jointly obtained in (1) - intuitively, a detector should know the type of watermark to be observed in order to pose a feasible detection model. The discussion focuses on \"fixed detector\" and \"fixed watermark\" cases only, but it is not clear how an iterative optimization would be initialized, and no discussion of how to set up the problem appears in the manuscript. \n\nThe motivation for the paper is also unclear. The authors discuss the need to be able to detect deep fakes, which is a relevant and meritorious problem. However, it seems that the application of the proposed approach in this setting would require deep fake creators to add watermarks to their creations, which would be a naive assumption. Watermarks are used to demonstrate provenance, but deep fake creators would not want to have such provenance verified.\n\nThe watermark transferability premise seems to rely on the random initialization of the underlying deep learning watermark detector. One could posit that if a third party is interested in creating false positives, they could train the deep learning network using a database of watermarked and non-watermarked images. Another concern here is that the formulation provided in (3) would potentially lead to low transferability for the specific classifiers provided there, but one does not know which classifier an adversary would construct - or why an adversary would construct an adversary that does not have knowledge of the specific watermark being used, or of samples of the watermark. It would have been interesting to see if it is possible to construct a good watermark detector this way (given enough training data being available). It would also be good to have some intuition as to why this robustness formulation also improves \"certified robustness\".\n\nFinally, the performance comparison only considers a 10-year old watermarking algorithm; some discussion as to why this is sufficient should have been included.\n\nMinor comments\nThe norm infinity subindex is missing from the minimization conditions in (1).\nFigure 3 caption: watermark's -> watermarks\nIt is not clear why no numerical results were given for Section 4.2.4.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "rklOwGk0YH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors introduce ReSWAT, a method for transformation-resilient watermarking of images via adversarial training. The high level idea is to learn a watermark/detector pair (W,D). W can be any transformation (in this paper, an l-infty bounded perturbation) that imputes an imperceptible distortion to a given input, while D is a detector that distinguishes watermarked from non-watermarked images. There is an additional requirement that the detector should be robust to simple transformations such as rotations, cropping, flipping, and contrast enhancement. \n\nThe authors pose a min-max style learning problem for learning (W,D) that leads to a natural adversarial training scheme. In particular, it can be viewed as creating an adversarial defense to the Expectations-over-transformations attack of Athalye et al. Experimental results on a bunch of different datasets confirm that the method works,\n\nThe paper is well-written and the contributions are clear. In terms of conceptual or theoretical novelty, the paper is limited (the method essentially boils down to regular adversarial training with a slightly non-standard loss function) but the connection to watermarking seems novel and is nicely executed.\n\nMy only concern is the lack of comparisons with baselines. I am not a digital forensics expert -- so I don't know what the state of the art is -- but the only comparisons made are with the Broken Arrows (BA) watermarking , which seems to be over 10 years old, so I am not sure how to evaluate the results.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "SyeSWv57cB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is about a novel method to add watermarks to images and audio that is highly robust to several transformations that is closely related to gan methods. The idea is that the watermark signal is learned concurrently to the detector network, which share similarities to a generator and detector networks. Five standard attack transformations are considered and a specific optimization to reduce the transferability of the watermark is considered. The method is compared against Broken arrows on Cifar10 and Imagenet. It shows similar or better performance for Gaussian noise attack given the same amount of perturbance allowed in a signal while very much better performance for the other attacks. Going beyond the five attacks, the paper also includes an estimation of the probability of confidence of finding watermarked images given a fixed l2 norm radius. Finally the method is also tested on audio on a proprietary dataset with a deepspeaker architecture which still shows very good performance and it is confirmed by a human evaluation where participants found the watermarked audio not significantly worse or degraded.\n\nThe paper is well written and of excellent presentation. The proposed method is novel and goes in the interesting direction of learning watermarks using adversarial training techniques. Experiments shows that the method has good performance but it is only compared to Broken Arrows (i.e. a zero bit watermarking) which is the state of the art of this type of watermarking. It would have been interesting a comparison to other key based watermarking methods to also deeply evaluate the signal transformation attacks. All in all, I found the paper to be significant and I would like to see it accepted.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper offers an interesting and potentially useful approach to robust watermarking.  The reviewers are divided on the significance of the method.  The most senior and experienced reviewer was the most negative.  On balance, my assessment of this paper is borderline; given the number of more highly ranked papers in my pile, that means I have to assign "reject".
Decision: Reject