Adversarial Attacks on Copyright Detection Systems
Authors: Parsa Saadatpanah
Ali Shafahi
Tom Goldstein
Venue:  (2020)

Abstract: It is well-known that many machine learning models are susceptible to adversarial attacks, in which an attacker evades a classifier by making small perturbations to inputs. This paper discusses how industrial copyright detection tools, which serve a central role on the web, are susceptible to adversarial attacks. We discuss a range of copyright detection systems, and why they are particularly vulnerable to attacks.  These vulnerabilities are especially apparent for neural network based systems.  As proof of concept, we describe a well-known music identification method and implement this system in the form of a neural net. We then attack this system using simple gradient methods. Adversarial music created this way successfully fools industrial systems, including the AudioTag copyright detector and YouTube's Content ID system. Our goal is to raise awareness of the threats posed by adversarial examples in this space and to highlight the importance of hardening copyright detection systems to attacks.

Under review as a conference paper at ICLR 2020
ADVERSARIAL ATTACKS ON COPYRIGHT DETECTION SYSTEMS
Anonymous authors Paper under double-blind review
ABSTRACT
It is well-known that many machine learning models are susceptible to adversarial attacks, in which an attacker evades a classifier by making small perturbations inputs. This paper discusses how industrial copyright detection tools, which serve a central role on the web, are susceptible to adversarial attacks. We discuss range of copyright detection systems, and why they are particularly vulnerable to attacks. These vulnerabilities are especially apparent for neural network based systems. As proof of concept, we describe well-known music identification method and implement this system in the form of neural net. We then attack this system using simple gradient methods. Adversarial music created this way successfully fools industrial systems, including the Audio copyright detector and YouTube's Content ID system. Our goal is to raise awareness of the threats posed by adversarial examples in this space and to highlight the importance of hardening copyright detection systems to attacks.
INTRODUCTION
Machine learning systems are easily manipulated by adversarial attacks, in which small perturba- tions to input data cause large changes to the output of a model. Such attacks have been demonstrated on a number of potentially sensitive systems, largely in an idealized academic context, and occasion- ally in the real-world (Tencent. 2019 Kurakin et al. 2016 Athalye et al. 2017: Eykholt et al. 2017
Yakura Sakuma 2018 Qin al.] 2019).
Copyright detection systems are among the most widely used machine learning systems in industry, and the security of these systems is of foundational importance to some of the largest companies in the world. Despite their importance, copyright systems have gone largely unstudied by the ML se- curity community. Common approaches to copyright detection extract features, called fingerprints, from sampled video or audio, and then match these features with a library of known fingerprints. Examples include YouTube'ss ss Content ID, which flags copyrighted material on YouTube and enables copyright owners to monetize and control their content. At the time of writing this paper, more than 100 million dollars have been spent on Content ID. which has resulted in more than 3 billion dollars in revenue for copyright holders (Manara 2018). Closely related tools such as Google Jigsaw detect and remove videos that promote terrorism or jeopardized national security. There is also regula- tory push for the use of copyright detection systems; the recent EU Copyright Directive requires any service that allows users to post text, sound, or video to implement a copyright filter.
wide range of copyright detection systems exist, most of which are proprietary. It is not possible to demonstrate attacks against all systems, and this is not our goal. Rather, the purpose of this paper to discuss why copyright detectors are especially vulnerable to adversarial attacks and establish how existing attacks in the literature can potentially exploit audio and video copyright systems.
As proof of concept, we demonstrate an attack against real-world copyright detection systems for music. To do this, we reinterpret simple version of the well-known "Shazam" algorithm for music fingerprinting as a neural network and build a differentiable implementation of in TensorFlow (Abadi al. 2016). By using gradient-based attack and an objective that is designed achieve good transferability to black-box models, we create adversarial music that is easily recognizable to human, while evading detection by machine. With sufficient perturbations, our adversarial music

---

Under review as a conference paper at ICLR 2020
successfully fools industrial systems including the Audio Tag music recognition service 2009). and YouTube's Content ID system Google 2019).
(Audio Tag
WHAT MAKES COPYRIGHT DETECTION SYSTEMS VULNERABLE TO ATTACKS?
Work on adversarial examples has been focused largely on imaging problems, including image clas- sification, object detection, and semantic segmentation (Szegedy et al. 2013: Goodfellow et al. Moosavi-Deztooli et al., 2016 2017, Shafahi et al.) 2018 Xie et al.] 2017] Fischer et al, More recently, adversarial examples have been studied for non- applications such as
2014
2017
speech recognition (i.e., speech-to-texp (Carlini & Wagner] 2018 Alzantot et al. 2018: Taori et al.)
2018 Yakura & Sakuma (018). Attacks on copyright detection systems are different from these
applications in a number of important ways that result in increased potential for vulnerability.
First, digital media can be directly uploaded to server without passing through microphone or camera. This is drastically different from physical-world attacks, where adversarial perturbations must survive data measurement process. For example, perturbation to a stop sign must be ef- fective when viewed through different cameras, resolutions, lighting conditions, viewing angles, motion blurs, and with different post-processing and compression algorithms. While attacks exist that are robust to these nuisance variables (Athalye et al. 2017), this difficulty makes even white- box attacks difficult, leaving some to believe that physical world attacks are not a realistic threat model (Lu et al. 2017a b). In contrast, a manipulated audio file can be uploaded directly to the web without passing through a microphone that may render perturbations ineffective.
Second, copyright detection is an open-sel problem, in which systems process media that does not fall into any known class (i.e., doesn't 't correspond to any protected audio/video). This is different from the closed-set detection problem in which everything is assumed to correspond to class. For example, a mobile phone application for music identification may solve a closed-set problem; the developers can assume that every uploaded audio clip corresponds to known song, and when results are uncertain there is no harm in guessing. By contrast, when the same algorithm is used for copyright detection on server, developers must solve the open-set problem; nearly all uploaded content is not copyright protected, and should be labeled as such. In this case, there is harm in "guessing" an ID when results uncertain, as this may bar users from uploading non-protected material. Copyright detection algorithms must be tuned conservatively to operate in an environment where most content does not get flagged.
Finally, copyright detection systems must handle a deluge of content with different labels despite strong feature similarities. Adversarial attacks are known succeed easily in environment where two legitimately different audio/video clips may share strong similarities at the feature level. This has been recognized for the ImageNet classification task (Russakovsky al. 2015)) where feature overlap between classes (e.g., numerous classes exist for different types of cats/dogs/birds) makes systems highly vulnerable to untargeted attacks in which the attacker perturbs an object from its home class into different class of high similarity. As a result, state of the art defenses for untargeted attacks on ImageNet achieve far lower robustness than classifiers for simpler tasks (Shafahi et al., 2019 Cohen et al.] (019). Copyright detection systems may suffer from a similar problem; they must discern between protected and non protected content even when there is a strong feature overlap between the two.
3 TYPES OF COPYRIGHT DETECTION SYSTEMS
Fingerprinting algorithms typically work by extracting an ensemble of feature vectors (also called a "hash" in the case of audio tagging) from source content, and then matching these vectors to a li- brary of known vectors associated with copyrighted material. If there are enough matches between source sample and a library sample, then the two samples are considered identical. Most audio, im- age, and video fingerprinting algorithms either train neural network extract fingerprint features, or extract hand-crafted features. In the former case, standard adversarial methods lead to immediate
A Affected parties were notified before publication of this article.
2

---

Under review as a conference paper at ICLR 2020
susceptibility. In the latter case, feature extractors can often be re-interpreted and implemented as shallow neural networks, and then attacked (we will see an example of this below).
For video fingerprinting, successful approach by Saviaga & Toxtli (2018) is to use object detec- tors to identify objects entering/lcaving video frames. An extracted hash then consists features de- seribing the entering/leaving objects, in addition the temporal relationships between them. While effective at labeling clean video, recent work has shown that object detectors and segmentation en- gines are easily manipulated adversarially place/remove objects from frames (Wang al. 2019 Xie et al. 2017).
Works such Li et al. (019) build "robust" fingerprints by training networks commonly used distortions (such as adding border, adding noise, or flipping the video), but do not consider adver- sarial perturbations. While such networks are robust against pre-defined distortions, they will not be robust against white-box (or even black-box) adversarial attacks.
Similarly, recent plagiarism detection systems such as Yasaswi et al. 2017) rely on neural networks to generate a fingerprint for a document. While using the deep feature representations of a document as a fingerprint might result higher accuracy for the plagiarism model, it potentially leaves the system open adversarial attacks.
Audio fingerprinting might appear to be more secure than the domains described above because practitioners typically rely on hand-crafted features rather than deep neural nets. However, we will see below that even hand-crafted feature extractors are susceptible to attacks.
CASE STUDY: EVADING AUDIO FINGERPRINTING
We now describe a commonly used audio fingeprrnttingddetcction algorithm and show how one can build a differentiable neural network resembling this algorithm. This model can then be used to mount black-box attacks real-world systems.
4.1 AUDIO FINGERPRINTING MODELS
An acoustic fingerprint is a feature vector that is useful for quickly locating a sample or finding sim- ilar samples in audio database. Audio fingerprinting plays central role in detection algorithms such as Content ID. Therefore, in this section, we describe a generic audio fingerprinting model that will ultimately help us generate adversarial examples.
4.1.1 IMPORTANT FINGERPRINTING GUIDELINES FROM SHAZAM
Due to the financially sensitive nature of copyright detection, there are very few publicly available fingerprinting models. One of the few widely used publicly known models from the Shazam team (Wang et al., (003). Shazam is popular mobile phone app for identifying music. According to the Shazam paper, a good audio fingerprint should have the following properties:
Temporally localized: every fingerprint hash is calculated using audio samples that span a short time interval. This enables hashes to be matched to short sub-sample of a song.
Translation invariant: fingerprint hashes are (nearly) the same regardless of where in the song sample starts and ends.
Robust: hashes generated from the original clean database track should be reproducible from a degraded copy of the audio.
4.1.2 THE HANDCRAFTED FINGERPRINTING MODEL
The spectrogram of signal, also called the short-time Fourier transform, is plot that shows the frequency content (Fourier transform) of the waveform over time. After experimenting with various features for fingerprinting, Wang al. (2003) chose to form hashes from the locations of spectro- gram peaks. Spectrogram peaks have nice properties such as robustness in the presence of noise and approximate linear superposability.
In the next subsection, we build a shallow neural network that captures the key ideas of Wang et al. 2003), while adding extra layers that help produce transferable adversarial examples. In particular,
3

---

Under review as a conference paper at ICLR 2020
mxm Max Pool
Fingerprint
y 1xN, Conv,
1xN, Conv,
Figure I: An audio fingerprinting model with two convolution layers and a max pooling layer. This model produces binary fingerprints by finding local maxima of the spectrogram of the input signal.
add an extra smoothing layer that makes our model difficult attack and helps us craft strong attacks that can transfer to other black-box models.
4.2 INTERPRETING THE FINGERPRINT EXTRACTOR AS A CNN
Here we describe the details of the generic neural network model we use for generating the audio fingerprints. Each layer of the network can be seen transformation that is applied to its input. We treat the output representation of our network as the fingerprint of the input audio signal. Ideally, we would like to extract features that uniquely identify a signal while being independent of the exact start or end time of the sample. Convolutional neural networks maintain the temporally localized and translation invariant properties mentioned in section 4.1.1 and so we model the fingerprinting procedure using fully convolutional neural networks.
The first network layer convolves with normalized Hann function, which is filter of the form
sin
fi(n)
(1)
E sin? (N)
where N is the width of the kernel. Convolving with a normalized Hann window smooths the udversarially perturbed audio waveform and the output of this layer is perturbed but smooth audio sample that is then fingerprinted. This layer removes discontinuities and bad spectral properties that may be introduced into the signal during adversarial optimization and also makes the block-box attacks more efficient by preventing perturbations that do not transfer well to other models.
The next convolutional layer computes the spectrogram (aka Short Term Fourier Transform) of the waveform and converts the audio signal from its original domain representation in the frequency domain. This is accomplished by convolving with an ensemble of N Fourier kernels of different frequencies, each with N' output channels. This convolution has filters of the form
f2(k, n) 122kk//N
where l E 0,1,...,, N filter coefficient, After this convolution is computed, we apply |ar] on the output to get the magnitude of the STFT.
is an output channel index and n E 1, ,N-
is the index of the
After the convolutional layers, we get feature representation of the audio signal. We call this feature representation ((x), where x is the input signal. This representation is susceptible to noise and slight perturbation in the audio signal can change it. Furthermore, this representation is very dense which makes it relatively hard to store and search against all audio signals in the database. To address these issues, Wang et al. (2003) suggest using the local maxima of the spectrogram as features.
We can find local maxima within neural net framework by applying max pooling function
the feature representation ((x). We then find the places where the output of the maxpool equals the original feature representation (i.e., the locations where ((x) marpool (o(x))). The resulting binary map of local maxima locations is the fingerprint of the signal and can be used to search for signal against database of previously processed signals. We will refer this binary fingerprint as ab (x) where x is the input signal. Figure [ depicts the 2-layer convolutional network we use in this work for generating signal fingerprints.
4

---

Under review as a conference paper at ICLR 2020
4.3 FORMULATING THE ADVERSARIAL LOSS FUNCTION
To craft an adversarial perturbation, we need a differentiable surrogate loss that measures how well an extracted fingerprint matches a reference. The CNN described section 4.2 uses spectrogram peaks to generate fingerprints, but we did not yet specify a loss for quantifying how close two finger- prints are. Once we have such a loss, we can use standard gradient methods to find a perturbation that can be added to an audio signal to prevent copyright detection. To ensure the similarity between perturbed and clean audio, we bound the perturbation 8. That is, we enforce ||8|, S c. Here ll- ll-IIp the f,-norm of the perturbation and is the perturbation budget available the adversary. In our experiments, we use the fx -nom as our measure of perturbation size.
The simplest similarity measure between two binary fingerprints simply the Hamming distance. Since the fingerprinting model outputs a binary fingerprint (ar), we can simply measure the number of local maxima that the signals z and y share by (x)) To make a differentiable loss function from this similarity measure, we use
(x, = (0(z) (xx) (b)).
In the white box case where the fingerprinting system is known, attacks using the loss (3) are ex- tremely effective. However, attacks using this loss are extremely brittle and do not transfer well; one minimize this loss by changing the locations of local maxima in the spectrogram by just one pixel. Such small changes in the spectrogram are unlikely to transfer reliably to black-box industrial systems.
To improve the transferability our adversarial examples, we propose a robust loss that promotes large movements in the local maxima of the spectrogram. We do this by moving the locations of local maxima in ((x) outside of any neighborhood the local maxima of o(y). To efficiently implement this constraint within neural net framework, use two separate max pooling layers, one with a bigger width wy (the same width used in fingerprint generation). and the other with a smaller width u2. If location in the spectrogram yields output of the wi pooling strictly larger than the output of the W2 pooling we can be sure that there is no spectrogram peak within radius U2 of that location.
Equation 4 describes a loss function that penalizes the local maxima of æ that are in the neigh- borhood of local maxima of y. This loss function forces the output of the max pooling layers be different by at least margin
(xx = Σ ReLU
(c-(AER,, max +j (11SS
max <(i + )i:z)) )) l
(((w))
(4)
Finally, make our loss function differentiable by replacing the maximum operator with the smoothed max function
So (I1, In) i== T.
Σ
(5)
eaz.
where a is a smoothing hyper parameter. As - oc, the smoothed max function more accurately approximates the exact max function. For simplicity, we chose a = 1 for all experiments.
4.4 CRAFTING THE EVASION ATTACK
We solve the bounded optimization problem
min [x + 6, 8.1. 1:0 Se
(6)
where is the benign audio sample, and J is the loss function defined in equation 4 with the smoothed max function. Note that unlike common adversarial example generation problems from the literature, our formulation is minimization problem because of how we defined the objective. We solve 6 using projected gradient descent (Goldstein et al.] 2014) in which each iteration updates the perturbation using Adam (Kingma & Ba 2014), and then clips the perturbation to ensure that the foo constraint is satisfied.
"The first maxpool layer's output always greater than equal the output of the second maxpool layer.
5

---

Under review as a conference paper at ICLR 2020
4.5 REMIX ADVERSARIAL EXAMPLES
The optimization problem defined in equation tries to create an adversarial example with a finger- print that does look like the original signal's fingerprint. While this approach can trick the search algorithm used in copyright detection systems by lowering its confidence, it can result in unnatural sounding perturbations. Alternatively, we can try to enforee the perturbed signal's 's fingerprint to be similar to different audio signal. Due to the approximate linear superposability characteristic of the spectrogram peaks, this will make the adversarial example sound more natural and like the target signal audio.
To achieve this goal, we will first introduce loss function that tries make two signals look similar rather than different. As described in equation 7 such loss can be obtained by replacing the order of max over big and small neighborhoods equation 4 Note that we will still use the smooth maximum from equation
Jremin (a, y) Σ [ReLU (c- max o(i j;z) [5lS
max o(i b(i+;ta)) ((:w))
(7)
bil
Using this loss function, we define the optimization problem equation 8 which not only tries to make the adversarial example different from the original signal x, but also forces similarity to another signal y.
min +6 + AJmmii (x + 6, s.t. 10 S.
(8)
Here À is scale parameter that controls how much enforce the similarity between the finger- prints of x + and y. We call adversarial examples generated using equation "remix" adversarial examples as they sound more like remix, and refer to examples generated using equation 6 as default adversarial examples. While a successful attack's 's adversarial perturbation may be larger in the case of remix adversarial examples (due to the additional term in the objective function), the perturbation sounds more natural.
5 EVALUATING TRANSFER ATTACKS ON INDUSTRIAL SYSTEMS
We test the effectiveness our black-box -bx attacks on two real-world audio search/copyright detection systems. The inner workings of both systems are proprictary, and therefore it is necessary attack these systems with black-box transfer attacks. Both systems claim to be robust against noise and other input signal distortions.
We test our system on a dataset containing the top billboard songs from the past 10 years. We ex- tract 30-second fragment of these songs and craft both our default and remix adversarial examples for them. Although both types of adversarial examples can dodge detection, they have very dif- ferent characteristics. The default adversarial examples (equation (6 work by removing identifiable frequencies from the original signal, while the remix adversarial examples (equation g) work by introducing new frequencies to the signal that will confuse the real-world systems.
5.1 WHITE-BOX ATTACK RESULTS
Before evaluating black-box transfer attacks against real-world systems, we evaluate the effective- ness of white-box attack against our own proposed model. Doing so will allow us to have baseline of how effective adversarial example can be if the details of model are ever released leaked.
To create white-box attacks against our model, we use the loss function defined equation 3 By optimizing this function, perturbations that are unnoticeable by humans. Table [ shows the norms of the perturbations re- quired to remove 90%, 95%, and 99% of the fingerprint hashes.
remove almost all of the fingerprints identified by model with
5.2 TRANSFER ATTACKS ON AUDIOTAG
AudioTag is a free music recognition service with millions of songs in its database. When a user uploads a short audio fragment on this website, AudioTag compares the audio fragment against
hhtps://audiotag.nnoo/

---

Under review as a conference paper at ICLR 2020
Percentage of removed hashes
90%
95%
99%
Perturbation norm (f
0.012
0.023
0.038
Perturbation norm (62)
0.004
0.005
0.006
Table 1: Norms of the perturbations for white-box attacks. Before computing the norms, we have normalized the signals have samples that lie in [0. 1].
database of songs and identifies what song this audio fragment belongs to. AudioTag claims to be "robust to sound distortions, noises and even speed variation, and will therefore recognize songs even low quality audio recordings" Therefore, one would expect that low-amplitude non-adversarial noise should not affect this system.
As shown in Figure 2 AudioTag can accurately detect the songs corresponding to the benign signal. However, the system fails to detect both the default and remix adversarial examples built for them. During our experiments with AudioTag, we realized that this system is relatively sensitive to our proposed attacks and it can be fooled with relatively small perturbation budgets. Qualitatively, the magnitude the noise required to fool this system is small and it is not easily noticeable by humans. Based this observation, suspect that the architecture of the fingerprinting model used in Tag may have similarities to our surrogate model in section
(13%) Time interval: 0 10 sec
Artist: Radio 538 (CD Series)
Recognition start time-point: 0 sec Recognized interval duration: 32 sec
Track: Ke$ha Tik
Album: 538 Hitzone
No results found
Year: 2010
No cover available
Find this track on YouTube
Start over
Find same style tracks
Other albums with this track
Figure 2: AudioTag can identify the benign audio signals, but fails to detect adversarial examples.
Table 2 shows the for and le norms of the perturbations required to fool AudioTag 90% of the songs in our dataset. We also verified Audio claim of being robust to input distortions by applying random perturbations to the audio recordings. To fool AudioTag with random noise, the magnitude (foc) of the noise must be roughly times larger than the noise we craft using equation
Target model Type of perturbation
AudioTag
YouTube
default
remix
random noise
default
remix
random noise
Perturbation norm (foo)
0.03
0.03
0.12
0.10
0.10
0.32
Perturbation norm (62)
0.02
0.02
0.06
0.07
0.08
0.19
Table 2: Norms of the perturbations in adversarial examples that can evade each real-world system. Before computing the norms, we have normalized the signals to [0. 1].
5.3 YOUTUBE
YouTube video sharing website that allows users to upload their own video files. YouTube has developed system called "Content iD automatically tag user-uploaded content that con- tains copyrighted material. Using this system, copyright owners can submit their content and have YouTube scan uploaded videos against it.
As shown the screenshot Figure YouTube Content ID can successfully identify the benign songs use in our experiment. At the time of writing this paper both our default and remix attacks
htlps://audiotagiinfo/iaa

---

Under review as a conference paper at ICLR 2020
Video
Visibiiity
Draft
Add description
Copyright
0:33
adversarial
Add description
content your below, along with actions.
Content used
Claim type
Impact on the video
Actions
TIK TaK Kečha
J'
Video connot monetized
SELECT ACTION
Ad
Figure 4: YouTube can successfully identify the benign/original audio signal while it fails detect the adversarial examples.
successfully evade Content ID and go undetected. However, YouTube Content ID is significantly more robust to our attacks than AudioTag. To fool Content ID, we had to use larger value for 6. This makes perturbations quite noticeable, although songs still immediately recognizable by human. Furthermore, perturbation with non-adversarial random noise must have an foo norm times larger than our adversarial perturbations to successfully avoid being detected.
We repeated our experiments with identi- cal hyper-parameters on the songs from our dataset. Table 2 shows the foo norms (i.e., the parameter and l norms of the perturbations required to fool YouTube on 67% of the songs in our dataset. Furthermore, figure shows the recall of YouTube's copyright detection tool our dataset for different magnitudes of pertur- bations.
0.75
Recall
0.5
6 CONCLUSION
0.25
Copyright detection systems are important category of machine learning methods, but the robustness of these systems to adversarial at- tacks has not been addressed yet by the machine learning community. We discussed the vulner- ability of copyright detection systems, and ex- plain how different kinds of systems may vuinerable to attacks using known methods. As a proof of concept, we build simple song iden- tification method using neural network primitives and attack it using well-known gradient methods. Surprisingly, attacks on this model transfer well to online systems.
0
5
10-2
0.1
0.15
0.2
Ex (c)
Figure 3: YouTube's copyright detection recall against the magnitude of noise on top Billboard songs dataset
Note that none of the authors of this paper are experts in audio processing or fingerprinting systems. The implementations used this study are far from optimal, and we expect that attacks can be strengthened using sharper technical tools, including perturbation types that are less perceptible to the human car. Furthermore, we are doing transfer attacks using fairly rudimentary surrogate models that rely hand-crafted features, while commercial system likely rely full trainable neural nets.
Our goal here not to facilitate copyright evasion, but rather to raise awareness of the threats posed by adversarial examples in this space, and to highlight the importance of hardening copyright detection and content control systems to attack. A number defenses already exist that can be utilized for this purpose, including adversarial training.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: system for large- scale machine learning. In 12th {USENIX) Symposium on Operating Systems Design and Imple- mentation ((OSDI) 16), pp. 265-283, 2016.
Moustafa Alzantot, Bharathan Balaji, and Mani Srivastava. Did you hear that? adversarial examples against automatic speech recognition. arXiv preprint arXiv: 1801.00554, 2018.
Anish Athalye, Logan Engstrom, Andrew Ilyas, and Kevin Kwok. Synthesizing robust adversarial examples. arXiv preprint arXiv:/707.07399 2017.
AudioTag, Audiotag - freee music recognition robot, 2009. URL
info/
Nicholas Carlini and David Wagner. Audio adversarial examples: Targeted attacks on speech-to- text. In 2018 IEEE Security and Privacy Workshops (SPW), pp. 1-7. IEEE, 2018.
Jeremy M Cohen, Elan Rosenfeld, and J Zico Kolter. Certified adversarial robustness via randomized smoothing. arXiv preprint 1902.029 2019.
Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song. Robust physical-world attacks on deep learning models. arXiv preprini PrXiiv/707.08900 2017.
Volker Fischer, Mummadi Chaithanya Kumar, Jan Hendrik Metzen, and Thomas Brox. Adversarial examples for semantic image segmentation. arXiv preprint arXiv: 101, 2017.
Tom Goldstein, Christoph Studer, and Richard Baraniuk. field guide with a fasta implementation. arXiv preprint arXiv: 2014.
splitting
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6577 2014.
Google. How content id works youtube help, 2019. UtL/ttpps//auppott.ooggg... /yottubbe/amssee//77777777171
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. arXiv preprint 11X7:1667.022200 2016.
Y. Li, D. Wang. and L. Tang. Robust and secure image fingerprinting learned by neural network. IEEE Transactions on Circuits and Systems for Video Technology, pp. 1-1, 2019. ISSN 1051- 8215. doi: 10. N0.109//CSTT2229999999000
Jiajun Lu, Hussein Sibai, Evan Fabry, and David Forsyth. No need worry about adversarial examples in object detection in autonomous vehicles. arXiv preprint arXiv: 2017a.
Jiajun Lu, Hussein Sibai, Evan Fabry, and David Forsyth. Standard detectors aren't (currently) fooled by physical adversarial stop signs. arXiv preprint arXiv:1710.033:: 2017b.
Cedric Manara. Protecting what we love about the internet: our efforts to stop online piracy, 2018. prreeeacccciiaa.eeeeeeeeeeetttte
[Accessed: 05/21/2019).
Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. Deepfool: simple and accurate method to fool deep neural networks. In Proceedings of the IEEE conference on com- puter vision and pattern recognition, pp. 2574-2582, 2016.
Seyed-Mohsen Moosavi-Des -ezaool,,, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Universal adversarial perturbations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1765-1773, 2017.

---

Under review as a conference paper at ICLR 2020
Yao Qin, Nicholas Carlini, Ian Goodfellow, Garrison Cottrell, and Colin Raffel. Imperceptible, robust, and targeted adversarial examples for automatic speech recognition. arXiv preprint arXiv:1903.1036.. 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-255 2015.
Claudia Saviaga and Carlos Toxtli.
Deepiracy: by using longest common subsequence and deep learning. 2018. https:
Video piracy detection system
wow----ww--o-o------------------ www.ovvvaaaootteeeeeeeeeeeeeeeee htt.:://ww..................oo.. [Accessed: 05/21/2019).
Ali Shafahi, Mahyar Najibi, Zheng Xu, John Dickerson, Larry S Davis, and Tom Goldstein. Uni- versal adversarial training. arXiv preprini arXiv:1811.11304. 2018.
Ali Shafahi, Mahyar Najibi, Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! arXiv preprint arXiv: 12843, 2019.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruma, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint 1312.6/999 2013.
Rohan Taori, Amog Kamsetty, Brenton Chu, and Nikita Vemuri. Targeted adversarial examples for black box audio systems. arXiv preprint 1rX5v:1805.074 1805.07820, 2018.
Tencent. Experimental security research of tesla autopilot, 2019.
Avery Wang al. An industrial strength audio search algorithm. In Ismir, volume 2003, pp. 7-13. Washington, DC, 2003.
Derui Wang, Chaoran Li, Sheng Wen, Surya Nepal, and Yang Xiang. Daedalus: Break- ing non-maximum suppression in object detection via adversarial examples. arXiv preprint arXiv:1902.02067, 2019.
Cihang Xie, Jianyu Wang, Zhishuai Zhang, Yuyin Zhou, Lingxi Xie, and Alan Yuille. Adversarial examples for semantic segmentation and object detection. In Proceedings of the IEEE Interna- tional Conference on Computer Vision, pp. 1369-1378, 2017.
Hiromu Yakura and Jun Sakuma. Robust audio adversarial example for physical attack. arXiv preprint A.I0:1810117000 2018.
J. Yasaswi, S. Purini, and C. V. Jawahar. Plagiarism detection in programming assignments using deep features. In 20/7 4th IAPR Asian Conference on Putlern Recognition (ACPR), pp. 652-657, Nov 2017. doi: 10. 1099//CPR.201..14
10


Reviews:
[{"review_id": "ryxuDm2AKr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper points out that the industrial copyright detection systems are susceptible to adversarial attacks and discuss the reason why these systems, especially the neural-network-based systems, are vulnerable to attacks. The authors describe a widely used music identification method and propose a simple gradient method to attack this system in this paper.  Specifically, the authors build a simple song identification method using neural network primitives and propose a differentiable surrogate loss to measure how close two fingerprints of audio are, which is used to generate the adversarial examples. And then they attack a song identification using the adversarial examples. \nThe authors conduct experiments to evaluate the proposed method. The white-box attack on their own model and the experiments on two real-world systems (AudioTag and YouTube) validate the effectiveness of the proposed method. The experimental results on AudioTag and YouTube seem impressive.\n\nWeak points:\nFrom my point of view, this paper is more like an application-oriented paper than a research paper. Because the main contribution of this paper is the results of experiments on a real-world system. Though the experimental results seem to support the justification of the model, this paper has limitations in terms of Model and Experiments.\n1.\tModel\na.\tThe description of the proposed model is not clear enough. The whole architecture of this model is not presented. The authors should describe how to generate the adversarial example more clearly.\nb.\tIn section 4.1.2, “we build a shallow neural network that captures the key ideas of Wang et al. [2003] while adding extra layers that help produce transferable adversarial examples”. The Audio fingerprinting models proposed in the paper are quite incremental.\nc.\tThe proposed differentiable surrogate loss, which measures how close two fingerprints of audio are, should have more theoretical analysis to illustrate its reasonability or provide some analysis of the difference between adversarial examples and real examples.\n2.\tExperiment\na.\tThe results of the paper are hard to reproduce.  The detail of the model used in the experiment is not presented, such as how to train the model.\nb.\tThe authors should add more baseline or other intuitive methods to make the results of both the white-box attack and real-world more convincing.\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "H1lUxX-H9H", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Thank you to the authors for clarifying that they disclosed the vulnerability to YouTube. It would be good to surface this in the main text rather than a footnote, and to include some details as to what the response from YouTube was.\n\nThe paper demonstrates that adversarial examples are a practical scheme for evading copyright infringement detection tools. The study of this application is well motivated: adversarial examples need not be played in the physical domain (since the audio/video is uploaded directly in a digital format) and copyright detection is a bit more complicated than object classification (because false positives are expensive), small margin between classes. \n\nCould you motivate the choice of the L_inf norm as an appropriate metric for the magnitude of the perturbation added to audio samples? Would the perturbation affect a listener’s ability to follow the audio that the attacker is trying to avoid detection of? In Section 4.5, why is inserting a perturbation that is close to a different audio sample a successful attack? How would the baseline of playing the target audio sample at a very low volume perform in comparison?\n\nWhile the paper considers a system that was designed by the authors, because they demonstrate transferability to commercial products, this is not a limitation of their work and demonstrates a realistic threat model. The description of experimental results is however missing important details required to evaluate the results presented:\n\n* What are the details of the dataset used to evaluate the approach? How many songs were included in the dataset? \n* Did you conduct a study to verify that the perturbation added did not affect legitimate listeners?\n\nThe paper is well written and easy to follow. \n\nA simple nitpick on page 2: do you have any references to point to for the following claim? “Most audio, im- age, and video fingerprinting algorithms either train a neural network to extract fingerprint features, or extract hand-crafted features.” \n\nAnd on page 7, the following sentence might be a typo given that the perturbations introduced in this paper are adversarial: “Therefore, one would expect that low-amplitude non-adversarial noise should not affect this system.”", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Bkx-RNuUqH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "[Summary] This paper proposed a black(white)-box attacking algorithms to attack industrial copyright detectors. Specifically, a simple un-targeted gradient-based method can successfully fool commercial system like YouTube and AudioTag. This paper has generalized the concept of “adversarial examples” to a new application area, which is indeed a very important one, to raise awareness for the potential threats. \n\nThere are a couple of reasons of why I like this paper: (1) copyright attack and especially defense (missing in this paper) is a more important application compared to others widely studied vision tasks like classification. To increase the copyright detectors' accuracy, companies must have invested a huge amount of money on deep learning based methods. As this paper suggests, these more accurate detectors also come with severe security problems, and such problems are rarely studied before. In addition, the experimental results in this paper show that two commercial systems can be easily compromised in a black-box manner. (2) This paper is clear in writing and easy to follow. I am not an expert in acoustic/audio recognition, but this paper shows that a not very complicated approach can work in real-world without too much domain/expert knowledge. \n\nMy rating for this paper is “3: weak reject”. I think it’s below the line for two reasons: technical novelty and experimental settings. I will elaborate these two points in the following sections.\n\nAs the author mentioned and generally believed, almost every machine learning model (black and white-box) can be compromised with adversarial examples, especially in un-targeted manner. “Copyright detector” is just another machine learning model regardless its important usage. The technical approach proposed in this paper is straightforward and limited. First, section 4.2 is very similar to Wang et al. (2003). What’s the biggest improvement or novelty here? All the filters are handcrafted and fixed as suggested by Eq.1 and Eq.2. Why don’t train a network or take a pre-trained DNNs to predict spectrogram?  The biggest advantage of deep learning (end2end trainable as feature extractor) is not utilized.  Second, the solution of Eq.5 is award. Usually we don’t use a softmax with temperature to estimate max, better ways should be tried like gumbel-softmax. Third, besides the optimization methods used for Eq.6, Carlini&Wagner is also widely used for optimizing this objective. What’s the reason that C&W is missed in this paper? Fourth, Eq.8 can be used for targeted attacks. However, this part is missing. Un-targeted attack is considered as easy especially in white-box cases. Targeted attack is a better and harder task for evaluation of algorithms. Fifth, the surrogate loss Eq.4 is basically a hinge loss between two quantized feature maps with mask \\psi. It’s pretty standard to me and I am surprised this is the only loss proposed and studied. \n\nThe experiments in this paper is more like a “proof-of-concept” rather than “serious evaluation”. First problem is that the norm is used to evaluate the perturbation. Unlike the norm in image domains which can be visualized and easily understood, I don’t have a clear understanding of how “big” the perturbations are for audios. A cognitive study or something like a user study should be conducted. Another question related to this, the random noise is 3x bigger in terms of norm, does this make huge difference when listening to it? Are these two perturbations both very obvious or both unnoticeable? Second, it seems like a dataset is built but the stats are missing. How big is this dataset? How many songs are used to generate results in Tab.1, Tab.2, and Fig.3? What’s the attack accuracy on AudioTag and Youtube? Third, no baseline methods are compared to in this paper, not even an ablation study. The proposed two methods (default and remix) seem to perform similarly. I don’t think the best thing people can do previously to attack copyright system is just random noise. The authors could argue that there are no previous papers, but I think more realistic baselines should be studied. \n\nOne question, the inline equation at the bottom of Page.4  should be \\phi(x) == maxpool(\\phi(x)) right? And \\psi(x) = ( \\phi(x) == maxpool(\\phi(x)) ) ?\n\nOverall, I think this paper is not in good shape to be published even though the targeted problem has already become a big concern right now. \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Bkeg58SwqS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors of this work bring to light the security vulnerabilities of copyright detection systems to (DL style) adversarial attacks. The work summarizes the basics of copyright detection systems for audio, and notes that recent methodologies for feature extraction incorporate neural networks, as opposed to hand-crafted features. Following which the authors embark on designing a simple copyright system based on a neural network. With the newly proposed system, it is shown that one can construct adversarial examples to obstruct copyright detection systems using differentiable programming. The constructed adversarial examples are shown to be successful in evading popular copyright detection systems for audio (YouTube Content ID, AudioTag) as black box attacks.\n\nOverall the paper brings about an interesting and pressing issue in a timely manner that seems to be of broad interest to the security and ML community. The paper is very well written and I enjoyed reading it. Further, the construction of the adversarial objective and attacks seems novel. However, I am not as familiar with the literature with adversarial attacks for audio. Overall I am giving this work a weak accept, which I am willing to change if the authors provide additional insight into their experiments. \n\nIn particular, it is difficult for me to assess the success of the imperceptibility of the perturbation in the audio domain from l2 and l_infinity norms, so it is hard for me to judge whether such adversarial examples are competitive and useful. For this reason I would like the authors share excerpts from their attack experiments for multiple examples for the different results presented.\n\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper shows a case study of an adversarial attack on a copyright detection system. The paper implements a music identification method with a simple convolutional neural network, and shows that it is possible to fool such CNN with an adversarial learning. After the discussion period, two among three reviewers incline to the rejection of the paper. Although the majority of the reviewers agree that this is an interesting problem with an important application, they also find many of their concerns remain unaddressed. These include the generality of the finding as the current paper is more like a proof-of-concept that black/white-box attack can work for copyright system. The reviewers are also concerned that the technique solution/finding is not novel as it is very similar to prior work in other domains (e.g., image classification). One reviewer was particularly concerned about that the user study is missing, making it difficult to judge whether the quality of the modified audio is reasonable or not.
Decision: Reject