Active Image Indexing
Authors: Pierre Fernandez
Matthijs Douze
Herve Jegou
Teddy Furon
Venue: ICLR 2023 poster (2023)
Keywords: Indexing
Copy detection
Image similarity search
Watermarking

Abstract: Image copy detection and retrieval from large databases leverage two components. First, a neural network maps an image to a vector representation, that is relatively robust to various transformations of the image. Second, an efficient but approximate similarity search algorithm trades scalability (size and speed) against quality of the search, thereby introducing a source of error. 
This paper improves the robustness of image copy detection with active indexing, that optimizes the interplay of these two components.  We reduce the quantization loss of a given image representation by making imperceptible changes to the image before its release. The loss is back-propagated through the deep neural network back to the image, under perceptual constraints. These modifications make the image more retrievable. 
Our experiments show that the retrieval and copy detection of activated images is significantly improved. For instance, activation improves by $+40\%$ the Recall1@1 on various image transformations, and for several popular indexing structures based on product quantization and locality sensitivity hashing.

Published as a conference paper at ICLR 2023
ACTIVE IMAGE INDEXING
Pierre Fernandez!.2. Matthijs Douze', Hervé Jégou', Teddy Furon", Meta AI, FAIR "Centre Inria de L'Unvvrriiéé de Rennes
"Correspondence: pfz meta.com.
ABSTRACT
Image copy detection and retrieval from large databases leverage two components. First, a neural network maps an image to a vector representation, that is relatively robust to various transformations of the image. Second, an efficient but approxi- mate similarity search algorithm trades scalability (size and speed) against quality of the search, thereby introducing source of error. This paper improves the robustness of image copy detection with active indexing, that optimizes the inter- play of these two components. We reduce the quantization loss of given image representation by making imperceptible changes to the image before its release. The loss is back-propagated through the deep neural network back to the image, under perceptual constraints. These modifications make the image more retriev- able. Our experiments show that the retrieval and copy detection of activated images is significantly improved. For instance, activation improves by +40% the Recalll on various image transformations, and for several popular indexing structures based on product quantization and locality sensitivity hashing.
INTRODUCTION
The traceability of images on a media sharing platform is challenge: they are widely used, easily edited and disseminated both inside and outside the platform. In this paper, we tackle the corre- sponding task of Image Copy Detection (ICD), i.e. finding whether an image already exists in the database; and so, give back its identifier. ICD methods power reverse search engines, photography service providers checking copyrights, or media platforms moderating and tracking down malicious content (e.g. Microsoft's 's PhotoDNA (2009) or Apple's NeuralHash (2021)). Image identification systems have to be robust to identify images that edited (cropping, colorimetric change, JPEG compression.... ...) after their release (Douze et al., 2021; Wang et al., 2022).
The common approach for content-based image retrieval reduces images high-dimensional vec- tors, referred to as representations. Early representations used for retrieval were hand-crafted fea- tures such color histograms (Swain & Ballard, 1991), GIST (Oliva & Torralba, 2001), or Fisher
Original image
Latent space
Activated image
Outside world
(((+))
Publication
Re(L))
Activation
Approximate Search
Indexing
Index
Image editing
Scarch results
Figure 1: Overview of the method and latent space representation. We start from original image Lo that be edited ((-) in various ways: its feature extraction f(((L.) spawns the shaded region in the embedding space. The edited versions should recoverable by nearest neighbor search on quantized representations. In the regular (non-active) case, I J'Io) is quantized by the index as e. When the image is edited, I(Io) switches cells and the closest neighbor returned by the index is the wrong one o. In active indexing: Io is modified in imperceptible way generate I such that f(I") is further away from the boundary. When edited copies f(t(1") are queried, retrieval errors are significantly reduced.

---

Published as a conference paper at ICLR 2023
Vectors (Perronnin et al., 2010). As of now, large body of work on self-supervised learning focuses on producing discriminative representations with deep neural networks, which has inspired recent ICD systems. In fact, all submissions to the NeurIPS2021 Image Similarity challenge (Papakipos et al., 2022) exploit neural networks. They are trained to provide invariance to potential image transformations, akin to data augmentation in self-supervised learning.
Scalability is another key requirement of image similarity search: searching must be fast on large- scale databases, which exhaustive vector comparisons cannot do. In practice, ICD engines leverage approximate neighbor search algorithms, that trade search accuracy against scalability. Approxi- mate similarity search algorithms speed up the search by not computing the exact distance between all representations in the dataset (Johnson et al., 2019; Guo et al., 2020). First they lower the num- ber of scored items by partitioning the representation space, and evaluate the distances of only a few subsets. Second, they reduce the computational cost of similarity evaluation with quantization or binarization. These mechanisms make indexing methods subject to the curse of dimensional- ity. In particular, in high-dimensional spaces, vector representations lic close to boundaries of the partition (Böhm et al., 2001). Since edited versions of an original image have noisy vector represen- tations, they sometimes fall into different subsets or are not quantized the same way by the index. All in all, it makes approximate similarity search very sensitive to perturbations of the edited image representations, which causes images evade detection.
In this paper, we introduce method that improves similarity search on large databases, provided that the platform photo provider can modify the images before their release (see Fig. 1). We put the popular saying "attack is the best form of defense" into practice by applying image perturbations and drawing inspiration from adversarial attacks. Indeed, representations produced with neural networks are subject to adversarial examples (Szegedy et al., 2013); small perturbations of the input image can lead to very different vector representations, making it possible create adversarial queries that fool image retrieval systems (Liu et al., 2019; Tolias et al., 2019; Dolhansky & Ferrer, 2020). In contrast, we modify an image to make it more indexing friendly. With minimal changes in the image domain, the image representation is pushed towards the center of the indexing partition, rising the odds that edited versions will remain in the same subset. This property is obtained by minimizing an indexation loss by gradient descent back to the image pixels, like for adversarial examples. For indexing structures based on product quantization (Jegou et al., 2010), this strategy amounts to pushing the representation closer to its quantized codeword, in which case the indexation loss is simply measured by the reconstruction error. Since the image quality is an important constraint here, the perturbation is shaped by perceptual filters remain invisible to the human eye.
Our contributions are:
new approach improve ICD and retrieval, when images can be changed before release;
an adversarial image optimization scheme that adds minimal perceptual perturbations to images in order to reduce reconstruction errors, and improve vector representation for indexing; experimental evidence that the method significantly improves index performance.
2 PRELIMINARIES: REPRESENTATION LEARNING AND INDEXING
For the sake of simplicity, the exposure focuses on image representations from SSCD net- works (Pizzi et al., 2022) and the indexing technique IVF-PQ (Jegou et al., 2010), since both are typically used for ICD. Extensions to other methods can be found in Sec. 5.4.
2.1 DEEP DESCRIPTOR LEARNING
Metric embedding learning aims to learn mapping f Rexhxw + Rd. such that measuring the similarity between images I and I' amounts to computing the distance ()) - f(T')l Il. In recent works, f is typically a neural network trained with self-supervision on raw data to learn metri- cally meaningful representations. Methods include contrastive learning (Chen et al., 2020), self- distillation (Grill et al., 2020; Caron et al., 2021), or masking random patches of images (He et al., 2022: Assran et al., 2022). In particular, SSCD (Pizzi et al., 2022) is a training method specialized for ICD. It employs the contrastive self-super method SimCLR (Chen et al., 2020) and entropy regularization (Sablayrolles et al., 2019) to improve the distribution of the representations.
2

---

Published as a conference paper at ICLR 2023
2.2 INDEXING
Given a dataset X' = {ri)i=1 C I of d-dimensional vector representations extracted from n images and query vector zig. we consider the indexing task that addresses the problem:
argmin =EX
|| - xg
(1)
This exact nearest neighbor search is tractable over large-scale databases. Approximate search algorithms lower the amount of scored items thanks to space partitioning and/or accelerate the com- putations of distances thanks quantization and pre-computation.
Space partitioning and cell-probe algorithms. As a first approximation, nearest neighbors are sought only within a fraction of X: at indexing time, X' is partitioned into X' = -U_= X, At search time, an algorithm Q: : Rd {1, 67Kd determines a subset of k buckets which search, such that k' = (1(g)) < b, yielding the approximation:
argmin || - ry argmin xEX
||---
A well known partition is the KD-tree (Bentley, 1975) that divides the space along predetermined directions. Subsequently, locality sensitive hashing (LSH) (Indyk & Motwani, 1998; Gionis et al., 1999) and derivative (Datar et al., 2004; Paulevé al., 2010) employ various hash functions for bucket assignment, which implicitly partitions the space.
We focus on the popular clustering and Inverted Files methods (Sivic Zisserman, 2003), herein denoted by IVF. They employ codebook C {c C I2d of l centroids (also called "visual words" in local descriptor context), for instance learned with k-means over training set of repre- sentations. Then, Q associates J to its nearest centroid ge(ar) such that the induced partition is the set of the Voronoř cells. When indexing v, the IVF stores a in the bucket associated with ej = (c(x)) When querying T.. IVF searches only the k buckets associated to centroids c, nearest to T.q.
Efficient metric computation and product quantization. Another approximation comes from compressed-don -omaan distance estimation. Vector Quantization (VQ) maps representation Rd to a codeword gr (x) E C = {C:) The function qe is often referred to quantizer and C; as reproduction value. The vector æ is then stored The distance between & and query is approximated by []ee(z) - Izgll which an "asymmetric" distance computation (ADC) because the query is not compressed. This leads to:
integer in (1, K) corresponding (+(x).
argmin | zgll = argmin [1q)(x) - -rglll 2EX
(3)
zEX'
Binary quantizers (a.k.a. sketches, Charikar (2002) lead efficient computations but inaceu- rate distance estimates (Weiss et al., 2008). Product Quantization (PQ) (Jegou et al., 2010) or derivatives Ge al. (2013) offer better estimates. In PQ. a vector E R split into m subvectors in Rd/m;
(u', ...., )") The product quantizer then quantizes the subvectors: x + (q² (w²), ((."),...,"""" (..","""""""") If each subquantizer q² has K, reproduction values, the resulting
quantizer qe has high K (K,) The squared distance estimate is decomposed
[r((x) x1111
Σ Ell - u2l1? j=1
This is efficient since x is stored by the index as (x) which has m log, K2 bits, and since sum- mands can be precomputed without requiring decompression at search time.
ACTIVE INDEXING
Active indexing takes input an image Io, adds the image representation to the index and outputs an activated image with better traceability properties for the index. makes the feature repre- sentation produced by the neural network more compliant with the indexing structure. The activated image is the one that disseminated on the platform, therefore the alteration must not degrade the perceived quality of the image.
3

---

Published as a conference paper at ICLR 2023
Images activated by an optimization on their pixels. The general optimization problem reads:
== argmin C (1;
(5)
IEC(1.)
where L is an indexation loss dependent on the indexing structure, C(L)) is the set of images per- ceptually close to Lo. Algorithm and Figure provide an overview active indexing.
3.1 IMAGE OPTIMIZATION DEDICATED TO IVF-PQ ("ACTIVATION")
The indexing structure IVF-PQ involves coarse quantizer ge built with k-means clustering for space partitioning, and a fine product quantizer q on the residual vectors, such that a vector I E R2 is ap- proximated by ((x) = ge (x) + qe (a (x)))
Algorithm Active indexing for IVF-PQ
Input: I;: original image; f: feature extractor; Add x f((o) to Index, get g(zo);
Initialize 6g (exxxxw)) for f. D. .,... N do
4 +- L. + a HoND((o) @ tanh(6;)
We solve the optimization problem (5) by itera- tive gradient descent, propapatogg through the neural network back to the image. The method classically used in adversarial example genera- tion (Szegedy et al., 2013; Carlini & Wagner, 2017) and watermarking (Vukotić et al., 2020; Fernandez et al., 2022).
Tt - f(I()
C + (z(x1,9(2x)) + (6,
644 < 6; + x Adam(C)
end for
Output: 1 In activated image
Given an original image Io, the loss is an aggregation the following objectives:
Li(2, q(x))) || 9( Ci(T. = 1
with 2, f(o), f(1)
(6)
C is a regularization on the image distortion. & is the indexation loss that operates on the represen- tation space. C; is the Euclidean distance between r and the target g(xx) and its goal is push the image feature towards (Ix)). With IVF-PQ as index, the representation of the activated image gets closer the quantized version of the original representation, but also closer to the coarse centroid. Finally, the losses are combined as C(I; = Li(x, ((x.)) + Io,
3.2 PERCEPTUAL ATTENUATION
common to optimize a perturbation added to the image, rather than the image itself. The adversarial example literature often considers perceptual constraints in the form of an nn-norm bound applied on (Madry et al. (2018) use ||ō|| < E = 8/255). Although a smaller æ makes the perturbation less visible, this constraint is not optimal for the human visual system (HVS), e.g. perturbations are more noticeable on flat than on textured areas of the image (see App. A.2).
We employ handcrafted perceptual attenuation model based Just Noticeable Difference (JND) map (Wu et al., 2017), that adjusts the perturbation intensity according to luminance and contrast masking. Given an image I, the JND map HJND(!) E [exxxx models the minimum difference perceivable by the HVS at each pixel and additionally rescales the perturbation channel-wise since the human eye is more sensible to red and green than blue color shift (see App. A for details).
The relation that links the image I sent to f, å being optimized and the original Lo, reads:
- 1, a H.ND O tanh(o),
(8)
with global scaling parameter that controls the strength of the perturbation and @ the point- wise multiplication. Coupled with the regularization C₁ (6), enforces that the activated image is perceptually similar, i.e. 1* C(T.) as required in (5).
3.3 IMPACT ON THE INDEXING PERFORMANCE
Figure illusstrtes that the representation of the activated image gets closer to the reproduction value ([)))) and farther away from the Voronoř boundary. This is expected to make image similarity search more robust because (1) it decreases the probability that x = f(((L,)) "falls" outside the bucket; and (2) it lowers the distance between & and q(x), improving the PQ distance estimate.
4

---

Published as a conference paper at ICLR 2023
Besides. by design, the representation stored by the index is invariant to the activation. Formally stated, consider two images I, J. and one activated version J together with their representations I, y". When querying is oblivious to the change J + J This means that the structure can index passive and activated images at the passive images does not change. This compatibility property makes it possible to select only subset of images to activate, but also activate already-indexed images at any time.
f the distance estimate is Ilq(x)) - = ll( æ||, the index
time. Retrieval of activated images is more accurate but the performance
4 ANALYSES
We provide insights on the method for IVF-PQ, considering the effects of quantization and space partitioning. For image whose representation is f(1) E RR, å denotes the representation of a transformed version: = f(t(Z)) E RR, and x² the representation of the activated image I*. For details the images and the implementation used in the experimental validations, see Sec. 5.1.
4.1 PRODUCT QUANTIZATION: IMPACT ON DISTANCE ESTIMATE
We start by analyzing the distance estimate considered by the index:
| [x))2 = 2z [(x)]]2
=1|² + 2(x q(z)) (8 x).
(9)
The activation aims reduce the first term, i.e. the quantization error | g(x))11 which in turn reduces || (zx) Figure 3 shows in blue the empirical distributions of || - q(x)]l (passive) and ||.- - ((x) 1| (activated). As expected the latter has lower mean, but also a stronger variance. The variation of the following factors may explain this: i) the strength of the perturbation (due to the HVS modeled by HIND (8)), ii) the sensitivity of the feature extractor V (x))] (some features are easier to push than others), iil) the shapes and sizes of the Voronoï cells of PQ.
The second term of (9) models the impact of the image transformation in the feature space. Com- paring the orange and blue distributions in Fig. 3, we see that it has a positive mean, but the shift is bigger for activated images. We can assume that the third term has null expectation for two reasons: the noise å x is independent of g(x)) and centered around O, ii) in the high definition regime, quantification noise x ((x) is independent of x and centered on Thus, this term only increases the variance. Since x= q(x)) has smaller norm, this increase is smaller for activated images.
All in all, |+* (2x) has a lower mean but a stronger variance than its passive counterpart || ((x)]| Nevertheless, the decrease of the mean is large that it compensates the larger variance. The orange distribution in active indexing is further away from the green distribution for negative pairs, iee. the distance between indexed vector q(x) and independent query y.
Passive
bl abxll?
Passive (wAP=0.130)
qboll
0.8
Active (AAP=0.308)
| q(dllp
1 1-A0 p/l = 0.133
Precision
1 - ply = 0.323
0.4
Active
lly
atxil|
0.2
|
aixll
|| q(x]l
0.0
0.0
0.1
0.2
0.3
0.4
0.5
Recall
0.4
0.6
0.8
1.0
1.2
1.4
Figure Precision-Rece Recall curve for ICD with 50k queries and LM reference images (more details for the experimental setup in Sec. 5.1). pfff is the probability of failure of the IVF (Sce. 4.2).
Figure 3: Distance estimates histograms (sec. 4.1). With active indexing, (==((>))l² is reduced (+),, inducing shift (+) in the distribution of || q(x)]] where t(I) a hue-shifted version y a random query.
5

---

Published as a conference paper at ICLR 2023
4.2 SPACE PARTITIONING: IMPACT ON THE IVF PROBABILITY OF FAILURE
We denote by pr := F(g-(#) + (z)) the probability that á is assigned to a wrong bucket by IVF assignment Go- In the single-probe search (k' 1), the recall (probability that pair detected when it is a true match, for given threshold on the distance) is upper-bounded by Pr:
R = P (gee() = (=)))) n {Il (x))) < S P (( = de (x())) = 11 Pr- (10)
In other terms, even with a high threshold T + oo (and low precision); the detection misses represen- tations that ought be matched, with probability pr- It explains the sharp drop at recall R 0.13 in Fig. 2. This is why it is crucial to decrease pt. The effect of active indexing is to reduce || - qo (x))
therefore reducing pr and increasing the upper-bound for R: the drop shifts towards R 0.32.
This explanation suggests that pushing towards ge (z) decreases even more efficiently pr. This makes the IVF more robust to transformation but this may jeopardize the PQ search because features of activated images are packed altogether. In way, our strategy, which pushes towards q(x),
dispatches the improvement over the IVF and the PQ search.
EXPERIMENTAL RESULTS
5.1 EXPERIMENTAL SETUP
Dataset. We use DISC21 (Douze et al., 2021) a dataset dedicated to ICD. It includes 1M reference images and 50k query images, 10k of which true copies from reference images. A disjoint 1M- image set with same distribution as the reference images is given for training. Images resolutions range from x00 200 1024x 1024 pixels (most of the images are around x024x pixels).
The queries used in our experiments are not the queries in DISC21, since we need to control the im- age transformations in our experiments, and most transformations of DISC21 were done manually so they are not reproducible. Our queries are transformations of images after active indexing. These transformations range from simple attacks like rotation to more realistic social network transforma- tions which created the original DISC21 queries (see App. B.
Metrics. For retrieval, our main metric Recall 1 @@1 (R@1 for simplicity), which corresponds to the proportion of positive queries where the top-1 retrieved result is the reference image.
For copy detection, we use the same metric as the NeurlPS Image Similarity Challenge (Douze et al., 2021). We retrieve the k = 10 most similar database features for every query; and we declare a pair is a match if the distance is lower than a threshold т. To evaluate detection efficiency, we use the 10k matching queries database). We use precision and recall, as well as the area under the precision-recall curve, which is equivalent to the micro average precision (μAP). While R@ 1 only measures ranking quality of the index, μAP takes into account the confidence of a match.
mentionee together with 40k negative queries (i.e. not included in the
As for image quality metric, we use the Peak Signal-to-Noise Ratio (PSNR) which is defined as 10 log10 (255² MMS((T,T))))) 7))2), as well as SSIM (Wang et al., 2004) and the norm / - l' [o
Implementation details. The evaluation procedure is: (1) we train an index on the 1M training images, (2) index the IM reference images, (3) activate (or not) 10k images from this reference set. (4) At search time, we use the index to get closest neighbors (and their distances) of transformed versions from a query set made of the 10k images.
Unless stated otherwise. we use a IVF4096, index (IVF quantizer with 4096 centroids, and PQ with 8 subquantizers of 2 centroids), and use only one probe on IVF search for shortlist selection (K' = 1). Compared to realistic setting, we voluntarily use an indexing method that severely degrades learned representations to showcase and analyze the effect of the active indexing. For feature extraction, we use an SSCD model with a ResNet50 trunk (He et al., 2016). It takes image resized to 288 288 and generates normalized representations in 512 Optimization (5) is done with the Adam optimizer (Kingma & Ba, 2015), the learning rate is set to 1, the number of iterations to N 10 and the regularization to À 1. In (8), the distortion scaling is set = 3 (leading to an average PNSR around 43 dB). In this setup, activating 128 images takes around 6s (+= 40ms/image) with 32GB GPU. It can be sped-up at the cost of some accuracy (see App. C.2).

---

Published as a conference paper at ICLR 2023
Table I: Comparison of the index performance between activated and passive images. The search is done IM image set and R@ is averaged over 10k query images submitted different transformations before search. randomly apply 1 to 4 transformations. Avg. average on the transformations presented in the table (details in App. B.2). No index: exhaustive brute-force nearest neighbor search. IVF-PQ: IVF4096, (F4096,PQ8x8 index with ===1 (16 for IVF-PQ" "). IVF-PQ'; IVF512,PQ322. with k' 32.
Random:
Search (ms)
Bytes/vector
Activated
Idenrity
Cont 0.5
Contr 2.0
Bright. 0.5
Bright. 2.0
Hue 0.2
Blur 2.0
JPBG 50
Rot. 25
Rot. 90
Crop 0.5
Resi. 0.5
Meme
Random
Avg.
No index
252
2048
X
1.00
1,00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
0.90 0.99
X
1.00
0.73
0.39
0.73
0.28
0.62
0.48
0.72
0.07
0.14
0.14
0.72
0.14
0.13
0.45
IVT-PQ
0.38
8
V
1.00
1.00
0.96
1.00
0.92
1.00
0.96
0.99
0.10
0.50
0.29
1.00
0.43
0.32
0.75
IVF-PQ"6
0.42
8
X
1.00
1.00
0.90
1.00
0.78
0.99
0.95
0.99
0.35
0.57
0.57
1.00
0.56
0.39 0.79
1.00
1.00
1.00
1.00
0.98
1.00
1.00
1.00
0.43
0.88
0.75
1.00
0.84
0.50
0.88
X
1,00
1,00
0.99
1,00
0.95
1.00
0.99
1.00
0.72
0.87
0.88
1.00
0.87
0.61 0.92
IVF-PQU
1.9
1.00
1.00
0.99
1.00
0.98
1.00
1.00
1.00
0.75
0.92
0.91
1.00
0.92
0.63
0.94
5.2 ACTIVE VS. PASSIVE
This section compares retrieval performance of active and passive indexing. We evaluate R@] I when different transformations are applied the 10k reference images before search. The "Passive" lines of Tab. 1 show how the IVF-PQ degrades the recall. This is expected, but the IVF-PQ also accelerates search 500x and the index is 256x more compact, which is necessary for large-scale applications. Edited images of up to +60 R@ 1 for strong brightness and contrast changes, close to results of the brute-force search. We also notice that the performance of the active IVF-PO =1 is approximately the same as the one of the passive IVF-PO%==16 meaning that the search can be made more efficient at equal performance. For the IVF-PQ" that does less approximation in the search (but is slower and takes more memory), retrieval on activated images is also improved, though to lesser extent.
retrieved more often when they were activated for the index: increase
As for copy detection, Figure 2 gives the precision-recall curves obtained for a sliding value of T, and corresponding μAP, Again, we observe a significant increase (x2) in μAP with active indexing. Note that the detection performance is much weaker than the brute-force search even in the active case because of the strong approximation made by space partitioning (more details in Sec. 4.2).
Example of activated images are given in Fig. 5 (more in App. E), while the qualitative image metrics are as follows: PSNR= 43.8 + 2.2 dB, SSIM= 0.98 = 0.01, and ||! - ~l'll = 14.5 + 1.2. These results are computed on 10k images, the = indicates the standard deviation.
5.3 IMAGE QUALITY TRADE-OFF
For fixed index and neural extractor, the performance of active indexing mainly depends on the scaling that controls the activated image quality. In Fig. 4, we repeat the previous experiment for different values of a and plot the μAP against the average PSNR. As expected, lower PSNR implies better μAP. For instance, at PSNR 30 dB, the μAP is augmented threefold compared to the passive case. Indeed, for strong perturbations the objective function of (6) can be further lowered, reducing even more the gap between representations and their quantized counterparts.
5.4 GENERALIZATION
Generalization to other neural feature extractors.
We first reproduce the experiment of Sec. 5.1 with different extractors, that cover distinct training methods and architectures. Among them, we evaluate ResNext101 (Xie et al., 2017) trained with SSCD (Pizzi et al., 2022), a larger network than the ResNet50 used in our main experiments ; the winner of the descriptor track of the NeurIPS ISC, LYAkAAP-dtl (Yokoo, 2021), that uses an EfficientNetv2 architecture (Tan & Le, 2021): networks from DINO (Caron et al., 2021), either based on ResNet50 ViT (Dosovitskiy et al., 2021), like the VIT-S model (Touvron et al., 2021).
7

---

Published as a conference paper at ICLR 2023
Table 2: R@1 for different transformations before search. We method to activate images for indexing
with IVF-PQ, with different neural networks used as feature extractors.
Architecture
Name
Activated
Identity
Contr. 0.5
Contr: 2.0
Bright. 0.5
Bright 2.0
Hoe
Blur 20
JPG 50
Rot 25
Rot. 90
Crop 0.5
Resi. 0.5
Mome
Random
Avg
x 1.00 0.73 0.39
0.73
0.28
0.62
0.48
0.72
0.07
0.14
0.14
0.72
0.14
0.13
0.45
ResNet50
1.00 1.00
0.96
1.00
0.92
1.00
0.96
0.99
0.10
0.50
0.29
1.00
0.43
0.32
0.75
SSCD
X 1.00
0.88
0.68
0.88
0.57
0.84
0,46
0.79
0.46
0.63
0.53
0,80
0.48
0.28
0.66
ResNext101
1.00
1.00
0.96
1.00
0.90
0.99
0.77
0.97
0.53
0.85
0.64
1.00
0.74
0.37
0.84
x 1.00
0.66
0.65
0.65
0.52
0.71
0.52
0.82
0.07
0.20
0.51
0.84
0.62
0.18
0.57
ResNet50
1.00 0.99 0.88
0.99
0.75
0.93 0.72 0.94
0.08
0.25
0.57
0.99
0.82
0.23
0.72
DINO
X 1.00 0.89 0.71 0.86
0.64
0.75
0.74
0.90
0.14
0.18
0.57
0.88
0.61
0.25
0.65
ViT-s
1.00 0.99 0.94 0.99 0.92 0.98 0.89 0.99
0.15
0.28 0.63 0.99
0.77
0.32 0.77
x 1.00 0.25
0.08
0.16
0.01
0.51
0.54
0.84
0.18
0.16
0.23
0.79
0.16
0.18 0.36
ISC-dtl EffNetv2
1.00 0.57 0.16 0.33
0.01
0.88
0.79
0.97
0.20
0.24
0.29
0.97
0.26
0.26
0.49
Table presents the @@1 1 obtained on 10k activated images when applying different transforma- tions before search. The R@1 1 is better for activated images for all transformations and all neural networks. The average improvement on all transformations ranges from +12% for DINO ViT-s to +330% for SSCD ResNet50.
Generalization to other indexes.
The method easily generalizes other types of indexing structures, the only difference being in the indexation loss Le (6). We present some of them below:
PQ and OPQ. In PQ (Jegou et al., 2010), vector E I is approximated by (z).). Cr reads || - qr (I. In OPQ (Ge et al., 2013), vectors are rotated by matrix R before codeword assignment, such that RR 1. L. becomes || - RR gr (Rxx)]l
IVE. Here, we only do space partitioning. Employing L = ||- (xx)) ("pushing towards the cluster centroid") decreases the odds of x falling in the wrong cell (see Sec. 4.2). In this case, an issue can be that similar representations are all pushed together to same centroid, which makes them less discriminate. Empirically, we found that this does not happen because perceptual constraint in the image domain prevents features from getting too close.
LSH. Locality Sensitive Hashing maps æ E IR to a binary hash b(x) E R²_ It is commonly done with projections against set of vectors, which give for j E [1,.., L). (x( sign(w x:) The objective L. = -1/ Σ, sign(b(<r.)) wo] x, allows to push x along the LSH directions and to improve the robustness of the hash.
Table 3 presents the R@1 and μAP obtained on the 50k query set. Again, results are always better in the active scenario. We remark that active indexing has more impact on space partitioning tech-
1.0
0.7
Active [|x- obIl|2
Passive
Active
05001
Passive
0.7
0.4 2
0.0
0.4
60
50
40
30
0.1
PSNR (dB)
Figure 4: PSNR trade-off. As the PSNR decreases, the μAP (orange) gets better, be- cause the distance (blue) between activated representations and g(x) decreases.
Figure S: Activated images. Left: reference from DISC (R000643.jpp jpp and R000761 jpg), middle; activated image, right: pixel-wise difference.
8

---

Published as a conference paper at ICLR 2023
Table 3: H@ averaged transformations presented in Tab. 1 and μAP for different indexing structures
R@1 avg. Activated
μAP Passive
Index
Scarch time
Passive
Activated
IVF 1024
0.32 ms
0.66
0.89
0.16
0.43
PCA64, LSHI
0.99 ms
0.78
0.86
0.25
0.39
OPQ 8x8
5.71
0.92
0.94
0.48
0.55
niques: the improvement for IVF higher than with PQ and the LSH binary sketches. As to be expected, the impact is smaller when the indexing method is more accurate.
6 RELATED WORK
Image watermarking hides a message in host image, such that it can be reliably decoded even the host image is edited. Early methods directly embed the watermark signal in the spatial or transform domain like DCT or DWT (Cox et al., 2007). Recently, deep-learning based methods jointly train an encoder and decoder to learn how to watermark images (Zhu et al., 2018; Ahmadi et al., 2020; Zhang et al., 2020).
Watermarking an alternative technology for ICD. Our method bridges indexing and watermarking. where the image is modified before publication. Regarding retrieval performance, active indexing is more robust than watermarking. Indeed, the embedded signal reinforces the structure naturally present in the original image, whereas watermarking has to hide large secret keyed signal indepen- dent of the original feature. App. D provides a more thorough discussion and experimental results comparing indexing and watermarking.
Active fingerprint is more related to our work. As far as we know, this concept was invented by Voloshynovskiy al. (2012). They consider that the image R. is mapped to æ E E. by an invertible transform W such that WWT. The binary fingerprint is obtained by taking the sign of the projections of x against set of vectors bı..,by by E R la LSH). Then, they change x to strengthen the amplitude of these projections recover I* with W This scheme is applied to image patches (Kostadinov et al., 2016) where the performance is measured fingerprinting to indexing, with modern deep learning representations and state-of-ther of-thh--rtt indexing techniques. The range of transformations is also much broader and includes geometric transforms.
that their signs become more robust to noise. They
bit eтrог rate after JPEG compression. Our paper adapts this idea from
CONCLUSION & DISCUSSION
We introduce way to improve ICD in large-scale settings, when images can be changed before release. leverages an optimization scheme, similar to adversarial examples, that modifies images so that (1) their representations human eye. We provide grounded analyses on the method and show that it significantly improves retrieval performance of activated images, on a number of neural extractors and indexing structures.
better suited for indexing, (2) the perturbation is invisible the
Activating images takes time (in the order of 10 ms/image) but one advantage that the database may contain both active and passive images: active indexing does not spoil the performance of passive indexing and vere-versa. This is good for legacy compliance and also opens the door to flexible digital asset management strategies (actively indexing images particular importance).
The main limitation of the method is that images need to be activated before release. In the case of existing databases where images have already been released, images could still be activated for future releases (meaning that there would be 2 versions of the image online, a passive one that can be retrieved as long as is not transformed too strongly, and an activated one with better copy detection properties). Another one is that it is not agnostic to the indexing structure and extractor that are used by the similarity search. Finally, an adversary could still break the indexing system in several ways. In a black-box setting, adversarial purification (Shi et al., 2021) could get rid of the perturbation that activated the image. semi-white-he setting (knowledge of the feature extractor), targeted mismatch attacks against passive indexing like Tolias et al. (2019) may also degrade the retrieval.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
Teddy Furon thanks ANR-AID for funding Chaire SAIDA ANR-20-CHIA-0011-000
ETIICS STATEMENT
Societal impact statement. Content tracing is double-edged sword. On the one hand, allows media platforms to more accurately track malicious content (pornographic, terrorist, violent images, e.g. Apple's NeuralHash and Microsoft's PhotoDNA) and to protect copyright (e.g. Youtube's Con- tent ID). On the other hand it can be used as a means of societal and political censorship, to restrict free speech of specific communities. However, we still believe that research needs to be advanced to improve global moderation in the internet. We also believe that advantages that better copy detection could bring are more numerous than its drawbacks.
Environmental impact statement. We roughly estimated that the total GPU-days used for run- ning all our experiments to 200, or = 5000 GPU-hours. Experiments were conducted using private infrastructure and we estimate total emissions to be in the order of ton COzeq. Estimations were conducted using the Machinel Learning Impact calculator presented in Lacoste et al. (2019). We do not consider this approximation: memory storage, CPU-hours, production cost of GPUs/ CPUs, etc. well as the environmental cost of training the neural networks used as feature extractors. Al- though the cost of the experiments and the method is high, it could possibly allow a reduction the computations needed in large data-centers thanks to improved performance of indexing structures.
REPRODUCIBILITY STATEMENT
The implementation is available at feature extraction (SSCD, DINO, ISC-dtl) can be downloaded in their respective repositories. builds upon the open-source Pytorch (Paszke et al., 2019) and FAISS (Johnson et al., 2019) libraries.
Models used for
The main dataset used in the experiments (DISC21) can be freely downloaded on its webpage http://affcccoooooooooma/mttttta -datasee// Dataset processing is described in App. B.1.
REFERENCES
Mahdi Ahmadi, Alireza Norouzi, Nader Karimi, Shadrokh Samavi, and Ali Emami. Redmark: Framework for residual diffusion watermarking based on deep networks. Expert Systems with Applications, 2020.
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Michael Rabbat, and Nicolas Ballas. Masked siamese networks for lahel-efficient learning. arXiv preprint arXiv:2204.077 141, 2022.
Anish Athalye, Logan Engstrom, Andrew Ilyas, and Kevin Kwok. Synthesizing robust adversarial examples. In ICML. PMLR, 2018.
Jon Louis Bentley. Multidimensional binary search trees used for associative searching. Communi- cations of the ACM, 1899)5509-517, 1975.
Christian Böhm, Stefan Berchtold, and Daniel A Keim. Searching in high-dimensional spaces: Index structures for improving the performance of multimedia databases. ACM Computing Surveys (CSUR), 33(3):322-333333 -333322223322222222 2001.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy (SP). IEEE, 2017.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in sulf-supe vision transformers. In ICCV. IEEF, 2021.
Moses Charikar, Similarity estimation techniques from rounding algorithms. In Proceedings of the thiry-fourth annual ACM symposium Theory of computing, pp. 380-388, 2002.
10

---

Published as a conference paper at ICLR 2023
Ting Chen, Simon Korblithh, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. JCML. PMLR, 2020.
Chun-Hsien Chou and Yun-Chin Li. A perceptually tuned subband image coder based on the mea- sure of tust-noticcableediitttttnn profile. IEEE Transactions circuits and systems for video technology, 5(6):467-476, 1995.
Ingemar Cox, Matthew Miller, Jeffrey Bloom, Jessica Fridrich, and Ton Kalker. Digital watermark- ing and steganography. Morgan kaufmann, 2007.
Mayur Datar, Nicole Immorlica, Piotr Indyk, and Vahab S Mirrokni. Locality-sensittve hashing scheme based on p-stable distributions. In Proceedings of the twentieth annual symposium on Computational geometry, pp. 253-262, 2004.
Brian Dolhansky and Cristian Canton Ferrer. Adversarial collision attacks on image hashing func- tions. arXiv preprint wxXi:2001.094735 2020.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
Matthijs Douze, Giorgos Tolias, Ed Pizzi, Zoë Papakipos, Lowik Chanussot, Filip Radenovic, Tomas Jenicek, Maxim Maximov, Laura Leal-Taixé, Ismail Elezi, et al. The 2021 image similarity dataset and challenge. arXiv preprint Nit::2206.09672 2021.
Pierre Fernandez. Alexandre Sablayrolles, Teddy Furon, Hervé Jégou, and Matthijs Douze. Water- marking images in self-supervised latent spaces. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022.
Tiezheng Ge, Kaiming He, Qifa Ke, and Jian Sun. Optimized product quantization for approximate nearest neighbor search. In CVPR. IEEF, 2013.
Aristides Gionis, Piotr Indyk, Rajeev Motwani, et al. Similarity search in high dimensions via hashing. In Vldb, volume 99, pp. 518-529, 1999.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. NeurlPS, 2020.
Ruiqi Guo, Philip Sun, Erik Lindgren, Quan Geng, David Simcha, Felix Chern, and Sanjiv Kumar. Accelerating large-scale inference with anisotropic vector quantization. In ICML. PMLR, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR. IEEE, 2016.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022.
Piotr Indyk and Rajeev Motwani. Approximate nearest neighbors: towards removing the curse of dimensionality. In Proceedings of the thirtieth annual ACM symposium on Theory of computing, pp. 604-613, 1998.
Herve Jegou, Matthijs Douze, and Cordelia Schmid. Product quantization for nearest neighbor search. Transactions on Puttern Analysis and Machine Intelligence, 33(1):117-128, 2010.
Qiuping Jiang, Zhentao Liu, Shiqi Wang, Feng Shao, and Weisi Lin. Towards top-down just notice- able difference estimation of natural images. IEEE Transactions Image Processing, 2022.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. 1EEE Transactions on Big Data, 2019.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Dimche Kostadinov, Slava Voloshynovskiy, Maurits Diephuis, and Taras Holotyak. Local active content fingerprinting: Optimal solution under linear modulation. In ICIP, 2016.

---

Published as a conference paper at ICLR 2023
Lester E Krueger. Reconciling fechner and stevens: Toward unified psychophysical law. Behav- ioral and Brain Sciences, 12(2):251-267, 1989.
Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. Quantifying the carbon emissions of machine learning. arXiv preprint arXiv:1910.09700, 2019.
Cassidy Laidlaw, Sahil Singla, and Soheil Feizi. Perceptual adversarial robustness: Defense against unseen threat models. In ICLR, 2021.
Zhuoran Liu, Zhengyu Zhao, and Martha Larson. Who's afraid of adversarial queries? the im- pact of image modifications on content-based image retrieval. In Proceedings of the 2019 on International Conference Multimedia Retrieval, pp. 306-314, 2019..
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant adversarial attacks. In ICLR, 2018.
Sébastien Marcel and Yann Rodriguez. Torchvision the machine-vision package of torch. In Inter- national Conference on Multimedia. ACM, 2010.
NeuralHash. Apple. https//www..agl..ccom///lldlssss 2021.
Aude Oliva and Antonio Torralba. Modeling the shape of the scene: A holistic representation of the spatial envelope. International journal of computer vision, 42(3)::45-175, 2001.
Zoe Papakipos and Joanna Bitton. Augly:
Data augmentations for robustness. arXiv preprint
arXiv:2201.06494. 2022.
Zoë Papakipos, Giorgos Tolias, Tomas Jenicek, Ed Pizzi, Shuhei Yokoo, Wenhao Wang, Yifan Sun, Weipu Zhang, Yi Yang, Sanjay Addicam, et al. Results and findings of the 2021 image similarity challenge. In NeurlPS 2021 Competitions and Demonstrations Track, pp. 1-12. PMLR, 2022.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), NeurIPS. Curran Associates, Inc., 2019.
Loic Paulevé, Hervé Jégou, and Laurent Amsaleg. Locality sensitive hashing: A comparison of hash function types and querying mechanisms. Pattern recognition letters, 31(1)), 2010.
Florent Perronnin, Yan Liu, Jorge Sánchez, and Hervé Poirier. Large-scale image retrieval with compressed fisher vectors. In CVPR, pp. 3384-3391. IEEE, 2010.
PhotoDNA. Microsoft. http://www.miccnoof..o/eeeeeee/e
2009.
Ed Pizzi, Sreya Dutta Roy, Sugosh Nagavara Ravindra, Priya Goyal, and Mathhij Douze. self- supervised descriptor for image copy detection. In CVPR. IEEE, 2022.
Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, and Hervé Jégou. Spreading vectors for similarity search. ICML, 2019.
Changhao Shi, Chester Holtz, and Gal Mishne. Online adversarial purification based on self- supervised learning. In ICLR, 2021.
Richard Shin and Dawn Song. Jpeg-resistant adversarial images. In NeurlPS Workshop on Machine Learning and Computer Security, 2017.
Josef Sivic and Andrew Zisserman. Video google: A text retrieval approach to object matching in videos. In Computer Vision, IEEE International Conference on, volume 3, pp. 1470-1470. IEFE Computer Society, 2003.
Michael Swain and Dana H Ballard. Color indexing. International journal of computer vision, (1):1 1-32, 1991.
12

---

Published as a conference paper at ICLR 2023
Christian Szegedy, Wojeiech Zaremba, llya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In ICLR, 2013.
Mingxing Tan and Quoc Le. Efficientnetv2: Smaller models and faster training. In ICML. PMLR, 2021.
Giorgos Tolias, Filip Radenovic, and Ondrej Chum. Targeted mismatch adversarial attack: Query with a flower to retrieve the tower. In ICCV. IEEE, 2019.
Hugo Touvron, Matthicu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-cfficient image transformers & distillation through attention. In International Conference Machine Learning, 2021.
Sviatoslav Voloshynovskiy, Farzad Farhadzadeh, Oleksiy Koval, and Taras Holotyak. Active content fingerprinting: marriage of digital watermarking and content fingerprinting. In International Workshop on Information Forensics and Security (WIFS), pp. 175-180. IEEE, 2012.
Vedran Vukotić, Vivien Chappelier, and Teddy Furon. Are classification deep neural networks good for blind image watermarking? Entropy, 2020.
Wenhao Wang, Yifan Sun, and Yi Yang. A benchmark and asymmetrical similaary learning for practical image copy detection. arXiv preprint arXiv:2205.12355. 2022.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility structural similarity. IEEE Transactions on image processing, 2004.
Andrew B Watson. Det quantization matrices visually optimized for individual images. Human vision, visual processing, and digital display IV, volume 1913, pp. 202-216 SPIE, 1993.
Yair Weiss, Antonio Torralba, and Rob Fergus. Spectral hashing. NeurIPS, 21, 2008.
Jinjian Wu, Leida Li, Weisheng Dong, Guangming Shi, Weisi Lin, and C-C Jay Kuo. Enhanced just noticeable difference model for images with pattern complexity. IEEE Transactions on Image Processing, 2017.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In CVPR. IEEE, 2017.
XK Yang, WS Ling, ZK Lu, Ee Ping Ong, and SS Yao. Just noticeable distortion model and its applications in video coding. Signal processing; Image communication, 20(7):662-680, 2005.
Shuhei Yokoo. Contrastive learning with large memory bank and negative embedding subtraction for accurate copy detection. arXiv preprint arXiv:2112.04322 2021.
Chaoning Zhang, Philipp Benz, Adil Karjauv, Geng Sun, and In So Kweon. Udh: Universal deep hiding for steganography. watermarking, and light field messaging. NeurIPS, 2020.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness deep features as perceptual metric. In CVPR. IEEE, 2018.
Xiaohui Zhang, Weisi Lin, and Ping Xue. Just-noticeable difference estimation with pixels in im- ages. Journal Visual Communication and Image Representation, 19(1):30 41, 2008.
Jiren Zhu, Russell Kaplan, Justin Johnson, and Li Fei-Fei, Hidden: Hiding data with deep networks. In ECCV, 2018.
13

---

Published as a conference paper at ICLR 2023
Supplementary material Active Image Indexing
A DETAILS ON THE PERCEPTUAL ATTENUATION MODEL
A.1 JUST NOTICEABLE DIFFERENCE MAP
The maximum change that the human visual system (HVS) cannot perceive is sometimes referred the just noticeable difference (JND) Krueger (1989). It is used in many applications, such as image/video watemarking, compression, quality assessment (JND is also used in audio).
JND models in pixel domain directly calculate the JND each pixel location (i.e. how much pixel difference is perceivable by the HVS). The JND map that we use is based on the work of Chou & Li (1995). We use this model for its simplicity, its efficiency and its good qualitative results. More complex HVS models could also be used if even higher inperceptibility is needed (Watson (1993); Yang et al. (2005); Zhang et (2008); Jiang al. (2022) cite a few). The JND map takes into account two characteristics of the HVS, namely the luminance adaptation (LA) and the contrast masking (CM) phenomena. We follow the same notations as Wu et al. (2017).
The CM map Mc is function of the image gradient magnitude Cj (the Sobel filter of the image);
Mc(x)- = 0.115 x a C-C(z))2.
C1 8a with C = Vv
V [xx]2 VyT(x)?
(11)
where x is the pixel location, I (xx) the image intensity, a 16, and 3 26. It is increasing function of C, meaning that the stronger the gradient is at æ. the more the image is masking local perturbation, and the higher the noticeable pixel difference is.
LA takes into account the fact that the HVS presents different sensitivity to background luminance (e.g. it less sensible in dark backgrounds). is modeled as:
(xx)
17 -
if B(x) < 127
CA(z)=
127
(12)
(B(x) 127)
+3
if 2 127,
128
where B(x) the background luminance, which calculated as the mean luminance value of local patch centered on x.
Finally, both effects are combined with nonlinear additivity model:
HIND = LA + MM - C min(4A,MMc,,
(13)
where C' is set to 0.3 and determines the overlapping effect. For color images, the final RGB heatmap HIND = n H. ag H, HgH,, where (an, ag. QB) are inversely proportional to the mixing co- efficients for the luminance: (a.R ag, ag) (.072/(0.29)) (558,00114)
Figure 6: A reference image from DISC21 (R002815 jpg), and the associated perceptual heatmap HIND(I).
14

---

Published as a conference paper at ICLR 2023
A.2 COMPARISON WITH & CONSTRAINT EMBEDDING
Figure 7 shows the same image activated using either the to2 constraint (commonly used in the adversarial attack literature) our perceptual constraint based the JND model explained above. Even with very small (4 over 255 in the example bellow), the perturbation is visible especially in the flat regions of the images, such as the sea sky.
Laidlaw et al. (2021) also show that the too is not good perceptual constraint. They use the LPIPS loss (Zhang al., 2018) as surrogate for the HVS develop more imperceptible adversarial attacks. Although a similar approach could be used here, we found that at this small level of image distortion the LPIPS did not capture CM and LA as well as the handcrafted perceptual models present in the compression and watermarking literature.
(a) 60 4, PSNR 36.4 dB, SSIM 0.91
(b) foc = 23, PSNR 34.4 dB, SSIM 0.94
Figure 7: Activated images, either with (a) the & < constraint with (b) perceptual model (best viewed screen). We give the corresponding measures between the original and the protected image, well the pixel-wise difference. perturbation on the right is much less perceptible thanks to the perceptual model, even though its &oo distance with the original image is much higher.
B EXPERIMENTS DETAILS
This section describes the details omitted in experimental sections.
B. DATASET
The dataset DISC 2021 was designed for the Image Similarity Challenge (Douze et al., 2021) and be downloaded in the dataset webpage: https://ai. https://a.faceeboo.cco wrweccooo..com/ttttsssstiiieieed -at.aasttttdisstt2asssseeee 1-dataset/.
We want to test performance on edited versions of activated images but DISC query set transfor- mations already applied to images. Therefore the query set cannot be used as it is.
We create first test set "Ref10k" by selecting the 10k images from the reference set that were originally used to generate the queries (the "dev queries" from the downloadable version). We also re-create a query set "Query50k". To be as close as possible, we use the same images that were used for generating queries DISC. Edited images are generated using the AugLy library (Papakipos & Bitton, 2022), following the guidelines given in the "Automatic Transformations" section of the DISC paper. Therefore, the main difference between the query set used in our experiments and the original one is that ours do not have manual augmentations.
B.2 TRANSFORMATIONS SEEN AT TEST TIME
They cover both spatial transformations (crops, rotation, etc.), pixel-value transformations (contrast, hue,jpee,, etc.) and "everyday life" transformations with the AugLy augmentations, All transforma- tions are illustrated in Fig. 4. The parameters for all transformations are the ones of the torchvision library (Marcel & Rodriguez, 2010), except for the crop and resize that represent area ratios. For the Gaussian blur transformation we use alternatively ar, the scaling factor in the exponential, or the ker- nel size K, (in torchvision hip (a 0.35)/0.15). The "Random" transformation is the one used to
15

---

Published as a conference paper at ICLR 2023
Table 4: Illustration of all transformations evaluated in Tab. 1.
Identity
Contrast 0.5
Contrast 2.0
Brightness 0,5
Brightness 2.0
Hue 0.2
Blur 2.0
JPEG 50
Rotation 25
Rotation 90
Crop 0.5
Resize 0.5
Meme LOL
Random
0000
develop the 50k query set. A series of simple 1-4 AugLy transformations are picked at random, with skewed probability for a higher number. Among the possible transformations, there are pixel-level, geometric ones, as well as embedding the image as screenshot of social network GUI.
C MORE EXPERIMENTAL RESULTS
C.1 DETAILED METRICS ON DIFFERENT IMAGE TRANSFORMATIONS
On Fig. 8, evaluate the average R@ 1 over the 10k images from the reference dataset. The experimental setup is the same as for Tab. but higher number of transformation parameters
Rotation
Hue
Resize
Brightness
0.8
0.6
0.4
0.4
0.4
0.2
0.2
02
0.2
0.0
-50
0.0-0.0
0.0
-0.25
0.00
0.25
0.50
00
0.2
3.4
0.6
3.8
1.0
0.0
0.5
1.0
1.5
Ipeg
Center
Blur
Contrast
1.0
0.8
0.8
0.8
0.6
0.6
0.6
0.6
0.9
6.4
0.6
0.4
0.2
02
02
0.2
0.0
20
40
60
RG
100
0.0
02
0.4
2.6
0.8
3.0
0.0
20
0.0
0.5
1.0
1.5
Passive
Artive
Figure 8: Average R@l comparison between active passive indexing with IVF-PQ.
16

---

Published as a conference paper at ICLR 2023
evaluated. As expected, the higher the strength of the transformation, the lower the retrieval performance is. The decrease in performance is significantly reduced with activated images.
C.2 ADDITIONAL ABLATIONS
Speeding-up the optimization. In our experiments, the optimization is done using 10 iterations of gradient descent, which takes approximately 40ms/image. If the indexation time is important (often, this is not the case and only the search time is), it can be reduced at the cost of some accuracy.
We activated 10k reference images, with the same IVF-PQ indexed presented in Sec. 5.2 with only one step of gradient descent with higher learning rate. Activation times are computed on average. The @@ 1 results in Tab. 5 indicate that the speed-up in the image optimization has small cost in retrieval accuracy. Specifically, reduces the R@@ for unedited images. The reason that the learning rate is too high: it can cause the representation to be pushed too far and to leave the indexing cell. This is why a higher number number of steps and a lower learning rate are used in practice. activation time is a bottleneck, can however be useful to use less optimization steps.
Table 5: R@ for different transformations applied before search, with either step at learning rate 10, or 10 steps at learning rate 1. Results are averaged on 10k images.
Actvvttin
Identity
Couur. 0.5
Contr 2.0
Bright 0,5
Bright. 2.0
Mue 02
Blur 2.0
JPBG 50
Rot 25
Rot. 90
Crop
Resi. 0.5
Meme
Random
Avg.
Passive
1.00 0.73 0.39
0.73
0.28
0.62
0.48
0,72
0.07
0.14
0.14
0.72
0.14
0.13
0.45
Adam.Ir=1 10 steps
39.8 msfing
1.00
1.00
0.96
0.92
1.00
0.96
0.99
0.10
0.50
0.29
1.00
0.43
0.32
0.75
Adam,Ir=10- step
ms/ing
0.92
0.84
0.99
0.95
0.99
0.10
0.27
0.72
Data augmentation at indexing time and EoT. Expectation over Transformations (Athalye et al., 2018) was originally designed create adversarial attacks robust set of image transformations. We follow a similar approach to improve robustness of the marked image against a set of augmenta- tions T. At each optimization step, we randomly sample A augmentations {t, J in T and consider the average loss: & = Σ C4,,,;; /oA In our experiments, T encompasses rotations, Gaus- sian blurs, color jitters and differentiable approximation of the JPEG compression Shin & Song (2017). A is set to 8 and we always take the un-augmented image in the chosen set of augmentations.
We activated 10k reference images, with the same IVF-PQ Sec. 5.2 with without using EoT. Table 6 shows the average R@1 performance over the images submitted to different transformations before search. BoT brings small improvement, specifically on transformations where base perfor- mance is low (e.g. rotation or crops here). However, it comes at a higher computational cost since each gradient descent iteration needs A passes through the network, and since fewer images can be jointly activated due to GPU memory limitations (we need to store and mnok-propagate through A transformations for every image). If the time needed to index or activate an image not a bottleneck, using EoT can therefore be useful. Otherwise, it is not worth the computational cost.
Table 6: R@ 1 for different transformations applied before search, with or without EoT when activating the images. Results are averaged on 10k images,
Activation
Identity
Conte 0.5
Contr: 2.0
Bright. 0.5
Bright. 2.0
Hue 0,2
Blur 2.0
JPE 50
Rat. 25
Rot 90
Crop 0.5
Resi 0.5
Mome
Random
Avg.
Without BOT
40
1.00
1.00
0.96
1.00
0.92
1.00
0.96
0.99
0.10
0.50
0.29
1.00
0.43
0.32
0.75
With EOT
870 ms
1.00
1.00
0.95
1.00
0.92
1.00
0.95
0.99
0.14
0.64
0.33
1.00
0.45
0.33
0.76
17

---

Published as a conference paper at ICLR 2023
D ACTIVE INDEXING VS. WATERMARKING
Discussion. Watermarking and active indexing both modify images for tracing and authentication, however there are significant differences between them. Watermarking embeds arbitrary information into the image. The information can be a message, a copyright, a user ID, etc. In contrast, active improve the efficiency of the search engine. Watermarking also focuses on the control over the False Positive Rate of copyright detection, i.e. a bound on the probability that a random image has the same message as the watermarked one (up to a certain distance).
indexing modifies
Although watermarking considers different settings than indexing methods, it could also be lever- aged to facilitate the re-identification of near-duplicate images. In this supplemental section, we consider it to address a use-case similar to the one we address in this paper with our active indexing approach. In this scenario, the watermark encoder embeds binary identifiers into database images. The decoded identifier is then directly mapped to the image (as the index of a list of images).
Experimental setup. In the rest of the section, we compare active indexing against recent water- marking techniques based on deep learning.
For indexing, we use the same setting as in Sec. 5.1 (IVF-PQ index with 1M reference images). When searching for image, look up the closest neighbor with the help of the index.
For watermarking, we encode 20-bit messages into images, which allows to represent 2 10 images (the number of reference images). When searching for an image, we use the watermark decoder to get back an identifier and the corresponding image the database.
Like before, we use R@1 as evaluation metric. For indexing, it corresponds to the accuracy of the top-1 search result. For watermarking, the R@ 1 also corresponds to the word accuracy of the decoding, that is the proportion of images where the message is perfectly decoded. Indeed, with 20t encoding almost all messages have an associated image in the reference set, so an error on a single bit causes mis-identification (there is no error correction)) )
We use two state-of-the watermarking methods based on deep learning: SSL Watermarking (Fer- nandez al., 2022), which also uses an adversarial-lin optimization to embed messages, and HiD- DeN (Zhu et al., 2018), which encodes and decodes messages thanks to Conv-BN-ReLU networks. The only difference with the original methods is that their perturbation ō is modulated by the hand- crafted perceptual attenuation model presented in App. A. This approximately gives the same image quality, thereby allowing for direct comparison between active indexing and watermarking.
Results. Tab. 7 compares the R@1 1 when different transformations are applied before search or decoding. Our active indexing method overall the best by a large margin. For some transforma- tions, watermarking methods are not effective as passive indexing, yet for some others, like crops for HiDDeN, the watermarks are more robust.
Table 7: R@ 1 for different transformations applied before scarch, when using either watermarking active indexing. Results are averaged Ik images. Best result is in bold and second best in italic.
Identity
Contr. 0.5
Contr: 2.0
Bright. 0.5
Bright, 2.0
Hue 0.2
Blur 2.0
JPEG 50
Rat. 25
Rot. 90
Crop 0.5
Resi, 0.5
Mome
Random
Avg.
Passive indexing
1.00
0.73
0.39
0.73
0.28
0.62
0.48
0.72
0.07
0.14
0.14
0.72
0.14
0.13
0.45
Active indcxing (ours)
1.00
1.00
0.96
1.00
0.92
1.00
0.96
0.99
0.10
0.50
0.29
1.00
0.43
0.32
0.75
SSL Watermarking
1.00
0.98
0.53
0.98
0.63
0.85
0.13
0.00
0.00
0.15
0.11
0.00
0.46
0.07
0.42
(Femmnndez et al., 2022)
HiDDeN
0.94
0.87 0.36 0.85
0.55 0.00 0.81 0.00 0.00 0.00 0.92 0.44 0.77 0.16 0.48
(Zhu et al., 2018)
'In order to provide error correction capabilities, one needs longer messages. This makes it more difficult to insert bits: in our experiments, with 64 bits we observe drastic increase of the watermarking bit error rate. reported other papers from the literature, results of the original paper are hard to reproduce. Therefore to make it work better, our model is trained on higher resolution images (224x 224). with a payload of 20-bits. instead of 30 bits embedded into 128 x 128. Afterwards. the same network is used on images of arbitrary resolutions, to predict the image distortion which is later rescaled as in Eq. (8). In this setting the watermark can not always be inserted (6% failure).
"Our implementation.
18

---

Published as a conference paper at ICLR 2023
E MORE QUALITATIVE RESULTS
Figure 10 gives more examples of activated images from the DISC dataset, using the same parame- ters as in Sec. 5.2. The perturbation is very hard to notice (if not invisible), even in flat areas of the images because the perceptual model focuses on textures. We also see that the perturbation forms a regular pattern. This due to the image (bilinear) resize that happens before feature extraction.
Figure 9 gives example of an image activated at several values of perturbation strength a of Eq. (8) (for instance, for a 20 the image has PSNR 27dB and for a 1 the image has PSNR 49dB). The higher the the more visible the perturbation induced by the activation is. Nevertheless, even with low PSNR values (< 35dB), it hard to notice if an image is activated or not.
2020. PSNR-27dB
15.00 PSNR-29dB
1-1,. PSNR-31dB
8.43, PSNR-340B 6.32, PSNR-36dB
a-4.7, PSNR-39dB
3=5.5,, PSNR-4IdB
2-6.66 PSNR
2-2.0 PSNR-46dB
a- 1.500 PSNR-49dB
Figure 9: Example of one activated image at different levels of a.
Figure 10: Example of activated images for 3.0. (Left) original images, (Middle) activated images, (Right) pixel-wise difference.
19

---

Published as a conference paper at ICLR 2023
Figure 10: Example of activated images for = 3.0. (Left) original images, (Middle) activated images, (Right)
pixel-wise difference. Images are R000005. jpg, R000045.jpg, R000076.jpg, R000172.jpg and R000396.jpg.
20


Reviews:
[{"review_id": "b8ORUkzYe3", "reviewer": "Reviewer_gK1Y", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper addresses the problem of image copy detection through image retrieval and proposes a method that optimizes jointly both representation learning and approximate similarity search. Large databases rely on indexes for approximate and efficient search, but the challenge comes when copies are significantly modified and then its representation switches indexing partitions and the closest neighbour cannot be returned. The authors address this problem by modifying the image in a way that its representation is pushed towards the center of the indexing partition, giving more \"room\" for edited images to still fall in the same partition as the original copy. This is done by minimizing and indextation loss back to the image pixels, assuming that the image can be modified before its release.", "summary_of_the_review": "The paper is well written, the idea is both interesting and novel, and the experimental analysis is extensive and thorough. The method has a few limitations (not only the adversial attacks discussed by the authors but the fact that images need to be activated before its release) but they do not have a large impact on my positive opinion towards the paper.", "strength_and_weaknesses": "Pros:\n- The paper is very well written and very well motivated by pointing out the challenges that indexing faces regarding copy detection and proposing a solution for it.\n- I really like the main idea proposed in this paper of imperceptibly modifying the image in order to push its represnation to the center of the indexing partition to make it more indexing friendly.\n- The paper contains a thorough experimental analaysis that justifies the authors claims and shows that their method significantly improves retrieval performance on a number of neural extractors and indexing structures\n\n\nCons:\n- The fact that the image has to be modified before its release is a big limitation since it cannot be applied to already existing databases where images have already been released. However, it can still be applied to new images released in the same database, benefiting all future releases.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JRPCQsCO8SG", "reviewer": "Reviewer_pXd8", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an active image indexing approach that allows image copy detection. The proposed solution applies some perturbations to the images during the indexing process which improves similarity search on large databases. Those perturbations are claimed to be invisible to the human eye. Evaluations on the DISC21 dataset showed that active indexing outperforms passive indexing and that the method generalizes to other feature extractors and other indexes.\n", "summary_of_the_review": "I like the proposed approach in this paper and the quality of the writing. The contributions of this paper are incremental and the paper lacks some comparison to some state of the art approaches for image copy detection (needed from my perspective to add credibility to how effective active indexing is). I also have some concerns regarding how useful such an approach is for real world applications, and even for solving research related problems. ", "strength_and_weaknesses": "Strengths of the paper:\n- Active indexing outperforms passive indexing by applying some perturbations on the original image before publication\n- Active indexing seems to generalize well with other feature extractors and other indexes\n\nAreas where the paper could improve:\n- The paper lacks some comparison to state of the art approaches for image copy detection\n- The queries used in the evaluation are not the queries from DISC21 dataset since the authors needed to control the image transformations. This makes the obtained results not comparable to other work using the same dataset. In my opinion a fair comparison to other existing work is needed to validate this approach, specifically that the contributions are incremental. \n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Bpw8EOHesh-", "reviewer": "Reviewer_DYH1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper addresses the issue of image copy detection and retrieval from large databases by incorporating a robust feature extractor and a scalable, efficient search algorithm. The feature extractor is a neural network that has been trained to extract descriptors that are robust to various transformations. The search algorithm is based on the Inverted Files Method followed by a Product Quantization technique to compress the images’ descriptors and significantly reduce retrieval time. Experiments are provided to demonstrate the applicability of the method.", "summary_of_the_review": "The paper marginally passes the acceptance threshold since the method assumes that the images can be processed before extracting the descriptors and defining the indexing structure. Consequently, the system cannot be used as a black box. Although the search speed is reduced considerably, the retrieval performance results are inconclusive. For example, in the case of IVF 512, PQ 32X8, the performance boost of active indexing is negligible.", "strength_and_weaknesses": "Pros:\n•\tThe paper is well-written and coherent, and the methods' explanation is illustrative.\n•\tThe experimental results showcase that the proposed method significantly reduces the time complexity of searching similar images without compromising the retrieval accuracy.\n•\tActive indexing can be used selectively in the database to enhance retrieval performance on important images without affecting the performance of passive indexing.\nCons:\n•\tThis method assumes that the images can be processed before extracting the descriptors and defining the indexing structure, and consequently, the system cannot be used as a black box.\n•\tAlthough the search speed is reduced considerably, the retrieval performance results are inconclusive. For example, in the case of IVF 512, PQ 32X8, the performance boost of active indexing is negligible.\n•\tThe neural network training time for active indexing is time-consuming.\n\nMinor details:\n•\tDifferent symbols could be used in equation (4), as the symbol x can be confusing because it refers to both the originally extracted descriptor and in the new split vector.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster