Title: MultiGrain: a unified image embedding for classes and instances

Abstract: We introduce MultiGrain, a neural network architecture that generates compact image embedding vectors that solve multiple tasks of different granularity: class, instance, and copy recognition. MultiGrain is trained jointly for classification by optimizing the cross-entropy loss and for instance/copy recognition by optimizing a self-supervised ranking loss. The self-supervised loss only uses data augmentation and thus does not require additional labels. Remarkably, the unified embeddings are not only much more compact than using several specialized embeddings, but they also have the same or better accuracy. When fed to a linear classifier, MultiGrain using ResNet-50 achieves 79.4% top-1 accuracy on ImageNet, a +1.8% absolute improvement over the the current state-of-the-art AutoAugment method. The same embeddings perform on par with state-of-the-art instance retrieval with images of moderate resolution. An ablation study shows that our approach benefits from the self-supervision, the pooling method and the mini-batches with repeated augmentations of the same image.

1. Introduction:

Image recognition is central to computer vision, with dozens of new approaches being proposed every year, each optimized for particular aspects of the problem. From coarse to fine, we may distinguish the recognition of (a) classes, where one looks for a certain type of object regardless of intra-class variations, (b) instances, where one looks for a particular object despite changes in the viewing conditions, and (c) copies, where one looks for a copy of a specific image despite edits. While these problems are in many ways similar, the standard practice is to use specialized, and thus incompatible, image representations for each case.

Consider for example image retrieval, where the goal is to match a query image to a large database of other images, whose applications include detection of copyrighted images and exemplar-based recognition of unseen objects. Often one would like to search the same collection with multiple granularities, by matching the query by class, instance, or copy. Adopting multiple image embeddings, narrowly optimized for each granularity, means multiplying the resource usage. Using a single embedding relevant to all these tasks reduces both the computing time and the storage space. However, this might come at the cost of a reduced accuracy.

In this paper we introduce MultiGrain, a compact embedding that, as illustrated in fig. 1 , can solve recognition tasks of different granularities while maintaining or surpassing the accuracy of specialized embeddings. MultiGrain is obtained by training a Convolutional Neural Network (CNN) jointly on the different tasks. CNNs trained for image classification are known to be good universal features extractors. However, authors (Babenko & Lempitsky, 2015) have noted that the intermediate layers of such CNNs are generally better for low-level tasks such as instance and copy recognition. In contrast, our work extracts a single global embedding at the top of the network. The key is to optimize this embedding simultaneously for classification and instance retrieval. In this manner, the same representation integrates different degrees of invariance. Indeed, by definition, copies of the same image contain the same instance, and images that contain the same instance also contain the same class. 

`class:

As an additional contribution, we show that MultiGrain can be learned using only class-level labels via self-supervised learning (Caron et al., 2018) . The instance recognition is learned for free, without labels specific to instance recognition: we use the identity of arbitrary images as labels, and data augmentation to generate different versions of each image. We also find that, unexpectedly, forming batches with multiple augmentations of the same image, improves the classifier performance, even for models trained only for classification. This contradicts the common knowledge that training batches should maximize diversity. Finally, we incorporate in MultiGrain a pooling layer inspired by image retrieval that boosts the classification accuracy for high-resolution images.

Overall, MultiGrain offers compelling performance both for classification and image retrieval, including outperforming the SoTA classification accuracy on ImageNet for ResNet-50. Image search. The objective of Image search is to find the images most similar to the query in a large image collection. It is usually evaluated for more specific problems such as landmark recognition (Philbin et al., 2007; Jégou et al., 2008) , particular object recognition (Nister & Stewenius, 2006) or copy detection (Douze et al., 2009) . In this paper image retrieval refers to instance-level retrieval, where object instances are as broad as possible, i.e., not restricted to buildings, as in the Oxford/Paris benchmark. Typically, a query image is described by an embedding vector, and the task amounts to searching the nearest neighbors of this vector in the embedding space. Refinement steps include as geometric verification ( Architectures for instance search are regular classification trunks, modified so the pooling stage gives more spatial locality, to cope with small objects and clutter. A competitive baseline for instance retrieval is the R-MAC image descriptor (Tolias et al., 2015) . It aggregates regionally pooled features extracted from an activation map. This pooling combined with PCA whitening (Jégou & Chum, 2012) leads to efficient many-to-many comparisons between image regions. Gordo et al. (2016; 2017) fine-tune this representation end-to-end on an external image retrieval dataset. Unlike their approach, we do not assume in this work that we have a domain-specific training set. Radenović et al. (2018) depart from regional pooling by adopting a generalized mean pooling (see section 3.2). It is a spatial pooling of the features raised to an exponent p over the whole image, which offers some benefits as analyzed by Boureau et al. (2010) with respect to noise-to-signal ratio and in simple image classification tasks.

Multi-task training stems from the observation that CNNs transfer to a wide range of vision tasks (Razavian et al., 2014) and exhibit a high level of compressibility (Han et al., 2015) . Despite some successes with multi-task networks such as UberNet (Kokkinos, 2017) , their design and training still involve numerous heuristics. Ongoing lines of work investigate efficient sharing of parameters (Rebuffi et al., 2018), and proper hyper-parameters settings to weight the gradients from different tasks (Guo et al., 2018) .

Data augmentation improves generalization and reduces over-fitting (Krizhevsky et al., 2012) . Traditionally, batches were made to contain random samples of the training set. The recently introduced batch augmented (BA) (Hoffer et al., 2019) sampling strategy consists in augmenting the size of the batches and filling them with data-augmented copies of the same image. This yields better generalization performance, and uses computing resources more efficiently through reduced data processing time. We show that this improvement can be obtained using the same batch size, i.e., , with a lower number of distinct images per batch. We see this repeated augmentations (RA) scheme as a way to boost the effect of data augmentation over the course of the optimization. Thus, RA is a technique of general interest, beyond large-scale distributed training.

3. Architecture design:

In the current best practices, the architectures and training procedures used for class and instance recognition differ significantly. This section describes the differences, summarized in table 1, and our solutions to bridge them, leading to the MultiGrain architecture in fig. 1 .

3.1. Training objective:

MultiGrain is jointly optimized for the classification and retrieval tasks, which is obtained by combining a classification loss and an instance retrieval loss in the optimization.

Classification loss. We adopt the standard cross-entropy loss. Given e i ∈ R d the output of eq. ( 4) for image i, w c ∈ R d the parameters of a linear classifier1 for class c = 1, . . . , C, and y i the ground-truth class for that image, then

EQUATION

)

Retrieval loss. The triplet loss (Schroff et al., 2015) imposes that a query image embedding must be closer to the embedding of an image that matches it than to other embeddings. The contrastive loss (Hadsell et al., 2006) imposes a stricter condition: all embedding distances between pairs of matching images must be smaller than all embedding distances between pairs of non-matching images. Optimizing both these losses depends on hard-to-tune hyper-parameters, including how pairs and triplets are sampled.

Some of these issues are solved by the approach of Wu et al. ( 2017), which starts from a batch of images and (1) normalizes their embeddings to the unit sphere, (2) samples negative pairs using the current embedding similarity, and (3) uses the pairs in a margin loss (that combines contrastive and triplet loss). Given images i, j ∈ B in a batch, the margin loss is given by: retr (e i , e j , β,

y ij ) = max{0, α + y ij (D(e i , e j ) -β)} (2)

where D(e i , e j ) = ei / ei -ej / ej is the Euclidean distance between the normalized embeddings, the label y ij is equal to +1 if the images match and to -1 otherwise, α > 0 is the margin (a hyper-parameter), and β > 0 is a learnable parameter controlling the volume of the space occupied embedding vectors. Due to the normalization, D(e i , e j ) is equivalent to a cosine similarity, which, up to whitening (section 3.4), is commonly used in retrieval.

We use distance-weighted sampling to sample pairs of images (see appendix A for details). This sampling is suited to our joint training: it tolerates relatively small batch sizes (|B| ∼ 80 to 120) and a small amount of positives images (3 to 5) of each instance in the batch, without the need for elaborate parameter tuning or offline sampling.

Joint loss. The joint loss on batch B is a combination weighted by a factor λ ∈ [0, 1]:

λ |B| • i∈B class (e i , w, y i ) + 1 -λ |P(B)| • (i,j)∈P(B)

retr (e i , e j , β, y ij ).

Note that the losses are normalized by the number of items in the corresponding summations.

3.2. Spatial pooling operators:

For recognition tasks, one requires to encode the whole image as a single vector. The latter is usually obtained by applying a global spatial pooling operator to the 3D activation tensor produced by the convolutional trunk of the network. This should be contrasted with local pooling operators, typically max pooling, that are found throughout the layers of CNNs to achieve local invariance to small translations.

The choice of global pooling operator has a significant effect on the representation. Recent architectures for classification, such as ResNet and DenseNet, use average pooling. Average pooling is permutation invariant and hence less sensitive to geometric transformations. It is also flexible as it allows the model to be applied to images of any size.

Image retrieval, on the other hand, requires more localized and fine-grained geometric information than the one captured by average pooling. This is because (i) the representation requires less invariance since object instances and landmarks are visually more similar and (ii) images are often more cluttered, with just a small distinctive part that warrants identification. Hence, the pooling should preserve local information. Next, we discuss the generalized mean pooling operator as a solution to this problem.

Let x ∈ R C×W ×H be the feature tensor computed by a convolutional neural network for a given input image. The tensor represents a feature map with C channels, height H and width W . Let u ∈ Ω = {1, . . . , H} × {1, . . . , W } be "pixel" in the map, c the channel, and by x cu the tensor element at location u and channel c, so that x = [x cu ] c=1,...,C,u∈Ω . The generalized mean pooling (GeM) layer computes the generalized mean of each channel:

EQUATION

where the exponent p > 0 is a parameter. Average pooling and max pooling are equivalent to GeM with p = 1, and p = ∞, respectively. Exponents in the range For a given learning rate, RA has lower performance than the standard i.i.d. scheme for small batch sizes, but outperforms it with larger batches. This is different from the observation of (Hoffer et al., 2019) , who also consider repeated samples in a batch, but simultaneously increase its size.

With standard sampling, two versions of the same image are seen only in different epochs.

We conjecture that correlated RA samples facilitate learning features that are invariant to the only difference between the repeated images -the augmentations. Appendix D shows this phenomenon in a simple artificial setting.

3.4. PCA whitening:

We apply a step of PCA whitening to the embeddings to use them for retrieval, in accordance with previous works (Gordo et al., 2017; Jégou & Chum, 2012) . The Euclidean distance between transformed features is equivalent to the Mahalanobis distance between the input descriptors. The PCA is trained at the end of the CNN training, using an external dataset of unlabelled images. The whitening operation Φ can be written as Φ(e) = S ( e / e -µ) given the whitening matrix S and centering vector µ.

The parameters of the classification layer have to be modified to take the whitened embeddings as input. For the classifier w c , e of eq. ( 1 

3.5. Input sizes:

In image classification, it is standard to resize and center-crop input images to a low resolution, e.g. 224 × 224 pixels (Krizhevsky et al., 2012) . The benefits are a smaller memory footprint, faster inference, and the possibility of batching the inputs if they are cropped to a common size. On the other hand, image retrieval depends on finer details in the images, as an instance can small or seen under a variety of scales. Feature extractors for image retrieval therefore commonly use input sizes of 800 (Gordo et al., 2017) or 1024 (Radenović et al., 2018) pixels, without cropping the image to a square. This is impractical for end-to-end training.

We train MultiGrain at the standard 224 × 224 resolution, and use larger resolutions at test time. Indeed, a network trained with a pooling exponent p and resolution s can be evaluated at a larger resolution s * > s using a larger pooling exponent p * > p, see section 4.3. The optimal p * obtained on IN-aug is a trade-off between retrieval and classification. Experimentally, we observed that other choices are suitable for setting this parameter: fine-tuning the p * using training inputs at a given resolution and back-propagating the cross-entropy loss provides similar results and values of p * (but is more complex).

4 Experiments and Results We now use larger images at test time, i.e., , we set s * > 224 and vary the exponents p * = p = 3. Figures 2a and 2b show the classification accuracy and the retrieval accuracy at different resolutions, for different values of the exponent p * . As expected, at s * = 224, the pooling exponent yielding best accuracy in classification is the exponent with which the network has been trained, p * = 3; instead, testing at larger scale requires an exponent p * > p, both for classification and for retrieval. In the following, we adopt the values obtained by our cross-validation on IN-aug, see section 3.5.

We defer to appendix C for the study on the weighting parameter λ. We set λ = 0.5 in our following experiments, as it gives the best classification accuracy at the practical resolutions s * = 224 and 500 pixels. As a reference, we also report a few results with λ = 1 (i.e., ignoring the retrieval loss).

4.3. Classification results:

From now on, our MultiGrain nets are trained at resolution s = 224 with exponent p = 1 or p = 3 in the GeM pooling. For each evaluation resolutions s * = 224, 500, 800, the same exponent p * is selected according to section 3.5, yielding a single embedding for classification and for retrieval. Table 2 presents the classification results. There is a large improvement in classification performance from our baseline ResNet-50 with p = 1, s = 224, "full" data augmentation (76.2% top-1 accuracy), to a MultiGrain model at p = 3, λ = 0.5, s = 500 (78.6% top-1). We identify four sources for this improvement:

1. The RA batch sampling (section 3.3) yields an improvement of +0.6% (p = 1).

2. The retrieval loss helps the generalizing effect of data augmentation: +0.2% (p = 1).

3. p = 3 pooling: GeM at training (section 3.2) allows the margin loss to have a much stronger effect thanks to increased localization of the features: +0.4%.

4. Expanding resolution: evaluating at resolution 500 adds +1.2% to the p = 3 MultiGrain network, reaching the 78.6 top-1 accuracy. The p = 3 training yields sparser features, more generalizable over different resolutions, and the p * pooling adaptation (without it the performance at this resolution is only 78.0%).

The p * selection for evaluation at higher resolutions has its limits: at 800 pixels, due to the large discrepancy between the training and testing scale for the feature extractor, the accuracy drops to 77.2% (76.2% without the p * adaptation).

AutoAugment (AA) is a reinforcement learning approach to find data augmentations that improve the accuracy of CNNs (Cubuk et al., 2018). We integrate the augmentations found on their ResNet-50 model. To give more impact to AA, we do 270 passes over the dataset, with batch size 512. MultiGrain with AA reaches 78.2% top-1 accuracy at s * = 224 (p = 3, λ = 0.5). To the best of our knowledge, this is the state-of-the-art for ResNet-50 when evaluating at this resolution: it outperforms AA alone (77.6%) and mixup (Zhang et al., 2018) (76.7%). Increasing the test resolution improves the accuracy to 79.4% at s * = 500.

We also experimented with other architecture, see appendix G. We observed that the GeM pooling substantially increase the accuracy of off-the-shelf networks, with only a tiny fine-Architecture λ data resol. train-time pooling aug.

s * p = 1 p = 3 tuning. For example, we obtain a top-1 accuracy of 83.6 with a PNASNet-5-Large, a +0.9% improvement over the original (Liu et al., 2018) .

4.4. Retrieval results:

Retrieval results are in table 3 , an ablation study and copy detection results are in the F. Our MultiGrain nets improve accuracies on all datasets with respect to the ResNet-50 baseline for comparable resolutions. Repeated augmentations (RA) is again a key ingredient in this context. We compare with reported accuracies in (Gordo et al., 2016; 2017) , without additional training data. MultiGrain compares favorably with their results at the same resolution (s * = 800). They reach accuracies above 93% mAP on Holidays but this requires a resolution s ≥ 1000 pixels.

Note that MultiGrain reaches a reasonable retrieval performance at resolution s * = 500, an interesting operating point compared to the traditional inference resolutions s = 800-1000 for retrieval. Indeed, a forward pass of ResNet-50 on 16 processor cores takes 3.80s at resolution 500, against 18.9s at resolution 1024 (5× slower). Because of this quadratic increase in timing, and the single embedding computed by MultiGrain, our solution is particularly apt in large-scale or low-resource vision applications. At resolutions 500 the results with margin loss (λ = 0.5) are slightly lower than without (λ =1). This is partly due to the limited transfer from the IN-aug task to the variations observed in retrieval datasets.

5. Conclusion:

MultiGrain is a unified embedding for image classification and instance retrieval. It relies on a classical CNN trunk, with a GeM pooling layer, topped with two heads at training time.

We have discovered that this pooling layer allows us to increase the resolution of images used at inference time, while maintaining a small resolution at training time. We have shown that MultiGrain embeddings can perform well on classification and retrieval. Interestingly, MultiGrain also sets a new state of the art on pure classification compared to all results obtained with the same convolutional trunk. Our approach will be open-sourced. 

D Data-augmented batches: toy model:

We have observed in section 3.3 and appendix C that training our architecture (ResNet-50 trunk) with data-augmented batches yields improvements with respect to the vanilla uniform sampling scheme, despite the decrease in image diversity. This observation holds even in the absence of ranking triplet loss, all things being equal otherwise: same number of iterations per epoch, number of epochs, learning rate schedule, and batch size. As an example, fig. D.1 shows the evolution of the validation accuracy of our network trained under cross-entropy with our training schedule and a p = 1 pooling, batches of size 512, with the data augmentation introduced in section 4.1, with uniform batches vs. with batch sampling. While initial epochs suffer from the reduced diversity of the batches compared to the uniformly-sampled variant, the reinforced effect on data augmentation compensates for this in the long run, and makes the batch-augmented variant reach a higher final accuracy.

Since we observe this better performance even for a pure image classification task, an interesting question is whether this benefit is specific to our architecture and training method (batch-norm, etc), or if it is more generally applicable? Hereafter we analyse a linear model and synthetic classification task that seems to align with the second hypothesis.

We consider an idealized model of the effect of including different data-augmented instances of the same image in one batch using standard stochastic gradient descent. We create a synthetic training set D of points pictured in fig. D.2 of N = 100 positive and N = 100 negative training points p i = (p i x , p i y ) by sampling from two 2D Gaussian distributions:

p i x ∼ N (µ = 0, σ = 1) p i y ∼ N (µ = y * i , σ = 1) (D.1)

with y * i = ±1 being the ground truth label. We sample a test dataset in the same manner. 

= max (1 -y * i f w (p i ), 0). (D.3)

We consider the symmetry across the x-axis

φ((p i x , p i y )) = φ((p i x , -p i y )) (D.4)

as a label-preserving data-augmentation suited to our synthetic dataset. We train the SVM (equation D.2) using one pass through the data-augmented dataset D of size 4N , using batches of size 2.

The only difference between the two optimization schedules is the order in which the samples are batched and presented to the optimizer. We consider two batch sampling strategies:

• Uniform sampling: we sample the elements of the batch randomly from D, without replacement; • Paired sampling: we generate a batch by pairing a random element from D and its data-augmentation, removing these two elements from D. This idealized experiment demonstrates that there are cases in which the repeated augmentation scheme provides an optimization and generalization boost, and reinforces the effect of data augmentation. As already reported in the main paper, for some datasets the choice of not using the triplet loss (λ = 1) is as good or better than our generic choice (λ = 0.5). Of course, then the embedding is not multi-purpose anymore. Overall, the different elements employed in our architecture (RA and the layers specific to Multigrain) still give a significant improvement over simply using the activations, and is competitive with the state of the art for the same resolution/complexity.

E Margin loss hyper-parameters:

Note, the AutoAugment data augmentation does not transfer well to the retrieval tasks. This can be explained by their specificity to Imagenet classification. This shows the limitation of a particular choice of data-augmentation if a single embedding for classification and retrieval datasets is desired. Learning AutoAugment specifically for the retrieval task would certainly help, but would probably also result in less general embeddings. Hence, data-augmentation is a limiting factor for multi-purpose embeddings: improving for one task like classification hurts the performance for other tasks. 

:

In practice, we also add a learnable bias term, which is equivalent to adding a constant feature channel to the feature vector ei.

Url: https://github.com/Cadene/pretrained-models.pytorch

Figures and Tables:

Figure 1: Top: Our goal is to extract an image descriptor incorporating different levels of granularity, so that we can solve, classification and particular object recognition tasks: The descriptor is either fed to a linear classifier, or directly compared with cosine similarity. Right: The MultiGrain architecture.

), we have w c , e = w c , Φ -1 (Φ(e)) = e ( w c , Φ(e) + b c ) where w c = S -w c and b c = w c , µ are the modified weight and bias for class c. We observed that inducing decorrelation via a loss(Cogswell et al., 2016) is insufficient to ensure that features generalize well, which concurs with prior works(Gordo et al., 2017;Radenović et al., 2018).

Proxy task to cross-validate p * . To select the exponent p * , suitable for all tasks, we create a synthetic retrieval task IN-aug: we sample 2,000 images from the training set of ImageNet, 2 per class, and create 5 augmented copies of each of them. We query all images using the retrieval embeddings and evaluate the retrieval accuracy on IN-aug by measuring how many of the first 5 augmentations of the image are ranked in top 5 positions. The best-performing p * ∈ {1, 2, . . . , 10} on IN-aug is shown in the table.

Figure 2: Retrieval and classification accuracies as a function of pooling exponent p * and the image resolution. At training time, the pooling was p = 3. Note the clear interaction between the resolution s * and the pooling exponent p * .

Figure B.1: An off-the-shelf ResNet-50 reacts strongly on channel 909 of the last activation map for class "racing car". The image on the left is a hard example for the class. We show channel 909 for that image, at several resolutions and with GeM parameters p * = 1 and p * = 3. In the low resolution version, the cars are too small to be visible individually on the activation map. In the full resolution version, the location of the cars is more clear. In addition, p * = 3 reduces the noisy detections relative to the true locations.

Figure D.1: Evolution of the validation accuracy on ImageNet-val with and without dataaugmented batches.

Figure D.2: Training set for the toy model in appendix D.

Figure D.3: Evolution of the test accuracy of the SVM trained on the synthetic data, averaged accross 100 runs.

Figure D.3 shows the evaluation of the accuracy with the iterations in both of these cases, averaged across 100 runs. It is clear that pairing the data-augmented pairs in one batch accelerates the convergence of this model.

Differences

Table content: <table><tr><td>between classifica-</td></tr><tr><td>tion and image retrieval: Retrieval archi-</td></tr><tr><td>tectures incorporate a final pooling layer</td></tr><tr><td>that is regionalized (RMAC of Tolias &amp;</td></tr><tr><td>Jégou (2014)) or magnifies activations</td></tr><tr><td>(GeM of Radenović et al. (2018)). The</td></tr><tr><td>triplet loss (Gordo et al., 2016) requires a</td></tr><tr><td>batching strategy with pairs of matching</td></tr><tr><td>images.</td></tr></table>

1 < p < ∞ are a trade-off between the two(Bo & Sminchisescu, 2009;Boureau et al., 2010;Dollár et al., 2009). GeM was introduced for image retrieval as a component of R-MAC that approximates max pooling(Dollár et al., 2009), but(Radenović et al., 2018) showed it is competitive on its own.(Boureau et al., 2010) studied this layer in the context of scene recognition/image classification. MultiGrain uses it to bridge the two worlds, as well as to dynamically adapt the network to varying image resolution. We introduce repeated augmentations, a sampling scheme for training with SGD and data augmentation. In RA we form an image batch B by sampling |B|/m different images, and transform them up to m times by a set of data augmentations. Thus, the instance level ground-truth y ij = +1 iff images i and j are two augmented versions of the same image. The key difference with the standard sampling scheme in SGD is that samples are not independent.

Table content: <table><tr><td>3.3 Batching with repeated augmentation (RA)</td></tr></table>

We build MultiGrain using ResNet-50 as convolutional trunk(He et al., 2016). The latter is optimized using SGD, starting with a learning rate of 0.2 which is reduced tenfold at epochs 30, 60, 90 for a total of 120 epochs (a standard setting(Paszke et al., 2017)). The batch size is |B| = 512 and an epoch "sees" a fixed number T = 5005 batches. With uniform sampling, one epoch does two passes over the training set; with RA and m = 3, one epoch sees ∼ 2/3 of the images of the training set. The baselines are trained using this longer schedule for a fair comparison.

Table content: <table><tr><td>Datasets. We train our networks on the ImageNet-2012 training set. Classification ac-</td></tr><tr><td>curacies are reported on the validation images. For image retrieval, we report the mean</td></tr><tr><td>average precision on the Holidays dataset (Jégou et al., 2008), with images rotated manually</td></tr><tr><td>when necessary, as in prior evaluations (Gordo et al., 2016). We also report the accuracy</td></tr><tr><td>on the UKB object recognition benchmark (Nister &amp; Stewenius, 2006), which shows 2,550</td></tr><tr><td>objects under 4 viewpoints each; each image is used as a query to find its 4 closest neigh-</td></tr><tr><td>bors in embedding space; the number of correct neighbors is averaged across all images</td></tr><tr><td>(i.e., the score is in [0, 4]). We report the performance of our network in a copy detection</td></tr><tr><td>setting, indicating the mean average precision on the "strong" subset of the Inria Copydays</td></tr><tr><td>dataset (Douze et al., 2009), combined with 10k distractor images randomly sampled from</td></tr><tr><td>YFCC100M (Thomee et al., 2016). We call the combination C10k. The PCA whitening</td></tr><tr><td>transformations are computed from the features of 20k images from YFCC100M, distinct</td></tr><tr><td>from the C10k distractors.</td></tr></table>



Table content: <table><tr><td>: ImageNet 2012 valida-</td></tr><tr><td>tion performance at top-1 / top-</td></tr><tr><td>5 accuracies (%). Resnet-50 is</td></tr><tr><td>a classification baseline trained</td></tr><tr><td>with cross-entropy with our train-</td></tr><tr><td>ing schedule, data augmenta-</td></tr><tr><td>tion, and uniform batch sampling.</td></tr><tr><td>MultiGrain uses the same Resnet-</td></tr><tr><td>50 trunk. At resolutions s  *  &gt; 224</td></tr><tr><td>we evaluate with exponent p  *  as</td></tr><tr><td>described in section 3.5. We com-</td></tr><tr><td>pare mixup (Zhang et al., 2018),</td></tr><tr><td>BA (Hoffer et al., 2019), and Au-</td></tr><tr><td>toAugment (Cubuk et al., 2018).</td></tr></table>

Instance search results and baselines, on Holidays (% mAP) and UKB (/4). We set p = 3 pooling at training time for our MultiGrain models, and p * set as given in section 3.5. We compare Fisher vectors(Jégou et al.,  2012), neural codes(Babenko et al., 2014), RMAC(Gordo et al., 2016), and GeM(Radenović et al., 2018). † GeM is fine-tuned at resolution 362×362 on additional retrieval data and uses multi-scale input processing at an extra cost.

Table content: <table/>



Table content: <table><tr><td colspan="2">.1: Margin loss and data-augmentation parameters</td></tr><tr><td>parameter</td><td>value</td></tr><tr><td>margin α</td><td>0.2</td></tr><tr><td>initial β 0</td><td>1.2</td></tr><tr><td colspan="2">β learning rate 0.1</td></tr><tr><td>We consider the SGD training of an SVM</td><td/></tr><tr><td colspan="2">f w (p i ) = w p i</td><td>(D.2)</td></tr><tr><td>using the Hinge loss</td><td/></tr><tr><td>hinge</td><td/></tr></table>

Table D.1: Full results including Copydays + 10k distractors (CD10k, % mAP), and ablation study for the MultiGrain models. The Pytorch model simply extract the last activation layer as a descriptor (Babenko et al., 2014). Resnet-50 corresponds to features extracted from a classification baseline with p = 1 or p = 3 GeM pooling, trained with cross-entropy with our training schedule, data augmentation, and uniform batch sampling.F Additional results and ablation study for Multigrain in retrievalTableD.1 reports additional results of the MultiGrain architecture, with an ablation study analyzing the effect of each component.

Table content: <table><tr><td colspan="6">Table E.2: full data-augmentation transforms and parameters</td></tr><tr><td/><td colspan="3">transformation</td><td colspan="2">parameter range</td></tr><tr><td/><td colspan="3">horizontal flip</td><td/></tr><tr><td/><td colspan="3">random resized crop</td><td colspan="2">scale ∈ [0.08, 1.0] ratio ∈ [3/4, 4/3]</td></tr><tr><td/><td/><td/><td/><td colspan="2">brightness 0.3</td></tr><tr><td/><td colspan="2">color jitter</td><td/><td colspan="2">contrast 0.3</td></tr><tr><td/><td/><td/><td/><td colspan="2">saturation 0.3</td></tr><tr><td/><td colspan="3">lighting transform</td><td colspan="2">intensity 0.1</td></tr><tr><td/><td/><td/><td colspan="2">Holidays</td><td>UKB</td><td>CD10k</td></tr><tr><td>Method</td><td>λ</td><td colspan="3">s  *  = 224 500 800</td><td>224 500 800</td><td>224 500 800</td></tr><tr><td>PyTorch model zoo</td><td/><td/><td colspan="2">85.5 86.6 82.8</td><td>3.71 3.85 3.80</td><td>61.5 61.1 43.0</td></tr><tr><td colspan="3">Resnet-50 trained with p = 1 pooling</td><td colspan="2">83.5 88.8 87.1</td><td>3.60 3.79 3.82</td><td>59.2 69.9 66.2</td></tr><tr><td colspan="3">Resnet-50 trained with p = 3 pooling</td><td colspan="2">86.8 90.0 90.4</td><td>3.73 3.87 3.89</td><td>70.6 78.9 75.7</td></tr><tr><td>MultiGrain</td><td>1</td><td/><td colspan="2">88.9 91.8 91.6</td><td>3.78 3.89 3.91</td><td>75.1 81.2 82.5</td></tr><tr><td>MultiGrain</td><td>0.5</td><td/><td colspan="2">88.3 91.5 92.5</td><td>3.78 3.90 3.91</td><td>74.1 80.7 78.6</td></tr><tr><td>MultiGrain + AA</td><td>0.5</td><td/><td colspan="2">86.5 90.3 89.4</td><td>3.75 3.89 3.90</td><td>69.7 77.8 76.1</td></tr></table>

1: Additional top-1/top-5 validation classification accuracies obtained by finetuning p * for higher evaluation scales from off-the-shelf networks: NASNet(Zoph et al., 2018), SENet(Hu et al., 2018) and PNASNet(Liu et al., 2018). The first column indicates the training resolution s and the accuracy we measured at this resolution, with standard evaluation (resize of the largest scale to s • 256/224 + center crop). The subsequent columns show the accuracy measured at higher resolutions s * = 350, 400, 450, 500 without cropping, together with the p * found by finetuning for these resolutions (appendix G).

Table content: <table><tr><td colspan="4">original evaluation s  NASNet-A-Mobile 224 74.1/91.7 1.7 75.1/92.5 2.1 74.2/92.1 2.4 71.8/90.9 2.6 68.4/89.0</td></tr><tr><td>SENet154</td><td>224</td><td>81.3/95.5</td><td>1.6 82.6/96.2 1.6 83.0/96.5 1.6 83.1/96.5 1.7 82.7/96.3</td></tr><tr><td colspan="2">PNASNet-5-Large 331</td><td>82.7/96.0</td><td>1.0 81.</td></tr></table>

Peer review (iclr2020):
The paper proposes to jointy train a deep neural net for image classification, instance and copy recognition. This is achieved by i) combining a classification (cross-entropy) loss with a retrieval loss based on multiple copies/augmentations of the training image, and ii) a generalized mean pooling (GeM) layer allowing to interpolate between average and max pooling. The network obtains improved image classification accuracy on ImageNet over the same model trained without additional losses, and retrieval results competitive with the state of the art. The paper is well-written and the different components are well explained. The improvements obtained on ImageNet are considerable, and I like the idea of using the GeM layer pooling exponent to adapt the network to higher resolutions. Also, it is desirable to have models that work well on multiple tasks e.g. for deployment on mobile devices. I have several questions and concerns: - Repeated augmentation together with a retrieval loss as the one presented in (2) is a known supervised learning technique called exemplar [1], which was originally proposed with a classification loss but recently extended to a to a triplet loss as the one in (2) in [2]. The same loss was also used together with labels for semi-supervised learning in [3], and similar improvements in image classification accuracy as in the proposed paper were observed for training a rotation-based self-supervised loss with all labels (the MOAM model in [3]). I think the paper would benefit from a review of recent work on self-supervised learning. - It was previously observed that increasing the resolution can improve image classification accuracy, see, e.g. in [5, 4]. It is also to be expected that adding AutoAugment to the mix will help. - It would be interesting to see a simple ImageNet pretrained retrieval baseline without bells and whistles in the retrieval experiments. I’m not a retrieval expert, but it seems that retrieval networks are typically trained on other data sets and do additional tricks than just using a pretrained ImageNet embedding. All in all I think the paper is solid and presents good results, but a very similar method was previously proposed by [3]. I’m therefore not convinced that the paper presents enough novelty. Minor comment: -Using a different letter for the supervised labels y_j and the pairwise labels y_ij might improve readability. [1] Dosovitskiy A, Springenberg JT, Riedmiller M, Brox T. Discriminative unsupervised feature learning with convolutional neural networks. In Advances in Neural Information Processing Systems 2014 (pp. 766-774). [2] Kolesnikov A, Zhai X, Beyer L. Revisiting Self-Supervised Visual Representation Learning. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2019 (pp. 1920-1929). [3] Zhai X, Oliver A, Kolesnikov A, Beyer L. S4L: Self-Supervised Semi-Supervised Learning. arXiv preprint arXiv:1905.03670. 2019 May 9. [4] Tan M, Le QV. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. arXiv preprint arXiv:1905.11946. 2019 May 28. [5] Touvron H, Vedaldi A, Douze M, Jégou H. Fixing the train-test resolution discrepancy. arXiv preprint arXiv:1906.06423. 2019 Jun 14.