It Takes Two to Tango: Mixup for Deep Metric Learning
Authors: Shashanka Venkataramanan
Bill Psomas
Ewa Kijak
laurent amsaleg
Konstantinos Karantzalos
Yannis Avrithis
Venue: ICLR 2022 Poster (2022)

Abstract: Metric learning involves learning a discriminative representation such that embeddings of similar classes are encouraged to be close, while embeddings of dissimilar classes are pushed far apart. State-of-the-art methods focus mostly on sophisticated loss functions or mining strategies. On the one hand, metric learning losses consider two or more examples at a time. On the other hand, modern data augmentation methods for classification consider two or more examples at a time. The combination of the two ideas is under-studied.

In this work, we aim to bridge this gap and improve representations using mixup, which is a powerful data augmentation approach interpolating two or more examples and corresponding target labels at a time. This task is challenging because, unlike classification, the loss functions used in metric learning are not additive over examples, so the idea of interpolating target labels is not straightforward. To the best of our knowledge, we are the first to investigate mixing both examples and target labels for deep metric learning. We develop a generalized formulation that encompasses existing metric learning loss functions and modify it to accommodate for mixup, introducing Metric Mix, or Metrix. We also introduce a new metric---utilization---to demonstrate that by mixing examples during training, we are exploring areas of the embedding space beyond the training classes, thereby improving representations. To validate the effect of improved representations, we show that mixing inputs, intermediate representations or embeddings along with target labels significantly outperforms state-of-the-art metric learning methods on four benchmark deep metric learning datasets.

Published as a conference paper at ICLR 2022
IT TAKES TWo TO TANGO: MIXUP FOR DEEP METRIC LEARNING
Shashanka Venkataramanan'
Bill Psomas
Ewa Kijak'
Laurent Amsaleg
Konstantinos Karantzalos
Yannis Avrithis
Inria, Univ Rennes, CNRS, IRISA
Athena RC
National Technical University of Athens
ABSTRACT
Metric learning involves learning a discriminative representation such that em- beddings of similar classes are encouraged to be close. while embeddings of dis- similar classes are pushed far apart. State-of-the-ar art methods focus mostly on sophisticated loss functions or mining strategies. On the one hand, metric learn- ing losses consider two or more examples at a time. the other hand, modern data augmentation methods for classification consider two more examples at time. The combination the two ideas under-studied.
In this work, we aim to bridge this gap and improve representations using mixup, which is powerful data augmentation approach interpolating two more exam- ples and corresponding target labels at a time. This task is challenging because unlike classification, the loss functions used in metric learning are not additive over examples, so the idea of interpolating target labels is not straightforward. To the best of our knowledge, we are the first investigate mixing both examples and target labels for deep metric learning. We develop generalized formulation that encompasses existing metric learning loss functions and modify it to accom- modate for mixup, introducing Metric Mix, or Metrix. We also introduce new metric-uullization- demonstrate that by mixing examples during training, we exploring areas of the embedding space beyond the training classes, thereby improving representations. To validate the effect of improved representations, we show that mixing inputs, intermediate representations embeddings along with target labels significantly outperforms state-of-the-ee metric learning methods on four benchmark deep metric learning datasets.
INTRODUCTION
Classification is one of the most studied tasks machine learning and deep learning. It is common source of pre-trained models for transfer learning learning to other tasks (Donahue et al., 2014; Kolesnikov et al., 2020). It has been studied under different supervision set- tings (Caron et al., 2018; Sohn et al., 2020), knowledge transfer (Hinton et al., 2015) and data augmentation (Cubuk al., 2018), in- cluding the recent research on mixup (Zhang al., 2018; Verma al., 2019), where em- beddings and labels are interpolated.
class label interpolation
anchor
positive negative mixed
Deep metric learning is about learning from pairwise interactions such that inference re- lies on instance embeddings, e.g. for near- est neighbor classification (Oh Song et al.,
Figure 1: Metrix (= Metric Mix) allows an- chor to interact with positive (same class). neg- ative (different class) and interpolated examples, which also have interpolated labels.
"cqual contribution

---

Published as a conference paper at ICLR 2022
2016), instance-level retrieval (Gordo al., 2016). few-shot learning (Vinyals et al., 2016), face recognition (Schroff et al., 2015) and semantic textual similarity (Reimers & Gurevych, 2019).
Following (Xing et al., 2003), it is most often fully supervised by one class label per example, like classification. The two most studied problems are loss functions (Musgrave et al., 2020) and hard example mining (Wu et al., 2017; Robinson et al., 2021). Tuple-based losses with example weighting (Wang et al., 2019) can play the role of both.
Unlike classification, classes (and distributions) at training and inference are different in metric learning. Thus, one might expect interpolation-basee data augmentation like mixup to be even more important in metric learning than in classification. Yet, recent attempts are mostly limited to special cases of embedding interpolation and have trouble with label interpolation (Ko & Gu, 2020). This raises the question: what is a proper way to define and interpolate labels for metric learning?
In this work, we observe that metric learning is not different from classification, where examples are replaced by pairs of examples and class labels by "positive" or "negative", according to whether class labels of individual examples are the same or not. The positive or negative label of an example, or a pair, is determined in relation to a given example which is called an anchor. Then, as shown in Figure 1, straightforward way is to use binary (two class) label per pair and interpolate it linearly as in standard mixup. We call our method Metric Mix, or Metrix for short.
To show that mixing examples improves representation learning, we quantitatively measure the properties of the test distributions using alignment and uniformity (Wang & Isola, 2020). Align- ment measures the clustering quality and uniformity measures its distribution over the embedding space; a well clustered and uniformly spread distribution indicates higher representation quality. We also introduce new metric, utilization. to measure the extent to which test example, seen query, lies near any of the training examples, clean or mixed. By quantitatively measuring these three metrics, we show that interpolation-based data augmentation like mixup is very important in metric learning, given the difference between distributions at training and inference.
In summary, we make the following contributions:
1. We define a generic way of representing and interpolating labels, which allows straightforward extension of any kind of mixup deep metric learning for large class of loss functions. We develop our method on a generic formulation that encapsulates these functions (section 3).
2. We define the "positivity" of a mixed example and we study precisely how it increases as a function of the interpolation factor, both in theory and empirically (subsection 3.6).
3. We systematically evaluate mixup for deep metric learning under different settings, including mixup at different representation levels (input/manifold), mixup of different pairs of examples loss functions and hard example mining (subsection 4.2).
4. We introduce new evaluation metric, utilization, validating that a representation more appro- priate for test classes is implicitly learned during exploration of the embedding space in the presence of mixup (subsection 4.3).
5. We improve the state of the art on four common metric learning benchmarks (subsection 4.2).
2 RELATED WORK
Metric learning Metric learning aims to learn metric such that positive pairs of examples are nearby and negative ones are far away. In deep metric learning, we learn an explicit non-linear mapping from raw input to a low-dimensional embedding space (Oh Song et al., 2016), where the Fuclidean distance has the desired properties. Although learning can be unsupervised (Hadsell et al., 2006), deep metric learning has mostly followed the supervised approach, where positive and negative pairs are defined as having the same or different class label, respectively (Xing et al., 2003).
Loss functions can be distinguished into base- and based (Musgrave et al., 2020). Pair- based losses use pairs of examples (Wu et al., 2017; Hadsell et al., 2006), which can be defined over triplets (Wang et al., 2014; Schroff et al., 2015; Weinberger & Saul, 2009; Hermans et al., 2017), quadruples (Chen al., 2017) or tuples (Sohn, 2016; Oh Song et al., 2016; Wang et al., 2019). Proxy-based losses use one or more proxies per class, which are learnale parameters in the embedding space (Movshovitz-Attias et al., 2017; Qian et al., 2019; Kim et al., 2020c; Teh et al.,
2

---

Published as a conference paper at ICLR 2022
2020: Zhu et al., 2020b). Pair-based losses capture data-to-data relations, but they are sensitive to noisy labels and outliers. They often involve terms where given constraints are satisfied, which produce zero gradients and do not contribute to training. This necessitates mining of hard examples that violate the constraints, like semi-hard (Schroff et al., 2015) and distance weighted (Wu et al,, 2017). By contrast, proxy-based losses use data-to-proxy relations, assuming proxies can capture the global structure of the embedding space. They involve less computations that are more likely to produce nonzero gradient, hence have less or no dependence on mining and converge faster.
Mixup Input mixup (Zhang et al., 2018) linearly interpolates between two or more examples in the input space for data augmentation. Numerous variants take advantage of the structure of the input space interpolate non non-linearly, e.g. for images (Yun et al., 2019; Kim et al., 2020a; 2021; Hendrycks et al., 2020; DeViies & Taylor, 2017; Qin et al., 2020; Uddin et al., 2021). Manifold mixup (Verma et al., 2019) interpolates intermediate representations instead, where the structure is learned. This can be applied to or assisted by decoding back to the input space (Berthelot et al., 2018; et, 2018; 8; Beckham et al., 2019; Zhu al., 2020a; Venkataramanan et al., 2021). In both cases, corresponding labels are linearly interpolated too. Most studies are limited to cross-entropy loss for classification. Pairwise loss functions have been under-studied, as discussed below.
Interpolation for pairwise loss functions As discussed in subsection 3.3, interpolating target labels is not straightforward in pairwise loss functions. In deep metric learning, embedding expan- sion (Ko & Gu, 2020), HDML (Zheng et al., 2019) and symmetrical synthesis (Gu & Ko, 2020) interpolate pairs of embeddings in deterministic way within the same class, applying to pair-based losses, while proxy syuthesis (Gu et al., 2021) interpolates between classes, applying to proxy-based losses. None performs label interpolation, which means that (Gu et al., 2021) risks synthesizing false negatives when the interpolation factor À is close to 0 or 1.
In contrastive representation learning. MoCHi (Kalantidis et al., 2020) interpolates anchor with negative embeddings but not labels and chooses X E 0.5) to avoid false negatives. This resembles thresholding of À at 0.5 in OpTrransMix (Zhu et al., 2020a). Finally, i-mix (Lee et al., 2021) and MixCo (Kim et al., 2020b) interpolate pairs of anchor embeddings as well as their (virtual) class labels linearly. There is only one positive, while all negatives are clean, so it cannot take ad advantage of interpolation for relative weighting of positives/negaatves per anchor (Wang et al., 2019).
By contrast, Metrix is developed for deep metric learning and applies to a large class of both pair- based and proxy-based losses. It can interpolate inputs, intermediate features or embeddings of anchors, (multiple) positives or negatives and the corresponding two-class (positive/negative) labels per anchor, such that relative weighting of positivess/negativs depends on interpolation.
3 MIXUP FOR METRIC LEARNING
3.1 PRELIMINARIES
Problem formulation We are given a training set X C. X, where X' is the input space. For each anchor X, we are also given a set P(a) X of positives and set N(a) X of negatives. The positives are typically examples that belong to the same class as the anchor, while negatives belong a different class. The objective is train the parameters 0 of model f X Rd that maps input examples to a d-dimensional embedding, such that positives are close to the anchor and negatives are far away in the embedding space. Given two examples x;, E X, we denote by s(x, 2²) the similarity between z' in the embedding space, typically a decreasing function of Euclidean distance. is common to éz-normalize embeddings and define s(x, x²) == (f()). f faa)), which is the cosine similarity. To simplify notation, we drop the dependence of J, on 0.
Pair-based losses (Hadsell et al., 2006; Wang et al., 2014; Oh Song et al., 2016; Wang et al., 2019) use both anchors and positives/neggaiies in X, as discussed above. Proxy-hased losses define one or more learnable proxies E R2d per class, and only use proxies as anchors (Kim et al., 2020c) or as positives/neegative (Movshovitz-Attias et al., 2017; Qian et al., 2019; Teh et al., 2020). To ac- commodate for uniform exposition, we extend the definition of similarity as s(V, x) == (v. f()) X, Rd (proxy pos-
for Rd, x E X' (proxy anchors) and s(z, v) (f(z), v) for itives/negatives). Finally, to accommodate for mixed embeddings in subsection 3.5. we define s(v, v') (v,'") for v. E IRd. Thus, we define s (X U Rdj2 - R over pairs of either
3

---

Published as a conference paper at ICLR 2022
inputs in X' embeddings in R We discuss few representative loss functions below, before deriving a generic form.
The contrastive loss (Hadsell et al., 2006) encourages positive examples to be pulled towards the anchor and negative examples to be pushed away by margin R. This loss is additive over positives and negatives, defined as:
Contrastive
eomm (a; 0) = Σ s(a,p)++
Σ nEN(a)
[s(a, m]++
PEP(a)
The multi-similarity loss (Wang al., 2019) introduces relative weighting to encourage positives (negatives) that are farthest from (closest to) the anchor to be pulled towards (pushed away from) the anchor by higher weight. This loss not additive over positives and negatives:
Multi-Similarity
(ms(a; 0) = log 1
1
Σ e (-((((op))mm)
+ log 14
Σ nEN(a)
er(olan)-m)
(2)
PEP(a)
Here, 3, IR are scaling factors for positives, negatives respectively.
Proxy Anchor The proxy anchor loss (Kim et al., 2020c) defines a learnable praxy in Rd for each class and only uses proxies as anchors. For a given anchor (proxy) a E Rd. the loss has the same form as (2), although similarity is evaluated on Rd x X.
3.2 GENERIC LOSS FORMULATION
We observe that both additive (1) and non-additive (2) loss functions involve sum over positives P(a) and over negatives N (a). They also involve a decreasing function of similarity s(a, p) for each positive p E P(a) and an increasing function of similarity s(a, n) for each negative n E ((a). Let denote by pt this function for positives, negatives respectively. Then, non-additive functions differ from additive by the use of a nonlinear function ot, it, on positive and negative terms respectively, well possibly another nonlinear function their sum:
(0+
E(a; 0) = a+
pt (s((a.))) to PEP(a)
Σ
Σ Σn p (s(a, nEN((a)
With the appropriate choice for T, r,ato . . this definition encompasses contrastive (I), multi-similarity (2) proxy-anchor as well many pair-based proxy-basee loss functions, as shown in Table 1. It does not encompass the triplet loss (Wang et al., 2014), which operates on pairs of positives and negatives, forming triplets with the anchor. The triplet loss is the most challenging in terms of mining because there is a very large number of pairs and only few contribute to the loss. We only use function т to accommodate for lifted structure (Oh Song al., 2016; Hermans al., 2017), where Tzz) == [a]- is reminiscent of the triplet loss. We observe that multi multi-similarity (Wang al., 2019) differs from binomial deviance (Yi et al., 2014) only in the weights of the positive and negative terms. Proxy anchor (Kim et al., 2020c) is proxy version of multi multi-similarity (Wang et al., 2019) on anchors and ProxyNCA (Mosshoviz--Atias et al., 2017) a proxy version of NCA (Gold- berger et al., 2005) on positives/negatives.
This generie formulation highlights the components of the loss functions that are additive over pos- itives/negatives and paves the way towards incorporating mixup.
3.3 IMPROVING REPRESENTATIONS USING MIXUP
To improve the learned representations, we follow (Zhang et al., 2018; Verma et al., 2019) in mixing inputs and features from intermediate network layers, respectively. Both are developed for classifi- cation.
Input mixup (Zhang al., 2018) augments data by linear interpolation between pair of input examples. Given two examples x, x' X we draw X ~ Beta(a, a) as interpolation factor and mix with x' using the standard mixup operation mix (3r, x²) Xx (1 A)a'.

---

Published as a conference paper at ICLR 2022
Loss
ANCHOR
POS/NGG r(x)
(1))))
(x)
('(x)
p (x)
Contrastive (Hadsell et al., 2006)
X
-x
[ ml
structure (Hermans 2017)
[x]+
log(z)
log(x)
Binomial deviance (Yi 2014)
X
log(1 +a)
log(1 + a))
-(x -B(x-m) m)
7(x
Multi Multi-similae similarity (Wang al, 2019)
X
X
4 log(1 +
log(1 +a)
e-((o-m)
27(0-m)
Proxy anchor (Kim et al,, 2020c)
proxy
# log(1 + x)
log(1 x)
e-l(omm)
e2(0mm)
(Goldbberger
2005)
X
lzg(x)
log(z)
ProxyNCA (Movshovitz-Atiis
al., 2017)
X
log(z)
log(z)
ProkyNCA+ + + (Teh et al., 2020)
X
log(z)
log(z)
Table I: Loss functions. Anchor/psstiveenegaaive X: embedding of input example from training
set X by f: proxy: learnable parameter in IR T temperature. All loss functions are encompassed by (3) using the appropriate definition of functions .attaa"" a pptpp _pt,p" as given here.
Manifold mixup (Verma et al., 2019) linearly interpolates between intermediate representations (fea- tures) of the network instead. Referring to 2D images, we define Im X Rex the mapping from the input to intermediate layer m of the network and fm Rxx - R as the mapping from intermediate layer m to the embedding, where is the number channels (feature dimensions) and h is the spatial resolution. Thus, our model can be expressed as the composition f = J Jm-
For manifold mixup, we follow (Venkataramanan et al., 2021) and mix either features of intermedi- ate layer or the final embeddings. Thus, we define three mixup types in total:
f (mixxxx x')))
input mixup
[x(x,))) = Imiimxx (x), 9m (x')),, (fxx) f(x), f(x'),
feature mixup
(4)
embedding mixup.
Function fx :2 + Rd performs both mixup and embedding. We explore different mixup types in subsection B.4 of the Appendix.
3.4 LABEL REPRESENTATION
supervised classification. each example x E X is assigned an one-hot encoded label 13 where C is the number of classes. Label vectors also linearly interpolated: given two labeled examples (ar, y). (x', yy), the interpolated label is mix (y: y'). The loss (cross- entropy) is continuous function of the label vector. We extend this idea to metric learning.
Classification
Metric learning Positives P(a) and negatives ((a) of anchor are defined as having the same or different class label as the anchor, respectively. To every example in P(a) U N(),, we assign a binary (two-class) label {0, 1). such that y 1 for positives and y for negatives:
U- (a) {(p 1) e P(a))
(5)
U (a) = (m,
N(a))
(6)
Thus, we represent both positives and negatives by U(a) = U (t) (a). We now rewrite the generic loss function (3) as:
()a ; (0+ (
(++
)) (7)
Σ upt(s(a, + to- (x.))EU(())
Σ (1 - v) (s(a,
(o (o(a,,)))
(x.g)EU((a)
Here, every labeled example (z, y) in U(a) appears in both positive and negative terms. However, because label y is binary, only one of the two contributions nonzero. Now, in the presence of mixup, we can linearly interpolate labels exactly as in classification.
3.5 MIXED LOSS FUNCTION
Mixup For every anchor a, given set (a) of pairs of examples to mix. This is subset of (S(a) Ua)) x (a) where S(a) == (a, 1). That is, we allow mixing between positive-negative positive-positive and negative-negative pairs, where the anchor itself is also seen as positive. We
5

---

Published as a conference paper at ICLR 2022
define the possible choices of mixing pairs (a) in subsection 4.1 and assess them in subsec- tion B.4 of the Appendix. Let V(a) the set of corresponding labeled mixed embeddings
V(a) == {(fx(x, mix (v. (v.y') y²)) ((r, v), (z', v/) E M(a), A ~ Beta(a, a)) (8)
where fA is defined by (4). With these definitions in place, the generie loss function é over mixed examples takes exactly the same form as (7), with only (aa replaced by V(a):
((a;0)==+ = (r=0)==(00
(1- y1- (s(a, 2)) )) (9)
Σ (v.g)EV(a)
upt(s(a, ()), to
Σ (o.))V(())
where similarity s is evaluated on x 1dd for pair-based losses and on 12 x Id for proxy anchor. Now, every labeled embedding (v. y/ in V(a) appears in both positive and negative terms and both contributions are nonzero for positive-negative pairs, because after interpolation, y E [0, 1].
Error function Parameters are learned by minimizing the error function, which a linear com- bination of the clean loss (3) and the mixed loss (9), averaged over all anchors
E(
1
Σ OE
((a:0) + wl(a;)),
(10)
X
where w 2 O is the mixing strength. At least for manifold mixup, this combination comes at little additional cost, since clean embeddings readily available.
3.6 ANALYSIS: MIXED EMBEDDINGS AND POSITIVITY
Let Pos(a, v) be the event that mixed embedding v behaves as "positive" for anchor i.e., min- imizing the loss E(a; 8) will increase the similarity s(a, v). In subsection A.2 of the Appendix, explain that this "positivity" is equivalent to δε(a; 0)/Os(a, v) s O. Under positive-negative mix- ing, i.., M(a) C (a) x U-(a), we then estimate the probability of Pos(a, v) as a function of X the case of multi-simillaity (2) with a single mixed embedding v:
P(Pos(a, v)) Fx (
A
B8+ In
1- +
(11)
where F,, is the CDF of similarities s(a, v) between anchors a and mixed embeddings v with in- terpolation factor X. In Figure 2, we measure the probability of Pos(a, v) as function of X in two ways, both purely empirically and theoretically by (11). Both measurements are increasing func- tions of À of sigmoidal shape, where mixed embedding is mostly positive for X close to 1 and mostly negative for À close
EXPERIMENTS
4.1 SETUP
Datasets We experiment on Caltech-UCSD Birds (CUB200) (Wah et al., 2011), Stanford Cars (Cars (Krause et al., 2013), Stanford Online Products (SOP) (Oh Song et al., 2016) and In-Shop Clothing retrieval (In-Shop) (Liu et al., 2016) image datasets. More details are in subsection B.1.
Network, features and embeddings We use Resnet-50 (He et al., 2016) (R-50) pretrained on Im- ageNet (Russakovsky et al., 2015) as backbone network. We obtain the intermediate representation (feature), 7 x 7 x 2048 tensor, from the last convolutional layer. Following (Kim et al., 2020c), combine adaptive average pooling with max pooling, followed by a fully-connected layer to obtain the embedding of d 512 dimensions.
Loss functions We reproduce contrastive (Cont) (Hadsell et al., 2006), multi-similarity (MS) (Wang et al., 2019), proxy anchor (PA) (Kim et al., 2020c) and ProxyNCA+ (Teh et al., 2020) and evaluate them under different mixup types. For MS (2), following Musgrave et al. (2020), we use 3 = 18, 7 = 75 and m = 0.77. For PA, we use 3 = 32 and m = 0.1, as reported by the authors. Details training are in subsection B.I of the Appendix.

---

Published as a conference paper at ICLR 2022
Methods We compare our method, Merrix, with proxy synthesis (PS) (Gu al., 2021), i-mix (Lee et al., 2021) and MoCHi (Kalantidis et al., 2020). For PS, we adapt the official code' to PA on all datasets, and use it with PA only, because it is designed for proxy-based losses. PS has been shown superior to (Ko & Gu, 2020; Gu & Ko, 2020), although meant for contrastive representation learning. We evaluate using Recall K (Oh Song al., 2016) For each test example taken as a query, we find its K -nearesst neighbors in the test set excluding itself in the embedding space. We assign a score of if an example of the same class is contained in the neighbors and ( otherwise. Recall @ K is the average of this score over the test set.
different networks. MoCHi and i-mix are
Mixup settings For input mixup, we use the hardest negative examples for each anchor (each example in the batch) and mix them with positives or with the anchor. We use k = 3 by default. For manifold mixup, we
0.8
P (a, u))
0.6
focus on the last few layers instead, where features and embeddings are compact, and we
0.4
empirical
mix all pairs. We use feature mixup by de-
0.2
theoretical
fault and call it Metrix/feature or just Metrix, while input and embedding mixup are called Metrix/inpul and Metrix/embed, respectively. For all mixup types, we use clean exam- ples as anchors and we define set ((a of pairs of examples to mix for each anchor a, with their labels (positive negative). By default, we mix positive-negatio or anchor- negative pairs, by choosing uniformly at ran- dom between M(u) == (a) x (a) and ((a) := S(a) x U (a), respectively, where U- (a) is replaced by hard negatives only for input mixup. More details are in subsection B.2.
0
0.2
0.4
0.6
0.8
Figure 2: "Positivity" of mixed embeddings vs.
X. We measure P(Pos(a, v)) empirically as
v) I 0) and theoretically by (11), where F) is again measured from data. We use emhedding mixup on MS (2) on CUB200 epoch 0, based on the setup of subsection 4.1.
4.2 RESULTS
Improving the state of the art As shown in Table 2, Metrix consistently improves the perfor- mance of all baseline losses (Cont, MS, PA, ProxyNCA++) across all datasets. More results in subsection B.3 of the Appendix reveal that the same true for Metrix/input and Metrix/embed too. Surprisingly, MS outperforms PA and PCoxyNCA++ under mixup all datasets but SOP, where the three losses are on par. This is despite the fact that haseline PA outperforms MS on CUB200 and Cars-196, while ProxyNCA++ outperforms MS on SOP and In-Shop. Both contrastive and MS are significantly improved by mixup. By contrast, improvements on PA and ProxyNCA++ are marginal, which may be due the already strong performance of PA, or further improvement is possible by employing different mixup methods that take advantage of the image structure.
In terms of Recall @ our MS+Metrix best overall, improving by 3.6% (67.8 + 71.4) on CUB200, 1.8% (87.8 - 89.6) on Cars 96, 4.1% (76.9 - 81.0) on SOP and 2.1% (90.1 - 92.2) on In-Shop. The same solution sets new state of the art, outperforming the previously best PA by 1.7% (69.7 + 71.4) on CUB200, MS by 1.8% (87.8 + 89.6) on Cars196, ProXyNCA++ by 0.3% (80.7 + 81.0) on SOP and SoftTriple by 1.2% (91.0 + 92.2) on In-Shop. Importantly, while the previous state of the art comes from a different loss per dataset, MS+Metrix is almost consistently best across all datasets.
Alternative mixing methods In Table 3, we compare Metrix/input with i-Mix (Lee et al., 2021) and Metrix/embed with MoCHi (Kalantidis et al., 2020) using contrastive loss, and with PS (Gu et al., 2021) using PA. MoCHi and PS mix embeddings only, while labels are always negative. For i-Mix, mix anchor-negative pairs (S(a) (a)). For MoCHi, the anchor is clean and mix negative-negative (U) and anchor-negative (S(a)) x X U-(a)) pairs, where U- (a) is replaced by k 100 hardest negatives and À E (0, 0.5) for anchor-negative. PS mixes embeddings of different classes and treats them as new classes. For clean anchors, this corresponds to positive-negative (U+ (a) U U- (a)) and negative-negative (U-(a)²) pairs, but PS also supports mixed anchors.
'https://ee thub. www.///////eeeeeee navervis in wrwxyyy---ooeeeeooeeeeeooooeoorm htt.rrssssessppooooossssteeee synthesis

---

Published as a conference paper at ICLR 2022
CUB200
CARSI96
SOP
IN-SHOP
MATHOD
R@i
R@2
R@4
R@I
R@2
R@4
R@:
R@ 10
R@
100
R@1
RØ 10
R@20 20
Triplet (Weinberger & Saul, 2009)
63.5
75.6
84.4
77.3
85.4
90.8
70.5
85.6
94.3
85.3
97.8
LittedStucture (Oh Song et 2016)
65.9
75.8
84.5
81.4
88.3
92.4
76.1
88.6
95.2
88.6
97.6
98.4
PCa (Movshovitz-Alies
al., 2017)
65.2
75.6
83.8
81.2
87.9
92.6
73.2
87.0
94.4
86.2
95.9
97.0
Margin (Wu et al., 2017)
65.0
76.2
84.6
82.1
88.7
92.7
74.8
87.8
94.8
88.6
97.0
97.8
SoftTriple (Qian ot al., 2019)
67.3
77.7
86.2
86.5
91.9
95.3
79.8
91.2
96.3
91.0
97.6
98.3
D&C (Sanakoycn al. 2019)*
65.9
76.6
84.4
84.6
94.1
75.9
88.4
94.9
85.7
96.9
EPSHN (Xuan et al., 2020)*
64.9
75.3
83.5
82.7
89.3
93.0
78.3
90.7
96.3
87.8
95.7
96.8
Cont (Hadsell et al., 2006)
64.7
75.9
84.6
81.6
88.2
92.7
74.9
87.0
93.9
86,4
94.7
96.2
+ +Metrix
67.4
77.9
85.7
85.1
91.1
94.6
77.5
89.1
95.5
89.1
95.7
97.1
+2.7
+2.0
+1.1
+3.5
+2.9
+1.9
+2.6
+2.1
+1.5
+2.7
+0.9
(Wang et al, 2019)
67.8
77.8
85.6
87.8
92.7
95.3
76.9
89.8
95.9
90.1
98.4
+Metrix
71.4
80.6
86.8
89.6
94.2
96.0
81.0
92.0
97.2
92.2
98.5
98.6
+3.6
+2.8
+1.2
+1.8
+1.5
+0.7
+4.1
+2.2
+1.3
+2.1
+0.99
+0.22
PA (Kim at, 2020g)
69.7
80.0
87.0
87.7
92.9
95.8
PA (Kim at. 2020c)
69.5
79.3
87.0
92.3
95.5
79.1
90.8
96.2
90.0
97.4
98.2
+ Metrix
71.0
81.8
88.2
89.1
93.6
96.7
81.3
91.7
96.9
91.9
98.2
98.8
+1.2
+1.44
+0.77
+0.9
+2.2
+0.9
+0.7
+1.9
+0.8
NCA+ (Teh al., 2020)*
69.0
79.8
87.3
86.5
92.5
95.7
80.7
92.0
96.7
90.4
98.1
98.8
ProxyNCA++ (Teh 2020)
79.5
87.7
86.6
92.1
95.4
80.4
91.7
96.7
90.2
97.6
98.4
+ +Metrix
70.4
80.6
88.7
88.5
93.4
96.5
81.3
92.7
97.1
91.9
98.1
98.8
+1.3
+0.8
+1.0
+1.9
+0.9
+0.8
+0.6
+0.7
40.4
+1.5
+0.0
+0.0
Gain over SOTA
+1.7
+1.8
+0.5
+1.8
+1.3
+0.9
40.6
+0.7
+0.5
+1.2
+0.4
+0.0
Table 2: Improving the SOTA with our Metrix (Metrix/featture using Resnet-50 with embedding size d = 512. R@ K (%): Recall K; higher is better. *: reported by authors. Bold black: best baseline (previous SOTA, one per column). Red: Our new SOTA. Gain over SOTA is over best baseline. MS: Multi-Similarity, PA: Proxy Anchor. Additional results are in subsection B.3 of the Appendix.
In terms of Recall @1 1, Metrix/input outperforms i-Mix with anchor-negative pairs by 0.5% (65.8- 66.3) on CUB200, 0.9% (82.0 + 82.9) on 196.196,6 0.6% (75.2 + 75.8) and 0.6% (87.1 - 87.7) on In-Shop. Metrix/embed outperforms MoCHI with mehor--eeative pairs by 1.2% (65.2 + 66.4) on CUB200, 1.4% (82.5 - 83.9) on Cars196, 0.9% (75.8 + 76.7) and 1.2% (87.2 - 88.4) on In-Shop. The gain over MoCHi with negative-negative pairs is significantly higher. Metrix/embed also outperforms PS by 0.4% (70.0 + 70.4) on CUB200, 1 1% (87.9 + 88.9) Cars196, 1% (79.6 + 80.6) on SOP and 1.3% (90.3- + 91.6) on In-Shop
Computational complexity
We study the computational complexity of Metrix subsection B.3.
Ablation study We study the effect of the number & of hard negatives, mixup types (input, feature and embedding), mixing pairs and mixup strength w in subsection B.4 of the Appendix.
4.3 HOw DOES MIXUP IMPROVE REPRESENTATIONS?
We analyze how Metrix improves representation learning, given the difference between distributions training and inference. As discussed in section 1, since the classes inference are unseen train- ing, one might expect interpolation-base than in classification. This is so because, by mixing examples during training, we are exploring areas of the embedding space beyond the training classes. We hope that this exploration would pos- sibly lead the model to implicitly learn a representation more appropriate for the test classes, if the distribution of the test classes lies near these areas.
data augmentation like mixup to be even more important
Aligmment and Uniformity In terms of quantitative measures of properties of the training and test distributions, we follow Wang & Isola (2020). This work introduces two measures - aalgnment and uniformity (the lower the better) to be used both as loss functions (on the training set) and as evaluation metrics (on the test set). Alignment measures the expected pairwise distance between positive examples in the embedding space. A small value of alignment indicates that the positive examples are clustered together. Uniformity measures the (log of the) expected pairwise similarity between all examples regardless of class, using a Gaussian kernel as similarity. A small value of uniformity indicates that the distribution is more uniform over the embedding space, which is particularly relevant to our problem. Meant for contrastive learning, (Wang & Isola, 2020) use the same training and test classes, while in our case they are different.
8

---

Published as a conference paper at ICLR 2022
CUB200
CARSI90
SOP
IN-SHOP
METROD
MIXING PAIRS
R@i
R@2
R@4
R@
R@2
R@4
R
RØ
10
R@ 100
R@
RØ
R
Cont (Hadsell 2006)
75.9
84.6
88.2
92.7
87.0
93.9
86.4
96.3
6-Mix (Lee et al.,
anc-neg
65.8
76,2
84,9
82.0
88.5
93.2
75.2
87.3
94.2
87.1
95.4
96.1
Metris/inpal
neg / anc-neg
66.3
77.1
85.2
82.9
89.3
93.7
75.8
87.8
94.6
87.7
95.9
96.5
MaCHi (Kalantidis et al. 2020)
neg-neg
63.1
74.3
81.8
76.3
84.0
89.3
68.9
83.1
91.8
81.8
91.9
91.9
-MoCHi (Kalantidis et al,, 2020)
anc-nog pos-teg/ / anc-neg
65.2
75.8
84.2
82.5
88.0
92.9
75.8
87.1
94.8
87.2
92.8
94.9
Motrix/embed
66.4
77.6
85.4
83,9
90.3
94.1
76.7
88.6
95.2
88.4
95.4
96.9
PA (Kim et al,, 2020c)
69.7
80,0
87.0
87.6
92.3
95.5
79.1
90.8
96.2
90.0
97.4
98.2
-PS (Gu 2021)
pos neg / neg neg
70.0
79.8
87.2
87.9
92.8
95.6
79.6
90.9
96.4
90.3
97.4
98.0
Metrix/emmbed
pos-neg/ anc-neg
70.4
81.1
87.9
88.9
93.3
96.4
80.6
91.7
96.6
91.6
98.3
98.3
Table 3: Comparison of our Metrix/embed with other mixing methods using R-50 with embedding
size d = 512. R@ K (%); Recall @ K; higher is better. PA: Proxy Anchor, PS: Proxy Synthesis.
By training with contrastive loss on CUB200 and then measuring on the test set, we achieve an alignment (lower the better) of 0.28 for contrastive loss, 0.28 for i-Mix (Lee et al., 2021) and 0.19 for Metrix/input. MoCHi (Kalantidis et al., 2020) and Metrix/embed achieve an alignment 0.19 and 0.17, respectively. We also obtain uniformity (lower the better) of -2.71 for contrastive loss, 2.11 for i-Mix and -3.13 for Metrix/input. The uniformity of MoCHi and Metrix/embed -3.18 and -3225, respectively. This indicates that Metrix helps obtain test distribution that is more uniform over the embedding space, where classes are better clustered and better separated.
The measures proposed by (Wang & Isola, 2020) are limited to a single distribution or dataset, either the training set (as loss functions) the test set (as evaluation metrics). It is more interesting to measure the extent to which a test example, seen as a query, lies near any of the training examples, clean or mixed. For this, introduce the measure of utilization u u(Q, : X) of the training set X by the test set Q as
Utilization
u (Q, = 1 Σ gEQ
min l[f(g) - f(z)]1 xex
(12)
Utilization measures the average, over the test set Q, of the minimum distance of query q to a training example x E X in the embedding space of the trained model f (lower is better). A low value of utilization indicates that there are examples in the training set that are similar test examples. When using mixup, we measure utilization as u(Q, X), where X is the augmented training set including clean and mixed examples over a number of epochs and f remains fixed. Because X C X, we expect X) < (Q, that is, the embedding space is better explored in the presence of mixup.
By using contrastive loss CUB200, utilization drops from 0.41 to 0.32 when using Metrix. This indicates that test samples are indeed closer to mixed examples than clean in the embedding space. This validates our hypothesis that a representation more appropriate for test classes is implicitly learned during exploration of the embedding space in the presence of mixup.
CONCLUSION
Based on the argument that metric learning is binary classification of pairs of examples into "pos- itive" and "negative", we have introduced a direct extension of mixup from classification to metric learning. Our formulation is generic, applying itives from negatives per anchor and involve component functions that are additive over examples. Those are exactly loss functions that require less mining. We contribute principled way of in- terpolating labels, such that the interpolation factor affects the relative weighting of positives and negatives. Other than that, our approach is completely agnostic with respect to the mixup method, opening the way to using more advanced mixup methods for metric learning.
large class of loss functions that separate pos-
We consistently outperform baselines using a number of loss functions on a number of bench- marks and we improve the state of the art using single loss function all benchmarks, while previous state of the art was not consistent in this respect. Surprisingly, this loss function, multi- similarity Wang et al. (2019), is not the state of the art without mixup. Because metric learning is about generalizing to unseen classes and distributions, our work may have applications to other such problems, including transfer learning, few-shot learning and continual learning.

---

Published as a conference paper at ICLR 2022
6 ACKNOWLEDGEMENT
Shashanka's work was supported by the ANR-19-CE23-0228 MEERQAT project and was per- formed using the HPC resources from GENCI-IDRIS Grant 2021 AD011011709RI. Bill's work was partially supported by the EU RAMONES project grant No. 101017808 and was performed using the HPC resources from GRNET S.A. project pr01 This work was partially done when Yannis was at Inria.
REFERENCES
Christopher Beckham, Sina Honari, Vikas Verma, Alex Lamb, Farnoosh Ghadiri, R Devon Hjelm, Yoshua Bengio, and Christopher Pal. On adversarial mixup resynthesis. arXiv preprint arXiv:1903.02209, 2019.
David Berthelot, Colin Raffel, Aurko Roy, and Ian Goodfellow. Understanding and improving interpolation in autoencoders via an adversarial regularizer. arXiv preprint arXiv: 2018.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsu- pervised learning of visual features. In ECCV, 2018.
Weihua Chen, Xiaotang Chen, Jianguo Zhang, and Kaiqi Huang. Beyond triplet loss: deep quadru- plet network for person re-identification. In CVPR, 2017.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation policies from data, arXiv preprint (XXi::880500901 2018
Terrance DeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint
2017.
Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In ICML, 2014.
Jacob Goldberger, Sam Roweis, Geoffrey Hinton, and Ruslan Salakhutdinov. Neighbourhood com- ponents analysis. In NIPS, 2005.
Albert Gordo, Jon Almazán, Jerome Revaud, and Diane Larlus. Deep image retrieval: Learning global representations for image search. ECCV, 2016.
Geonmo Gu and Byungsoo Ko. Symmetrical synthesis for deep metric learning. In AAAI, 2020.
Geonmo Gu, Byungsoo Ko, and Han-Gyu Kim. Proxy synthesis: Learning with synthetic classes for deep metric learning. In AAAI, 2021.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In CVPR, 2006.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: A simple data processing method to improve robustness and uncertainty. ICLR, 2020.
Alexander Hermans, Lucas Beyer, and Bastian Leibe. In defense of the triplet loss for person re- identification. arXiv preprint arXiv: 2017.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.0253/ 2015.
Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning. NewrIPS, 2020.
10

---

Published as a conference paper at ICLR 2022
Hyun Kim, Wonho Choo, and Hyun Oh Song. Puzzle mix: Exploiting saliency and local statistics for optimal mixup. In ICML, 2020a.
Jang- Hyun Kim, Wonho Choo, Hosan Jeong, and Hyun Oh Song. Co-mixup: Saliency guided joint mixup with supermodular diversity. In ICLR, 2021.
Sungnyun Kim, Gihun Lee, Sangmin Bae, and Se- Young Yun. Mixco: Mix-up contrastive learning for visual representation. NeurlPS Workshop on Self-Supervised Learning, 2020b.
Sungyeon Kim, Dongwon Kim, Minsu Cho, and Suha Kwak. Proxy anchor loss for deep metric learning. In CVPR, 2020c.
Byungsoo Ko and Geonmo Gu. Emhedding expansion: Augmentation in embedding space for deep metric learning. In CVPR, 2020.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung. Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In ECCV, 2020.
Jonathan Krause, Michael Stark, Jia Deng, and Fei-Fei Li. 3d object representations for fine-grained categorization. ICCVW, 2013.
Kibok Lee, Yian Zhu, Kihyuk Sohn, Chun-Liang Li, Jinwoo Shin, and Honglak Lee. I-mix: A domain-agnostic strategy for contrastive representation learning. In ICLR, 2021.
Xiaofeng Liu, Yang Zou, Lingsheng Kong, Zhihui Diao, Junliang Yan, Jun Wang, Site Li, Ping Jia, and Jane You. Data augmentation via latent space interpolation for image classification. In ICPR, 2018.
Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang. Deepfashion: Powering robust clothes recognition and retrieval with rich annotations. In CVPR, 2016.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. ICLR, 2019.
Yair Movshovitz-Attie Attias, Alexander Toshev, Thomas K Leung, Sergey loffe, and Saurabh Singh. No fuss distance metrie learning using proxies. In ICCV, 2017.
Kevin Musgrave, Serge Belongie, and Ser-Nam Lim. A metric learning reality check. In ECCV, 2020.
Hyun Oh Song. Yu Xiang, Stefanie Jegelka, and Silvio Savarese. Deep metric learning via lifted structured feature embedding. In CVPR, 2016.
Qi Qian, Lei Shang, Baigui Sun, Juhua Hu, Hao Li, and Rong Jin. Softtriple loss: Deep metric learning without triplet sampling. In ICCV, 2019.
Jie Qin, Jiemin Fang, Qian Zhang, Wenyu Liu, Xingang Wang, and Xinggang Wang. Resizemix: Mixing data with preserved object information and true labels. arXiv preprint arX1v:2012.11101, 2020.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. In EMNLP, 2019.
Joshua Robinson, Ching-Yar Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. ICLR, 2021.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. IJCV, 2015.
Artsiom Sanakoyeu, Vadim Tschernezki, Uta Buchler, and Bjor Ommer. Divide and conquer the embedding space for metric learning. In CVPR, 2019.
Florian Schroff. Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In CVPR, 2015.
11

---

Published as a conference paper at ICLR 2022
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. In NIPS, 2016.
Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. arXiv preprint urXiv:2001.0768. 2020.
Eu Wern Teh, Terrance DeVries, and Graham W Taylor. Proxynca*+: Revisiting and revitalizing proxy neighborhood component analysis. ECCV, 2020.
A. F. M. Shahab Uddin, Sirazam Monira Mst., Wheemyung Shin, TaeChoong Chung, and Sung-Ho Bae. Saliencymix: saliency guided data augmentation strategy for better regularization. In ICLR, 2021.
Shashanka Venkataramanan, Yannis Avrithis, Ewa Kijak, and Laurent Amsaleg. Alignmix: Improv- ing representation by interpolating aligned features. arXiv preprint arXiv:2103.15375, 2021.
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez- Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In ICML, 2019.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Match- ing networks for one shot learning, arXiv preprint arXiv:1606.040 2016.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The Caltech- UCSD Birds-200-2011 Dataset. Technical Report CNS-TR-2011-00 California Institute of Technology, 2011.
Jiang Wang, Yang Song, Thomas Leung, Chuck Rosenberg, Jingbin Wang, James Philbin, Bo Chen, and Ying Wu. Learning fine-grained image similarity with deep ranking. In CVPR, 2014.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. ICML, 2020.
Xun Wang, Xintong Han, Weilin Huang, Dengke Dong, and Matthew R Scott. Multi-similarity loss with general pair weighting for deep metric learning. In CVPR, 2019.
Kilian Q Weinberger and Lawrence K Saul. Distance metric learning for large margin nearest neigh- bor classification. JMLR, 2009.
Chao-Yuaa Wu, R. Manmatha, Alexander J. Smola, and Philipp Krähenbühl. Sampling matters in deep embedding learning. In ICCV, 2017.
Eric P Xing, Michael Jordan, Stuart Russell, and Andrew Ng. Distance metric learning with application to clustering with side-information. In NIPS, 2003.
Hong Xuan, Abby Stylianou, and Robert Pless. Improved embeddings with easy positive triplet mining. In WACV, 2020.
Dong Yi, Zhen Lei, and Stan 7. Li. Deep metric learning for practical person re-identification. arXiv preprint arXiv: 1703.07377 2014.
Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV, 2019.
Andrew Zhai and Hao- Yu Wu. Classification is a strong baseline for deep metric learning. arXiv preprint arXiv:1811.12644 2018
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018.
Wenzhao Zheng, Zhaodong Chen, Jiwen Lu, and Jie Zhou. Hardness- aware deep metric learning. In CVPR, 2019.
12

---

Published as a conference paper at ICLR 2022
Jianchao Zhu, Liangliang Shi, Junchi Yan, and Hongyuan Zha. Automix: Mixup networks for sample interpolation via cooperative barycenter learning. ECCV, 2020a.
Yuehua Zhu, Muli Yang, Cheng Deng, and Wei Liu. Fewer is more: A deep graph metric learning perspective using fewer proxies. NeurIPS, 2020b.
13

---

Published as a conference paper at ICLR 2022
A MORE ON THE METHOD
A.1 MIXED LOSS FUNCTION
Interpretation To better understand the two contributions labeled embedding (e, y) V(a) to the positive and negative terms of (9), consider the case of positive-negative mixing pairs, ((a C. U+ (a) x U (a). Then, for ((z, y), (x)) M(), the mixed label mix y/) = mixa (1, 0) À and (9) becomes
>p*(s((, to- Σ (1 (1- (1-A)ø A)p (s(a,))) )) (13)
f(a:0) = o+ Σ (v.A)EV(())
(eLA)EV(a)
Thus, the mixed embedding e is both positive (with weight A) and negative (with weight A). Whereas for positive-positive mixing, that is, for (a) (a)²², the mixed label is 1 and the negative term vanishes. Similarly, for negative-negative mixing, that is, for M(a) C U=(a)² the mixed label 0 and the positive term vanishes.
In the particular case of contrastive (1) loss, positive-negativv mixing (13) becomes
Ecome (a; 0) =
-) (o.A)EV(a)
Σ
E
(1 b))))a((, ml+
(14)
(v,))V((a)
Similarly, for multi-similarity (2),
(ms(a: a;; 0)
log
1+
Σ
Ae ((=((((a)))m) +
(v,A)EV(a)
(15)
1
log
1+
Σ
(1- (1-e)e=(o
(v,A)EV(a)
A.2 ANALYSTS: MIXED EMBEDDINGS AND POSITIVITY
Positivity Under positive-negative mixing, (13) shows that mixed embedding with interpo- lation factor À behaves as both positive and negative to different extents, depending on A: mostly positive for À close to 1, mostly negative for close to 0. The net effect depends the derivative of the loss with respect the similarity Ōē(a: 6) /as(a, v): the derivative is negative, then behaves as positive and vice versa. This is clear from the chain rule
aā(a; 0) dē(a; 0) Ov
Ōs(a, v)
(16)
Os(a, v) Ov
because Ōs(a, 00 vector pointing direction that makes a, u more similar and the loss is being minimized. Let Pos(a, v) be the event that v behaves as "positive", i.e,, ()(aaa0) v) s 0 and minimizing the loss will increase the similarity s(a,v) v).
Multi-sinilarity We estimate the probability of Pos(a, u) as function of in the case of multi- similarity with a single embedding e obtained by mixing a positive with negative:
1 } Ms (a; log (1 de A(s(a,)) + 7
log (1+ 2)ex((la)))-m)
(17)
In this case, Pos(a, v) occurs if and only if
0lms(a;0) - -e A(s(a,))
(1 (0
50.
(18)
0s(a, (1 de -la(aa)))))
(1 + (1 1)
14

---

Published as a conference paper at ICLR 2022
Clean train examples
Mixed train examples
Test examples
(a)
(b)
Figure 3: Exploring the embedding space when using (a) only clean examples (b) clean and mixed examples. Given a query the distance d to its nearest training embedding (clean or mixed) is smaller with mixup (b) than without (a).
By letting f. == s(a, v) m, this condition is equivalent
(1 A)e
(1+ (1 - N) - (1+ de-
(19)
(1- (1+ de Ae- S de -0 (1 + (1 - ))
(20)
+ +A1 2)c(0-B)e de st +A(1 A))(
-
(21)
c(0ti)0 - 1-X
A
(22)
(8 (8+7)(s(a,v) m) S In 1-
(23)
s(a, v) S 8+7
In
1-
2
m.
(24)
Finally, the probability of Pos(a, v) as a function of is
P(Pos(a, v)) Fx ( 3 In
h
m
(25)
where F2 is the CDF of similarities s(a, v) between anchors a and mixed embeddings with inter- polation factor
In Figure 2, we measure the prohability of Pos(a, v)) as function of À in two ways. First, we measure the derivative Oiss (a; 0)/Os(a, v) for anchors a and mixed embeddings over the en- tire dataset and we report the empirical probability of this derivative being non non-positive versus A. Second, we measure ((Pos(a, v)) theoretically using (25), where the CDF of similarities s(a, v) is again measured empirically for a and v over the dataset, ing assumption of single positive and a single negative in deriving (25), we observe that the two measurements agree in general. They are both increasing functions of À of sigmoidal shape, they roughly yield ((Pos(a, v)) Z 0.5 for X 2 0.5 and they confirm that a mixed embedding mostly positive for À close to 1 and mostly negative for À close to O.
function of X. Despite the simplify-
A.3 MORE ON UTILIZATION
In subsection 4.3, we discuss that a representation more appropriate for test classes is implicitly learned during exploration of the embedding space in the presence of mixup. We provide an illus- tration of this exploration in Figure 3, where we visualize the embedding space using (a) only clean train examples and (b) clean and mixed train examples. In case (a), the model is trained using only clean examples, exploring smaller area of the embedding space. In case (b), it is trained using both
15

---

Published as a conference paper at ICLR 2022
DATASET
CUB200 (Wah et 201,
CARS196 (Krause 2013) SOP (Oh Soog et al,, 2016) IN-SHOP (Liu et al, 2016)
Objects
birds
household furniture
clothes
# classes
200
196
22, 634
7, 982
training images
894
60, 026
26,
testing images
5, 894
8, 099
60, 027
26, 356
training classes
100
98
11, 318
3991
lesting classes
100
98
11, 318
3991
sampling
random
random
balanced 5
balanced
samples per class
classes per batch
65
70
20
20
learning
10
x 10
3 10
10
Table Statistics and settings for the four datasets we use in our experiments. t; average.
mixed and clean examples, exploring a larger area. is clear that the distance between a query and its nearest training example (clean or mixup) is smaller in the presence of mixup. Utilization is the average of this distance over the test set. This shows that the model implicitly learns representation closer the test example in the presence of mixup during training and it partially explains why mixup leads to better performance.
B
MORE ON EXPERIMENTS
B.
SETUP
Datasets and sampling Dataset statistics are summarized in Table Since the number of classes is large compared to the batch size in SOP and In-Shop, batches would rarely contain a positive pair when sampled uniformly random. Hence, we use balanced sampling (Zhai Wu, 2018), i.e. a fixed number of classes and examples per class, as shown in Table 4. For fair comparison with baseline methods, images are randomly flipped and cropped to 224 x 224 training. At inference, we resize 256 256 and then center-crop to 224 x 224.
Training We train R-50 using AdamW (Loshchilov & Hutter, 2019) optimizer for 100 epochs with a batch size 100. The initial learning rate per dataset is shown Table 4. The learning rate decayed by 0.1 for Cont and by 0.5 for MS and PA on CUB200 and Cars196. For SOP and In-Shop, we decay the learning rate by 0.25 for all losses. The weight decay is set to 0.0001.
B.2 MIXUP SETTINGS
In mixup for classification, given a batch of n examples, standard form n pairs of examples by pairing the batch with random permutation of itself, resulting in n mixed examples, either for input or manifold mixup. metric learning, it is common to obtain n embeddings and then use all nn 1) pairs of embeddings in computing the loss. We thus treat mixup types differently.
Input mixup Mixing all pairs would be computationally expensive in this case, because we would compute fn(n- -1 embeddings. A random permutation would not produce as many hard examples as can be found in all pairs. Thus, for each anchor (each example in the batch), we use the k hardest negative examples and mix them with positives or with the anchor. We use it = 3 by default.
Manifold mixup Originally, manifold mixup (Verma et al., 2019) focuses on the first few layers of the network. Mixing all pairs would then be even more expensive than input mixup, because intermediate features (tensors) are even larger than input examples. Hence, we focus on the last few layers instead, where features and embeddings are compact, and mix all pairs. We use feature mixup by default and call Metrix/feature or just Metrix, while input and embedding mixup are called Metrix/inpul and Metrix/embedl, respectively. All options are studied in subsection B.4.
Mixing pairs Whatever the mixup type, clean examples anchors and define set (aa pairs of examples to mix for each anchor a, with their labels (positive or negative). By default, we mix positive-negative anchor-negative pairs, according M(a) == U U+(a) U- (a)
16

---

Published as a conference paper at ICLR 2022
CUB200
96
SOP
IN-SHOP
Method
2
100
20
Triplet (Weinberger& Saul, 2009)
75.6
84.4
85.4
90.8
85.6
94.3
97.8
LifeedStrucurre (Oh Song 2016)
65.9
75.8
84.5
81.4
88.3
92.4
76.1
88.6
95.2
88.6
97.6
98.4
PrA (Movohovitz-Atian etal. 2017) 2017)
65.2
75.6
83.8
81.2
87.9
92.6
73.2
87.0
94.4
86.2
95.9
97.0
Margin (Wu al., 2017)
65.0
76.2
84.6
82.1
88.7
92.7
74.8
87.8
94.8
88.6
97.0
97.8
SoftTriple (Qian et al, 2019)
67.3
77.7
86.2
86.5
91.9
95.3
79.8
91.2
96.3
91.0
97.6
98.3
D&C (Sanakoyeu al. 2019)*
76.6
84.4
90.7
94,1
88.4
94.9
85.7
96.9
EPSHN (Xuan et all,, 2020)*
64.9
75.3
83.5
82.7
89.3
93.0
78.3
90.7
96.3
87.8
95.7
96.8
ProxyNCA++ (Teb et al., 2020)*
79.8
92.5
95.7
96.7
90.4
Cont (Hadsell et at., 2006)
64.7
75.9
84.6
81.6
88.2
92.7
74,9
87.0
93.9
86.4
94.7
96.2
+Metrrx/inpur
66.3
77.1
85.2
82.9
89.3
93.7
75.8
87.8
94.6
87.7
96.3
+1.6
+1.2
+0.6
+1.3
+1.1
+1.0
+0.9
+0.88
+0.7
+1.3
+1.2
+0.3
+Metrrx
77.9
91.1
94.6
77.5
95.5
89.1
97.1
+2.7
+2.0
+1.1
+3.5
+2.9
+1.9
+2.6
+2.11
+1.5
+2.7
+1.0
+0.9
+Merrix/embbe
66.4
77.6
85.4
83.9
90.3
94.1
76.7
88.6
95.2
88.4
95.4
96.8
+1.7
+1.7
+0.8
+2.3
+2.1
+1.4
+1.8
+1.6
+1.3
+2.0
+0.7
+0.6
MS (Wang al,, 2019) -Metriz/input
67.8
77.8
85.6
87.8
92.7
95.3
76.9
89.8
95.9
90.1
97.6
98.4
69.0
79.1
86.0
89.0
93.4
96.0
77.9
90.6
95.9
91.8
98.0
98.9
+1.2
+1.3
+0.4
+1.2
+0.7
+0.7
+1.0
40.8
+0.0
+1.7
+0.4
+0.5
+Metrix
71.4
80.6
86.8
89.6
94.2
96.0
81.0
92.0
97.2
92.2
98.5
98.6
+3.6
+2.8
+1.2
+1.8
+1.5
+40.7
+4.1
+2.2
+1.3
+2.1
+0.9
+0.2
--Metrixeembed
70.2
80.4
86.7
88.8
92.9
95.6
78.5
91.3
96.7
91.9
98.3
98.7
+2.4
+2.6
+1.1
+1.0
+0.2
+0.33
+1.6
+1.55
+0.8
+1.8
+0.7
+0.3
PA (Kim all, 2020c)" (Kim 2020c)
69.7
80.0
87.0
87.7
92.9
95.8
79.3
87.0
92.3
95.5
79.1
90.8
96.2
90.0
98.2
+Metrri/input
70.5
81.2
87.8
88.2
93.2
96.2
79.8
91.4
96.5
90.9
98.1
98.4
+0.8
+1.2
+0.8
+0.5
+0.3
+0.4
+0.7
+0.6
+0.3
+0.9
+0.7
+0.2
+MMtrix
71.0
81.8
88.2
93.6
96.7
81.3
91.7
96.9
91.9
98.8
+1.3
+1.8
+1.2
+1.4
+0.7
+0,9
+2.2
+0.9
+0.7
+1,9
+0.8
+0.6
+-Merix/eembed
81.1
87.9
93.3
96.4
91.7
96.6
91.6
98.3
+0.7
+1.1
+0.9
+1.2
+0.4
+0.6
+1.5
+0.9
+0.4
+1.6
+0.9
+0.1
ProxyNCA++ (Teh et al., 2020)*
69.0
79.8
87.3
86.5
92.5
95.7
80.7
92.0
96.7
90.4
98.1
98.8
ProxNCA++ (Teb et al,, 2020)
69.1
79.5
87.7
86.6
92.1
95.4
80.4
91.7
96.7
90.2
97.6
98.4
+Metriiinnuut
69.7
79.9
88.3
87.5
92.9
96.0
80.9
92.2
96.9
91.4
98.1
98.8
+0.6
+0.1
+0.6
+0.9
+0.4
+0.3
+0.2
+0.2
+0.2
+1.0
+0.0
+0.0
+Merrix
70.4
80.6
$8.7
$8.5
93.4
96.5
81.3
92.7
97.1
91.9
98.1
98.8
+1.3
+0.8
+1.0
+1.9
+0.9
+0.8
+0.6
+0.7
+0.4
+1.5
+0.0
+0.0
+Metrix// embed
70,2
80.2
88.2
88.1
93.0
96.2
81.1
92.4
97.0
91.6
98.1
98.8
+1.1
+0.4
+0.5
+1.5
+0.5
+0.5
+0.4
+0.4
40.3
+1.2
+0.0
+0.0
Gain over
+0.5
+0.9
40.5
+1.2
Table Improving the SOTA with our Metrix (Metrix/feature) using Resnet-50 with embedding size d = 512. R@ K (%): Recall@ K; higher is better. ": reported by authors. Bold black: best baseline (previous SOTA, one per column). Red: Our new SOTA. Gain over SOTA is over best baseline. MS: Multi-Similarity, PA: Proxy Anchor
and M(a) == S(a) x (a), respectively, where U-(a) is replaced by hard negatives only for input mixup. The two options are combined by choosing uniformly at random in each iteration. More options are studied subsection B.4.
For any given mixup type or set of mixup pairs, the interpolation factor is drawn from Beta(a:, a)) with 2. We empirically set the mixup strength (10) to w 0.4 for positive-negative pairs and anchor-negative pairs.
Hyper-parameters
B.3 MORE RESULTS
Computational complexity On CUB200 dataset, using a batch size of 100 on an NVIDIA RTX 2080 Ti GPU, the average training time in ms/batch is 586 for MS and 817 for MS+Meurix. The 39% increase in complexity is reasonable for 3.6% increase in R@1 1. Furthermore, the average training time in ms/batch is 483 for baseline PA, 965 for PA+Metrix and 1563 for PS eGu et al., 2021). While the computation cost of PS is higher than Metrix by 62%, Metrix outperform PS by 0.4% and 1.3% in terms of R@1 and R @2 respectively (Table 3). At inference, the computational cost is equal for all methods.
17

---

Published as a conference paper at ICLR 2022
Query
Top-5 Retrieved Images
Cont.
Cont.
Metrix
Cont.
Cont.
Metrix
Figure 4: Retrieval results CUB200 using Contrastive loss, with and without mixup. For each query, the top-5 retrieved images are shown. Images highlighted green (red) are correctly (incor- rectly) retrieved images.
Improving the state of the art Table 5 is extension of Table 2 that includes all three mixup types (input, feature, embedding). It shows that not just feature mixup but all mixup types con- sistently improve the performance of all baseline losses (Cont, MS, PA, ProxyNCA++) across all datasets. also shows that across all baseline losses and all datasets, feature mixup works best, followed by embedding and input mixup. This result confirms the findings of Table Cars196.
Qualitative results of retrieval Figure 4 shows qualitative results of retrieval CUB200 using Contrastive loss, with and without mixup. This dataset has large intra-class variations such as pose variation and background clutter. Baseline Contrastive loss may fail retrieve the correct images due to these challenges. The ranking is improved in the presence of mixup.
Visualization of embedding space We visualize CUB200 test examples for 10, 15 and 20 classes in the embedding space using Contrastive loss, with and without mixup Figure 5. We observe that in the presence of mixup, the embeddings are more tightly clustered and more uniformly spread, despite the variations in pose and background the test set. This finding validates our quantitative analysis of alignment and uniformity in subsection 4.3.
B.4 ABLATIONS
We perform ablations on Cars using R-50 with d = 512, applying mixup on contrastive loss.
Hard negatives We study the effect of the number hard negatives using different mixup types. The set of mixing pairs is chosen from (positive-negattiv,, anchor-negative) uniformly at random per iteration. We choose = 3 for input mixup. For feature/embedding mixup, we mix all pairs a batch by default, but also study & E {20, 40). As shown in Table 6, for input and all pairs for feature/embedding mixup works best. Still, using few hard negatives for feature/embedding mixup on par outperforms input mixup. All choices significantly outperform the baseline.
18

---

Published as a conference paper at ICLR 2022
10 classes
15 classes
20 classes
+
Cont.
Cont.
+
Metrix
Figure 5: Embedding space visualization of CUB200 test examples of a given number of classes using Contrastive loss, with and without mixup.
Mixing pairs We study the effect of mixing pairs (a), in particular, (a) (positive-positive), U+ (a) x (-(a) (a) (positive-negative) and S(a) x (- (anchor-negative), again using different mixup types. As shown in Table 6, when using single set of mixing pairs during training, positive- negative and anchor- anchor-negative consistently outperform the baseline, while resitive-positive outperformed by the baseline. This may be due to the lack of negatives in the mixed loss (9), despite the presence of negatives in the clean loss (3). Hence, we only use positive-negative and anchor- negative by default, combined by choosing uniformly random in each iteration.
actually
Mixup types We study the effect of mixup type (input, feature, embedding), when used alone. The set of mixing pairs is chosen from (positive-negative, anchor-negative) uniformly at random per iteration. As shown in both "hard negatives" and "mixing pairs" parts of Table 6, our default feature mixup works best, followed by embedding and input mixup.
Mixup type combinations We study the effect of using more than one mixup type (input, feature, embedding), chosen uniformly at random per iteration. The set of mixing pairs is also chosen from (positive-negative, anchor-negative) uniformly at random per iteration. As shown in Table 6, mixing inputs, features and embeddings works best. Although this solution outperforms feature mixup alone by 0.2% Recall @ (85.1 + 85.3), it is computationally expensive because of using input mixup. The next best efficient choice is mixing features and embeddings, which however worse than mixing features alone (84.7 vs. 85.1). This is why we chose feature mixup by default.
Mixup strength w We study the effect of the mixup strength w in the combination of the clean and mixed loss (10) for different mixup types. As shown in Figure 6, mixup consistently improves the baseline and the effect of w is small, especially for input and embedding mixup. Feature mixup works best and slightly more sensitive.
Ablation on CUB200 We perform additional ablations on CUB200 using R-50 with d 128 by applying contrastive loss. All results are shown in Table 7. One may draw the same conclusions as from Table 6 on 196s196 with 512, which confirms that our choice of hard negatives and mixup pairs is generalizable across different datasets and embedding sizes.
In particular, following the settings of subsection B.4, we observe Table that using A = 3 hard negatives for input mixup and all pairs for feature/embedding mixup achieves the best performance in terms Recall@1. 1. Similarly, using a single set of mixing pairs, Dositive-negative and anchor- negative consistently outperform the baseline, whereas resitiv--osiitvve is inferior than the baseline. Furthermore, combining nestaiv and anchor-negee negative pairs by choosing uniformly at ran- dom in each iteration achieves the best overall performance.
19

---

Published as a conference paper at ICLR 2022
STUDY
HARD NEGITIVES
MIXING PAIRS
MIXUP TYPE
R@0
R@2
R@4
R
baseline
81.6
88.2
92.7
95.8
pos-neg lanc-neg
inpui
82.0
89.1
93.1
96.1
pos--eeg neg ane neg
ingut
82.5
89.2
93.4
96.2
pos-nig anc-neg
input
pos pos-neg neg
hard negatives
40
pos-neg/ / aanc-ne
feature
84.0
90.4
94.2
96.8
pos-ncg/ anc-noo
feature
85.1
91.1
94.6
97.0
20
pos-neg aanc-ne
embed
82.7
89.2
93.4
96.1
pos-ncg anc-nog
embed
83.0
96.4
all
pos-nig anc-nog
embed
83.4
89.9
94.1
96.4
pos-pos
input
81.0
88.2
92.6
95.6
pos-neg
input
82.4
89.1
93.3
95.6
neg
inpui
81.8
89.0
93.6
95.4
pos-pos
feature
mising pains
all
neg anc-neg
feature
84.0
90.2
94.2
96.6
feature
96.7
pus-pux
embed
78.3
85.7
90.8
94.4
pos-neg
embed
83.1
90.0
93.9
96.6
anc-nog
embed
82.7
89.5
93.5
96.3
[1, all)
pos-neg / lanc
(input, feature)
83.7
94.2
95.9
96.7
mixup type combisations
[3, all)
pos-neg/ anc-neg
(input, embod]
83.0
94.1
96.4
{all. all)
pos-neg anc-nog
{feature. embed]
84.7
90.6
94.4
96.9
(1, all, all)
pos-neg anc nog
{ finput, leature, embed)
85.3
94.9
96.2
97.1
Table Ablation study of our Metrix using contrastive loss and R-50 with embedding size 512 on C9rs196. @@ K (%). Recall @ K; higher is better.
90
bascline
input
88
embedding
feature
86
Recall
84
82
80
0.2
0.4
0.6
0.8
mixup strength
Figure 6: Effect of mixup strength for different mixup types using contrastive loss and R-50 with embedding size d = 512 on Cars196. Recall@ K (%): higher is better.
20

---

Published as a conference paper at ICLR 2022
STUDY
HARD NEGATIVES &
MIXING PAIRS
MIXUP TYPE
R@i
R@2
R@4
R@8
baseline
61.6
73.7
83.6
90.1
pos-neg neg lamc
ingut
62.4
73.9
83.0
89.7
pos-meg ane-neg
input
62.7
74.2
83.6
90.0
pos ney neg
input
63.1
74.5
83.5
90.3
pos-neg Lanc-neg
feature
63.9
75.0
83.9
89.9
hard negatives
40
pos-nig anc-nog
feature
63.5
75.2
83.5
89.8
pos-neg/ anc-neg
feature
pos-ncg/ anc-nog
embed
63.1
83.1
90.0
40
pos-neg
embed
63.5
74.7
83.6
90.1
pos neg une neg
embeed
64.0
75.1
84.8
90.9
pos-pus
input
58.7
70.7
80.1
87.1
pos neg
ingut
input
62.8
74.7
83.6
90.1
pus pus-pus
feature
61.0
73.1
82.5
89.7
mixing pains
pos-neg anc-nog
feature
63.9
75.0
83.9
89.9
feature
63.8
74.8
90.2
pos-poss
embed
all
pos-neg
embed
63.8
75.1
83.3
90.5
anc-big
embed
63.5
75.0
83.9
90.5
[1, all)
pos-neg anncneg
(input, feature)
63.9
75.1
84.9
90.5
mixup type combinations
pos-ncg anc-nog
{input, embed)
63.4
[all, all)
pos-nig une anc-neg
[Teature, embed
64.2
75.2
84.1
90.7
(1. all, all)
pos-neg anc-nog
(inpus, feature, embed)
65.3
76.2
84.4
91.2
Table Ablation study of our Metrix using contrastive loss and R-50 with embedding size 128 on CUB200. R K (%). Recall @ K; higher is better.
We also study the effect of using more than one mixup type (input, feature, (eature,,mbeedding), chosen uniformly random per iteration. The of mixing pairs is also chosen from (positive-negative, anchor-negative) uniformly at random per iteration in this study. From Table 7, we observe that al- though mixing input, features and embedding works best with an improvement 0.8% over feature mixup alone (64.5 + 65.3), it is computationally expensive due to using input mixup. The next best choice is mixing features and embeddings, which is worse than using feature mixup alone (64.2 vs. 64.5). This confirms our choice of using feature mixup as default.
21


Reviews:
[{"review_id": "z2jhACE6I1Z", "reviewer": "Reviewer_LaZ6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposes a mixup strategy to augment the data, interpolate the labels and design a new loss function for deep metric learning. Overall, the paper is well written and easy to follow. And the results are improved compared with other models. \b\n\nI am not an expert in this area so the following concerns might not be important but I wish the authors could clarify. From my point of view, the results are impressive. The datasets used in the experiment are also popular enough for supporting the claims. \n\nHowever, there are two concerns:\n\n1) Although the authors propose the new general mixup loss function, the key elements of mixup from the paper follows from Zhang et al., 2018; Verma et al., 2019 and Venkataramananetal.,2021. In particular, the authors mentions \"the idea of interpolating target labels is not straightforward.\" However, it seems that label interpolation is natraully derived from Zhang et al., 2018 in the paper. Could the authors please elaborate the key contributions from the paper? Is it the new generic loss function for mixup?\n\n2) The authors propose a utilization method to support the claim of exporing new space using L2 distance in embedding space, which is not very convicing to me since L2 distance hardly captures information at the distribution level. From my point of view, the lower of the utilization score does not necessarily mean the model is exploring new and meaningful space. So could the author please elaborate on what is the key reason of using L2 distance in this case? \n", "summary_of_the_paper": "The paper mentions the missing of studying both metric loss function and data augmentation techniques for the metric learning problem and proposes to use a mixup strategy for the improvement.  The authors claim the better results over the state-of-the-art using the mixup and use a new metric (utilization) to claim their method is exploring new space. ", "summary_of_the_review": "The authors propose a new and generic loss function in mixup scenario for deep metric learning. The experiments on popular datasets mostly support the claims in the paper with the needs of explaining some of ideas mentioned in the paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "g7YsmoAP6u", "reviewer": "Reviewer_gPed", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1) This is a good paper which has techincal novelty  - generalized loss for metrics learning, which allows to incorporate mixap augmentation. The authors also illustrate the effectiveness of the suggested technique for various losses previously used for Metric Learmning(Table 2). \n2) The paper also suggest theoretical analysis of the \"positivity\", which is used for loss definition, (Section 3.6 and Figure 1) which aligns with empirical results.\n3) The paper also performs comparison to different mixup methods and shows its superiority (Table 3)\n\n\nCons:\n1) The paper is hard to read because many important formulations are found in the appendix (e.g evaluation used for Table 2 and Table 3 Recall @ K, and what K stands for is only explained in appendix B1, the discussion about influence of the approach on the training time is only found in B3,  a more eleborative discussion on positivity is presented in A2 and there are more examples). So the paper might benefit from  substantial revision or maybe  a resubmission to a journal might be considered.\n2) The paper introduces new evaluation metric, utilization, validating that a representation more appropriate for test classes is implicitly learned during exploration of the embedding space in the presence of mixup. What is not clear to me is if this metrics is measured on the classes that particiapte in training (After eq.12 -  Utilization measures the average, over the test set Q, of the minimum distance of a query q to a training example x ∈ X in the embedding space of the trained model f (lower is better)). If this is the case this metrics is not really interesting with respect to the actual goal of metrics learning (to be able to generalize for unseen classes) -- it would be helpful if the authors can clarify this point and also elaborate why this metrics is more interesting than other available metrics. \n3) Minor issue: In Table 4 SOP dataset - how can we get 5 examples per class if there are 22, 634 classes and 60, 026 images in training (so if we assume balanced dstribution of classes we will get ~2.65 images per class) -- are you using the same images?\n4) Are you normalizing embeddings ? \n \n\n", "summary_of_the_paper": "The paper deals with the problem of metrics learning. It proposes to extend Mixap data augmentation from calssification to metrics learning. This Mixap data augmentation approach interpolates two or more examples (either directly in the input or the corresponding embeddings - eq.4) and corresponding target labels at a time. This task is challenging because unlike classification, the loss functions used in metric learning are not additive over examples and hence the authors claim that the idea of interpolating target labels is not straightforward.\nThe authors suggest a general formulation to accomodate mixup for many losses currenly used in metrics learning (see eqs. 3, 7, 9, 10 for generic formulation  and  eqs. 13, 14,15 for formulation of specific losses). The authors provie extensive experiments that show the effectiveness of their techqniue (Table 2)  on 4 different dataset (Table 4) \n\n", "summary_of_the_review": "I recommend to accept this paper because I think that the paper is relevant to the ICLR community and has some innovation supported by experiments (and some minor theory).  It also serves as a good survey for the community with many useful details for AI practicionaries in the field of metric learning, which are outlined in appendix. However, my concern is that the paper is not well organized and maybe better suited for a journal venue. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "e35alfO8hoL", "reviewer": "Reviewer_EXEY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n1. The paper is very clearly written and easy to follow. I really like the part on building the loss function in (9) step-by-step. \n2. I really appreciate the extensive experiments that have been done in the paper. \n3. The generalization of the loss function seems to be pretty standard, but the simple solution turns out to be very effective.\n\n**Weaknesses**\n1. My main concern is in terms of the error function in (10). Is there any explanation why the first clean loss must be there? What if only the mixed loss term is used as an objective function.\n2. I appreciate the authors’ effort on the analysis in section 3.6 although I am not convinced what is the message we can get from (11). Moreover, the objective function also has the clean loss term and the analysis is only in terms of the mixed loss.\n\nMinor Issue\n1. In Section 4.1 in mixup settings, M(a) only uses positive-negative and anchor-negative pairs, while in section 3.5 in the description of M(a) positive-positive pairs are also defined as a subset of M(a), would it hurt the experiment results if positive-positive pairs are also used?\n2. In the utilization section, the word \"significantly\" is used which is very dangerous in statistics. The uncertainty is not quantified, therefore there isn't enough evidence to use this word.\n", "summary_of_the_paper": "The paper proposes a generalized loss function for the purpose of using mixup in deep metric learning, which extends the some existing loss functions for deep metric learning without mixup. Extensive experiments are conducted to show the superior performance of the proposed method.\n", "summary_of_the_review": "The paper is overall very clearly written and easy to follow, the authors made a great effort to conduct the experiments in the paper. I am not an expert in this area and may have missed some related work in this area. My main concern is in terms of the error function in (10) as described in the main review. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "LEMgcvlhhQ9", "reviewer": "Reviewer_c8hM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros.\n1) It adapts a generalized formulation of loss function to the mixup technique, such that both examples (inputs) and target labels (outputs) can be integrated to yield better embeddings or representations.\n2) It proposes a new metric named utilization to examine the improvements of representation in the embedding space.\n3) It provides convincing empirical results on benchmark data sets.\n\n\nCons.\n1) Previous works have shown that the effectiveness of the mixup for interpolating embeddings and labels (Zhang et al., 2018; Verma et al., 2019). This work borrows the same idea on input samples and labels rather than using embeddings. Such exploration should be encouraged, but the novelty is a bit weak. Other mixup variants might be better choices to improve novelty.\n2) More analysis is required on error function in Eq. (10).\n3) Some qualitative comparison results can be shown to further verify the improvement brought by mixup.", "summary_of_the_paper": "This work explores the way of mixing both examples and target labels for deep metric learning. It considers metric learning loss and data augmentation technique (e.g., mixup) together when handling two or more examples at a time. A generalized formulation of loss function was modified to accommodate for the mixup technique. Also, a new metric called utilization is introduced for evaluating representation improvements. Some theoretical analysis are provided. A number of experiments were conducted on four benchmarks to show the superiority of the proposed method.", "summary_of_the_review": "This paper is generally well written and easy to read. It provides sound technique details with theoretical analysis. Also it presents lots of experimental results to support its claim that the mixup of inputs and outputs improves the representation quality. However, the novelty is of a bit limited. Overall, it is an acceptable paper that will attract attention from relevant researchers.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "3-gfCuBS-Wh", "reviewer": "Reviewer_S9kx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n------------\n1) A general form of loss function for DML that can be easily converted to specific losses like Contrastive, Proxy-anchor, Multi-similarity etc. + incorporating mixup to this.\n2) Good experimental validation showing the effectiveness of the proposed approach compared to the baseline and the SOTA \n\nWeaknesses: \n-----------------\n1) Limited contribution/significance: The two main contributions of the paper are representing the loss function in a general form and incorporating mixup into this loss function (using continuous labels instead of discrete labels as used in the existing dml). I think the idea of using a general dml loss function is inspired from Wang et. al 2019 which uses Multi-similarity loss as a general dml loss function. The loss function in Eqn. (3) is a modified version of MS loss Eqn. (2). Mixup has been extensively used in classification tasks and three different types (input/feature/embedding) have also been proposed previously. Although this is the first time mixup has been applied to dml, the contribution of the paper seems to be limited.\n\n2) Limited scope: The proposed mixup approach is tailored to the dml problem in this work. Hence, its scope beyond dml is limitted. Although the paper mentions that the proposed approach can be extended to transfer/few-shot/continual learning, \"our work may have applications to other such problems, including transfer learning, few-shot learning and continual learning\", it would be difficult because the method assumes a particular form of loss function which may not hold true in other cases.\n\n3) Experiments: It would interesting to see some images for the following cases:\na) the mixup training (for input mixup) images for different values of \\lambda. \nb) how the tsne plots for the mixup samples would look like? \nc) the nearest neighbors for different values of \\lambda (obtained by comparing the features in the embedding space)\n\n", "summary_of_the_paper": "The paper presents a technique for using mixup augmentation in deep metric learning training. Specifically, the dml loss function is represented in a general form so that mixup loss can be easily computed for different pairs. This loss is combined with regular dml loss for training the network. The method is evaluated on CUB200, CARS196, SOP and IN-SHOP datasets where it outperforms it's baselines and also achieves SOTA results.", "summary_of_the_review": "Although the proposed approach seems to be novel (applying mixup to dml), the scope of the technique is limited to the dml applications only and the contributions are marginal. Also, showing some images of the retrieval, tsne plots would be helpful and can provide more insights into the problem.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper adapts the mixup data augmentation strategy to the case of metric learning. The main challenge addressed is the fact that in metric learning, the loss function does not treat each example as an IID sample. The paper takes the view of metric learning as learning over positive and negative pairs (those belonging to the same/different classes) and uses this to develop a fairly general metric-mixup formulation. To measure the effectiveness of the approach for metric learning, the paper introduces a new measure called utilization that looks at the distance of a query point to its nearest training point in embedding space.

The reviewers (5 of them) all favour acceptance on the grounds of novelty, and the performance of the method. During the discussion, some issues were raised around whether utilization is a useful measure, improvements to the paper clarity, whether the clean loss in eq. 10 is necessary, and potential limitations on the generality of the approach. However, additional experiments and clarification during the discussion period has resolved these issues to their satisfaction.
Decision: Accept (Poster)