INSTANCE CROSS ENTROPY FOR DEEP METRIC LEARNING
Authors: Xinshao Wang
Elyor Kodirov
Yang Hua
Neil M. Robertson
Venue:  (2020)
Keywords: Deep Metric Learning
Instance Cross Entropy
Sample Mining/Weighting
Image Retrieval

Abstract: Loss functions play a crucial role in deep metric learning thus a variety of them have been proposed. Some supervise the learning process by pairwise or tripletwise similarity constraints while others take the advantage of structured similarity information among multiple data points. In this work, we approach deep metric learning from a novel perspective. We propose instance cross entropy (ICE) which measures the difference between an estimated instance-level matching distribution and its ground-truth one. ICE has three main appealing properties. Firstly, similar to categorical cross entropy (CCE), ICE has clear probabilistic interpretation and exploits structured semantic similarity information for learning supervision. Secondly, ICE is scalable to infinite training data as it learns on mini-batches iteratively and is independent of the training set size. Thirdly, motivated by our relative weight analysis, seamless sample reweighting is incorporated. It rescales samples’ gradients to control the differentiation degree over training examples instead of truncating them by sample mining. In addition to its simplicity and intuitiveness, extensive experiments on three real-world benchmarks demonstrate the superiority of ICE.

Under review as a conference paper at ICLR 2020
INSTANCE CROSS ENTROPY FOR DEEP METRIC LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Loss functions play a crucial role in deep metric learning thus a variety of them have been proposed. Some supervise the learning process by pairwise triplet- wise similarity constraints while others take advantage of structured similarity information among multiple data points. In this work, we approach deep metric learning from a novel perspective. We propose instance cross entropy (ICE) which measures the difference between an estimated instance-leve matching distribution and its ground-trune -tuuth one. ICE has three main appealing properties. Firstly, sim- ilar to categorical cross entropy (CCE), ICE has clear probabilistic interpretation and exploits structured semantic similarity information for learning supervision. Secondly, ICE is scalable to infinite training data as it learns on mini-batches it- eratively and is independent of the training set size. Thirdly, motivated by our relative weight analysis, seamless sample reweighting is incorporated. It rescales samples' gradients to control the differentiation degree instead of truncating them by sample mining. In addition to its simplicity and intuitiveness, extensive experiments on three real-world benchmarks demonstrate the superiority of ICE.
training examples
INTRODUCTION
Deep metric learning (DML) aims to learn non-linear embedding function (a.k.a. distance metric) such that the semantic similarities over samples are well captured in the feature space (Tadmor al, 2016 |Shnn 2016). Due to its fundamental function of learning discriminative representations, DML has diverse applications, such as image retrieval (Song et al. (0166), clustering (Song et al.) (017)) verification (Schroff et al. 2015). few-shot learning (Vinyals et al. 2016) and zero-shot learning
(Bucher et al. 2016).
A key to DML to design an effective and efficient loss function for supervising the learning pro- cess, thus significant efforts have been made (Chopra al. 2005 Schroof et al. 2015 Sohn [016
Song et al. 2016 2017 Law et al., 2017, Wu al. 2017). Some loss functions learn the embedding function from pairwise or triplet-wise relationship constraints (Chopra et al. 2005 Schroff al., 2015 Tadmor et al. 2016). However, they are known to not only suffer from an increasing num- ber of non- non-informative samples during training, but also incur considering only several instances per loss computation. Therefore, informative sample mining strategies are proposed (Schroff et al.) 2015 Wu et al., [017 Wang et al.) 2019b). Recently, several methods consider semantic relations among multiple examples to exploit their similarity structure (Sohn) 2016 [ong et al. 2016)] 2017 Law et al., 2017). Consequently, these structured losses achieve better performance than pairwise and triple- wise approaches.
In this paper, we tackle the DML problem from a novel perspective. Specifically, we propose a novel loss function inspired by CCE. CCE well-known in classification problems owing to the fact that it has an intuitive probabilistic interpretation and achieves great performance, e.g., ImageNet classification (Russakovsky et al. 2015). However, since CCE learns a decision function which predicts the class label of input, it learns class-level centres for reference (Zhang et al. 2018 Wang et al., 2017a). Therefore, CCE not scalable to infinite classes and cannot generalise well when it is directly applied to DML (Law al. (017).
With scalability and structured information in mind, introduce instance cross entropy (ICE) for DML. learns an embedding function by minimising the cross entropy between predicted instance-level matching distribution and its corresponding ground-truth. In comparison with CCE,

---

Under review as a conference paper at ICLR 2020
given query, CCE aims to maximise its matching probability with the class-level context vector (weight vector) of its ground-truth class, whereas ICE targets at maximising its matching probability with it similar instances. As ICE does not learn class-level context vectors, it is scalable to infinite training classes, which is an intrinsic demand of DML. Similar to (Sohn, 2016) Song et al., 2016) 2017 Law al., 2017 Goldberger et al. 2005 Wu et al. 2018). ICE is structured loss as also considers all other instances in the mini-batch of a given query. We illustrate ICE with comparison to other structured losses in Figurel!
A common challenge of instance-basee losses is that many training examples become trivial as model improves. Therefore, we integrate seamless sample reweighting into ICE, which functions similarly with various sample mining schemes (Sohn) 2016, [Schroff et al.) 2013] Shi et al. 2016
Yuan et al. 2017] Wu et al. (017). Existing mining methods require either separate time-consuming
process, e.g. class mining (Sohn 2016), or distance thresholds for data pruning (Schroff et al.
2015 Shi et al., 2016 Yuan et al., 2017 Wu et al., (017). Instead, our reweighting scheme works without explicit data truncation and mining. is motivated by the relative weight analysis between two examples. The current common practice of DML is to learn angular embedding space by projecting all features to a unit hypersphere surface (Song et al. 2017] Law et al. 2017 Movshovitz- Attias al. 2017). We identify the challenge that without sample mining, informative training examples cannot be differentiated and emphasised properly because the relative weight between two samples is strictly bounded. We address it by sample reweighting, which rescales samples' gradient to control the differentiation degree among them.
Finally, for intraclass compactness and interclass separability, most methods (Schroff' et al. 2015 Song et al. 2016 Tadmor al.] 2016) Wu al. 2017) use distance thresholds decrease intraclass variances and increase interclass distances. In contrast, we achieve the target from a perspective of instance-level matching probability. Without any distance margin constraint, ICE makes no sumptions about the boundaries between different classes. Therefore, ICE is easier to apply in applications where we have no prior knowledge about intraclass variances.
Our contributions are summarised: (I) We approach DML from a novel perspective by taking in the key idea of matching probability in CCE. We introduce ICE, which is scalable to an infinite number of training classes and exploits structured information for learning supervision.
(2) seamless sample reweighting scheme is derived for ICE to address the challenge of learning an embedding subspace by projecting all features a unit hypersphere surface. (3) We show the superiority of ICE by comparing with state-of-the-an art methods on three real-world datasets.
2 RELATED WORK
2.1 STRUCTURED LOSSES BY QUERY VERSUS CLASS CENTRES
Heated-up. NormFace, TADAM, DRPR, Prototypical Networks, ProXy-NCA. These methods calculate the similarities between a query and class centres (a.k.a. proxies or prototypes) instead of other instances (Zhang et al. 2018 Wang al. 2017 Oreshkin et al., 2018 Law et al. 2019 Snell et al.] 2017; Movshovitz- Attias et al. 2017). In Heated-up and NormFace, the class centres are learned parameters of fully connected layer, which is similar Center Loss (Wen al. 2016). While in TADAM. DRPR, and Prototypical Networks, a class centre is the mean over all embeddings of a class. By comparing& a sample with other examples other than class centres, more informative instances can contribute more in ICE.
2.2
STRUCTURED LOSSES BY QUERY VERSUS INSTANCES
NCA GGoldbrger et al., 2005), S-NCA (Wu et al., 2018). NCA learns similarity relationships between instances. Since original NCA learns the whole training data and its time complexity is quadratically proportional to the scale of training data, S-NCA is proposed recently with linear time complexity with respect to the training data size. Instead, ICE is scalable to infinite training data by iterative learning on randomly sampled small-scale instances matching tasks. S-NCA and NCA share the same learning objective. However, they treat the event of all similar instance being correctly recognised as a whole by a sum accumulator. Instead, we maximise the probability of every similar sample being correctly identified individually. Therefore, ICE's optimisation task is harder, leading to better generalisation.
N N-pair-mc (Sohn) 2016). The aim of N N-pair-me is to identify one positive example from N negative examples of N classes (one negative example per class). In other words, only one
2

---

Under review as a conference paper at ICLR 2020
SN
P2
CE
SN
pt
CE
p
mini-barch
Learning class centroids
Dot product
distribation
Matching
GT
mini-barch
Computing class means
Dot product
distribution
Matching
dim
GT
(a) A query versus learned parametric class cen- troids. All T classes in the training set consid- ered. Prior work: CCE, Heated-up (Zhang al. 2018), NormFace (Wang al. 2017a).
(b) A query versus cam--aaammtric class means. Only classes in the mini-batch considered. Representative
work: TADAM (Oreshkin et al.) 2018), DRPR (Law
at 2019). Prototypical Networks (Snellet al. 2017).
SN
pi
pu
prtpz pa p
CE
3
p2
pu
pa
A mini-batch
Sampling and dot product
Matching
GT
distribution
dim =
mini batch
Dat product
Matching distribution
Neighbourhood grouping
GT 2005) and S-NCA
query versus one instance per class. mini-batch has to be exam- plcs per class rigidly. Only one instance per nega- tive class is randomly sampled out of 2.
(c) N N-pair-ma me (Sohn 2016):
(d) NCA
Goldherger et al.
Wu al.
2018) A query versus the rest instances.
A query of circle class
pi
CE
Instances of circle and rectangle, respectively.
3
pa
Statistically computed class means of circle, rectangle, respectively, Only classes in the mini-batch are considered.
pi
Learned class centroids (weight vectors) of circle, rectangle, triangle, respectively.
mini-batch
3
ps
Triangie class in the training set but examples sampled this mini-batch.
product
distribution
Matching
dlim
= Step forward
g: Dot product between two representations.
(c) Our ICE: query versus one positive and all negatives per distribution. A query's number of matching distributions is defined by the number of its positive examples.
SN
Softmax normalisation (SN) after doc product.
CE
The cross entropy (CE)
distribution p relative to ground-truth
distribution one-hot
Figure Our ICE and related losses. The first row shows prior work of a query versus class centres/means while the second row displays the work of query versus instances. Note that the cross entropy computation and interpretation are different in different losses. For mini-batch, show two classes, i.e., circle and rectangle, with 3 examples per class except N-pair-me which requires samples per class. The icons are at the right bottom. GT means ground-truth matching distribution. When illustrating the losses of query versus instances in (c), (d) and (e), index those instances with numbers for clarity, except the query.
positive and one negative instance per class are considered per loss computation by simulating CCE exactly. Instead, ICE exploits all negative examples benefit from richer information. When constructing mini-batches, N-pair-me requires expensive offline class mining and samples images per class. According to (Sohn 2016) N-pair-me is superior NCA.
Hyperbolic (Nickel & Kiela 2018). It aims to preserve the similarity structures among instances as well. However, it learns a hyperbolic embedding space where the distance depends only on norm of embeddings. Instead, learn an angular space where the similarity depends only on the angle between embeddings. Besides, Hyperbolic requires separate sampling of semantic subtrees when the dataset is large.
2.3 SAMPLE MINING AND WEIGHTING
Mining informative examples or emphasising on them are popular strategies in DML: 1) Min- ing non-trivial samples during training is crucial for faster convergence and better performance.
3

---

Under review as a conference paper at ICLR 2020
Therefore, sample mining is widely studied in the literature. In pairwise triplet-wise approaches (Schroff et al. 2013: Wu et al., [017]] Huang et al.] 2016)] Yuan et al., 2017). data pairs with higher losses emphasized during gradient backpropagation. As for structured losses, Lifted Struct (Song et al., 2016) also focuses on harder examples. Furthermore, (Sohnj 2016) and (Suh et al.) 2019) propose mine hard negative classes construct informative input mini-batches. Proxy- NCA (Movshovitz-Attias et al. (0177) addresses the sampling problem by learning class proxies. 2) Assigning higher weights to informative examples is another effective scheme (Wang et al. 2019a Beyond, there are some other novel perspectives to address sample mining or weight- ing, e.g. hardness-aware examples generation (Zheng et al., 2019) and -nnncuu of the embedding space (Sanakoyeu et al. 2019).
Our proposed ICE has a similarity scaling factor which helps to emphasise more on informative examples. Moreover, as described in (Schroff' et al., 2013). very hard negative pairs are likely to be outliers and it is safer to mine semi-hard ones. In ICE, the similarity scaling factor is flexible in that it controls the emphasis degree on harder samples. Therefore, a proper similarity scaling factor can help mine informative examples and alleviate the disturbance of outliers simultaneously. What makes ours different is that we do not heuristically design the mining or weighting scheme. Instead, it is built-in in and simply scale it as demonstrated in Section 3.4
2.4 DISCUSSION
We remark that Prototypical Networks, Matching Networks (Vinyals et al. 2016 and NCA are also scalable and do not require distance thresholds. Therefore, they are illustrated and differentiated in Figure| Matching Networks are designed specifically for one-shot learning. Similarly, (Tn- antafillou et al., 2017) design mAP-SSVM and mAP-DLM for few-shot learning, which directly optimises the retrieval performance mAP when multiple positives exist. FastAP (Cakir et al. 2019) is similar to (Triantafillou et al., 2017) and optimises the ranked-based average precision. Instead, ICE processes one positive at time. Beyond, the setting of few-shot learning is different from deep metric learning: Each mini-batch is complete subtask and contains support set as training data and a query set as validation data in few-shot learning. Few-shot learning applies episodic training in practice.
Remarkably, TADAM formulates instances versus class centres and also has a metric scaling pa- rameter for adjusting the impact of different class centres. Contrastively, ICE adjusts the influence of other instances. Furthermore, ours is not exactly distance metric scaling since we simply apply naive cosine similarity as the distance metric at the testing stage. That is why we interpret it as a weighting scheme during training.
INSTANCE CROSS ENTROPY
an input mini-batch, where x; E Axxwx3 and y: E {I....,(] C'} represent i-th image and the corresponding label, respectively; {x{) is a set of No images belonging to c-th class, Vc, .N 2 2. The number of classes C' is generally much smaller than the total number of classes T in the training set (C & T). Note that T is allowed to be extremely large in DML. Given sufficient number of different mini-batches, our goal is to learn an embedding function f that captures the semantic similarities among samples in the feature space. We represent deep embeddings of X as {(fe = [ - Given a query, 'positives' and 'negatives' refer to samples of the same class and different classes, respectively.
Notation. X [(x,, 9:)) ((x0) No
3.1 REVISITING CATEGORICAL CROSS ENTROPY
CCE is widely used in variety of tasks, especially classification problems. As demonstrated in (Liu et al., 2016). a deep classifier consists two joint components: deep feature learning and linear classifter learning. The feature learning module is transformation (i.e., embedding function composed of convolutional and non-linear activation layers. The classifier learning module has one neural layer, which learns T class-level context vectors such that any image has the highest compatibility (logit) with its ground-truth class context vector:
p(Wy, (x:) exp(f, ( w
and LCOR W) = ΣN =
logp(w,,, (xi),
(1)
Ik=I exp(f, Wx)
4

---

Under review as a conference paper at ICLR 2020
where f f(x)) E Rd is d-dimensional vector, p(w,, [x,) ) is the probability (softmax normalised logit) of x; matching wy. W = { I is the learned parameters of the classifier. During training. the goal is to maximise the joint probability of all instances being correctly classified. The identical form is to minimise the negative log-likelihood, i.e., LOCe (X; W). Therefore, the learning objective of CCE is:
arg max J.W
p( v. x.) = arg min (x; f.).
(2)
J.W
3.2 INSTANCE CROSS ENTROPY
In contrast to CCE, ICE is a loss for measuring instance matching quality (lower ICE means higher quality) and does not need class-level context vectors. We remark that an anchor may have multiple positives, which are isolated in separate matching distributions. There is a matching distribution for every anchor-positive pair versus their negatives as displayed in Figure
Let E. be a random query, we compute its similarities with the remaining points using dot product. We define the probability of the given anchor x,, matching one of its positives xf (i o a) as follows:
exp(fe ff)
p(x
exp(ff (?) + Eofe exp(ts fÿ)"
where f. f is the similarity between x and x in the embedding space, Lote E, exp(f; fj) the sum of similarities between x and its all negatives. Similarly, when the positive is xf, the probability of one negative point x2(0 # e) matching the anchor is:
exp(fe T f9) exp(fe f9) Eoje E, exp(f P9)
p(x}) x x,)
We remark: (1) Dot product measures the similarity between two vectors; (2) Eq. represents the probability of a query matching a positive while Eq. ( is the probability of query matching its ground-truth class. To maximise p(x; xx.) and minimise p(x;) X xf) simultaneously, we minimise the Kullback-Leibler divergence (Kullback & Leibler 1951) between the predicted and ground-truth distributions, which is equivalent to minimise their cross entropy. Since the ground-truth distribution log (x)
one-hot encoded, the cross-entropy
To be more general, for the given anchor x there may exist multiple matching points when N: > 2, i.e. [x] hizel = NNe 1 > 1. In this case, we predict one matching distribution per positive point. Our goal is to maximise the joint probability of all positive instances being correctly identified, i.e., Pxg
Teza pxxfpxx)) (x)) A case of two positives matching given query is described in Figure le
In terms of mini-batch, each image in serves as the anchor iteratively and we aim to maximise the Equivalently, we can achieve this by minimising the
joint probability of all queries {pxx))=1 J.N. IC J& sum of all negative log-likelihoods. Therefore, our proposed ICE on X is as follows:
LICE (X; = ΣEE No.
=- EE E
logPx; Px5
logp(x{(x))
(5)
3.3 REGULARISATION BY Ta FEATURE NORMALISATION
Following the common practice in existing DML methods, we apply Læ-normalisation to feature embeddings before the inner product. Therefore, the inner product denotes the cosine similarity.
The similarity between two feature vectors is determined by their norms and the angle between them. Without L2 normalisation, the feature norm can be very large, making the model training unstable and difficult. With L2 normalisation, all features are projected to a unit hypersphere sur- face. Consequently, the semantic similarity score is merely determined by the direction of learned representations. Therefore, T.2 normalisation can be regarded as a regulariser curing training Note that the principle is guite different from recent hyperspherical learning methods (Liu et al. 2017a Wang et al., 2018b at Liu et al.) 20176: 2018b a). They enforce the learned weight parameters to a unit hypersphere surface and diversify their angles. In contrast, feature normalisation is output
'The training without L2 feature normalisation leads to the norm of features becoming very large easily and the dot product becoming INF.
5

---

Under review as a conference paper at ICLR 2020
regularisation and invariant the parametrisation of the underlying neural network Pereyrr al. 2017). In summary, our learning objective is:
arg max IT
II No
Px: arg min LXce(X: s.t. Va, || 1.
The feature In-normalisation layer is implemented according to Wang et (2017a) It is differ- entiable layer and can be easily inserted at the output of neural net.
3.4 SAMPLE REWEIGHTING OF ICE
Intrinsic sample weighting. We find that ICE emphasises more on harder samples from the perspec- tive of gradient magnitude. We demonstrate this by deriving the partial derivatives of LICE (X: f) with respect to positive and negative examples.
Given the query x the partial derivative of its any positive instance is derived by the chain rule:
OLICE (X:
f. Lofo E, exp(f. fÿ)
= (1-
Of.
+))EEtt exp(fé f9) + Eofe Exex((f fÿ)
(1- p(xx)xxx5 ( be viewed the weight of f when the anchor is x Thus, ICE focuses more on harder positive samples, whose pxx](x) is lower.
[ Since ||C5ll2 1, ((()x())
OF
Similarly, the partial derivative of its any negative sample is:
@LicE(X: 8fg
t exp f2)
Eifo exp(fs f.) Eopo E, exp(fe 17)
=fa- E'fe (xx))xx.xx)) (8)
is the matching probability between x3 and x given that the ground-truth ample is x5. The weight of x w.r.t. x is: W(xxxxs) = || [ |2 = Eijo p(x3xx xf).
where p(x}) xe; xf)
Clearly, the harder negative samples own higher matching probabilities and weights.
Relative weight analysis. In general, the relative weight (Tabachiick et al. 2007) is more notable the exact weight will be rescaled during training, e.g., linear post-processing by multiplying the learning rate. Therefore, we analyse the relative weight between two positive points of the same anchor (i + l + a):
1 (xx(x))
1 1 p(x x
exp(f EX) + Eofe E, exp(Fx (ff) TP
Txp(f.TTg)
(9)
W (x))
p(x x5
exp(fe f9) + Eofe E, exp(ps 19))
Similarly, the relative weight between two negative points of the same anchor (o # c, 1 c) is:
((xe(())) (xc) Eigo p(x x2, x5)
Pi x x()
exp(f: f9)
(10)
exp(fE f.
Note that the positive relative weight in Eq. 9 is only decided by r f and f f while the negative relative weight in Eq. (10) is only determined by fg f9 and f. f. The relative weight is merely determined by the dot product, which is in the range of (-1, 1] and strietly bounded.
Non-linear scaling for controlling the relative weight. Inspired by (Hinton et al. 2015), intro- duce a scaling parameter to modify the absolute weight non-linearly:
Eoto E, exp(s f f9)
i (x(
= =1 -1-ppxp -xx(xxxx))
(11)
exp(s E f.) + Eofe E3 exp(s sxpp(s-a'' P9)
exp(s P TET 19)
(((xx()) Σ igta exp(s fo f.) Eote E, exp(s (s- fc fCT F9)
Enjo xxx)Ixx..)) (12)
where s 2 1 is the scaling parameter. In contrast to p and w, P and i represent the rescaled matching probability and partial derivative weight, respectively. We remark that scale the absolute weight

---

Under review as a conference paper at ICLR 2020
Algorithm Learn by minimising ICE stochastically
Batch setting: C' classes, N₂ images from e-th class, batch size N = Ees1 No.
Hyper-setting- The scaling parameter and the number of iterations T. Input: Initialised embedding function f, iteration counter iter = 0.
Output: Updated f.
for iler < T do
iter = iter + 1.
Sample one mini-batee randomly X
((x))
Step 1: Feedforward X into f to obtain feature representations (ff)iae 1
Step 2: Compute the similarities between an anchor and the remaining instances. Every exam- ple serves as the anchor iteratively.
for f
N. -C {[E]]= 11=1f=1 do
for ff E [ff) Compute p(x xx) using Eq.
Jipa a do
//
We do not need to compute Eq. (4
end for end for
Compute LICE f) using Eq.
Step 3: Gradient back- pook-propagation to update the parameters of f using Eq. 15 end for
non-linearly, which is an indirect way of controlling the relative weight. We do not modify the relative weight directly and Eq. 4 and Eq. (TOp are only for introducing our motivation.
Our objective is to maximise anchor's matching probability with its any positive instance com- peting against its negative set. Therefore, we normalise the rescaled weights based on each anchor:
1
(x(x)))
1 x(x{(xx)
(13)
N Σ Eippa (x)(x))) + Eope E, th (xe(xx))
2N
Σ ip
(1 p(x; xa))
ia p(x;) X2 x;)
(14)
N
Lifa (xF(ox)) + Losf Ej (xp(xx))
2N
Σija (1 ((x)
Note that the denominators in Eq. (13) and (1) are the accumulated weights of positives and neg- atives w.r.t X respectively. Although there are much more negatives than positives, the negative set and positive set contribute equally as a whole, as indicated by 1/2. N = === No is the total number of instances in X. We select each instance as the anchor iteratively and treat all anchors equally, as indicated by 1/N.
is worth noting that during back poeck-rrpaggtion, the magnitudes of partial derivatives in Eq. 7 and Eq. 18,,ie. W(xf(xe) and ((x(igg), > replaced by (((x(x))) and (xfxx;) respectively. The direction of each individual partial derivative is unchanged. However, since weights are rescaled non-linearly, the final partial derivative of each sample is changed better weighted combination of multiple partial derivatives. Final partial derivatives of (X; f) w.r.t. positives and negatives are:
(X: -f% (x(x)) and OLICE fX: Of.
-h ((gxx))
(15)
ary
3.5 CASE STUDY AND INTUITIVE EXPLANATION OF ICE
To make more clear and intuitive for understanding, we now analyse a naive case of ICF, where there are two samples per class in every mini-batch, i.e., Vc, Nc = 2, {x] = N 1 1. In this case, for each anchor (query), there is only one positive among the remaining data points. As result, the weighting schemes in Eq. (13 for positives and Eq. (14) for negatives can be simplified:
1 p(x xx))
1
Th (x(:)))
2N
E (1 p(x)(xa)
N 2'
(16)
Σiea ((x) X x9)
1 x(x]IxX, x()
(17)
2N
p(x{(x)) -(xx(1)))
N 2 1 p(x{(x)))
Firstly, we have N anchors that are treated equally as indicated by 1/N. Secondly, for each anchor, aim to recognise its positive example correctly. However, there is sample imbalance problem

---

Under review as a conference paper at ICLR 2020
Table 1: A summary of three fine-grained datasets. Training and test classes are disjoint. '#' refers to the number of each item. There are only 5.3 images per class on average in SOP.
Table 2:
The results of different reweighting parameters on SOP in terms of Recall @ K (%). There are 90 classes and 2 images per class in a mini- batch, i.e, the batch size 180.
Datasets
CARSI96 CUB-28 (UB-200-200 201-2011
SOP
Reweighting
R@i
@ 10
R 100
Context
Cars
Birds
Products
#Total classes
196
200
22,634
42.0
58.1
74.1
#Total images
16,185
11,788
120.053
$ 16
71.0
85.6
93.8
#Training classes
98
100
11,318
8 32
73.6
87.5
94.7
#Training images
8,054
5,864
59,551
8 48
76.9
89.7
95.5
#Test classes
98
100
11,316
64
77.3
90.0
95.6
#Test images
8,131
5,924
60,502
80
75.4
88.7
94.9
because each anchor has only one positive and many negatives. ICE addresses it by treating the
positive set (single point) and negative set (multiple points) equally, i.e., 1/2 in Eq. (16) and Eq. (17)
Finally, as there are many negative samples, aim to focus more informative ones, I.e., harder negative instances with higher matching probabilities with a given anchor. The non-linear transformation can help control the relative weight between two negative points.
The weighting scheme shares the same principle as the popular temperature-based categorical cross entropy (Hinton al., 2015 Oreshkin et al. 2018). The key is that we should consider not only focusing on harder examples, but also the emphasis degree.
3.6 COMPLEXITY ANALYSIS
Algorithm [ summarises the learning process with ICE. As presented there, the input data format of ICE is the same CCE, i.e., images and their corresponding labels. In contrast to other methods which require rigid input formats (Schroff et al. 2015 Sohn 2016). e.g., triplets and n-pair tuplets, ICE is much more flexible. We iteratively select aim to maximise its matching probabilities with its positive samples against its negative examples. Therefore, the computational complexity over one mini-batch is O(N2), being the same as recent online metric learning approaches (Song et al.] 2016: Wang et al. 2019b). Note that in FaceNet (Shhroff et al. (2155) and N N-pair-me (Sohn) 2016), expensive sample mining and class mining are applied, respectively.
image the anchor. For each anchor,
4
EXPERIMENTS
4.1
IMPLEMENTATION DETAILS AND EVALUATION SETTINGS
For data augmentation and preprocessing, we follow Song et al 2016 2017). In detail, we first resize the input images 256 256 and then crop it 227 227. We use random cropping and horizontal mirroring for data augmentation during training. To fairly compare with the results reported in (Song et al.) 2017), we use centre cropping without horizontal flipping in the test phase. For the embedding size, we set it to 512 on all datasets following (Sohn 2016 Law et al.) 2017 Wang et al. 2019a). To compare fairly with (Song et al.) 2017) Law et al. 2017: Movshovitz- Attias et al. 2017) we choose GoogLeNet V2 (with batch normalisation) (loffe & Szegedy 2015) as the backbone architecture initialised by the publicly available pretrained model on ImageNet (Russakovsky et al.] 2015). We simply change the original 1000-neuron fully connected layers tollowed by softmax normalisation and CCE to 512-neuron fully connected layers followed by the proposed ICE. For faster convergence, we randomly initialise the new layers and optimise them with 10 times larger learning rate than the others as in (Song et al. 2016).
We implement our algorithm in the Caffe framework (Jia et al. 2014). The source code will be available soon.
Datasets. Following the evaluation protocol in (Song et al. 2016 2017). test proposed
method on three popular fine-grained datasets including CARSI9 (Krause et al. 2013). CUB- 200-2011 (Wah et al. 2011) and SOP (Song et al. 2016). summary of the datasets is given in
"The weight sum of negatives: Eofc E, p(x] x x() = -({[x() Eosc E, ((xxx))
1 (x = 12(2N).
8

---

Under review as a conference paper at ICLR 2020
Comparison with the state-of-the-art methods on CARSI96, CUB-200-2011 and SOP in terms of Recall K (%). All the compared methods use GoogLeNet V2 as the backbone architec- ture. '_' means the results which are not reported in the original paper. The best results in the first block using single embedding are bolded.
Table 3:
CARS196
CUB-200-2011
SOP
K
2
8
10
100
Without fine-tuning
35.6
47.3
59.4
72.2
40.1
53.2
66.0
76.6
43.7
60.8
76.5
Finc-tuned with CCE
48.8
58.5
71.0
78.4
46.0
58.0
69.3
78.3
51.7
69.8
85.3
Triplet Semihard
51.5
63.8
73.5
82.4
42.6
55.0
66.4
77.2
66.7
82.4
91.9
Lifted Struct
53.0
65.7
76.0
84.3
43.6
56.6
68.6
79.6
62.5
80.8
91.9
N-pair-me
53.9
66.8
77.8
86.4
45.4
58.4
69.5
79.5
66.4
83.2
93.0
Struct Clust
58.1
70.6
80.3
87.8
48.2
61.4
71.8
81.9
67.0
83.7
93.2
Spectral Clust
73.1
82.2
89.0
93.0
53.2
76.7
85.3
67.6
83.7
93.3
Proxy NCA
73.2
82.4
86.4
88.7
49.2
61.9
67.9
72.4
73.7
RLL
74,0
83.6
90.1
94.1
57.4
69.7
79.2
86.9
76.1
89.1
95.4
ICE
77.0
85,3
91.3
94.8
58.3
69.5
79.4
86.7
77.3
90.0
95.6
(LL-HL.M.H) ICE-(L, M. H)
82.1
89.3
93.7
96.7
61.3
72.7
82.7
89.4
79.8
91.3
96.3
82.8
89.5
93.7
96.4
61.4
73.2
82.5
89.2
80.1
91.8
96.6
We also keep the same train/test splits. We remark that to test the generalisation and transfer capability of the learned deep metric, the training and test classes are disjoint.
Table I
Evaluation protocol. We evaluate the learned representations on the image retrieval task in terms of Recall @ K performance (Song et al. 2016). Given a query, its K nearest neighbours are retrieved from the database. Its retrieval score is one if there is an image of the same class in the K nearest neighbours and zero otherwise. Recall @ K is the average
of all queries.
Training settings. All the experiments are run on a single PC equipped with Tesla V100 GPU with 32GB RAM. For optimisation, we use the stochastic gradient descent (SGD) with a weight decay of le and momentum of 0.8. The base learning rate is set 1e 3 The training converges at 20k iterations on SOP while 4k iterations on CARS196 and CUB-200-2011. As for the hyper- parameters, study their impacts in Sec. 4.3 and supplementary material. The mini-batch size is 60 for small datasets CARSI 196 and CUB-200-2011 while 180 for the large benchmark SOP. Additionally, we set
N₂ 10 CARS196 and CUB-200-2011 while C 90, N. on SOP. The design reasons are: 1) SOP has only 5.3 images per class on average. Therefore N₂ cannot be very large: 2) It helps to simulate the global structure of deep embeddings, where the database is large and only few matching instances exist.
The analysis of batch content, batch size and embedding size is presented in the supplementary material.
4.2
QUANTITATIVE RESULTS
Remarks. For fair comparison, remark that the methods group (Ustinova & Lempitsky 2016
Harwood et al. 2017 Wang et al. 2017b Duan et al. 2018 Lin et al. Suh et al. 2019 Zheng
al. 2019) using GoogLeNet V1 (Szegedy al. 2015) and another group (Wu al. 2017 Cakir
et al. 2019 Sanakoyeu et al. 2019) using ResNer- 50 (He et al. 2016) are not benchmarked Besides,
ensemble models Yuan considered. HTL (Ge et al. 2018) also uses Googl Net V2, but it constructs a hierarchical similarity
2017 Opitz al. 2017 Kim
2018 Xuan al. 2018)
tree over the whole training set and updates the tree every epoch, thus being highly unscalable and expensive in terms of both computation and memory. That is why HTL achieves better performance on small datasets but performs worse than ours on the large dataset SOP. Finally, there are some other orthogonal deep metric learning research topics that are worth studying together in the future, e.g., a robust distance metric (Yuan et al. 2019) and metric learning with continuous labels (Kim) et al. 2019). In GoogLeNet V2, there are three fully connected layers of different depth. We refer them based on their depth: L for the low-level layer ionepttion-ccouuppee finception-4e/oupge
for the mid-level layer and H for the high-level layer (inception5b/oupput) By default, 'H'. We also report the results of their combination (L, M, H) for reference following RLL (Wang et al. 2019a).
only

---

Under review as a conference paper at ICLR 2020
Competitors. All the compared baselines. Triplet Semihard (Schroff et al.
2015). Lifted Struct
(Song et al. 2016), --pair-me (Sohn, 2016), Struct Clust (Song et al. 2017). Spectral Clust (Law)
et a 2017), ProXy-NCA (Movshovitz-Atuue
et al. 2017). RLL (Wang al. 2019a) and our ICE are trained and evaluated using the same settings: (1) GoogLeNet V2 serves as the backbone net- work; (2) All models are initialised with the same pretrained model ImageNet; (3) All apply the same data augmentation during training and use a centre-cropped image during testing. The results of some baselines (Schroff et al. 2015 SSng et al., 2016, [ohnn] 2016) are from (Song al.) 2017), which means they are reimplemented there for fair comparison. In addition, the results of vanilla GoogLeNet V2 pretrained on ImageNet without fine-tuning and with fine-tuning via min- imising CCE are reported in (Law et al. 2017), which can be regarded as the most basic baselines. Among these baselines, Proxy NCA is not scalable as class-level proxies are learned during training- Struct Clust and Spectral Clust are clustering- motiivaed methods which explicitly aim to optimise the clustering quality. We highlight that clustering performance Normalised Mutual Information (NMI) (Schütze et al., 2008) is not a good assessment for SOP (Law et al.) 2017) because SOP has a large number of classes but only 5.3 images per class on average. Therefore, we only report and compare Recall @ K performance.
Results. Table 3 compares the results of our ICE and those of the state-of-the-art DML losses. ICE achieves the best Recall@1 performance on all benchmarks. We observe that only RLL achieves comparable performance in few terms. However, RLL is more complex since it has three hyper- parameters in total: one weight scaling parameter and two distance margins for positives and nega- tives, respectively. In addition, its perspective is different since it processes the positive set together similarly with (Trintatiilouu et al., 2017, Wang et al., 2019c). We note that (Wang et al. 2019c) is also complex in designing weighting schemes and contains four control hyper-parameters. However, our Recall 1 on SOP is 77.3%, which is only 0.9% lower than 78.2% of (Wang et al. 2019c). It is also worth mentioning that among these approaches, except fine-tuned models with CCE, only our method has clear probability interpretation and aims to maximise the joint instance-level matching probability. As observed, apart from being unscalable, CCE's performance much worse than the state-of-the-art methods. Therefore, ICE can be regarded as successful exploration of softmax regression for learning deep representations in DML. The t-SNE visualisation (Van Der Maaten, 2014) of learned embeddings are available in the supplementury material.
4.3 ANALYSIS OF SAMPLE REWEIGHTING IN ICE
We empirically study the impact of the weight scaling parameter 8, which is the only hyper- parameter of ICF It functions similarly with the popular sample mining or example weighting (Wang et al., b/ widely applied in the baselines in Table 3 Generally, different s corre- sponds to different emphasis degree on difficult examples. When larger, more difficult instances are assigned with relatively higher weights.
In general, small datasets are more sensitive to minor changes of hyper-settings and much easier to overfit. Therefore, the experiments are conducted on the large dataset SOP. The results are shown in Table 2 Note that when is too small, e.g., == 1, we observe that the training does not converge, which demonstrates the necessity of weighting/mining samples. The most significant observation is that focusing on difficult samples is better but the emphasis degree should be properly controlled. When 8 increases from 16 to 64, the performance grows gradually. However, when 80, we observe the performance drops lot. That may be because extremely hard samples, e.g., outliers, are emphasised when is too large.
CONCLUSION
In this paper, we propose a novel instance-level softmax regression framework, named instance cross entropy, for deep metric learning. Firstly, the proposed ICE has clear probability interpretation and exploits structured semantic similarity information among multiple instances. Secondly, ICE is scalable to infinitely many classes, which is required by DML. Thirdly, ICE has only one weight scaling hyper- parameter, which works as mining informative examples and can be easily selected via cross-vall validatio... Finally, distance thresholds and interclass separability. This indicates that ICE makes no assumptions about intraclass variances and the boundaries between different classes. Therefore ICE owns general applicability.
hyper-parameter,
not applied to achieve intraclass compactness
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Maxime Bucher, Stéphane Herbin, and Frédéric Jurie. Improving semantic embedding consistency by metric learning for zero-shot classification. In ECCV, 2016.
Fatih Cakir, Kun He, Xide Xia, Brian Kulis, and Stan Sclaroff. Deep metric learning to rank. In CVPR, 2019.
Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning similarity metric discriminatively, with application to face verification. In CVPR, 2005.
Yueqi Duan, Wenzhao Zheng, Xudong Lin, Jiwen Lu, and Jie Zhou. Deep adversarial metric learn- ing. In CVPR, 2018.
Weifeng Ge, Weilin Huang, Dengke Dong, and Matthew R Scott. Deep metrie learning with hierar- chical triplet loss. In ECCV, 2018.
Jacob Goldberger, Geoffrey E Hinton, Sam T Roweis, and Ruslan R Salakhutdinov. Neighbourhood components analysis. In NeurlPS, 2005.
Ben Harwood, BG Kumar, Gustavo Carneiro, Ian Reid, Tom Drummond, et al. Smart mining for deep metric learning. In ICCV, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. In NewrlPS Deep Learning and Representation Learning Workshop, 2015.
Chen Huang, Chen Change Loy, and Xiaoou Tang. Local similarity-aware deep feature embedding. In NeurIPS, 2016.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Ser- gio Guadarrama, and Trevor Darrell. Caffe: Convolutional architecture for fast feature embed- ding. In ACMMM, 2014.
Sungyeon Kim, Minkyo Seo, Ivan Laptev, Minsu Cho, and Suha Kwak. Deep metric learning beyond binary supervision. In CVPR, 2019.
Wonsik Kim, Bhavya Goyal, Kunal Chawla, Jungmin Lee, and Keunjoo Kwon. Attention-based ensemble for deep metric learning. In ECCV, 2018.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In ICCV Workshop, 2013.
Solomon Kullback and Richard Leibler. On information and sufficiency. The Annals of Mathe- matical Statistics, pp. 79-86, 1951.
Mare T Law, Raquel Urtasun, and Richard S Zemel. Deep spectral clustering learning, In ICML, 2017.
Mare T Law, Jake Snell, Amir-massoud Farahmand, Raquel Urtasun, and Richard S Zemel. Dimen- sionality reduction for representing the knowledge of probabilistic models. In ICLR, 2019.
Xudong Lin, Yueqi Duan, Qiyuan Dong, Jiwen Lu, and Jie Zhou. Deep variational metrie learning. In ECCV, 2018
Weiyang Liu, Yandong Wen, Zhiding Yu, and Meng Yang. Large-margin softmax loss for convolu- tional neural networks. In ICML, 2016.
Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. Sphereface: Deep hypersphere embedding for face recognition. In CVPR, 2017a.
11

---

Under review as a conference paper at ICLR 2020
Weiyang Liu, Yan-Ming Zhang, Xingguo Li, Zhiding Yu, Bo Dai, Tuo Zhao, and Le Song. Deep hyperspherical learning. In NeurlPS, 2017b.
Weiyang Liu, Rongmei Lin, Zhen Liu, Lixin Liu, Zhiding Yu, Bo Dai, and Le Song. Learning towards minimum hyperspherical energy. In NeurIPS, 2018a.
Weiyang Liu, Zhen Liu, Zhiding Yu, Bo Dai, Rongmei Lin, Yisen Wang, James M Rehg, and Le Song. Decoupled networks. In CVPR, 2018b.
Yair Seeshovviz-Attiasee Alexander Toshev, Thomas K Leung, Sergey loffe, and Saurabh Singh. No fuss distance metric learning using proxies. In ICCV, 2017.
Maximillian Nickel and Douwe Kiela. Learning continuous hierarchies the lorentz model of hyperbolic geometry. In /CML, 2018.
Michael Opitz, Georg Waltner, Horst Possegger, and Horst Bischof, Bier-boosting independent embeddings robustly. In 1,C,,C,,111
Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. In NeurlPS, 2018.
Gabriel Pereyra, George Tucker, Jan Chorowski, Eukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions. In ICLR Workshop, 2017.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, pp. 211-252 2015.
Artsiom Sanakoyeu, Vadim Tschernezki, Uta Buchler, and Bjorn Ommer. Divide and conquer the embedding space for metric learning. In CVPR, 2019.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In CVPR, 2015.
Hinrich Schütze, Christopher D) Manning, and Prabhakar Raghavan. Introduction information retrieval. Cambridge University Press, 2008.
Hailin Shi, Yang Yang, Xiangyu Zhu, Shengcai Liao, Zhen Lei, Weishi Zheng, and Stan Z Li. Embedding deep metric for person re-identification: study against large variations. In ECCV, 2016.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In NeurIPS, 2017.
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. In NeurIPS, 2016.
Hyun Oh Song. Yu Xiang, Stefanie Jegelka, and Silvio Savarese. Deep metric learning via lifted structured feature embedding. In CVPR, 2016.
Hyun Oh Song, Stefanie Jegelka, Vivek Rathod, and Kevin Murphy, Deep metric learning via facility location. In CVPR, 2017.
Yumin Suh, Bohyung Han, Wonsik Kim, and Kyoung Mu Lee. Stochastic class-based hard example mining for deep metric learning. In CVPR, 2019.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, 2015.
Barbara G Tabachnick. Linda S Fidell, and Jodie B Ullman. Using multivariate statistics. Pearson Boston, MA, 2007.
Oren Tadmor, Tal Rosenwein, Shai Shalev- Shwaar,, Yonatan Wexler, and Amnon Shashua. Learning metric embedding for face recognition using the multibatch method. In NeurIPS, 2016.
12

---

Under review as a conference paper at ICLR 2020
Eleni Triantafillou, Richard Zemel, and Raquel Urtasun. Few-shot learning through an information retrieval lens. In NeurIPS, 2017.
Evgeniya Ustinova and Victor Lempitsky. Learning deep embeddings with histogram loss, In NeurIPS, 2016.
Laurens Van Der Maaten. Accelerating t-sne using tree-based algorithms. The Journal of Machine Learning Research, pp. 3221-3245, 2014.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In NeurlPS, 2016.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.
Feng Wang, Xiang Xiang, Jian Cheng, and Alan Loddon Yuille. Normface: 12 hypersphere embed- ding for face verification. In ACMMM, 2017a.
Feng Wang, Jian Cheng, Weiyang Liu, and Haijun Liu. Additive margin softmax for face verifica- tion. In ICLR Workshop, 2018a.
Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Zhifeng Li, Dihong Gong, Jingchao Zhou, and Wei Liu. Cosface: Large margin cosine loss for deep face recognition. In CVPR, 201 2b.
Jian Wang, Feng Zhou, Shilei Wen, Xiao Liu, and Yuanqing Lin. Deep metrie learning with angular loss. In ICCV, 2017b.
Xinshao Wang, Yang Hua, Elyor Kodirov, Guosheng Hu, Romain Garnier, and Neil M Robertson. Ranked list loss for deep metric learning. In CVPR, 2019a.
Xinshao Wang, Yang Hua, Elyor Kodirov, Guosheng Hu, and Neil M. Robertson. Deep metric learning by online soft mining and class-aware attention. In AAAI, 2019b.
Xun Wang, Xintong Han, Weilin Huang, Dengke Dong, and Matthew Scott. Multi-similarity loss with general pair weighting for deep metric learning. In CVPR, 2019c.
Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao. A discriminative feature learning approach for deep face recognition. In ECCV, 2016.
Chao- Yura Wu, Manmatha, Alexander J Smola, and Philipp Krähenbühl. Sampling matters in deep embedding learning. In ICCV, 2017.
Zhirong Wu, Alexei A Efros, and Stella X Yu. Improving generalization via scalable neighborhood component analysis. In ECCV, 2018.
Hong Xuan, Richard Souvenir, and Robert Pless. Deep randomized ensembles for metric learning. In ECCV, 2018.
Tongtong Yuan, Weihong Deng, Jian Tang, Yinan Tang. and Binghui Chen. Signal-to-noise ratio: A robust distance metric for deep metric learning. In CVPR, 2019.
Yuhui Yuan, Kuiyuan Yang, and Chao Zhang. Hard-aware deeply cascaded embedding. In ICCV, 2017.
Xu Zhang, Felix Xinnan Yu, Svebor Karaman, Wei Zhang, and Shih-Fu Chang. Heated-up softmax embedding. arXiv preprint arXiv:1809.0415: 2018.
Wenzhao Zheng, Zhaodong Chen, Jiwen Lu, and Jie Zhou. Hardness-awarc deep metric learning. In CVPR, 2019,
13

---

Under review as a conference paper at ICLR 2020
Supplementary Material for Instance Cross Entropy for Deep Metric Learning
A MORE ABLATION STUDIES
A.1 BATCH CONTENT
We evaluate the impact of batch content which consists of C' classes and k images per class, i.e., k
Ve, .v. k. The batch size N
180. In our experiments, we change the number of classes C' from 36 to 90, and the number of images k from 2 to 5, while keeping the batch size unchanged. Thoos the results on SOP dataset. We observe that when there are more classes in the mini-batch, the performance is better. We conjecture that as the number of classes increases, the mini-batch training becomes more difficult and helps the model to generalise better.
Table 4: The impact of batch content C' x & on SOP in terms of Recall (%). The batch size is N 180 and the scaling parameter is 64.
180, 8
R@D I
R@
R@ 100
C k: 90
90.0
95.6
C x 60
75.2
88.7
95.2
C X A: 45 x
74.9
88.7
95.3
C X h: 36 x
74.6
88.7
95.4
A.2 BATCH SIZE
To explore different batch size N, fix k and only change C. In this case, N C 2. Table 3 shows that as the number of classes increases, the performance grows. In detail, when the number of classes increases from 50 90, the performance raises from 74.4% 77.3% accordingly. One reason may be that as the number of classes increases, it fits the global structure of the test set better, where there are a large number of classes but only few positive examples. In addition, the increasing difficulty of mini-batch training can help the model to generalise better.
Table 5: The results of different batch size N on SOP in terms of Recall K (%). While changing C, we fix and 8 = 64. Therefore, N C x 2.
h: 2, s 64
R @ I
R@ 10
R@ 100
N = 180
77.3
90.0
95.6
N 160
75.4
88.8
95.1
N 140
75.1
88.7
95.2
120
75.1
88.6
95.2
N 100
74.4
88.2
95.1
A.3 EMBEDDING SIZE
The dimension of feature representations is an important factor in many DML methods. We conduct experiments on SOP see the influence of different embedding size. The results are presented in Table 6 We observe that when the embedding size is very small, e.g., 64, the performance is much worse. The performance increases gradually as the embedding size grows.
B T-SNE VISUALISATION
The t-SNE visualisation (Van Der Maaten 2014) of learned embeddings is available Figures 2
14

---

Under review as a conference paper at ICLR 2020
Table 6: The results of different embedding size SOP in terms of Recall K (%). In all experi-
ments: 8 - 64, C = 90, k = 2. N - 90 x 2.
180
90 2,
64
R@1
R@ 10
R@ 100
64
72.6
87.1
94.0
128
74.3
87.9
94.5
256
75.2
88.6
94.8
512
77.3
90.0
95.6
50
a
Figure 2: t-SNE visualisation Van Der Maaten 2014 on the SOP test set. Best viewed on a monitor when zoomed in.
15

---

Under review as a conference paper at ICLR 2020
Figure 3: t-SNE visualisation Van Der Maaten 2014) the CUB-200-200 test set. Best viewed on a monitor when zoomed in.
16

---

Under review as a conference paper at ICLR 2020
Figure 4: t-SNE visualisation Van Der Maaten 2014) the CARS196 test set. Best viewed monitor when zoomed in.
17


Reviews:
[{"review_id": "ryxSBCNAFr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a method to measure the difference between an\nestimated instance-level matching distribution and its ground-truth\none, based on instance cross entropy (ICE).\nThe goal is to learn an embedding that captures the semantic\nsimilarities among samples. \nIn particular, with ICE they try to maximize the matching probability\nof an instance with similar instances (same class).\nThe authors also use sample re-weighting into ICE and show the benefits\nof the approach against other state-of-the-art methods on three\ndatasets. The authors performed several experiments with convincing\nresults.\n\nThe positive aspect of the paper is introducing this instance-based\nmeasure which is shown to perform well on 3 challenging datasets.\n\nIt is not clear how is the scaling parameter (s) determined. Are there\nany guidelines for fixing its value?\nThe authors should explain in more detail how is the non-linear\ntransformation achieved.\n\nThe algorithm goes through all the examples of a class on each\nmini-batch, which seems a computational expensive procedure. The paper\nwill benefit if time results are reported for different approaches.\n\nThe paper is sometimes difficult to follow and needs a careful\nrevision. \n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Hkg88mzdsr", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overview  \n\nThis paper proposes a new objective function called Instance Cross Entropy (ICE) for metric learning. Compared to the triplet loss (or contrastive loss) and its many variants, the distance between points in feature space is defined to be the dot/inner product, rather than computing the euclidian distance between feature vectors. Since the L2 norm of the final features is constrained  to be 1, the dot product represents the cosine distance between 2 feature vectors. Compared to the softmax/categorical loss, this objective has the advantage that there is no need to learn a per-class weight vector in the output softmax layer and therefore this method can be used even when the number of classes during training is unbounded. This is a very useful design feature, since the output softmax layer grows linearly with the size of the classes and can quickly become prohibitively large. Furthermore, mini batches for training can be randomly sampled without requiring expensive negative data mining strategies which are necessary when using the triplet loss. This is another very useful property. \n\nOverall, I think the ideas presented are interesting and the paper provides a useful summary of existing approaches for metric learning. However, I think the technical presentation needs to be improved significantly before  the paper can be accepted for publication. I found the paper quite difficult to follow. I had to re-read the motivation and the description of the loss function many times before being able to understand what is going on and how the proposed loss compares and contrasts with existing approaches. I also think some of the terminology and notation used in the paper is very confusing and should be updated to help the reader get to the main argument quickly. \n\nComments\n\n\n1. Title: What exactly does instance cross entropy (ICE) mean? And how is it different from categorical cross entropy (CCE)? Isn’t categorical cross-entropy also calculating the entropy between the predicted distribution for *each instance* and the ground truth? I think the authors should reconsider the name for the proposed loss and choose a more descriptive name for the algorithm.\n\n2.  I also found the use of the term “matching distribution” a bit confusing since it has a well-defined meaning in statistics. However ML is full of overloaded terms and I understand if the authors want to keep this description. \n\n3. None of the references in the main text have brackets around them. I think this makes the paper appear very cluttered and makes parsing each paragraph quite difficult. I would highly recommend the paper be reformatted and the authors use brackets around references. \n\n4. In the abstract, the authors mention that the proposed method has a “clear probabilistic interpretation”. From my understanding, I see ICE as a blend of the categorical cross-entropy loss and the triplet loss, retaining useful properties of each without increasing the complexity of the loss computation. However, I do not see a clear probabilistic interpretation of the loss function. The softmax computation yields a probability distribution like output given a query and an anchor point, which is hand-designed in the objective function. I  think the authors should more clearly motivate what the probabilistic angle for the loss function is. \n\n5. I found Figure 1 to be very difficult to interpret. This is a missed opportunity, since this figure alone could communicate some of the key ideas in the paper to the reader. However, there are many details that have not been explicitly mentioned. What do the colours (white, blue, yellow) mean? The figure shapes probably mean classes. What do the terms p_* mean? What do i,j mean in Figure 1 e and how are they different from the previous figures. These details should all be present in the title for the figure. Even with these details, the figure could still be a bit more explicit. I found the yellow arrow with cross entropy and the notion of “ground truth” in the figure difficult to follow. \n\n6. How was the constrained optimisation performed? In Equation 6, we see that there is a constraint associated with each example in the mini batch which says the norm of the feature vector should be equal to 1. However I could not find any implementation details of how this constraint was satisfied. Did the authors use a Lagrangian formulation? I think the paper is irreproducible without this detail.  \n\nSummary\n\nI think this is an interesting proposal to combine the useful features of the softmax and triplet losses. However, I think the technical presentation needs to be improved significantly in order for the paper to be accepted for publication. \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HJew1MGhFr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a method inspired by the categorical cross entropy (CCE). In a similar way as many metric learning approaches, a softmax approach based on cross entropy is used to enforce similar examples to have smaller distances than dissimilar distances. \nNonetheless, instead of considering a centroid-based deep metric learning approach (e.g. like Snell et al. (2017)), only one anchor is considered for each training example, and the distance of an anchor with one of its positive is learned that its distance is smaller than the distance with examples belonging to different categories.\n\nI vote for reject for the following reasons:\n- The paper is too similar to NCA and S-NCA (introduced in Section 2.2). In the same way as ICE, S-NCA also considers learning l2-regularized representations. The main difference with S-NCA is the way negative examples are sampled. \nS-NCA proposes a framework based on augmented memory, ICE proposes a sampling strategy similar to [A] and used in Nickel et al. (2018) to subsample negative pairs.\n- Another contribution of ICE is the use of some hyperparameter s in Equations (11) and (12). This hyperparameter s plays the role as the inverse of the temperature and is in fact learned in ICE. S-NCA also plays with the temperature but does not learn it. \nOn the other hand, learning the temperature in a similar way has been proposed in the deep metric learning literature (e.g. TADAM).\n\nOn the other hand, the paper has some nice contributions:\n- The main \"novelty\" of ICE seems to be the analysis in Section 3.4 of the partial derivatives and their impact on the sample reweighting. Although the explanation is simple, it helps understand what's happening during optimization.\n- The experimental results on different transfer learning benchmarks seem convincing.\n\n\n[A] Jean et al. On usingvery large target vocabulary for neural machine translation. ACL 2015", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a new objective function called ICE for metric learning.

There was a substantial discussion with the authors about this paper. The two reviewers most experienced in the field found the novelty compared to the vast existing literature lacking, and remained unconvinced after the discussion. Some reviewers also found the technical presentation and interpretations to need improvement, and this was partially addressed by a new revision.

Based on this discussion, I recommend a rejection at this time, but encourage the authors to incorporate the feedback and in particular place the work in context more fully, and resubmit to another venue.
Decision: Reject