Generalized Sum Pooling for Metric Learning
Authors: Yeti Z. Gürbüz
Ozan Sener
A. Aydın Alatan
Venue: Submitted to ICLR 2023 (2023)
Keywords: Metric learning
feature selection
global average pooling
zero-shot regularization

Abstract: A common architectural choice for deep metric learning is a convolutional neural network followed by global average pooling (GAP). Albeit simple, GAP is a highly effective way to aggregate information. One possible explanation for the effectiveness of GAP is considering each feature vector as representing a different semantic entity and GAP as a convex combination of them. Following this perspective, we generalize GAP and propose a learnable generalized sum pooling method (GSP). GSP improves GAP with two distinct abilities: i) the ability to choose a subset of semantic entities, effectively learning to ignore nuisance information, and ii) learning the weights corresponding to the importance of each entity. Formally, we propose an entropy-smoothed optimal transport problem and show that it is a strict generalization of GAP, \ie a specific realization of the problem gives back GAP. We show that this optimization problem enjoys analytical gradients enabling us to use it as a direct learnable replacement for GAP. We further propose a zero-shot loss to ease the learning of GSP. We show the effectiveness of our method with extensive evaluations on 4 popular metric learning benchmarks. Code is available at: GSP-DML Framework

Under review as a conference paper at ICLR 2023
GENERALIZED SUM POOLING FOR METRIC LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
A common architectural choice for deep metric learning is a convolutional neural network followed by global average pooling (GAP). Albeit simple, GAP is a highly effective way to aggregate information. One possible explanation for the effective- ness of GAP is considering each feature vector as representing different semantic entity and GAP as a convex combination of them. Following this perspective, we generalize GAP and propose a learnable generalized sum pooling method (GSP). GSP improves GAP with two distinct abilities: i) the ability to choose a subset of semantic entities, effectively learning to ignore nuisance information, and ii) learning the weights corresponding to the importance each entity. Formally, we propose entropy-smootie generalization of GAP, i.e., a specific realization of the problem gives back GAP. We show that this optimization problem enjoys analytical gradients enabling us use it as a direct learnable replacement for GAP. We further propose a zero-shot loss ease the learning of GSP. We show the effectiveness of our method with extensive evaluations on 4 popular metric learning benchmarks. Code is available GSP-DML Framework
optimal transport problem and show that it is a strict
INTRODUCTION
Distance metric learning (DML) addresses the problem of finding an embedding function such that the semantically similar samples embedded close to each other while the dissimilar ones are placed relatively apart in the Euclidean sense. Although the prolific and diverse literature of DML includes various architectural designs (Kim et al.) 2018 Lin al. 2018: Ermolov et al., (022). loss functions (Musgrave et al., 2020), and data data-augmentation techniques (Roth et al. 2020] Venkataramanan et al. 2022), many of these methods have shared component: a convolutional neural network (CNN) Tollowed by global pooling layer, mostly global average pooling (GAP) (Musgrave et al.) 2020).
Common folklore to explain the effectiveness of GAP is considering each pixel of the CNN feature map corresponding to separate semantic entity. For example, spatial extent of one pixel can correspond to "tire" object making the resulting feature a representation for "lireness" of the image. If this explanation is correct, the representation space defined via output of GAP is convex combination of semantically independent representations defined by each pixel in the feature map. Although this folklore is later empirically studied in (Zeiler & Fergus 2014 7hou et al. 2016 2018 and references therein) and further verified for classification in (Xu et al. (0020) its algorithmic implications are not clear. If each feature is truly representing a different semantic entity, should we really average over all of them? Surely, some classes belong to the background and should be discarded as nuisance variables. Moreover, is uniform average of them the best choice? Aren't 't some classes more important than others? In this paper, we try to answer these questions within the context of metric learning. We propose learnable and generalized version of GAP which learns choose the subset of the semantic entities to utilize as well as weights to assign them while averaging.
In order to generalize the GAP operator to be learnable, we re-define it as solution of an optimization problem. We let the solution space to include O-weight effectively enabling us to choose subset of the features as well as carefully regularize it to discourage degenerate solution of using all the features. Crucially, we rigorously show that the original GAP is a specific case of our proposed optimization problem for a certain realization. Our proposed optimization problem closely follows optimal transport based top-k operators (Cuturi et al. 2019) and we utilize its literature solve it. Moreover, we present an algorithm for an efficient computation of the gradients over this optimization problem enabling learning. A critical desiderata of such an operator is choosing subset of features

---

Under review as a conference paper at ICLR 2023
which are discrimantive and ignoring the background classes corresponding to nuisance variables. Although supervised metric learning losses provide guidance for seen classes, they carry no such information to generalize the behavior to unseen classes. To enable such a behavior, we adopt a zeno- shol prediction loss as regularization term which is built on expressing the class label embeddings as convex combination of attribute embeddings (Demirel et al. 2017 Xu al. 2020).
In order to validate the theoretical claims, we design a synthetic empirical study. The results confirm that our pooling method chooses better subsets and improve generalization ability. Moreover, our method can be applied with any DML loss as GAP is a shared component of them. We applied our method on 6 DML losses and test on 4 datasets. Results show consistent improvements with respect to direct application of GAP as well as other pooling alternatives.
2 RELATED WORK
We discuss the works which are most related ours. Briefly, our contributions include that i) we introduce a general formulation for weighted sum pooling, it) we formulate local feature selection as an optimization problem which admits closed form gradient expression without matrix inversion, and iii) we propose a meta-learning based zero-shot regularization term to explicitly impose unseen class generalization to the DML problem.
DML. Primary thrusts in DML include t) tailoring pairwise loss terms (Musgrave et al.
2020) that
penalize the violations of the desired intra- and inter-class proximity constraints, 2i) pair mining Rothh
ct al. 2020). i8z) generating informative samples (Ko & Gu 2020 Liu et al. 2022]] Gu et al. 2021
Venkataramanan et al.) 2022). and ir) augmenting the mini- batches with virtual embeddings ca called
proxies (Wang et al. 2020; Tch et al.] 2020). To improve generalization; learning theoretic ideas
(Dong et 2020 Lei Lei et al. [2021] Gurbuz et al. 2022), disentangling class- daass-iicrrmmiativee and class- shared features (Lin al. 2018 Roth al. 20T9), intra-batch feature aggregation (Seidenschwarz
cet al. 2021), and further regularization terms (Jacob et al., 2019 /Zhang et al., [0200] Kim & Park 2022) are utilized. To go beyond of single model, ensemble (Xuan al. [2018]
2021 Roth et al.
Kim et al.] 2018 Sanakoyeu et al., 2019 Zheng et al. 202 by and multi-task based approaches
Miilbich et al.] 2020 Roth et al. 2021) are also used. Different to them, we propose a learnable
pooling method for the global feature extraction generalizing GAP, a shared component of all of the
mentioned works. Hence, our work is orthogonal to all of these and can be used jointly with any of
them.
Prototype-based pooling. Most related to ours are trainable VLAD (Arandjelovic et al. 2016) and optimal transport based aggregation (Mialon et al.) (021). Such methods employ similaritis to the prototypes to form a vector of aggregated local features for each prototype and build ensemble of representations. Similar to us, Mialon et al. (2021) uses optimal transport formulation to select local features to be pooled for each prototype. That said, such methods map a set of features to another set of features without discarding any and do not provide a natural way aggregate the dlas-discriminative subset of the features. On the contrary, our pooling machine effectively enables learning to select discriminative features and maps a of features single feature that is distilled from nuisance information.
Attention-based pooling. Among the methods that reweights the CNN features before pooling, CroW (Kalantidis et al. 2016). Trainable-SMK (Tolias et al. 2020), and CBAM (Woo et al. 2018) build on feature magnitude based saliency, assuming that the backbone functions must be able to zero-out nuisance information. Yet, such a requirement is restrictive for the parameter space and annihilation of the non-discriminative information might not be feasible in some problems. Similarly, attention- based weighting methods DeL.F (Noh et al. 2017). GSoP (Gao et al. 2019) do not have explicit control on feature selection behavior and might result in poor models when jointly trained with the feature extractor (Noh et al., 2017). Differently, our method unifies attention-based feature masking practices (e.g. convolution, correlation) with an dfficient-to-solly optimization framework and lets do away with engineered heuristics in obtaining the masking weights (e.g. normalization, sigmoid, soft-plus) without restricting the solution space unlike magnitude based methods.
Optimal transport based operators. Optimal transport (OT) distance (Cuturi 2013) match local features is used as the DML distance metric instead of '2 in (Zhao et al.) 2021). Despite effective, replacing f2 with OT increases memory cost for image representation well as computation cost for
2

---

Under review as a conference paper at ICLR 2023
the distance computation. Different to them, we shift OT based computation in pooling (i.e., feature extraction) stage while having OT's merits and hence, do not affect the memory and computation costs of the inference by sticking to f2 metric. Moreover, our feature selection and aggregation formulation has close relation to optimal transport (Cuturi 2013) based top-k (Xie et al. 2020). ranking (Cuturi et al. 2019) and aggregation (Mialon al. (021) operators. What makes our method different is the unique way we formulate the feature selection problem to fuse aggregation into it. Our formulation allows computationally appealing and matrix inversion free gradient computation of the selection operator unlike optimal transport plan based counterparts (Luise et al. 2018
3 PRELIMINARIES
Consider the data distribution Paxy over Xx where X' is the space of data points and y is the space of labels. Given zid. samples from Pxxy as {2,, y)). distance metric learning problem aims to find the parameters 0 of an emhedding function e(:; 0) space of embeddings is consistent with the label information where d ís the embedding dimension. More specifically, [|e(,,;; 0) elry; 0)ll small whenever = Y/j and large whenever y/ * yj- In order enable learning, this requirement represented via loss function (((2)) (z 2/ 0) (e.g. contrastive (Wu et al. 2017). tripler (Schroff et al., 2015), multi-similarity (Wang et al. 2019)).
Ht" such that the Euclidean distance the
The typical learning mechanism is gradient descent of an empirical risk function defined over a batch of data points B. To simplify notation throughout the paper, we will use 6 = {b(i) 24, E B} to index the samples in batch. Then, the typical empirical risk function defined as:
(b;) = To EΣ'((E (
(3.1)
KEbjEb
We are interested specific class embedding functions where global average pooling is used. Specifically, consider the composite function family e = go I such that g is pooling and f is feature computation. We assume a further structure over the functions and ʃ. The feature function f maps the input space X' into ipuxhxdd where w and are spatial dimensions. Moreover, g performs averaging as;
f.. g gf((r;0)) w iE(w-h]
(3.2)
where [m] 1,.... and we let fiElt denote 7th spatial feature of [xx 0) to avoid convoluted notation. In the rest of the paper, we generalize the pooling function g into a learnable form and propose an algorithm to learn it.
METHOD
Consider the pooling operation in Eq. (3.2). it is a simple averaging over pixel-level feature maps (fi). As we discuss in |[ one explanation for the effectiveness of this operation is considering each as corresponding to different semantic entity corresponding to the spatial extend of the pixel, and the averaging as convex combination over these semantic classes. Our method based on generalizing this averaging such that specific subset of pixels (correspondingly subset of semantic entities) are selected and their weights are adjusted according to their importance.
We generalize Eq. 3.2 in 4.11by formulating feature selection problem in which we prioritize subset of the features that are closest to some trainable prototypes. If feature is to be selected, its weight will be high. We then formulate our pooling operation as differentiable layer so that the prototypes can be learned along with the rest of the embedding function parameters in $ We learn the prototypes with class-level supervision, however in metric learning, learned representations should generalize to unseen classes. Thus, we introduce a zero-shot prediction loss to regularize prototype training for zero-shot setting in 4.3
3

---

Under review as a conference paper at ICLR 2023
4.1 GENERALIZED SUM POOLING AS LINEAR PROGRAM
Consider the pooling function, with adjustable weights as ( 0); - Eie[n] Pifa where w.. h. Note that, Pi = 1/n corresponds to average pooling. Informally, we want to control the weights to ease the metric learning problem. Specifically, we want the weights corresponding to background classes to be 0 and the ones corresponding to discriminative features to be high.
we were given representations discrimantive semantic entities, we could simply compare them with the features (f,) and choose the ones with high similarity. Our proposed method is simply learning these representations and using them for weight computations. We first discuss the weight computation part before discussing learning the representations of prototypes.
Assume that there are m discrimantive semantic entities which we call protorypes with latent repre- sentations = (es)) of appropriate dimensions (same as fi). Since we know that not all features (E)))) are relevant, we need to choose subset of (Ji)e We perform this top-k selection process by converting it into an optimal transport (OT) problem.
Consider cost map Cūj ||;; fulle which is
(number of prototypes) by (number of features) matrix representing the closeness of prototypes w; and features fy after some normalization (1, We would like to find transport map æ which re- re-distributes the uniform mass from features to prototypes. Since we do not have any prior information over features, we also consider its marginal distribution (importance of each feature begin with) to be uniform. As need to choose a subset, we set μÆ(0, (0, 1] ratio of mass to be transported. The resulting OT problem is:
P arg min Tay
(PI)
P.T 00
P5 P. 1// bn
Different to typical OT literature, we introduce decision variables, to represent residual weights to be discarded. Specifically modelling discarded weight instead of enforcing another marginalization constraint is beneficial beyond stylistic choices as it allows us to very efficient compute gradients. When the introduced transport problem is solved, we perform weighting using residual weights as:
((f(x;)):w))
P.f: = Σ. 1/-Pi
(4.1)
Given set of prototypes (w: [imm)) solving the problem in (PI) is strict generalization of GAP since setting μ = 1 recovers the original GAP. We formalize this cquivalence in the following claim.
Claim 4.1. If μ the operation Eq. (4.1) reduces to global average pooling in Eq. (3.2).
We defer the proof to Appendix. Having generalized GAP to learnable form, we introduce method learn the prototypes [wi] [W])ee(m] in the next section.
4.2 GENERALIZED SUM POOLING AS A DIFFERENTIABLE LAYER
pooling weights fcature map
Consider the generalized form of pooling, defined as solution of PI), as layer of a neural network. The input is the feature vectors (ff fie(]. the learnable parameters are prototype representations (witie(m) and the output is residual weights p". To enable leamn ing, we need partial derivatives of pa with respect to {wi hejm]- However, this function is not smooth. More importantly requires the μ parameter to be known a priori.
linear programming
(e so--mheehed
We use a toy example to set the stage for rest of the formula- tion. Consider a 10x 10x3 feature map visualized as RGB-image in Fig. and corresponding two prototypes with representations (1, O, 0) (red) and (0, 0, 1) (blue). The true μ 0.5 since the half of the image corresponds to red and blue, and other half background class of green. Consider an under-estimation of μ 0.2, the global solution (shown as linear programming) is explicitly ignoring in- formative pixels (part of red and blue region). To solve this issue,
Figure 1: Feature map and the resultant pooling weights (higher the darker) of different problems.
4

---

Under review as a conference paper at ICLR 2023
entropy smoothing which is first introduced in (Cuturi 2013) to enable fast computation of optimal transport. Consider the entropy smoothed version of the original problem in (PI) as:
ple), (() arg min ij
CiyTiy + Lis miy log miy + +E, Ps log p5),
(P2)
0
+ 2/// De
and obtain pooling weights by replacing p" with ((=) in Eq. (4.1) When smoothing is high (s++0), the resulting solution is uniform over features similar to GAP. When it is low. the result is similar to top-k like behavior. For us, controls the trade-off between picking i portion of the features that are closest to the prototypes and including as much features possible for weight transfer.
We further visualize the solution of the entropy smoothed problem in Fig. showing desirable behavior even with underestimated
Beyond alleviating the under-estimation of μ problem, entropy smoothing also makes the problem strictly convex and smooth. Thus, the solution of the problem enables differentiation and in fact, admits closed-form gradient expression. We state the solution of (P2) and their corresponding gradients in the following propositions and defer their proofs to Appendix.
Proposition 4.1. Given initialization (()) = 1, consider the following iteration:
plt+1) 1/ + ((k) exp(-Ec)T1.m)) 1,,) ((k+1) = (1 (xp(-
where exp and (-)- are element-wise and 1m is m-dimeensional vector of ones. Then, (pl ,100) converges to the solution of (P2) defining transport map via n<() = ((k) exp(=se)Diag(pft)
Proposition 4.2. Consider any differentiable loss function as a function of (p. TT. Given gradients DC
DC and ac Tytey Setee, with q = p($) G BC
+ (π((5) ④
T1 and 1j (ple) Sptes 1 n ple).
Bates
the gradient with respect to c reads:
dL =
OC
no (x) Diagla - I-u--plee )ple))
(4.2)
dc
OT(())
where 3 denotes element-wise multiplication.
Proposition and suggest that our feature selective pooling can be implemented as a differen- tiable layer. Moreover, Proposition 4.2 gives matrix inversion free computation of the gradient with respect to the costs unlike optimal transport based operators (Luise et al. 2018). Thus, the prototypes, w, can be jointly learned with the feature extraction efficiently.
4.3 CROSS-BATCI ZERO-SHOT REGULARIZATION
Until now, we formulate a prototype based feature pooling and can learn the prototypes using class labels with any DML loss. To this end, specializing to classes is a feasible behavior for the prototypes. On the other hand, we rather want the prototypes to capture transferable attributes so that the learning can be transferred the unseen classes as long as the attributes are shared. In other words, learning with prototype based pooling shapes the embedding geometry in such a way that we have clusters corresponding to the prototypes in the embedding space. We want such clusters have transferable semantics rather than class-specific information. To enable this, we now formulate a mechanism to predict class embedding vectors from the prototype assignment vectors and use that mechanism to tailor a loss regularizing the prototypes to have transferable representations.
Our feature selection layer should learn discriminative feature prototypes, w, using top-down label information. Consider two randomly selected batches, (b,, b2). of data sampled from the distribution. the prototypes are corresponding to discrimantive entities, the weights transferred to them (i.e., marginal distribution of prototypes) should be useful in predicting the classes and such behavior should be consistent between batches for zero-shot prediction. Formally, if one class in b2 does not exist in bj. a predictor class labels based on marginal distribution of prototypes for each class of by should still be useful for b2- Sadly, DML losses do not carry such information. We thus formulate zero-shot prediction loss enforce such zero-shot transfer.
We consider that we are given semantic embedding vector for each of c-many class labels, [us)eeld] We are to predict such embeddings from the marginal distribution of the prototypes. In
5

---

Under review as a conference paper at ICLR 2023
particular, linear predictor, A, to predict label embeddings as 6 A where 2 is the normalized distribution of the weighs on the prototypes;
J. E
where
t. +(+)
[ kE(M)
(4.3)
we consider the prototypes as semantic vectors some auxiliary labels such as aftributes commonly used in zero-shot learning (ZSL) (Demirel et al.) 2017; Xu et al. (0200] Huynh & Elhamifar (020).
then we can interpret z as pseudo-atrribute predictions. Given pseudo-attribute predictions, {2 ribb and corresponding class embeddings for batch, b, we fit the predictor as;
A
arg min Eieo || Vy. + Eie(m) |a:11. A=(axhc(mm)
(P3)
which admits a closed form expression enabling back propagation As = Y, (2 + + where Y, ket Zg = [ai practice, we are not provided with the label embeddings, T = [us]ee[e]- Nevertheless, having closed-form expression for As enables to exploit meta- learning scheme like (Bertinctto et al. 2018) to formulate a zero-shot prediction loss learn them jointly with the rest of the parameters.
Specifically, we split a batch, b, into two as b and by such that classes are disjoint. We then estimate attribute embeddings, Ab according to (P3) using set and use that estimate predict the label embeddings of the other set to form zero-shir shot prediction loss. Formally, our loss becomes:
Cbs(b; 0) b2
Σ log (1 + Σ e (. + 1b VEbs
Σ log (1 iE
Σ %E(0
Lv, A2 (4.4)
jEld
i.., rearranged soft-max cross-entropy where Ag=Ab, with the abuse notation, and {0,, r
(i.e., CNN parameters, prototype vectors, label embeddings).
We learn attribute embeddings (i.e., columns of A) as sub-task and can define such learning as a differentiable operation. Thus, cross- cross-batch zero-shot prediction loss, Lzs, is to achieve learning to learn attribute embeddings for zero-shot prediction. Intuitively, such a regularization should be useful in better generalization of pooling operation unseen classes since pseudo-atribute predictions are connected to prototypes and the local features. We combine this loss with the metric learning loss using mixing (i.e., (1-A)CDML + ALzs) and jointly optimize.
4.4 IMPLEMENTATION DETAILS
Embedding function. For the embedding function, f(;; we use ResNet20 (He et al. 2016) for Cifar (Krizhevsky & Hinton| 2009) experiments, and ImageNet (Russakovsky et al. 2015) pretrained BN-Inception (loffe & Szegedy 2015) for the rest. We exploit architectures until the output before the global average pooling layer. We add a per-pixel linear transform (i.e., 1x1 convolution). the output to obtain the local embedding vectors of size 128.
Pooling layer. For baseline methods, we use global average pooling. For our method, we perform parameter search and set the hyperparameters accordingly. Specifically, we use 64- or 128-many prototypes depending on the dataset. We use ==0.5 for proxy-based losses and 5=5.0 for non-proxy losses. For the rest, we set μ=0.3, e00.55, A= and iterate until 1=100 in Proposition 4.1 The embedding vectors upon global pooling are normalized to have unit norm.
label enihadkings T
Gaatures
costs
pscudo-stttile veular
minn [T-2A13 3 d
sumooltt prodictio
Decuco araibute
uniform
pooling weights
vecioes
CNNI6
Sollure
dir cless dl....
(DMI
generalized sum pooling
enibedanng
labels
Input booch (Ks.
trunable perametens: (9,.4,
training flow
inference flow
Figure 2: Sketch of the method.

---

Under review as a conference paper at ICLR 2023
Global Average Pooling
Generalized Sum Pooling
evaluation Collage
(Pai Fakk Nirge] 1ass0, duns y1 21
Piat rgit Mangel the so, so aus, stand
baseline
camples
+25
+CSP
77.61
+GSP425P
NAW WA
2
best samples
2.23
Dong specn
shilec Coline
shandd sclens
Iurndd precppo
kurrer puddgges
ProxyNCA++
(a.2)
(b.1)
(b.2)
Figure 3: (a): GAP GSP in aggregating features, where tokens denote learned embedding vectors and samples are obtained by aggregating them. (b): Evaluation on Cifar Collage dataset (b.1), 1), and (b.2) sample train and test images with their attention maps in terms of pooling weights. Distilled denotes baseline performance on non-collage dataset (i.e., excluding the shared classes).
5 EXPERIMENTS
We start our empirical study with synthetic study validating the role of GAP in learning and the impact of GSP on the feature geometry. We further examine the effectiveness of our generalized sum pooling in metric learning for various models and datasets. We further perform ablation studies for the implications of our formulation as well as effects of the hyperparameters. We share the implementation details as well complete Tensorflow (Abadi et al. 2016) code base in the supplemental materials.
with acro-shol proliction loss attribute
without acro-slox preliciion loss attribute
5.1 SYNTHETIC STUDY
plano
plano
We design a synthetic empirical study to evaluate GSP a fully controlled manner. We consider 16- class problem such that classes are defined over trainable tokens. In this setting, tokens correspond semantic entities but we choose to give a specific working to emphasize that they are trained as part of the learning. Each class is defined with dis- tinct tokens and there are also 4 background tokens shared by all classes. For example, "car" class would have tokens like "tire" and "window" as well as background tokens of "Iree" and "road", We sam- ple class representations from both class specific and background tokens according to a mixing ratio NN(055, 0.1). Such 50-many feature collec- tion will correspond to a training sample (i.e., we are mimicking CNN's output with trainable tokens). We then obtain global representations using GAP and GSP. We visualize the geometry of the embed- ding space in Fig. (a). With GAP, we observe overlapping class convex hulls hence classes are not well discriminated. In other other hand, GSP gives well separated class convex hulls further validation that it learns to ignore background tokens.
decr-
deer
dog-
dog
frog
frgg
ship
truck
truck
attributes
attributes
MAPpy 33.71 MM Row 64.47
M MARp Pa 23.79 /MAP
30.16
semplo imeges oon their pooting wengar matted with persebbe protiction loss
Figure 4: Comparing the distributions of the learned 8 prototypes across classes of Ci- far 0 dataset with and without Lzs. Pooling weights are coloured according the domi- nant prototype at that location.
We further extend this study image domain. We consider the super-classes of Cifar100 dataset where each has 5 sub-classes. For each super-class, we split the sub-classes for train (2), validation (1), and test (2). We consider 4 super-classes as the shared classes and compose 4x4-stitched collage images for the rest 16 classes. In particular, we sample an image from class and then sample images from shared classes. We use ResNet20 backbone pretrained on Cifar 100 classification task and follow the implementation explained in 4.4. We provide the evaluation results in Fig, (b). GSP and the proposed zero shot loss effectively increase MAP@R R. We also provide sample train and test images to showcase that our pooling transfer well to unseen domain.
7

---

Under review as a conference paper at ICLR 2023
We also evaluate the zero-shot prediction performance of the pseudo-attribut vectors. We train on Cifarl0 dataset with 8 prototypes using ProxyNCA++ Teh et al. (2020) (PNCA) loss with and without Lzs. We then extract pseudo-attribute histograms for each class and visualize them in Fig. 4 We observe transferable representations with Lzs and we visually show in Fig. E that the semantic entities represented by the prototypes transfer across classes. We quantitatively evaluate such behavior by randomly splitting the classes into half and apply cross-batch zero-shot prediction explained in 4.3 Namely, we fit in P3) for one subset and use it to predict the class embeddings for the other set. We pre-compute class embeddings from the dataset as the class mean. To this end, our evaluation assesses generalization of both the features and the prototypes. We use MAP with both /2 distance and cosine similarity in our evaluation. We repeat the experiment 1000 times. We observe in Fig. that zero-shot performance of the prototypes learned with Lzs is substantially superior. We also see that our feature aggregation method enables approximate localization of the semantic entities. Recent ZSL approaches (Huynh Elhamifar 2020; Xu et al. 2020) can provide attribute localization and share similar spirit with our method. However, attribute annotations must be provided for those methods whereas we exploit only class labels to extract attribute-like features. Our method can be considered as attribute- srrribbte-unspervissee version of these methods.
5.2
DEEP METRIC LEARNING EXPERIMENTS
5.2. SETUP
In order to minimize the confounding of factors other than our proposed method, we keep the comparisons as fair as possible following the suggestions of recent work explicitly studying the fair evaluation strategies for metric learning (Roth et al., 2020 Musgrave et al. 2020 Fehervari et al.) 2019). Specifically, we mostly follow the procedures proposed in (Musgrave et al. 2020) to provide fair and unbiased evaluation of our method as well as comparisons with the other methods. We additionally follow the relatively old-fashione conventional procedure (Oh Song al. 2016) for the evaluation of our method and provide those results in the supplementary material. We provide full detail of our experimental setup in the supplementary material for complete transparency and reproducibility.
Datasets. We use CUB-200-2011 (CUB) (Wah et al. 2011), Cars196 (Krause & Golovin 2014), In-shop (Liu et al. 2016), and Stanford Online Products (SOP) (Oh Song et al. 2016) with the data augmentation from (Musgrave al. 2020).
Evaluation metrics. We report mean average precision (MAP@R) at R where R is defined for each query and is the total number of true references of the query.
Hyperparameters, We use Adam (Kingma & Ba, 2014) optimizer with learning rate 10-5, weight decay 10 batch size 32 (4 per class). We train fold: models (1 for each 3/1 train set partition).
Evaluation. Average performance (128D) where each of 4-fold model is trained 3 times resulting in realization of 311 different model collections. In our results we provide mean of 81 evaluations.
Baselines. We implement our method on top of and compare with Contrastive (C2): Contrastive with positive margin (Wu al.] (2017). MS: Multi-similari (Wang et al. (2019). Triplet: Triplet (Schroff) et al. 2015). XBM: Cross-batch memory (Wang et al.) 2020) with contrastive loss (Hadsell et al. PNCA: ProxyNCA++ (Teh et al. 2020). PAnchor: ProxyAnchor (Kim et al., 2020).
2006
5.2.2
RESULTS
We compare our method (GSP) against direct application of GAP with 6 DML methods in 4 datasets. We also evaluate 13 additional pooling alternatives on Ciffar Collage and CUB datasets. We provide the results in supplementary material, Tab. 4 Based on CUB performances, we pick generalized mean pooling (GeMean) (Radenović et al. 2018) and DeLF (Noh et al., 2017) to compare against in 4 DML benchmarks. We also evaluate max pooling (GMP) and its combination with GAP we typically observe GAP+GMP in the recent works /Venkataramanan et al.) 2022 Teh et al. 2020 Kim et al. 2020 Wang et al., 2020). We also apply our method with GMP (GMP+GSP) and with GeMean (GeMean+GSP) to show that per channel selection is orthogonal to our approach and thus,
GSP can marginally improve those methods as well.
8

---

Under review as a conference paper at ICLR 2023
Ingrovements Posting
Improvements BML Losses
36.3
14.9
30.5
MS
16.0
MAAIS Cars Plo
HA HAngB in Inaahsp
Triplet
PMCA
(Andhor
F2-GMP
-SP
PUCA GAP
PUCA Gvp
-DAP
DDeLF
-Geteea
-COP
01.7
663
21.8
400
45.0
16.5
20.9 21.5
Figure 5: Summary of relative improvements recorded in Table
We provide the detailed evaluation results in supplementary material, Tab. and we summarize the
relative MAP@R orderings of the methods with 128D embeddings Fig. 5 We observe consistent improvements upon direct application of GAP in all datasets. On the average, we consistently improve the baselines x1% points MAP@R Our improvement margins are superior to ones of attention based DeLF pooling. We improve state-of-the-an (SOTA) XBM method up to 2% points, which is good evidence that application of GSP is not limited to loss terms but can be combined with different DML approaches. We also consistently improve GMP and GeMean pooling methods in all datasets, yet another evidence that our method can be combined with max pooling based methods.
We additionally evaluate method with different architectures and methods in conventional setting (Oh Song et al. (0166) for the comparison with SOTA. The results are provided in supplementary material, Tab. 3 where observe that achieve SOTA performances with XBM (Wang et al.)
2020 and LIBC (Seidenschwarz et al. 2021 methods.
5.2.3 ABLATIONS
Effect of Czs. We empirically show the effect Lzs on learned represen- tations in & [.11 We further examine the effect Lzs quantitatively by en- abling/disabling it in 4 datasets. We also evaluate its effect without GSP by setting 4=1 where use GAP with pseudo- attribute vectors. The results are summa- rized in Tab. [ showing that both com- ponents improves the baseline and their combination brings the best improvement. We observe similar behavior in Cifar Collage experiment
Table Effects of the two components: Zero-shot pre- diction loss (ZSP) and Generalized Sum Pooling (GSP)
Rase Method: C2
MAP@R
Component
SOP
In-shop
CUB
Cars196
2SP
GSP
512D
128D
512D
128D
512D
128D
512D
128D
46.78
42.66
59.46
55.50
26.25
20.85
25.54
17.88
16.60
12.55
$9.38
55.43
26.19
21.08
25.54
17.67
46.81
42.84
60.01
55.94
27.12
21.52
26.25
18.31
(Fig. B (b)) where the effect of Lzs is more substantial.
Effect of μ. As we discuss in $ 4.2, GSP is similar top-k operator with an adaptive thanks to entropy smoothing. We empirically validate such behavior in CUB dataset with C2 loss by sweeping μ parameter controlling top-k behavior. We plot the performances in Fig. [ Relatively lower values of per- forms similarly. As we increase μ4, the performance drops towards GAP due to possibly overestimating the foreground ratio.
effect of (CUB-C2)
#2.02
77.32
25.95
baseline:C2-512
CONCLUSION
MAP@I
C2 GSP 512D
baseline:C2-122
C2 C2-G5D-128D
Building on perspective explaining the success of GAP, we pro- posed learnable and generalized version. Our proposed general- ization is a trainable pooling layer that selects the feature subset and re-weight it during pooling. To enable effective learning of the proposed layer, we also proposed a regularization loss to improve zero-shot transfer. With extensive empirical studies, validated the effectiveness the proposed pooling layer in various metric learning benchmarks.
0.1 0.3 06 0.8
Figure 6: Effect of μ in CUB dataset with C2 loss. Shaded regions represent Fstd.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Martín Abadi, Paul Barham, Jiammin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Chemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: a system for large-scale machine learning. In OSDI, volume 16, pp. 283, 2016.
Relja Arandjeelvic,, Petr Gronat, Akihiko Torii, Tomas Pajdla, and Josef Sivie. Netvlad: Cnn architecture for weakly supervised place recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition,, pp. 5297-5307, 2016.
Heinz Bauschke and Adrian S Lewis. Dykstras algorithm with bregman projections: convergence proof. Optimization, 48(4):409-427 427, 2000.
Luca Bertinetto, Joao F Henriques. Philip Torr, and Andrea Vedaldi. Meta-learning with differentiable closed- form solvers. In International Conference Learning Representations, 2018.
Lev M Bregman. The relaxation method finding common point of convex sets and its application to solution of problems in convex programming. USSR computational mathematics and mathematical physics, 7 (3):200-217, 1967.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013.
Marco Cuturi, Olivier Teboul, and Jean-Philippe Vert. Differentiable ranking and sorting using optimal transport. Advonces neural information processing systems, 32, 2019.
Berkan Demirel, Ramazan Gokberk Cinbis, and Nazli Ikizler-Cinbis. Attribute2classmame:: discrimina- tive model for attribute- based unsupervised zero-shot learning. In Proceedings of the IEEE international conference computer vision, pp. 1232-1241, 2017.
M Dong, X Yang, Zhu, Wang, and J Xue. Generalization bound of gradient descent for non-convex metric learning. Neural Information Processing Systems Foundation, 2020.
Thibaut Durand, Nicolas Thome, and Matthieu Cord. Weldon: Weakly supervised learning of deep convolutional ncural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 4743-4752, 2016.
Aleksandr Ermolov, Leyla Mirvakhabova, Valentin Khrulkov, Nicu Sebe, and Ivan Oseledets. Hyperbolic vision transformers: Combining improvements in metric learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2022.
Istvan Fehervari, Avinash Ravichandran, and Srikar Appalaraju. Unbiased evaluation of deep metric learning algorithms. arXiv preprint arXiv: /9/ 1. 125228, 2019.
Zilin Gao, Jiangtao Xie, Qilong Wang, and Peihua Li. Global second-order pooling convolutional networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3024-3033, 2019.
Geonmo Gu, Byungsoo Ko, and Han-Gyu Kim, Proxy synthesis: Learning with synthetic classes for deep metric learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 1460-1468, 2021.
Yeti Z Gurbuz, Ogul Can, and Aydin Alatan. Deep metric learning with chance constraints. arXiv preprint arXiv:2209.08s 090000 2022.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Puttern Recognition (CVPR 06), volume 2, pp. 1735-1742. IEEE, 2006.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pp. 630-645. Springer, 2016.
Huynh and Ehsan Elhamifar. Fine-grained generalized zero-shot learning via dense attribute-based attention. In Proceedings of the IEEE/CVF conference on computer vision and puttern recognition, pp. 4483-4493, 2020.
Sergey Toffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448 456. PMI.R, 2015.
10

---

Under review as a conference paper at ICLR 2023
Pierre Jacob, David Picard, Aymeric Histace, and Edouard Klein. Metric learning with horde: High-order regularizer for deep embeddings. In The IEEE International Conference on Computer Vision (ICCV), October 2019.
Yannis Kalantidis, Clayton Mellina, and Simon Osindero. Cross-dimensional weighting for aggregated deep convolutional features. In European conference on computer vision, pp. 685-701. Springer, 2016.
Sungyeon Kim, Dongwon Kim, Minsu Cho, and Suha Kwak. Proxy anchor loss for deep metric learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3238-32-8 3247, 2020.
Wonsik Kim, Bhavya Goyal, Kunal Chawla, Jungmin Lee, and Keunjoo Kwon. inrention-basee ensemble for deep metric learning. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 736-751, 2018.
Yonghyun Kim and Wonpyo Park. Multi-leve distance regularization for deep metric learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 1827-1835, 2021.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
Byungsoo Ko and Geonmo Gu. Embedding expansion: Augmentation in embedding space for deep metric learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7255-7264, 2020.
Andreas Krause and Daniel Golovin. Submodular function maximization. In Tractability: Practical Approaches Hard Problems, pp. 71-104. Cambridge University Press, 2014.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Yunwen Lei, Mingrui Liu, and Yiming Ying. Generalization guarantee of sgd for pairwise learning. Advances in Neural Information Processing Systems, 34, 2021.
Xudong Lin, Yueqi Duan, Qiyuan Dong, Jiwen Lu, and Jie Zhou. Deep variational metric learning. In
Proceedings of the European Conference on Computer Vision (ECCV), pp. 689-704, 2018.
Chang Liu, Han Yu, Boyang Li, Zhiqi Shen, Zhanning Gao. Peiran Ren, Xuansong Xie, Lizhen Cui, and Chunyan Miao. Noise-resistant deep metric learning with ranking-based instance selection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6811-6820, 2021.
Ziwei Liu, Ping Luo. Shi Qiu, Xiaogang Wang, and Xiaoou Tang. Deepfashion: Powering robust clothes recognition and retrieval with rich annotations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1096-1104, 2016.
Tzon-Tzer and Sheng-Hua Shiou. Inverses 2x block matrices. Computers & Mathematics with Applications, 43(1-2): 13-1-9):199-12, 19-129, 2002.
Giulia Luise, Alessandro Rudi, Massimiliano Pontil, and Carlo Ciliberto. Differential properties of sinkhom approximation for learning with wasserstein distance. Advances in Neural Information Processing Systems, 31, 2018.
Grégoire Mialon, Dexiong Chen, Alexandre d' A'ppremon,, and Julien Mairal, A trainable optimal transport embedding for feature aggregation and its relationship attention. ICLR 2021-The Ninth International Conference on Learning Representations. 2021.
Timo Milbich, Karsten Roth, Homanga Bharadhwaj,, Samarth Sinha, Yoshua Bengio, Bjöm Ommer, and Joseph Paul Cohen. Diva: Diverse visual feature aggregation for deep metric learning. European Conference on Computer Vision, pp. 590-607. Springer, 2020.
Naila Murray, Hervé Jégou, Florent Perronnin, and Andrew Zisserman. Interferences in match kemelss. IEEE transactions on pattern analysis and machine intelligence, 3999) 2016.
Kevin Musgrave, Serge Belongie, and Ser-Nam Lim. A metric learning reality check. In European Conference Computer Vision, pp. 681 169.. Springer, 2020.
Tony Ng. Vassileios Balntas, Yurun Tian, and Krystian Mikolajezyk. Solar: second-order loss and attention for image retrieval. In European conference computer vision, pp. 253-270. Springer, 2020.
11

---

Under review as a conference paper at ICLR 2023
Hyeonwoo Noh, Andre Araujo, Jack Sim, Tobias Weyand, and Bohyung Han. Large-scale image retrieval with attentive deep local features. Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
Hyun Oh Song, Yu Xiang, Stefanic Jegelka, and Silvio Savarese. Deep metric learning via lifted structured feature embedding. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pp- 4004-4012, 2016.
Filip Radenović, Giorgos Tolias, and Ondiej Chum. Fine-tuning enn image retrieval with no human annotation. IEEE transactions on pattern analysis and machine intelligence, 41(())1655- 1668, 2018.
Karsten Roth, Biagio Brattoli, and Bjorn Ommer. Mic: Mining interclass characteristics improved metric learning. In The IEEE International Conference on Computer Vision (ICCV), October 2019.
Karsten Roth, Timo Milbich, Samarth Sinha, Prateek Gupta, Bjom Ommer, and Joseph Paul Cohen. Revisiting training strategies and generalization performance in deep metric learning. In International Conference on Machine Learning, pp. 8242-8252. PMLR, 2020.
Karsten Roth, Timo Milbich, Bjorn Ommer, Joseph Paul Cohen, and Marzyeh Ghassemi. S2sd: Simultaneous similarity-hased self-distillation for deep metric learning. In ICMI, 2021: 38th International Conference on Machine Learning, pp. 9095-9106, 2021.
Karsten Roth, Oriol Vinyals, and Zeynep Akata. Non-isotropy regularization for proxy-based deep metric learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7420-7430, 2022.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh. Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-25 2015.
Artsiom Sanakoyeu, Vadim Tschernezki, Uta Buchler, and Bjom Ommer. Divide and conquer the embedding space for metric learning. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 815 823, 2015.
Jenny Seidenschwarz, Ismail Elezi, and Laura Leal-Taixé. Learning intra-batch connections for deep metric leamning. In Proceedings of the 38th International Conference on Machine Learning. ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 9410-991999 9421. PMLR, 2021.
Eu Wern Teh, Terrance Deiriss, and Graham W Taylor. Proxynca++: Revisiting and revitalizing proxy neighborhood component analysis. In European Conference on Computer Vision (ECCV). Springer. 2020.
Giorgos Tolias, Tomas Jenicek, and Ondicj Chum. Learning and aggregating deep local descriptors for instance- level recognition. In European Conference on Computer Vision. pp. 460-474 (60. Springer, 2020.
Shashanka Venkataramanan, Bill Psomas, Ewa Kijak, laurent amsaleg, Konstantinos Karantzalos, and Yannis Avrithis. It takes two to tango: Mixup for deep metric learning. In International Conference on Learning Representations, 2022.
net www.mm wot.ouummdddyyyy3333-
Catherine Wah, Steve Branson, Peter Welinder, Pictro Perona, and Serge Belongie. The caltech-uesd birds-200- 2011 datasct. 2011.
Xun Wang, Xintong Han, Weilin Huang, Dengke Dong, and Matthew R. Scott. Multi-similarity loss with general pair weighting for deep metric learning. In The IEFF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Xun Wang, Haozhi Zhang, Weilin Huang, and Matthew R Scott. Cross-batch memory for embedding learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6388 6397, 2020.
Sanghyun Woo, Jongchan Park, Joon- Young Lec, and In So Kweon. Cbam: Convolutional block attention module. In Proceedings of the European conference on computer vision (ECCV), pp. 3-19, 2018.
Chao-Yumn Wu, R Manmatha, Alexander J Smola, and Philipp Krahenbuhl. Sampling matters in deep embedding learning. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2840-2848, 2017.
12

---

Under review as a conference paper at ICLR 2023
Yujia Xie, Hanjun Dai, Minshuo Chen, Bo Dai, Tuo Zhao, Hongyuan Zha, Wei Wei, and Tomas Plister. Differentiable top-k with optimal transport. Advances Neural Information Processing Systems, 33:20520- 20531, 2020.
Wenjia Xu, Yonggin Xian, Jiuniu Wang, Bert Schiele, and Zeynep Akata. Attribtee prototype network for zero-shot learning. Advances Neural Information Processing Systems, 33:21969-21980, 2020.
Hong Xuan, Richard Souvenir, and Robert Pless. Deep randomized ensembles metric leaming. Proceedings of the European Conference on Computer Vision (ECCV), pp. 723-734- 234-,34,22 2018.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. European conference on computer vision, pp. 818-833. Springer, 2014.
Dingyi Zhang, Yingming and Zhongfei Zhang. Deep metric learning with spherical embedding. Advances Neural Information Processing Systems, 33, 2020.
Wenliang Zhao, Yongming Rao, Ziyi Wang, Jiwen and Zhou. Towards interpretable deep metric learning with structural matching. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp- 9887-998 9897,999,,222 2021.
Wenzhao Zheng, Chengkun Wang. Jiwen Lu, and Jie Zhou. Deep compositional metric learning. I Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 9320-932029 2220-9329.0 2021a.
Wenzhao Zheng, Borui Zhang, Jiwen Lu, and Jie Zhou. Deep relational metric learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12065-12074, 2021b.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2921-2929, 2929, 2016.
Bolei Zhou, David Bau, Aude Oliva, and Antonio Torralba. Interpreting deep visual representations via network dissection, IEEE transactions pattern analysis and machine intelligence. 2018
Yuehua Zhu, Muli Yang, Cheng Deng, and Wei Liu. Fewer is more: A deep graph metric learning perspective using fewer proxies. Advances in Neural Information Processing Systems, 33:17792 17803, 2020.
13

---

Under review as a conference paper at ICLR 2023
SUPPLEMENTAL MATERIAL FOR "GENERALIZED SUM POOLING FOR METRIC LEARNING"
EXTENDED EMPIRICAL STUDY FOR DEEP METRIC LEARNING
In the following sections, we explain our empirical study in detail and provide additional experiments on effect of hyperparameters well as evaluation with the conventional experimental settings.
REPRODUCIBILIIY
We provide full detail of our experimental setup and recapitulate the implementation details for the sake of complete transparency and reproducibility. Code is available at: GSP-DML Framework
1.1 SETUP
1.1.1 DATASETS
We perform our experiments on widely-used benchmark datasets: Stanford Online Products (SOP) (Oh Song al., 2016), In-shop (Liu et al. 2016). Cars196 (Krause & Golovin 2014) and, CUB-200-2011 (CUB) (Wah et al. 2011).
SOP has 22,634 classes with 120,053 product images. The first 1 11,318 classes (59,551 images) are split for training and the other 11,316 (60,502 images) classes are used for testing.
In-shop has 7,986 classes with 72,712 images. We use 3,997 classes with 25,882 images as the training set. For the evaluation, we use 14,218 images of 3,985 classes as the query and 12,612 images of 3,985 classes as the gallery set.
Cars196 contains 196 classes with 16,185 185 images. The first 98 classes (8,054 images) are used for training and remaining 98 classes 18.131 images) reserved for testing.
CUB-200-2011 dataset consists of 200 classes with 11,788 images. The first 100 classes (5,864 images) split for training, the rest of 100 classes (5,924 images) are used for testing.
Data augmentation follows (Musgrave et al. 2020). During training, we resize each image so that its shorter side has length 256, then make random crop between 40 and 256, and aspect ratio between 8/1 and 4/3. We resize the resultant image to 227x227 and apply random horizontal flip with 50% probability. During evaluation, images are resized to 256 and then center cropped 227x227.
1.1.2 TRAINING SPLITS
Fair evaluation. We split datasets into disjoint training, validation and test sets according to (Musgrave et al. (020). In particular, we partition 50%/50% for training and test, and further split training data to 4 partitions where models are be trained by exploiting 1/4 as validation while training on 3/4.
Conventional evaluation. Following relatively old-fashioned conventional evaluation (Oh Song et al. 2016), we use the whole train split of the dataset for training and we use the test split for evaluation as well monitoring the training for early stopping.
Hyperparameter tuning. For the additional experiments related to the effect of hyperparameters, we split training set into 5 splits and train a single model on the 4/s of the set while using 1/s for the validation.
1.1.3 EVALUATION METRICS
We consider precision at 1 (P@ 1) and mean average precision (MAP@R) at R where R is defined for each query and is the total number of true references the query. Among those, MAP@@ R performance metric is shown to better reflect the geometry of the embedding space and to be less
A query is image for which similar images to he retrieved, and the references are the images in the searchable datahase.
14

---

Under review as a conference paper at ICLR 2023
noisy the evaluation metric (Musgrave et al. 2020). Thus, MAP@R to monitor training in our experiments except for conventional evaluation setting where we monitor P@ 1.
P@I: Find the nearest reference to the query. The score for that query is 1 if the reference is of the same class, otherwise. Average over all queries gives P@ metric.
P@ R: For query, i, find R3 nearest references the query and let r; be the number of true references in those R--neighbourhood. The score for that query P@R; = ri/n,. Average over all queries gives P@R metric, i.e., P@R
1 Σ P@R, R;, where n is the number of queries.
72 sEfn]
MAP@R: For query,
define MAP@R; = 1 Σ P(i), where P(i) P@R; if tth re-
SEIR.]
trieval correct or 0 otherwise. Average over all queries gives MAP@R metric, i.e., MAP@R = Σ MAP@R, (e. where is the number of queries. iE/n]
1.1.4 TRAINING PROCEDURE
Fair evaluation. We use Adam Kingma & Ba 2014) optimizer with constant 10-5 learning rate. 10-4 weight decay, and default moment parameters. 3 -9 and =2= We use batch size of 32 (4 samples per class). We evaluate validation MAP@R for every 100 steps of training in CUB and Cars196, for 1000 steps in SOP and In-shop. We stop training if no improvement is observed for 15 steps in CUB and 19rs and 10 steps in SOP and In-shop. We recover the parameters with the best validation performance. Following (Musgrave et al. 2020), we train 4 models for each 3/4 partition of the train set. Each model is trained 3 times. Hence, we have 3 81 many realizations of 4-model collections. We present the average performance well as the standard deviation (std) of such 81 models' evaluations.
Conventional evaluation. We Adam (Kingma & Ba 2014) optimizer with default moment parameters, =1= and 32= Following recent works (Kim et al. 2020), we use reduce on plateau learning rate scheduler with patience 4. The initial learning rate is 10"d for CUB, and 10-4 for Cars, SOP and In-shop. We use 10-4 weight decay for BNInception backbone and 4 10 wight decay for ResNet50 backbone. We use batch size of 128 (4 samples per class) for BNInception backbone and 112 (4 samples per class) for ResNet backbone (following (Roth et al. 2020).. We evaluate validation P@ I for every 25 steps of training CUB and 196s196, for 250 steps SOP and In-shop. We stop training if no improvement is observed for 15 steps in CUB and Cars 196, and 10 steps in SOP and In-shop. We recover the parameters with the best validation performance. We repeat each experiment 3 times and report the best result. For the evaluations on LIBC framework (Seidenschwarz et al.) 2021I, we follow their experimental setting.
Hyperparameter tuning. We use Adam (Kingma & Ba 2014) optimizer with constant 10-5 learning rate, 10-4 weight decay, and default moment parameters, 31= and 399 We use batch size of 32 (4 samples per class). We evaluate validation MAP@R for every 100 steps of training in CUB and Cars196, for 1000 steps in SOP and In-shop. We stop training if no improvement is observed for 10 steps in CUB and Cars196, and 7 steps in SOP and In-shop. We recover the parameters with the best validation performance. We train a single model on the 4/s of the training set while using l/s for the validation. We repeat each experiment times and report the averaged result.
1.1.5 EMBEDDING VECTORS
Fair evaluation. Embedding dimension is fixed to 128. During training and evaluation, the em- bedding vectors are (2 normalized. We follow the evaluation method proposed in (Musgrave et al. 2020) and produce two results: i) Average performance (128 dimensional) of 4-fold models and ii Ensemble performance (concatenated 512 dimensional) of 4-fold models where the embedding vector is obtained by concatenated 128D vectors of the individual models before retrieval.
Conventional evaluation. Embedding dimension is 512 in BNInception and ResNet50 experiments for both XBM and LIBC.
Hyperparameter tuning. Embedding dimension is fixed 128.
15

---

Under review as a conference paper at ICLR 2023
1.1.6 BASELINES WITH GSP
We evaluate our method with C1+XBM+GSP. Cross-batch memory (XBM) (Wagg et al., 2020) with original Contrastive loss (CI) (Hadsell et al. 2006), C2+GSP: Contrastive loss with positive margin (Wu et al., 2017). MS+GSP sulti-similarty (MS) loss (Wang et al. 2019), Triplet+GSP.
Triplet loss (Schroff et al. 2011)), PNCA+GSP: ProxyNCA++ loss (Tch et al. 2020 PAnchor+GSP. Proxy Aoxhhr loss (Kim et al. 2020).
1.1.7 HYPERPARAMETERS
For the hyperparameter selection, we exploit the recent work (Musgrave al. 2020) that has performed parameter search via Bayesian optimization on variety of losses. We further experiment the suggested parameters from the original papers and official implementations. We pick the best performing parameters. We perform no further parameter tuning for the baseline methods' parameters when applied with our method to purely examine the effectiveness of our method.
C1: We adopted XBM's official implementation for fair comparison. We use 0.5 margin for all datasets.
C2: C2 has two parameters, (m= m ") positive margin, m and negative margin. We set (m+ m to (0, 0.3841), (0.2652, 0.5409), (0.2858, 0.5130), (0. (0.2858, 0.5130) for CUB, Cars196, In-shop and SOP, respectively.
Triplet: We set its margin to 0.0961, 0. 1900, 0.0451, 0.0451 for CUB, Cars196, In-shop and SOP, respectively.
MS: MS has three parameters (ax, A). We set (a, A) to (2, 40, 0.5), (14.35, 75.83, 0.66). (8.49, 0.41), (2, 40, 0.5) for CUB, Cars196, In-shop and SOP, respectively.
a)) to (0.1,32) for all datasets. We use sample per class in batch setting (i.e., 32 classes with I samples per batch), perform I epoch warm-up training of the embedding layer, and we apply learning rate multiplier of 100 for the proxies during training. For SOP dataset, we use 10-6 learning rate.
ProxyAnchor: We set its two paremeters ())
We set its temperature parameter to 0.11 for all datasets. We use sample per class in batch setting (i.e., 32 classes with samples per batch), we perform epoch warm-up training of the embedding layer, and we apply learning rate multiplier of 100 for the proxies during training.
ProxyNCA++:
XBM: We evaluate XBM with C1 since in the original paper, contrastive loss is reported be the best performing baseline with XBM. We set the memory size XBM according to the dataset. For CUB and Cars 196, we use memory size of 25 batches. For In-shop, we use 400 batches and for SOP we use 1400 batches. We perform IK steps of training with the baseline loss prior to integrate XBM loss in order ensure XBM's slow drift assumption.
GSP: For the hyperparameters of our method, we perform parameters search, details of which are provided in $ [.6] We use 64-many prototypes CUB and Cars, and 128-many prototypes in SOP and In-shop. We use ==0.5 for proxy-based losses and ==5.0 for non-proxy losses. For the rest, we set 0=30.3, c=0.05, and we iterate until 1000 Proposition 4.1. For zero-shot prediction loss coefficient (i.e., (DL + XLzs), we set 0=0.1.
1.2 FAIR EVALUATION
We compare our method (GSP) against direct application of GAP with DML methods in datasets. We also compare our method with generalized mean pooling (GeMean) (Radenović et al. (018)) and DeLF (Noh et al. 2017). based on the results of evaluation 13 additional pooling alternatives on Ciffar Collage and CUB datasets (Tab. 4). We also evaluate max pooling (GMP) and its combination with GAP as we typically observe GAP+GMP in the recent works (Venkataramanan et al. 2022 Teh let al. 2020 Kim et al. 2020: Wang et al. 2020). We also apply method with GMP (GMP+GSP) and with GeMean (GeMean+GSPP) to show that per channel selection is orthogonal to our approach and thus, GSP can marginally improve those methods well.
We observe consistent improvements upon direct application of GAP in all datasets. Predominantly, achieve more improvement in MAP@ R metric than R @ (P@) 1) which is shown to be noisy
16

---

Under review as a conference paper at ICLR 2023
Table 2: Comparison with the existing methods for the retrieval task on SOP, In-shop, CUB, Cars datasets. Experimental setting follows $ [.J]Fair Fair evaluation. denotes I std margin. Red: the best, Blue: the second best, Bold: the loss term specific best.
Dumset
SOP
In-hop
CUB
C6
Dim
312D
125D
312D
128D
112D
128D
512D
128D
Method]
P81
MAP@R Pêi MAP@R
P@1 MAPER P&1 MAP@R
P@
MAPER P&1
MAP@R
Pê
MAPER P&1 MAPGR
Cl+
GAP
00:23
40.00
65:35
96,00
90.11
54.02
75,53
46.62
63.32
23.49
56.81
19:17
75,01
22.67
05.61
16,06
XBM+GAP
20.35
150.07
39..6
20.31
30.11
30..1
20.31
30.41
30.03
20.33
30.33
20.17
XBM+GSP
50.00
74.84
46.09
88.
58.55
00.00
26,
58.89
20,60
83.31
88
73.04
$46.1
-25.1
C2+
GAP
74.20
45.85
10.54
41.20
66,3
63.43
55.35
67.3
25.
58.59
20.58
60.95
21.08
00.3
05.55
17.00
14.35
-0.17
14111
-0.33
DeLP
46,54
45.50
55.00
05.00
27. 00.06
81.50
10.01
GeMean
74.02
46.00
71.53
43:12
96:2
50.12
63.53
55.70
68.3 70
12.12
60:97
21 20:00
82,43
25:07
70-25
17.41
GeMean+GSP
05
2011
231
2:11
24
22:31
2011
22:21
211
20
GMP
74.00
46.13
69.88
41.31
96:38
88.01
54.89
08.18
26
58.99
30.6
81.83
25.11 11
05:05
17.09
GMP+GAP
70.50
42.35
66
69.11
29
59:24
20.65
83.14
00,
17.99
20611
20.33
GMP+GSP
17.12
71:18
+1.1
+4.35
+6.35
89.54
+3.00
20:30
+0.88
15:38
MS+
GAP
7281
44.19
60.00
40.54
87 10
58.79
54.85
05.
24.05
57.57
20115
69.73
27. 01
72.54
18.09
GSP
44
40 80.87
65 26
85
65. 50
25.
57,99
20.34
28
19.91
20.12
20.17
20.10
Triplet-
GAP
16
15:14
GSP
75.59 59
47.35
70.05
41.08
86
60
55.54 82.74
66 09
24.80
57.13
19:35
78.03
23.42
65.81
16.14
13.10
19
GSP
75.68
20.38
47.74
12.50
110
54.08
65.
25. 19
58.20
19.20
12
DeLF
75.29
47.44
72/06
57.
54.13
25:31
57.95
20.01
71.10
18.81
12:75
2011
220/111
22
2013
20.30
22-21
GGS
76,89
45.17
72.91
44.61
87.0
87.84
58.13
85.08
55.25
67 39
26.3
50.39
21.81
27 96
75.95
19.74
GMP
46.33
79.290
42.24
26.79
50.99
65:74
97.01
20/33
19.9
$0.00
GMP4GAP
54.05
25.
89.00
20.18
20.11
20.25
10.30
$3.36
20.32
GMP+GSP
75.41
47.50 57.50
72.30
40.75
87.8 43
57.65
85,10
54.79
66.14
25 85
58.11 12
20.06
$5.46
27. 12
72.04
19:38
PAnchor+
GAP
73.50
59.0
19.15
20.00
$90.00
20.3
20141
26.31
GSP
49.00
45.07
88.9
85.07
80
86.
+9.28
+5.17
+5.1
+1..28
-0.31
+8..7
+3.29
+5.23
measure for DMI. evaluation (Musgrave et al., 2020). On the average, we consistently improve the baselines x1 % points in MAP@R. R. Our improvement margins are superior to ones of attention based DeLF pooling. We improve state-of-the-arr (SOTA) XBM method up to 2% points, which is good evidence that application of GSP is not limited to loss terms but can be combined with different DML approaches. We also consistently improve GMP and GeMean pooling methods in all datasets, yet another evidence that our method can be combined with max pooling based methods.
1.3 CONVENTIONAL EVALUATION
We additionally follow the relatively old-fashioned conventional procedure (Oh Song et al., 2016) for the evaluation of our method. We use BN-Inception (Toffe & Szegedy 2015) and ResNet50 (He et al. 2016) architectures as the backbones. We obtain 512D (BN-Inception and ResNet50) embeddings through linear transformation after global pooling layer. Aligned with the recent approaches (Venkataramanan al. 2022 Teh et al. 2020 Kim al. 2020 Wang et al. 2020), we use global max pooling as well as global average pooling. The rest of the settings are disclosed in 1.1
17

---

Under review as a conference paper at ICLR 2023
Table 3: Comparison with the existing methods for the retrieval task in conventional experimental settings with BN-Inception and ResNet50 backbones where superseripts denote embedding size. Red: the best. Blue: the second best. Bold: previous SOTA. RRssutss obtained from (Seidenschwarz et al.
1021
(a)
(b)
Backbone +
BN-Inception-512
Backhone
ResNet50
Dataset -
CUB
Cars196
SOP
In-shop
Dataset
CUB
Cars196
SOP
In-shop
Method
R@1
R@1
R@1
R@1
Method
R@1
R@1
R@1
R@1
C1+XBM
65.80
82.00
79.50
89.90
CI-XBM 126
poom
80.60
91.30
ProxyAncha
of
68.40
86.10
79.10
91.50
ProxyAnchor
69.70
87.70
80.00
92.10'
Kin 2007 2000
DiVA
2020
66.80
84.10
78.10
69.20
87.60
79.60
Milbich stal 2007
ProxyNCA++
66.30
85,40
80.20
88.60
Fewer
66.60
85.50
78.00
Ii all 20001
Zhu dl 20137
Margin+S2SDD
69.00
89.50
81.20
Margin+S2M
68.50
87.30
79.30
12th et 1121 H22
Ruch clal. 2021
LIBC
70.30
88.10
81.40
92.80
CI+XBM
64.32
77.63
79.29
90.16
Selkin22ear.cc dial p00
MS+Metrix M
71.40
89.60
81.00
92.20
(20.59)
(21.87)
(32.59)
(61.319
C1+XBM-GSP
64..99
79.07
79.
90.92
PAnclox+ DIML
66.46
86.13
79.35 79.22 (4)
(22:51)
(52.70)
[63.20
Claa of
(25.58)
(29 113
LIBC+GSP22
70.70
81.65
93.30
C1+XBMO12
66.68
82.83
$1.44
91.56
(25.38)
(25.34)
(64.00)
C1+XBM-GSP912
66.63
$2.60 60
91.75
We evaluate our method with XBM. We provide R@1 results in Tab. 3 for the comparison with SOTA. our evaluations, we also provide MAP@R scores in parenthesis under R@ scores. We also provide baseline XBM evaluation in our framework. The results are mostly consistent with the ones reported in the original paper [Wang et al. 2020) except for CUB and Cars datasets. In XBM (Wang et al., 2020), the authors use proxy-based trainable memory for CUB and Cars datasets. On the other hand, we use the official implementation provided by the authors, which does not include such proxy- based extensions.
We observe that our method improves XBM and XBM+GSP reaches SOTA performance in large scale datasets. With that being said, the improvement margins less substantial than the ones in fair evaluation. Such a result is expected since training is terminated by strly-stopping which is a common practice regularize the generalization of training (Dong al. 2020 Lei al.. (02I). In conventional evaluation, early-stopping is achieved by monitoring the test data performance, enabling good generalization to test data. Therefore, observing less improvement in generalization with GSP is something we expect owing to generalization boost that test data based early- carly-stopping already provides.
We also observe that in a few cases, the R R@| performance of GSP is slightly worse than the baseline. Nevertheless, once we compare the MAP@R R performances, GSP consistently brings improvement with no exception. We should recapitulate that R @ is a myopic metric to assess the quality of the embedding space geometry (Musgrave et al. 2020) and hence, pushing R@ does not necessarily reflect the true order of the improvements that the methods bring.
As we observe from MAP@R R comparisons in Table the methods sharing similar R@ 1 (i.e., P@ performances can differ in MAP@R performance relatively more significantly. In that manner, firmly believe that comparing MAP@R R performances instead of R@ technically sounds more in showing the improvements of method.
Finally, we also apply our method with LIBC (Seidenschwarz et al., 2021 to further show wide applicability of method. We use the official implementation of LIBC and follow their default experimental settings. The evaluations on 4 benchmarks show that GSP improve LIBC by ss 0.5pp @
18

---

Under review as a conference paper at ICLR 2023
Table 4: Evaluation feature pooling methods on Cifar Collage and CUB datasets with Contrastive and ProxyNCA++ losses for DML task. Red: the best, Blue: the second best, Bold: the third best.
128D MAP@R
+
Cifar Collage
CUB
Method. Loss ->
Contrastive
ProxyNCA++
Contrastive
ProxyNCA++
CBAM (Woo al. 2018
7.87
10.99
18.45
18.21
CroW (alanntdiı ct al.
2016
10.09
11.48
20.88
20.42
DeLF Noh ct al.
201
11.44
24.83
21.42
20.51
GeMax Murray cl a..
2016
7.04
7.83
18.85
17.83
GeMean Radenović
(2018)
10.97
10.60
21.50
20.71
GSoP (Gao et al.
11.15
17.73
20.52
15.72
ОТР Mialon et
2021
7.02
11.55
15.19 20.88
13.57 20.48
8x16 G-xx 128
SOLAR (Ng et al.
2020
17.30
20.36
19.89
20.14
T-SMK (Tolas et al.
2020)
9.21
13.15
21.01
20.23
VLAD Arandjeeoviic al.
2016
21.73
19.68
15.19 16.67
13.08 16.53
8x 64x 128
WELDON (Durand al. 2016
13.81
20.38
20.67
20.31
GAP GMP
8.09
10.68
20.58
20.63
9.53
11.25
20.66
20.33
GMP+GAP GSP
10.01
11.85
20.88
20.68
22.68
27.61
21.52
20.75
1.4 EVALUATION OF OTHER POOLING ALTERNATIVES
We evaluate 13 additional pooling alternatives on Ciffar Collage and CUB datasets with contrastive (C2) and Proxy-CCA+ (PNCA) losses. We pick contrastive since it is one of the best performing sample-base based loss. We pick Proxy-NCA++ since most of the pooling methods are tailored for landmark based image retrieval and use classification loss akin to Proxy-NCA++. We particularly consider Cifar Collage dataset since the images of different classes share considerable amount of semantic entities, enabling us to assess the methods with respect to their ability to discard the nuisance information.
In addition to our method (GSP) and global average pooling (GAP), we consider: i) global max pooling (GMP).
iz) GAP+GMP (Kim et al. 2020 tii CBAM (Woo et al. 2018 iv) CroW
(Kalantidis et al.
2016), v) DeLF (Noh al. 2017), vi) generalized max pooling (GeMax) (Murray
et al. 2016), utz) generalized mean pooling (GeMean) Radenović et al. 2018 viiz) GSoP (Gao
et al. 2019 ix) optimal transport based aggregation (OTP) (Mialon et al. 22222222
1221..0 x) SOLAR (Ng 2020). xi) trainable SMK (T-SMK) (Tolias et al. 2020). I2T NerVLAD (Arandjelovic et al.
et al. 2016), and miti) WELDON (Durand al. 2016). Among those, OTP and VLAD ensemble based
methods and typically necessitate large embedding dimensions. Thus, we both experimented their
128 dimensional version -68x16) (8 prototypes of 16 dimensional vectors) and 8192 dimensional version -(64x128) (64 prototypes of 128 dimensional vectors).
For CUB dataset, the experimental setting follows performance of the m-model average at 128 dimensional embeddings each. For Cifar Collage dataset, the experimental setting follows 2.2 and we report MAP@R performance. We provide the results in Tab.
Fair evaluation and report MAP@R R
Evaluations show that our method superior to other pooling alternatives including prototype based VLAD and ОTP. Predominantly, for 128 dimensional embeddings, our method outperforms prototype based methods by large margin. CUB dataset, the pooling methods either are inferior to or perform on par with GAP. The performance improvements of the superior methods are less than 1 1%, implying that our improvements in the order of 1-2% reported in Table is substantial. On the other hand, the methods that mask the feature map outperform GAP by large margin in Cifar Collage dataset. That being said, our method outperforms all the methods except for Contrastive+ VLAD by large margin in Cifar Collage dataset, yet another evidence for better feature selection mechanism of method.
19

---

Under review as a conference paper at ICLR 2023
For instance in CUB dataset, DeLF and GeMean par with method which has slightly better performance. Yet, our method outperforms both methods by large margin in Cifar Collage dataset.
Comparing to CroW, T-SMK and CBAM, our method outperforms those methods by large margin. Those methods are the built on feature magnitude based saliency, assuming that the backbone functions must be able zero-out nuisance information. Yet, such requirement is restrictive for the parameter space and annihilation of the non-discriminative information might not be feasible in some problems. We experimentally observe such a weakness of CroW T-SMK and CBAM in Cifar Collage dataset where the nuisance information cannot be zeroed-out by the backbone. Our formulation do not have such restrictive assumption and thus substantially superior those methods.
Similarly, attention-base based weighting methods. DeLF and GSoP, do not have explicit control on feature selection behavior and might result in poor models when jointly trained with the feature extractor (Noh et al.) 2017). which we also observe in Cifar Collage experiments. On the contrary, we have explicit control on the pooling behavior with parameter and the behavior of our method is stable and consistent across datasets and with different loss functions.
Moreover, attention-based methods DeLF, GSoP, and SOLAR typically introduce several con- volution layers to compute the feature weights. We only introduce an m-kernel 1x1 convolu- tion layer (i.e., m-many trainable prototypes) and obtain better results. We should note that pooling operation is as simple as performing a convolution (i.e., distance computation) and al- ternating normalization of vector and scalar. That being said, a zero-shot regularization loss into our problem naturally by using the prototype assignment weights. We can well incorporate such loss in DeLF which has 1xl convolution compute prototype similarities. However, we first need a mecha- nism aggregate the per prototype similarities (e.g. sum and normalization). Indeed, normalizing the similarities thannel-wise and spatially summing them correspond to solving our problem with /4 1.
able to incorporate
effect of in CUB with C2
batch size:
baseline
closed-form grad.
Other pooling methods, i.e., GAP, GMP, GAP+GMP. GeMax, GeMean, WELDON, VLAD, ОTР, do not build on discriminative feature selection. Thus, our method substantially outperforms those.
time per batch (ms)
auto-diff.
1.5 COMPUTATIONAL ANALYSIS
120
112b/
Forward and backward computation of proposed GSP method can be implemented using only matrix-vector products. Moreover, having closed-form matrix- inversion-free expression for the loss gradient enables memory efficient back propagation since the output of each iteration must be stored otherwise.
10
25
50
100
number of Iterations
batch 128
373
baseline
closed-form grad.
diff.
time per batch (ms)
We perform l iterations to obtain the pooling weights and at each iteration, only perform matrix-vector products, In this sense, the back propagation can be achieved using antomatii-aiiffreettaain One problem with automatic differentiation is that the computation load increases with increasing On the other hand.
302
with closed-form gradient expression, we do not have such issue and in fact we have constant back propagation complexity. Granted that the closed-form expression demands exact solution of the problem (i.e., k + oo), forward computation puts little computation overhead and is memory efficient since we discard the intermediate outputs. Moreover, our initial empirical study show that our problems typically converges for & > 50 and we observe similar performances with & 25.
278.67
25
50
100
number of Iterations
Figure 7: Comparing closed-form gra- dient with automatic differentiation through analyzing the effect of k com- putation CUB dataset with C2 loss. Shaded regions represent Fstd.
20

---

Under review as a conference paper at ICLR 2023
The choice of & is indeed problem dependent (f.e., size of the feature map and number of prototypes). Thus, it is important to see the effect of & on computation load. We analyze the effect & with automatic differentiation and with our closed-form gradient expression. We provide the related plots in Fig. 7 We observe that with closed-form gradients, our method puts little computation overhead and increasing & has marginal effect. On the contrary, with automatic differentiation, the computational complexity substantially increases.
1.6 HYPERPARAMETER SELECTION
prony
29.35
20
besslinacCa
presttiing lo)
ane number prototupos
21.6
mumber of protoryoos
PHCA-05P
PHCC+GSP
204
522
200
120
512
Fumber protigess I11
number of procaypes Dvo
(a)
(b)
Figure 8: Parameter search for number of prototoypes and e: entropy smoothing coefficient. We fix = 0.3 and À = 0.1. (a) Searching m space in CUB dataset. (b) Effect of m once we fix E 5 for Contrastive (C2) and 0.5 for Proxy NCA+ (PNCA).
We first perform screening experiment to the effect of the parameters. We design a 2-level fractional factorial (i.e., subset of the all possible combinations) experiment. We provide the results in Tab. [ In our analysis, we find that lower the better for À and μ.. Thus, we set μ = 0.3 and = 0.1. is observed to have the most effect and num- ber of prototypes, m, seems to have no significant effect.
Table Initial 2-1evel fractional facto- rial screening experiments for hyper- parameter tuning (conducted in CUB).
Setting
MAP@R
X
C2
PNCA
Nevertheless, we jointly search for m and E. To this end,
16
0.3
0.5
0.1
40.63
40.59
we perform grid search in CUB dataset with Contrastive
16
0.7
0.5
0.5
40.41
40.34
(C2) and Proxy NCA++ (PNCA) losses. We provide the
128
0.5
0.5
40.22
40.35
results in Fig. (a). We see that both losses have their best performance when works better for C2 while ε = 0.5 works better for PNCA. We additionally perform a small experiment
128
0.7
0.5
0.1
40.07
40.85
64. On the other hand, = 5.0
16
0.3
20
0.5
36.11
40.51
16
0.7
20
0.1
39.11
39,88
128
20
0.1
39.61
39.12
whether
128
0.7
20
0.5
35.36
39.92
E = 0.5 the case for Proxy Anchor loss as well and ob- serve that result of m-E search, we set = 5.0 for non-proxy based losses and = 0.5 for proxy-based losses.
0.5 is better choice over 5.0. As the
Baseline
39.77
39.90
Fixing = 0.3, x 0.1, g 0.5( or 5.0), we further experiment the effect of number of prototypes, m, in large datasets (i.e., SOP and In-shop). We provide the corresponding performance plots in Fig. (b). Supporting our initial analysis, m seemingly does not have a significant effect once is not small (e.g. m 64). We observe that any choice of m > 64 provides performance improvement. With that being said, increasing m does not bring substantial improvement over relatively smaller values. Considering the results of the experiment, we set m 128 for SOP and In-shop datasets since both C2 and PNCA losses perform slightly better with m 128 than the other choices of m.
DETAILS OF THE OTHER EMPIRICAL WORK
2.1 SYNTHETIC STUDY
We design synthetic empirical study evaluate GSP in fully controlled manner. We consider 16-class problem such that classes are defined over trainable tokens. In this setting, tokens correspond to semantic entities but choose give specific working to emphasize that they are trained as
21

---

Under review as a conference paper at ICLR 2023
part of the learning. Each class is defined with 4 distinct tokens and there are also 4 background tokens shared by all classes. For example, a "car" class would have tokens like "rire" and "window" as well as background tokens of "tree" and "road".
We sample class representations from both class specific and background tokens according to mixing ratio j (0.55, 0.1). We sample total of 50 tokens and such 50-many feature collection will correspond to a training sample (i.e., we are mimicking CNN's output with trainable
tokens). For instance, given class tokens for class-c, ple) {v1 1/2 23
w and shared tokens, (h) = { .12 2 v/ v1");; we first sample = 0.4 and then sample 20 tokens from vle) with replacement, and 30 tokens from ul)), forming a feature collection for a class-c, ie., ((o) {u]o) ( (1 (c)
(c) (o)
(b) (b] (b) 11 ..... We then obtain global representations ..... 3
using GAP and GSP.
We do not apply (2 normalization the global representations. We also constrain the range of the token vectors to be in between [-0.3, 0.3] to bound the magnitude the learned vectors. We use default Adam optimizer with 10 learning rate and perform early stopping with 30 epoch patience by monitoring MAP@R. In each batch, we use samples from 16 classes.
2.2 CIFAR COLLAGE
shared classes
We consider the 20 super-classes of Cifar100 dataset (Krizhevsky & Hinton, 2009) where each has 5 sub-classes. For each super-class, we split the sub-classes for train (2), validation (I), and test (2). We consider 4 super-classes as the shared classes and compose 4x4-stitched collage images for the rest 16 classes. In par- ticular, we sample an image from a class and then sample images from shared classes. We illustrate a sample formation process in Fig.
class-i
class-j
a random sample
random sample
3 random images
sample for class-i
sample for class-j
We should note that the classes exploited in training, validation and test are disjoint. For instance, if tree class used as a shared class in training, then that tree class does not ex- ist in validation or test set as shared feature. Namely, in our problem setting, both the back- ground and the foreground classes are disjoint across training, validation and test sets. Such setting useful to analyze zero-shot transfer capability of our method.
Figure 9: Illustration of sample generation for Cifar Collage dataset.
We use ResNet20 (i.e., stages, blocks) backbone pretrained on Cifar100 classification task. We use (2 normalization on global representations. We use default Adam optimizer with initial 0.001 learning rate. We use reduce on plateau with 0.5 decay factor and 5 epochs patience. For GSP, we set 128, / 0.2, 10, 0.5. We use 4 samples from 16 classes in batch.
2.3 EVALUATION OF ZERO-SHIOT PREDICTION Loss
We train on Cifar10 Krizhevsky & Hinton 2009) dataset with 8 prototypes using ProxyNCA++ Teh et al. (2200) (PNCA) loss with and without Lzs. We then use test set to compute pseudo-attribnn histograms for each class. Namely, we aggregate the marginal transport plans of each sample in class to obtain the histogram. Similarly, for each class, we compute the mean embedding vector (i.e. we average embedding vectors of the samples of a class). Our aim is to linear predictor to map attribute vectors to the mean embeddings.
To quantify the zero-shot prediction performance, we randomly split the classes into half and apply cross-batch zero-shot prediction. Specifically, we fit linear predictor using 5 classes and then use that transformation to map the other 5 classes to their mean embeddings. We then compute pairwise distance between the predicted means and the true means. We then evaluate the nearest neighbour classification performance. We use both 22 distance and cosine distance while computing the pairwise distances. We repeat the experiment 1000 times with different class splits.
22

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 PROOF FOR CLAIM 4.1
Proof. p' is obtained as the solution of the following optimal transport problem:
P aarg Ein Gij Tej-
2 =/y
Given solutions (p", x)), ), for ,=1, from the 3rd constraint, we have Mi =1. Then, using the 2nd constraint, we write:
Σ,
where jE[n] for n-many convolutional features. Hence, we have Σ, p² = 0 which implies p p²=0 =0
owing to non-negativity constraint. Finally, pooling weights becomes p: 1/- 1 1/n.
=1
A.2 PROOF FOR PROPOSITION 4.1
Before starting our proof, we first derive an iterative approach for the solution of (P2). We then prove that the iterations in Proposition 4.1 can be used obtain the solution.
We can write (P2) equivalently as:
((e) (e)) arg min Σij Ti + [ Tay log Kig + Ej logp;)
p.720
Σ, ku. =p
+ E, Opj - -is Tij - Ej + Eig Co0 + E, c-50
Rearranging the terms we get:
(()) m
-arg min Eij Téj log
Way e .aning
E3 Pj log
EEpjt 0- + E, e-c0
6
p +.,,,,=/4.. T
Pg
which is generalized Kullback-Leibler divergence (KID) between (p, m) and (exp (-e0), exp (-Ec)). Hence, we reformulate the problem as a KL.D prjoection onto a convex set, which can be solved by
Bregman Projections (i.e., alternating projections onto constraint sets) (Bregman 1967| Bauschke &
Lewis 2000). Defining C == {(p, π) O5 + Eij Fij- = 1/n Vj} and C2 = [p. 2ig Tis p). and omutting constants, we can write the problem as:
p(e), (()) arg min arg
Eng E. (log Tij. 1) + E, P, (log Lio 1)
(P2')
p.730 (e,
Given, (p(k) 7())), iteration k, KLD projection onto Ci. iee. ((l+t)),=((t))
PKL (kL((( ). reads:
(k+1) 1/m(p/ (K) + Σ, T. P. (++1) = 1/n(p) (K) + E, T - (1)
(k) (/0))=111 (k) (k)
Pj
where the results follow from method of Lagrange multipliers. Similarly, for (p(() ((i)), we have:
(k++)) = (R) π (k+1)
Eg
T
(c)
Given initialization, (p!0), = (1,, exp(-ec)), we obtain the pairs (p(1), (k1) for k: = 1,
as:
((++1) 1k/ + (b) )11 O (t) ((+1) pM
(A.1)
where A = Diag (/((p()) + (((T))))))))
23

---

Under review as a conference paper at ICLR 2023
Proof. We will prove by induction. From Proposition 4.1, we have
p(k+1) 1/n (1 + ((8) (000--e)_mmm?? ((k+1)
ja (I exp( exp(-ec))(at1))-
and T(k) ((h) exp(-Ec) (iag(p))))((())) It is easy to show that the expressions hold for the pair
(pl)), (p(),<(()) =(1)). Now, assuming that the expressions also holds for arbitrary (olk), x(()). We have
(K++1) 1/m(p!4) + +x (H)) o p())
Replacing x(²) {(K) exp( ( (Ecc) (iag(p get:
p((+1)) = 1/((p")) + Diag(p!")) (((()) exp(-Ec) 1mm) p(a))
where plk) terms cancel out, resulting in:
(k++1) 1/a(1 + {(4) 1xp(-
Similarly, we express से as:
म = (k)) exp(-ec) Diag (/((p(() + Diaglp"))) expl- =cm )m)
again (k') terms cancel out, resulting in:
= (k)) (ap(-ee)
((k) (m) =(()) (kk) Diag
Hence, N (k'+ +1) becomes:
π (1++1) u(1
(K)) (x') exp((xe) Dragg( Diag(p (1+)) 11)
(oW++1))
पंरन (11 exp(-Ec)p +1) ((0) exp(( Diag
+1)+11
(K' (1) exp(=e))Din(( Ding(p(k+)))
meaning that the expressions also hold for the pair (p (1N++))).
A.3 PROOF FOR PROPOSITION 4.2
Proof. We start our proof by expressing (P2') in a compact form as:
arg min e. + a (log (mm+1)m)
230
Ar=
where æ denotes the vector formed by stacking p and the row vectors of т,, E denotes the vector formed by stacking n-dimensional zero vector and the row vectors of c, and A and are such that Ax = b imposes transport constraints as:
Juxn
Tnx
A
b= 1-111 T
OT
1T
From Lagrangian dual, we have:
(e) exp(-e exp(-e(c+ATxx))
where X is the optimal dual Lagrangian satisfying:
Aexp(-e(++AATA')) b
Defining
Bag Bog
have;
(x -
0x* A) DiDgg(a))))
de
where I == 10 10mmneen [(men)x(((m+)))) Similarly, for the dual variable, we have:
-e 0x A)Diag(z(()) (
0
0x 86
-Diag(a (A) (AA
24

---

Under review as a conference paper at ICLR 2023
Putting back the expression for in dx
ax de
de we obtain:
dyle) - ) Diag(= (A (A))) ADiag(z!e)).
which includes (m++1) by n matrix inversion, H ADiag(z(e)) AT, We now show that H-1 can be obtained without explicit matrix inversion.
H can be expressed as:
1/1I 1/1 //--p pT
1/n
H
H is Hermitian and positive definite. (Corrolary 4.1 of (Lu & Shiou 2002), we obtain the inverse as;
Using block matrix inversion formula for such matrices
H 1 = nI k- PpT 5T
- k-1
where k = - 1- -np (c)) and X = 1 -np(1)
Given OC i.e., 0C
aptcy dx ), the rest of the proof to obtain
follows from right multiplying the
Ozic)+
de
Jacobian, i.e., 8C
(x(c) 8C and rearranging the terms.
Dr
Bc 8zie)
B OPTIMAL TRANSPORT BASED OPERATORS
In this section, we briefly discuss optimal transport based aggregation and selection operators. We provide their formulations show how our formulation differs from them.
B. FEATURE AGGREGATION
Given cost map Cjj = ||: fj ll2 which is an m (number of prototypes) by (number of features)
matrix representing the closeness of prototypes w/ and features fj. Mialon al. (202T) consider the following optimal transport problem:
arg min Taj
(P4)
720
E 1// 1,.m=11/m
and defines their aggregated feature as:
m f1 f2 ... |n]x
(B.1)
Namely, g an ensemble of m vectors each of which the weighted aggregation of (een)))))) with the weights proportional to the assignment weights to the corresponding prototype. In this ensemble representation, g. there is no feature selection mechanism and thus, all features are somehow included in the image representation.
we further sum these m vectors g to obtain single global representation, we end up with global average pooling: g
1m
/mlff f2 ... fT vm/n/fi fa ... fn]1, w.
Briefly, Mialon et al. (2021 map a set of features to another set features without discarding any and do not provide natural way aggregate the lass-discriminative subset of the features. Such representation useful for structural matching. On the contrary, our formulation effectively enables learning to select discriminative features and maps set of features single feature that is distilled from nuisance information.
B.2 TOP-k SELECTION
Given n-many scalars as x [allii and m-many scalars as y [y<bæ(() with y in an increasing family, i.e., y1 <y2 ...., Xie et al. (2020) consider the following optimal transport problem:
π
aarg min
City Tij
(P5)
=10
E
N, =P.
25

---

Under review as a conference paper at ICLR 2023
where Cij = 1b/s zy| and p is m-dimensional probability simplex, i.e., {p R Σ,p; = 1}. Then, top-k selection is formulated with the setting q = 1/nln, y = [0, and p = [ n- Similarly, sorted top-k selection is formulated with the setting y = [k] and p 1
n
m- J.
Solving the same problem in (P5), Cuturi al. (2019) formulate top-k ranking by letting q and p be any probability simplex of the proper dimension and be in an increasing family.
Such top-k formulations are suitable for selecting/ranking scalars. In our problem, the aim is to select subset of features that are closest to the prototypes which are representatives for the discriminative information. Namely, we have a problem of subset selection from set-to-set distances. we had our problem in the form of set-to-vector, then we would be able to formulate the problem using (P5). However, there no natural extension of the methods in (Xie et al., 2020) Cuturi et al.) 2019) to problem. Therefore, rigorously develop optimal transport based formulation express discriminative subset selection operation analytically in differentiable form.
With that being said, our formulation in (PI) differs from the typical optimal transport problem exploited in (P5). In optimal transport, one matches two distributions and transports all the mass from one to the other. Differently, we transport μ portion of the uniformly distributed masses to the prototypes that have no restriction on their mass distribution. In our formulation, we have portion constraint instead of target distribution constraint, and accounting for residual masses. If we do not have and set = 1, then the problem becomes a specific case of an optimal transport barycenter problem with 1 distribution.
an additional decision variable, p.
Our problem can be expressed in a compact form by absorbing P into with zero costs associated in the formulation, which indeed what we do in the proof of Proposition 4.2 (Appendix A..3 We choose to explicitly define in the problem (PI) to show its role and avoid convoluted notation. We believe its residual mass role is more understandable this way. The benefits of our formulation include that we can perform feature selection operation with matrix inversion free Jacobian and we can change the role of the prototypes as background representatives simply by using P to weight the features instead of 1/2- p in Eq. (4.1). Our specific formulation further allows us to tailor zero-shot regularization loss built on the learned prototypes within our pooling layer.
26

---

Under review as a conference paper at ICLR 2023
C IMPLEMENTATIONS WITH PSEUDO CODES
Algorithm 1 Deep Metric Learning Loss with GSP and ZSR
input: (X,Y) = (fz+), (ye // batch of image-label pairs
F+ Backbonc( X))
// CNN backbone such as BN-nnception ResNer
(Xp: Z) (GSP(f) (ff)))e))))
I/ get pooled features and attribute predictions, see Algorithm 2
LzSR + ZSR(Z, Y)
I/ compute ZSR loss, see Algorithm
LDMI. LXs, Y)
I/ DML loss such contrastive, triplet, XBM, LIBC,
++ (1-A)CDML + A229R
I/ set X=0.1
return L
Algorithm 2 GSP(f)
trainable parameters: (wikee(m)
|/ m--any prototypes
input: (filiejm)
I/ feature map. n wh (i.e., widthxheight)
Wi < 4./mmx(L.]]wwlll ViE(m].
[ t [1/max VjE[n]
Cty |00 fyll2
// cost map, [ci (13)e(ai
D. + WeightTranspon Transport(c)
// see Algorithm 3
f + 1-mp o
// re-weight features, @: element-wise multiplication
Tp - t Σ
I/ pooled feature, GSP for p=1, GeMean+GSP for p>1
iE[n]
2 t Σ
Tej ViElm]
|| pseudo-atriibute predictions, [=i)ie(m)
JE[n]
return Zp, 2
Algorithm 3 WeighTTranspor(()
hyperparameters: μ transport ratio, & entropy regularization weight, It number of iterations
forward: gets cost map, c, returns residual weights, P, and transport plan
input: c= (cagha.)))mmm
// cost map of m-many prototypes and n-many features
R < exp(-Ec). 4
// exp is element-wise
repeat & times
+ 1/n(1 + 1
1/ AT1m sum A along rows, (-)-1 is element-wise
t + (T
I| + tx Diag(p)
return t Diag(p)
backward: gets the solution (p, л) and the gradients G 8C
returns de
input: P. π, 0p" 8m
Oc ac
|/ results of forward pass and the loss gradient w.r.t them I1 ATIm: sum A along rows, @: element- multiplication
p 8p
0c
(TT @
ac T 85
+ (p 3 3C
T -
Bp
ac
Oc
no Diog( - )
- π ②
De
am
return Øc
de
27

---

Under review as a conference paper at ICLR 2023
Algorithm 4 ZZSRZ. Y)
trainable parameters: T (Ui)ie(Holnsee))
semantic embeddingg vector for each class label
input: Z (zi (ye // batch, b, of attribute prediction vectors, 2;, and their labels, yi (bi, b2) < split b into two class-disjoint halves s.t. {yi (eb, (yy = 0
Ta + |unleE for k=1,2 // label embedding matrix for batch-k, i.e., prediction targets
// attribute prediction matrix for batch-k, i.e., prediction inputs Ag + Ta(2, (xtel- 2 for k=1,2 // fit label embedding predictor for batch-k, c=0.05
ZĄ (allicby for k=1,2
Y. A2Z1, T2 AiZz
I/ use predictor for by to predict the label embeddings of by
T [2]
I/ concatenate predictions
S SoftMax(f
|/ similarity scores between predictions and label embeddings
LZSR + CrossEntropyyS,,))
return Lzsn
28


Reviews:
[{"review_id": "p-ig_sPYSv1", "reviewer": "Reviewer_ms1e", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new pooling technique named generalized sum pooling. It is claimed as a strict generalization of global average pooling. The experiments are conducted on four different datasets and achieve a SOTA performance.", "summary_of_the_review": "Based on my above point, I think this paper is novel and sound, but I do not fully check it through. \nWhile it still misses some key points which makes me confuse.", "strength_and_weaknesses": "I'm not an expert in this area, please correct me if I get it wrong. \n\nStrength:\n1. The proposed method seems to be sound, but I'm not fully checking it. The experimental results are outstanding \n2. The motivation seems to be reasonable.\n\nWeakness:\n1. While the motivation is reasonable, it still confuses me why we should redefine the learnable generalization of GAP as an optimization problem. What is the optimization objective?\n2. It seems like there are three categories of related works that are mentioned in this paper,  when selecting the baselines in your experiments, do you cover all of those three categories?\n3. Given the complicated structure of the GSP (i.e., Fig 2), how to can apply it to current existing metric learning or CV-related works?\n4. It is not clear to me, why we should bring the OT into this paper?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Ou2ntBdIh_", "reviewer": "Reviewer_twxc", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "An extension of the global average pooling (GAP), called generalized sum pooling (GSP), is presented in this paper: while GAP is a convex combination with \"flat weights,\" GSP instead learns the weights by solving an optimal transportation problem. GSP allows for the selection of features to be pooled by introducing a \"background prototype\" (rho) into the formulation. Experiments compared GSP with several existing metric learning and pooling methods. The results showed that GSP gives small improvements in several cases.", "summary_of_the_review": "The idea of using background prototypes is interesting and thorough experiments have been reported. \nThe major problems with this paper are the lack of novelty in the algorithm and the limited effectiveness of the proposed method. Also, the proposed method has many hyperparameters, which may be a drawback for its use in practical scenarios. Given that both novelty and effectiveness are important, I am leaning toward rejection for now.\n", "strength_and_weaknesses": "------------------- Strengths -------------------\n\n1. Introducing background prototype in optimal transportation problem is interesting.\n\n\n2. Thorough experiments are reported (in Supplementary Material).\n\n\n3. The paper is mostly well-written.\n\n\n4. Code is provided.\n\n------------------ Weaknesses ------------------\n\n1. Novelty\n\nAlgorithmic novelty may not be sufficient. As discussed in the paper, the idea of using optimal transportation formulation for feature pooling was proposed in Mialon et al. (2021), and the proposed problem (P1) is equivalent to a standard optimal transportation problem (by considering rho as a new pi_i, we can immediately get to the standard form). Given these, using entropy regularization, which is an efficient approach to solving the optimal transportation problem, is straightforward. (P2) is also equivalent to the standard form of the optimal transportation problem. The same is applied to the algorithm.\n\n\n2. Motivation\n\nWhile this paper focuses on metric learning, the application of pooling in general should not be limited to this. So far, I could not find a good reason for this limitation. I would expect an explanation if any.\n\n\n3. Evaluation of Rho\n\nThe point of GSP is to use rho, in other words, changing the range of i in (P1) 1<=i<=m to 1<=i<=m+1. So a before and after comparison between these two cases would be necessary.\n\n\n4. Overall Performance\n\nThe results of comparisons with various pooling methods reported in Supplementary Material (e.g., Table 2) show that the stand-alone performance of GSP is outperformed by several existing methods.\n\n\n5. Hyperparameter Setting\n\nGSP, together with zero-shot regularization, has many hyperparameters to be tuned (m, mu, epsilon, k, lambda). Since these were likely tuned for each dataset and task, it would not have been very difficult to achieve equal or better accuracy than other methods.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "1o5xNH1AidW", "reviewer": "Reviewer_MQSC", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a learnable and differentiable pooling operation to replace the global average pooling (GAP) for metric learning.   \n\nSpecifically, the pooling operation is formulated as the solution to an optimal transport problem with additional residual $\\rho$ and mass ration $\\mu$ parameters. $\\rho$ represents the per-element residual, and $\\mu$ stands for the total assignment mass.   \nThe authors derived an iterative solution for forward computation and an inversion-free computation of gradient for backward computation. The closed-form backward gradient formulation is efficient for network training.  \n\nMoreover, to improve the transferring property, cross-batch zero-shot regularization is proposed by predicting class embedding vectors from the prototype assignment vectors ($\\pi$). The sub-problem (P3) has a closed-form solution. Since real class embedding vectors are not given, a meta-learning scheme utilized a zero-shot prediction loss to tackle this. \n\nIn the experiments, toy experiments are designed to investigate the mechanism of the proposed GSP. In metric learning experiments, various pooling alternatives are compared, and GSP achieves the best performance. In addition, GSP is evaluated under different architectures and methods, verifying its effectiveness. Regarding the absolute improvements, the general boost is around or below 1% in most cases, but the improvements are consistent across all settings. \n\n  \n", "summary_of_the_review": "The overall quality and novelty of this paper are good, except for several concerns listed above. The experiments can verify the effectiveness of the proposed method.   \nI would like to see this paper accepted if the concerns and issues are solved by the authors. \n", "strength_and_weaknesses": "## Strength\n- The proposed OT optimization problem in (P1) is novel and reasonable as an alternative to GAP. The introduction of $\\rho$ and $\\mu$ are the keys to model nuisance features and dynamic re-weighting. \n- Both forward and backward solutions are given for an entropy-regularized version of (P1), i.e., (P2). Moreover, the closed-form and inversion-free formulation of the partial derivatives enables differentiable and efficient backward computation. \n- A zero-shot regularization auxiliary task is proposed to improve the transferring ability of metric learning. A simple formulation with a closed-form solution is devised, and a meta-learning procedure is proposed to enable training without ground truth label embeddings. \n- The toy experiments corroborate the design intuitions of GSP and zero-shot regularization, providing useful insights. The comparison experiments are conducted on various datasets, with 13 pooling alternatives and with various backbone/methods, verifying the general effectiveness. \n\n## Weakness\n- One major concern is the extra parameters and computation complexities induced by the proposed method. For example, extra parameters involve $\\{ \\omega, \\Upsilon \\}$ and extra computations involve Proposition 4.1 ($k=100$ forward iteration) and Proposition 4.2 (backward computation). \n- For the deriving of (4.2), important details are missing, e.g., $\\frac{\\partial \\mathcal{L}}{\\partial \\rho^{(\\epsilon)}}$. In $\\frac{\\partial \\mathcal{L}}{\\partial c}$, which kind of $\\mathcal{L}$ is used? In Proposition 4.2, the first $\\mathcal{1}_n$ should be $\\mathcal{1}_m$, and $\\gamma$ is not defined. All these make 4.2 incomplete. \n- I can infer the procedure of meta-learning in zero-shot regularization from Figure 2. However, Section 4.3 is a bit confusing, especially (4.4). It will be much better if an algorithm box is provided. \n- The author mentioned several times the similarity with the top-$k$ selection process. However, no discussion or formulation analysis is provided to show the relationship explicitly. \n- In Sec 4.2, \"this function is not smooth\" why? Is it because of the max operation in normalization? \n- There are several typos, especially in the appendix regarding Preposition 4.1. \n \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject