Supplementary Material: Hierarchical Proxy-based Loss for Deep Metric Learning

In this paper, we adopt two evaluation protocols to evaluate our method. The first evaluation protocol is adopted in many previous metric learning papers [1, 6, 9, 12] where each dataset is split into a training set (first half of classes) and a testing set (second half of classes) and best results on the testing set are reported; The second evaluation protocol was recently proposed in [10] featuring a more standardized and fairer evaluation procedure. Based on the evaluation protocol, the implementation (i.e., hyper-parameters selection) is slightly different. Note that like other methods (e.g., Proxy Anchor [15]) embeddings and proxies are L2 normalized in our method. All code is implemented using PyTorch [11]. Under the first evaluation protocol, we test our hierarchical proxy-based loss (HPL) with two backbone networks— ResNet-50 [4] and BN-Inception [5] (see main paper for results with ResNet-50 and Sec. 2.1 for results with BNInception). Following [6], we append a max pooling layer in parallel with an average pooling layer to the penultimate layer of the backbone network. The outputs from the max pooling layer and the average pooling layer are added and then passed through a embedding layer (i.e., fully-connected layer) which projects the network output to a desired embedding space. The embeddings are 512dimensional and L2 normalized before computing the loss, and during inference. Both backbone networks are pretrained on ImageNet [2] for the classification task. The batch size is set to 128 during training. Each model is trained for 30 epochs with learning rate 10−4 using the AdamW optimizer [7]. We train the baseline models with both Proxy-NCA loss and Proxy Anchor loss by following the standard hyper-parameters settings in [9] and [6], respectively. Namely, the scaling factor and margin for Proxy Anchor loss is set as 32 and 0.1, respectively. For our HPL loss, the same hyper-parameters are used across all levels of proxies. The second evaluation protocol [10] splits each dataset into a trainval set (first half of classes) and a testing set (second half of classes) as the first evaluation protocol. However, to mitigate the effect of hyper-parameter selections in different methods, the trainval set is further split into four equal-sized partitions for cross validation. Particularly, three partitions are used as the training set and the remaining partition is used as the validation set. Hyper-parameters are selected by optimizing the average validation performance (MAP@R) of four leave-one-out experiments. This hyper-parameter optimization is done by running M iterations of Bayesian Optimization (M = 50 for the CUB and Cars-196 datasets and M = 10 for the SOP dataset due to its high complexity). The hyper-parameters to be optimized in HPL include the learning rate of the proxies; the number of coarse proxies; the update frequency of the coarse proxies; and scaling factors and margins (if applicable) for each proxy level. The model parameters are trained using RMSprop with learning rate 10−6 and batch size 128. The training terminates when the validation accuracy plateaus. We refer the reader to [10] for further details of the evaluation protocol.