UDON: Universal Dynamic Online distillatioN for generic image representations
Authors: Nikolaos-Antonios Ypsilantis
Kaifeng Chen
Andre Araujo
Ondrej Chum
Venue: NeurIPS 2024 poster (2024)
Keywords: Universal Image Embeddings
Image Retrieval
Fine-grained recognition
Instance-level recognition
Primary area: machine_vision

Abstract: Universal image representations are critical in enabling real-world fine-grained and instance-level recognition applications, where objects and entities from any domain must be identified at large scale.
Despite recent advances, existing methods fail to capture important domain-specific knowledge, while also ignoring differences in data distribution across different domains.
This leads to a large performance gap between efficient universal solutions and expensive approaches utilising a collection of specialist models, one for each domain.
In this work, we make significant strides towards closing this gap, by introducing a new learning technique, dubbed UDON (Universal Dynamic Online distillatioN).
UDON employs multi-teacher distillation, where each teacher is specialized in one domain, to transfer detailed domain-specific knowledge into the student universal embedding.
UDON's distillation approach is not only effective, but also very efficient, by sharing most model parameters between the student and all teachers, where all models are jointly trained in an online manner.
UDON also comprises a sampling technique which adapts the training process to dynamically allocate batches to domains which are learned slower and require more frequent processing.
This boosts significantly the learning of complex domains which are characterised by a large number of classes and long-tail distributions.
With comprehensive experiments, we validate each component of UDON, and showcase significant improvements over the state of the art in the recent UnED benchmark.
Code: https://github.com/nikosips/UDON.

UDON: Universal Dynamic Online distillatioN for generic image representations
Nikolaos-Antonios Ypsilantis"
Kaifeng Chen?
André Araujo²
Ondřej Chum
VRG, FEE, Czech Technical University in Prague Google DeepMind
Abstract
Universal image representations are critical in enabling real-world fine-grained and instance-level recognition applications, where objects and entities from any domain must be identified at large scale. Despite recent advances, existing methods fail to capture important domain-specifie knowledge, while also ignoring differences in data distribution across different domains. This leads to a large performance gap between efficient universal solutions and expensive approaches utilising collection of specialist models, one for each domain. In this work, we make significant strides towards closing this gap, by introducing new learning technique, dubbed UDON (Universal Dynamic Online distillatioN). UDON employs multi-teacher distillation, where each teacher is specialized in one domain, to transfer detailed domain- specific knowledge into the student universal embedding. UDON's distillation approach is not only effective, but also very efficient, by sharing most model parameters between the student and all teachers, where all models are jointly trained an online manner. UDON also comprises sampling technique which adapts the training process to dynamically allocate batches to domains which are learned slower and require more frequent processing. This boosts significantly the learning of complex domains which are characterised by large number of classes and long- tail distributions. With comprehensive experiments, we validate each component of UDON, and showcase significant improvements over the state of the art in the recent UnED benchmark. Code: https: //te:///iii //github..b... com/nikosips /UDON.
Introduction
Imagine you point your cellphone at anything, and it tells you what it is, be it tangerine chicken with rice, Mki Volkswagen Rabbit Cabriolet, statue of Aquaman, Pasadena City Hall, Yorkshire Terrier. Such product is the ultimate goal of fine-graines and instance- visual recognition. The key component enabling such an application a general-purper purposs image representation, or equivalently image embedding, designed to handle imagery of varied domains at scale. Traditionally, image
embedding models have been developed for specific domains separately [29 34 17 9), such as
landmarks (32), products [41), clothes [22]. faces [40]. to name just few. However, as visual
recognition applications grow in popularity and scope [47 2 it is impractical to handle images of different object types with specialized, per-domain models. A potential solution for this problem is to leverage recent foundation models, such as CLIP 33 or DINOv2 [26]. which have been proposed to enable wide variety of multimodal applications. Even though these models possess broad visual understanding, they tend to lack detailed fine-grat -reiggaiied knowledge off-the-shelf [45), which is critical in practice. For this reason, recent efforts aim developing universal embedding solutions that can generalize to handle multiple fine -nn-ird object types with single model [4] 39) ensuring scalability in real-world scenarios.
*Corresponding author: upsilnikefel.cvtt
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Previous Work
Naive Multi-teacher Distillation
UDON Multi-teacher Distillation
Donain
Donoln
Comain N
Donalin
Donsin
Semsin
Donain
Donsln:
Conann
Univrrsol backbone
Univernel bac coona
Domain'1 bacibone
2omain bookbone
Domain N backbone
Unterrsa bax cbene
Universal
Jeiveroe Stadontt
Domen-1 Beacholl
Domain Biochh)
Domai- N (locher)
Uihheral baucenn))
Domrnn (eachil)
Domaan Goachell
DomenN Osachat
Training stages:
Training stages: N+1
Training stages:
Training of universal embedding on multiple fine aine-grained visual domains. The base- line approach of [45) (left) uses classification loss across training classes from all domains. It is prone to cancelling out contradicting cues from different domains. To overcome this issue, a naive multi-teacher distillation approach (middle) first trains one specialized teacher per domain (with classification loss) to capture domain specifies, then distils them to the universal embedding (student). Our proposed Universal Dynamic Online distillatioN UDON (right) jointly trains the specialized teacher embeddings and the universal embedding (student) with classification and, at the same time, distills the teacher embeddings to the universal embedding. Due to joint training of shared backbone, UDON scales to a large number of domains.
Figure 1:
There are two main challenges in training such universal models addressed in the paper. First, it is difficult to encode detailed knowledge about many image domains in a single model. In several cases, features that are helpful for one object type may be useless for others. One example is the importance of color: while for food recognition it is critical to discriminate red curry from green curry, when recognizing car models both red and green Toyota Corolla LE 2024 would belong the same class, This makes the training of universal models from data of different domains particularly hard, since the data may present conflicting peculiarities across domains, leading to sub-optimal learning. To address this issue, we propose novel knowledge distillation approach, where one teacher model is trained for each domain individually, and a student universal embedding model learns from the collection of teachers. This setup allows the specialized teachers to capture domain-specific knowledge, which is then transferred into the universal embedding. However, if this distillation setup deployed naively, one may incur substantial costs, separate teacher model would need to be trained for each domain. For this reason, we propose to share the backbone between all teachers and the universal student, reaching a solution that achieves high performance and incurs small additional training costs, as all models are jointly trained an online manner see Figure D
The second main challenge we highlight is that different domains may present data with vastly different distributions: e.g., while one domain may present moderate number of classes with roughly balanced number of samples, in others, the number of classes may be very large, and the distribution of samples long-tailed. This means that different domains may require different training curricula for model to properly learn their characteristics. This leads us to propose sampling technique that dynamically seleets which domains will be processed at each point in the learning process based the losses measured on the fly. With this method, demonstrate significant improvements to the performance of the most challenging domains, which are learned slower and require more frequent processing compared to other domains.
Contributions. To summarize, this work we introduce the following contributions. (1) We leverage knowledge distillation to infuse the universal embedding model with the learnings of specialist. per- domain teacher models. In our novel training setup, the model backbone for all teacher embeddings and the student universal embedding is shared, leading to an efficient method where all models are jointly trained an online manner. Our findings show that sharing the backbone facilitates the distillation process significantly, even reaching performance that surpasses the distillation from

---

separate specialist teachers. (2) To enable an appropriate training regime in a universal embedding setup, we propose to adapt the learning process dynamically, adjusting the sampling of image domains based on their losses, which appears suitable to visual knowledge spanning range of fine-grained domains. We show that this can help substantially with more complex domains that require more frequent model updates to enhance their performance. (3) We perform comprehensive experiments on the recent UnED [45] dataset, that highlight the value of each proposed component. as well as compare our technique against competitor approaches. Our complete method, named UDON (Universal Dynamic Online distillatioN), showcases state-of-the-art results that boost Recall by up to 2.3%.
2 Related Work
Knowledge distillation (KD). Initially proposed to transfer the knowledge of large and complex models to smaller and faster ones [14), standard KD trains light student to mimic the softmax outputs produced by a heavy teacher. Tailoring KD to representation learning, [27 (2 distill relations
between image representations and [9 focus on retrieval rankings. Online KD [12] lifts the need for separate two-stage training for the teacher and the student and trains them simultaneously. Multi- teacher KD
aims transfer the knowledge of multiple teachers into one student model. 24] performs multi-teacher KD for single domain visual retrieval, by aggregating the teacher relations single target that the student should mimic. Differently than [1), which proposes an online multi- teacher distillation approach that trains a different backbone for each teacher, we propose to share a common backbone between all the teachers and the student, showing improved performance while also being much more efficient. (50) combines online and multi-teacher knowledge distillation in single multi-branch network, training an ensemble of teachers on the fly. Differently from [50 24), each of the teachers in our KD approach is specialized to only a fraction of the data (a single domain), being relevant only to part of the universal embedding task. Similarly to (50), we also create an online multi-branch architecture, however the teachers are not updated by the other teachers' knowledge, as they are related to different visual domains. Additionally, our teachers transfer relational knowledge to the student, since our focus is on learning image embeddings, in contrast to [50] which only
20 13
focuses on classification.
Universal representation learning. Learning a representation that generalizes and can be reused efficiently across visual domains is a long- standing goal in computer vision. introduces domain- specific normalization to make classification networks generalize to multiple visual domains, while 335 introduces adapter modules to improve the accuracy of domain-specifie representations. (23 proposes multi-task vision and language universal representation trained on 12 different datasets. However, this body of work assumes knowledge of the test time domain, which does not hold in our setup. Recent large visual foundational models [26 3 that are trained on large amounts of data with diverse objectives show great zero-shot performance on number of downstream visual tasks, making them great candidates for universal embedding applications. However, [45] shows that these models, even though gencralizing to many diverse domains, cannot effectively handle instance-level and fine-graines domains (which are the focus of this work) without further fine-tuning. (37] shows that appending an MLP projector between the objective and the representation used for the downstream tasks improves the generalisability of the representation, inspiring a multi-domain variant we as a baseline in this work. In B. a multi-domain representation for fine-gratd -eeegga retrieval is learned, which utilizes no labels for training. Differently from it, we focus on the supervised task setup of [45] which constitutes a much larger-scale problem that additionally includes instance-level domains. 20 introduces distillation as a way to learn universal representations, while [10 tailors multi-teacher
distillation to universal embedding learning. Differently from 200 [0], we do not use task-specific
backbones that are costly to scale across a large number of domains. While [10] tackles a universal embedding setup, the effectiveness of their method is only assessed on a small dataset, where three small domains a time are distilled into universal representation. In contrast, we tackle learning in more practical large-scale setup, with an efficient approach that distills knowledge from eight diverse visual domains into the universal embedding. Recently, the UnED dataset was introduced in (45) as a new large-scale benchmark for universal embeddings. Their experiments considered the training of models only via classification objective, with different sampling and classifier configurations. Setting our approach apart is that we go beyond to capture detailed knowledge from diverse domains via distillation, besides proposing a more suitable training dynamic that can accommodate data from diverse domains. Concurrently, UNIC (38) proposes universal classification model using
3

---

multi- distillation. While our approach trains a student embedding from multiple teachers specialized in fine-grained visual domains, UNIC distills foundational models trained for diverse tasks, such as semantic segmentation and classification, with both supervised and self-supervised objectives.
Dynamic sampling. When training in a multi-task setting, the sampling frequency of the different domains can gready affect final performance. Poly-ViT (21) explored different samplings tailored to their multi-modal multi-task model, and concluded that sampling each domain with a weight proportional to the number of training steps that a corresponding specialized model needs to achieve maximum performance works the best, while [45] additionally comes to the same conclusion for the task of universal image embedding. This approach is costly with an increasing number of domains in hand, as one model for each domain needs to be trained, which inspires us to discover more efficient sampling strategies. [23] proposes Dynamic Stop-and-Goo sampling, which updates the sampling weight of each domain based on the validation set accuracy. Differently from them, we propose to calculate the sampling weights only based on training loss, which doesn't 't require the expensive feature extraction of the validation set but can happen on the fly. A similar idea has been explored by [31] in the context of pre-training vision-language models, which is far from this work's s, as we focus on learning multi-domain fine-grained image embeddings.
Proposed method
This section presents our proposed training method, Universal Dynamic Online distillatioN (UDON), to learn the universal image embedding. UDON utilizes a pretrained Vision Transformer [8] as the image encoder, which is further fine-tuned with a combination of classification and distillation objectives. First, some preliminaries concerning the backbone architecture that we build upon are introduced, and afterward, the complete training pipeline is presented in detail.
3.1 Preliminaries
The Vision Transformer g backbone produces the [CLS] token as a global representation of the image. Let 6 X' - RD denote the Vision Transformer function that takes an input image x E X and maps it to the [CLS] token es(x) E R,,, compactly denoted as cg. The dimensionality D is backbone dependent and usually higher than the one required in the downstream task, hence projection to lower dimensional space is introduced. The final vector after projection is the universal embedding, denoted as eu E IRd d < D. Following standard practice used in image retrieval architectures [1I] e, and e, are fg normalized. When referring to a batch of embeddings, we use capitalized notation, e.g., batch of embeddings e,, denoted E,, E gd B where B is the batch size. For training with a classification loss on top of the universal embedding in the multi-domain setup, a Separate Classifier (SC) per domain is employed, classifying across the classes of that specific domain, an option justified by (5). In the following, the word "head" denotes both the projection to the embedding space and the classifier to which the projected embedding is input.
3.2 Universal Dynamic Online distillatioN (UDON)
Our UDON training approach introduces an efficient multi-teacher distillation method, relying on shared feature extraction backbone. The entire training pipeline is presented in Figure 2 The backbone produces the initial high dimensional image embedding, eb, for the samples of all domains. Given es. in addition to projecting it into the universal embedding space e, that is used at test time (as in (45)), we also project e, to per-domain spaces, which constitute the teacher embeddings {ee, E RPI J. Teacher is only activated for samples of domain i. Both the universal and the Nomain-specitn (teacher) projections are realized by linear layers, and there is a domain-specific projection for each domain.
The universal embedding is trained with both classification and distillation objectives, calculated on batches containing single domain at time. While training with classification objective allows grasping broad knowledge for several domains, it may lead to a sub-optimal model due to contradictory cues when combining data from all domains. Therefore, we employ distillation from the domain-specific teachers to infuse the universal model with domain-specific knowledge. The loss functions are presented below.

---

domain/
i Teacher
By
Ed R
Ir E RA
Projettion
44
Batch x- dodaain
rc
R
C
Caaa fart
Beckbooe
Stucient
B B
1og
R 000 Universal Embeddiing
L. R
M Stap--Grad
Block diagram of UDON's training process. Each batch of size B contains images from single domain (e.g., cars, natural world, etc). When batch with domain is processed, the i-th teacher head is used. Both the teacher and the student employ a classification loss (C (a.) on top of their batched logits (L,,, L.). predicting among C classes. The student is additionally trained via distillation, by learning intra-batch relationships (Ctra) and logits (Ciog) with the domain teacher guidance. Note that the distillation losses backpropagated only through the student's head.
Figure 2:
Classification losses. The universal and the domain-specific embeddings are trained with classifica- tion losses (Normalized Softmax Loss (88), instantiated with separate classifiers for each domain. These losses update the backbone via gradients propagated through the teacher and the student heads, while teacher-specific or student-specific parameters only updated via gradients from their respective heads. The i-th teacher (of domain i) classification loss L and the universal embedding (student) classification loss Cas defined as:
B M y l
B Σ y log(ū") 1=1
Cas = B
and
Ca.
(I)
B
where B is the batch size, is the one-hot ground truth vector of sample j. 8/ is the predicted probability vector for sample produced from teacher of domain and 0) is the predicted probability vector for sample j produced from the universal embedding (student). It is important to note that the classifiers of the student and the teachers are different and that the student employs as many classifiers the number of domains (SC).
Distillation losses. The student is tasked to match the teacher embedding of the corresponding domain by enforcing two separate distillation losses. The first is relational distillation loss, which acts on batch similarity matrices. Given the student's 's batch embeddings E,, E d its batch similarity matrix is formed as E E,, E RBxB Similarly, for the i-th teacher's batch embeddings E,, E F D.x its batch similarity matrix is formed as E, E,, E Bx The goal is for the student to learn detailed intra-domain similarities from the more powerful domain-specifie teacher. Specifically, the student's intra-domain cosine similarities are encouraged to follow the i-th teacher's cosine similarities, when the batch of images comes from domain
FT F. F.J 12.
Pe!
Additionally, the student is tasked to match its logits to the teacher's s, minimizing their KL divergence, after scaling with identical temperature T and softmax normalization of both:
Cioy KL (softmax(L./TT) || softmax(4,/TT)
where L, the logit vector produced by the classifier on the student's 's side and 4, is the logit vector produced by the classifier on the i-th teacher's 's side. The temperature T is shared across all teachers
5

---

and the student. This loss provides more global context, as it captures the similarities between an embedding and all of the class prototypes in the domain (which exist in the classifier), instead of only relating embeddings in batch. Both distillation losses do not backpropagate through the Gomain-specifie teacher head, as only the student should try to learn from the teacher. Distillation starts at the beginning of the training and it happens in an online manner, at the same time that the universal student and the teachers are trained with the classification losses. The total loss for a training batch, containing images of domain i, is as follows:
Liotal = C.ds + Lds + Cir + Clog
(4)
Dynamic domain sampling, Each domain comes with its own training data, which differ in the number of classes and in the number of examples. Balancing of the domains is performed through sampling of the training data. In [45), training is performed with clean batches, i.e. each batch only contains examples from a single domain. Three different sampling schemes were compared in [45]. In Dataset Size sampling, the datasets are sampled proportionally to their size, biasing towards large datasets. In Round-RRobin (RR) sampling, the domains are sampled equally often in a eyclic order. The Specialist Steps sampling selects the domains proportionally to the number of steps the specialist at the particular domain needs to achieve maximum validation performance. Only the last approach takes into account the difficulty of the classification task in each domain. However, such sampling is static, does not reflect possible correlations in similar domains, and requires the training of a specialist for each domain prior to training the universal model.
We propose to sample each domain proportionally to the classification loss produced by the domain- specific embedding during training. relative to the corresponding loss of the other domain-speciffi embeddings. This way, the domains for which the training loss decreases slower than for other domains are sampled more for training. More specifically, we sample the domain of the next training batch out of the distribution which for each domain m assigns the probability:
P(m) train loss
2i=1 train loss;
where N is the number of domains, train lossm is the classification loss produced by the embedding of domain m, and the denominator represents the sum of the corresponding losses across all domains. This distribution is updated after every S steps, a hyperparameter that is tuned on the validation set.
Experiments
4.1 Experimental settings
Dataset. The proposed method is evaluated on the recent Universal Embeddings Dataset (UnED) [45)
the largest dataset for multi-domain fine-grained retrieval. It comprises 4.1M images, with 349k
classes distributed across 8 image domains: food (Food2k dataset) 225, cars (CARS 116 dataset) 18]
online products (SOP dataset) [4T) clothing (InShop dataset) [22] natural world (iNat dataset) 43
artworks (Met dataset) [46). landmarks (GLDv2 dataset) [44] and retail products (Rp2k dataset)
[30]. We follow the train-valis walidation-tess splits and the evaluation protocol defined in (45) brief review follows. Each index image and each query in the test set are described by a 64-D (universal) embedding, and Euclidean nearest neighbors in the embedding space are found for each query among the index set. The index contains images from all 8 domains combined; hence, cross-domain false positives are possible. Performance is measured by two metrics: Recall @ 1 (R 1). which is equivalent to the correctness of the top neighbor, and modified Mean Precision 5 (P@5) [5). The average performance over the queries in each domain is reported, as well as the balanced average of these values across all domains.
Compared methods. The universal embedding task is relatively new and only a few baselines were published in [45] 6). We extend these baselines by re- perppposiin the single-domain embedding method of B7 (Table l). Additionally, we also compare to two straight- multi-
distillation methods [10] TTblle B). The main baselines compared with this work are the best- best-performing methods from 45). namely USCRR, UJCRR, and USCSS, which vary the classifier setup (SC: Separate Classifier, JC: Joint Classifier) and the sampling (RR: Round Robin, SS: Specialist Steps). We also evaluate variant of USCRR, which uses the proposed dynamic sampling scheme, dubbed "USC w/ Dyn Sampler". The USCRR (45) method (USC w/ Dyn Sampler) is further expanded by

---

CARSTOG
Midel
21:1
52.9
31.0
7)8-D)
52.3
82.2
38.6
768-00
Std.IPP
768-D)
95.7
53.5
51.6
42.7
62.0
UACK IN
61.9
71.5
75.2
2044
24.9
45.1
73.6
87.1
34.9
65.6
CLIP SpecialsttDrocle
51.5
K3.4
58.5
68.0
73.0
23.6
52.9
66.77
34.4
39.6
70.3
63.9
71.3
7400
15.3
55
7.0
21.1
31.6
14.1
32.4
02.6
USCRR 85 USCSS
48.3
60.9
58.9
69.7
70.4
74.2
7.3
9.7
31.4
74.1
62.5
40.0
61.7
51.4
64.3
62.0
78.8
67.6
87.2
68.3
73.5
8.4
10.7
18.0
40.6
73.5
87.1
51.3
63.0
USC w/ Dyn Sampler
962
56.
56.5
67.6
785.4
653
86T
24.65
1,5.0
1556
25:55
377.5
75.2
36.55
31.3"
6,525
MLP bracling w/ Dyn Sumpler
47.5
60.1
51.3
63.0
61.8
7885
64.1
85.0
69.7
74.8
14.3
17.8
25.6
36.4
73.3
86.8
51.0
62.8
UDON (Ours)
49.6
62.2
61.3
71.2
64.2
$0.2
69.8
88.5
70.4
75.3
12.0
15.9
18.6
40.9
75.6
88.0
53.0
68.1
C1
IP pritraining
EST
501
62.0
89.8
65.4
68.6
82.7
77.0
91.1
63.7
09.5
4.6
5.8
15.5
56.0
76.1
84.1
35.00
64.6
<3
USCRR
40.5
61.4
79.0
84.0
63.6
$1.1
78.1
89.4
64.4
70.5
8.6
10.8
25.3
36.5
71.1
85.1
34.6
64.9
[S USCSS
49.8
62.0
76.4
K1.4
86.8
$1.3
21.0
88.5
65.3
71.4
9.
12.7
31.5
42.6
70.1
848
15.0
65.9
DDOR [Cuss)
363
6.5.4
966
85.8
36.
667
75.
155
166
365
334
75.
355
365
Table 1: Performance comparison of the universal embedding for the proposed UDON method against the previous state-of ofrthe-rrt and the proposed baselines, on the test set of UnED dataset. Off-the-shelf models are shown for reference, as they employ much higher dimensional descriptors (768-D) vs. 64-D) than the rest of the methods. For each type of pre-training, the best method is highlighted in bold. The Specialist+Oracle model constitutes non-realistic method that is presented in order to get an estimate of the maximum performance that can be achieved in each domain. All of the methods use the ViT-Base/16 backbone.
inserting an MLP projector 37 between the universal embedding and its a classifier for each domain.
The projectors consist of three hidden layers of sizes 256, 256, and 512 respectively. This new baseline method is referred to as the "MLP baseline". We compare with the off-the-shelf embeddings from ImageNer21 (IN) (2), finetuned ImageNet21k model with masked image modeling (IN+MIM) [6], CLIP 3 DINOv2 266.. and SigLIP [49], which utilize embeddings of much higher dimensionality (768D vs 64D). Lastly,
compare against the Specialist+Orace bascline, non-realistic model proposed in [45] to get a hypothetical estimate of the maximum performance that can be achieved on each individual domain, by choosing the specialist of the query's 's domain as the embedding for both the query and the index set. All of the methods (including UDON), use the ViT-Base/16 16 backbone, and additionally, apart from the off-the-shelf ones, are fine-tuned on the training set of UnED
Implementation details. For fair comparisons with the baselines of 45), identical values for common hyperparameters are used. The newly introduced hyperparameters are tuned based on performance on the validation set of UnED. For the KL divergence loss (3). the value of temperature T is set to T = 0.1 (a discussion regarding this choice can be found in the Appendix); the teacher embeddings have dimensionality of Dr 256; the four loss components contribute equally to the total loss Liotal (no weights need to be tuned). We set the universal student embedding dimensionality to = 64 for direct comparability against previous work. The batch size is set as B = 128. The hyperparameter S for the number of steps, after which the dynamic sampler is updated, is set to 1000. Each experiment is repeated 3 times with different seeds; the reported values are averaged over those runs. The standard deviations are reported in the Appendix. The ViT-Base/16 variant of the Vision Transformer is used as the backbone with ImageNet21k [42) and CLIP 33 initializations. The linear projections are initialized randomly, as well as the corresponding classifiers. Our implementation is based on the Scenic framework 77. a library based on Jax /5lxx [13. Experiments are executed on Google Cloud TPU v4s (16).
4.2 Main results
In Table we present the performance of UDON and the compared methods. For the full UDON method, we present results for two different pretrainings for more complete comparisons, namely ImageNet2 lk 42 and CLIP (33. For the "MLP baseline" and the [45] USC w/ Dyn Sampler" baseline, we present results for ImageNet21k lk pretraining only.
On average, as well as on most of the individual domains, the proposed UDON method achieves state- of-the-art performance, for both types of pretraining (ImageNet211 k and CLIP). For ImageNet2 achieves an improvement of 1.5% and 2.3% on mean P@5 and R@ 1, respectively, over the previous state-of-the-art. For CLIP, achieves an improvement of 1.9% and 1.8% on mean P@5 and R @ 1, respectively, over the previous state-of-the-art. The biggest improvements are observed in the Met.
7

---

GLDv2 and iNat domains, all of which are characterized by large number of classes and long-tail distribution. Our proposed method makes notable progress towards closing the performance gap the Specialist+Oracle baseline, coming close as 1%%%%%%% for the respective metrics, for ImageNer211 Ik pretraining. The MI.P baseline 37 with the addition of dynamic sampling ("MLP baseline Dyn Sampler") shows comparable performance methods [45] and the baseline method USCRR with Dyn Sampler instead of RR sampling ("USC w/ Dyn Sampler"), showing that it is not trivial to extend the conclusions of [37 to the multi-domain embedding setting. The proposed UDON method achieves an improvement of 2.9% and 2.5% on mean P@3 and R @ 1, respectively, over the "MLP baseline w/ Dyn Sampler", and 2.6% and 2. % on mean P@5 and R@ respectively, over the "USC w/ Dyn Sampler" baseline. We additionally perform experiment where combine the MLP baseline [37 with UDON, by appending MLP projector between the classifier of every domain and the universal emhedding. This underperforms the UDON method by 0.6% and 0.3% mean P@5 and R @ respectively, while bringing significant extra cost on the number of parameters of the model. In Figure ( qualitatiiee results for two queries of the UnED test set shown, for which the UDON universal embedding exhibits better retrieval performance compared to the USCRR [45 baseline embedding.
average with the previously reported state-of-the-art
ratk
2nd rank
3rd renk
4th rank
5th rank
Query
Baseline
Food2k
Food2k
Food2k
Food2k
Food2k
UDON
Food2k
Food2k
Food2k
Food2k
Food2k
1st rank
2nd rank
3rd rank
4th rank
Sth rank
Query
iNot
iNat
iNat
iNat
iNat
iNit
UDON
INiot
INat
INiot
INat
INat
nearest neighbours that retrieved by the baseline (USCRR) embedding (top row) and the proposed UDON embedding (bottom row), for queries that the proposed method improves over the bascline. Each image shows the domain it comes from (underneath it). The correct neighbors are in green border, the incorrect ones are in red.
Figure 3: Qualitative results for our UDON method. We present
4.3 Ablations
spectrum of methods employing various components of UDON are evaluated to examine the impact of each individual block. The methods range from the baseline USCRR (Table 2 row 1) to the full UDON (Table row 3) method. All ablations are initialized by ImageNet21k pretraining. Qualitative results comparing the full UDON method with the baseline "USCRR" presented in the Appendix. Dynamic sampler. Two methods are evaluated demonstrate the importance of the dynamic sampler (DS). The bascline with DS (Table 2 row 2 compare with row I) and UDON without DS (Table 2

---

FoodEk
CARST96
SOP
InShop
iNat
Met
GI.DV2
Ry2k
Mean
Model
Pas
Ros
Pars
Ree
Per5
Pa5.
Roe
Pers
Re1
P65
Par.
Roe
P65
R61
R
445 USCRR
48.3
60.9
58.9
69.7
61.9
78.7
70.
88.3
69.1
74.2
7.3
9.7
21.3
31.4
74.1
87.1
51.4
62.5
USC Dyn Sumpler
46.2
59.1
56.3
67.4
61.1
78.4
65.9
69.6
74.8
12.0
15.6
25.9
37.3
73.2
86.9
51.3
63.2
UDON (Ohurs)
3 Full UDON
49,6
62.2
61.3
71.2
64.2
80.2
69.8
88.5
70.4
75.3
12.0
15.9
28.6
40.9
15.6
88.0
33.9
66.3
w/o Dyn Sampler (RR)
50.3
62.9
62.9
72.5
67.4
82.2
75.4
90.8
70.4
75.4
7.6
9.4
23.3
33.4
76.8
88.6
54.3
64.4
66-D teachers
47.7
60.3
60.2
70.3
61.3
80.3
68.6
87.6
69.9
75.0
11.3
11.5
26.4
36.1
74.2
87.6
52.8
61.2
w/o logit distilaation
49.2
61.8
60:1
70.4
62.6
79.2
68.0
87.2
70.3
75.3
12.5
15.6
28.77
41.0
74.9
87.8
53.3
64.8
w/o any distiilarion
48.0
60.9
54.1
65.4
61.4
78.5
65.8
85.8
70.3
75.3
12.4
16.1
28
41,0
73.5
87.1
51.7
63.8
w/o
univ.
48.0
60.5
77.7
67.2
74.9
12.6
15.9
25.8
52.3
64.0
9 Dyn Sampler on univ.
48.2
60.8
60.6
70.8
61.3
80.4
69.5
88.3
70.1
75.2
13.2
16.5
27.2
39.5
75.2
87.7
53.6
61.9
Table 2: Ablation studies for the performance of the universal embedding, given different modifica- tions of the Full UDON approach. The comparison is performed on the test set of UnED.
row 4 compare with row 3). In both experiments, the dynamic sampler delivers a significant boost in the two most difficult (instance level) domains Met and GLDv2, similar performance in iNat, and drop in the other domains. All following ablations are performed with the dynamic sampler.
Distillation objectives. Two distillation losses are involved in training the full UDON method: the relational distillation loss (2) and the logit distillation loss 3 Both the losses improve the performance, as can be seen in Table Table@[coppriing the Full method (row 3), relational distillation only (w/o logit distillation, row 6), and no distillation loss (row 7).
Classification loss on the universal embedding. Removing the classification loss from the universal embedding ("w/o CE loss on univ.", row 8) results in loss of average performance. Interestingly. has a slightly positive impact on the Met domain.
Online teacher dimensionality. We perform experiment ("64-D teachers", row 5) with dimension- ality of the specific domain teachers reduced to 64D (i.e. the same dimensionality as of the universal student embedding) as compared to 256D) in the full method. This results in a performance drop, which aligns with previous observations that higher dimensional embeddings can be better teachers in distillation setting [36).
Scheduling the dynamic sampler. The sampler probability is updated every 1000 optimization steps (UDON needs ~ 120k steps to converge). Our experiments show that the method is not sensitive to the choice of this parameter. The probability is updated according to the training classification loss of the current model on each domain. In fact, there are two such losses in UDON. One provided by each domain teacher's is classifier, and one provided by the classification loss on the universal student's separate classifier for each domain. Using the latter update the sampler's weights incurs a small drop in performance, as seen in ("Dyn Sampler on univ.", row 9), compared to the Full UDON method, where the domain teacher's 's classification loss updates the sampler.
4.4 Other distillation approaches
The online distillation method of UDON provides a very efficient way of transferring domain-specific knowledge to the universal embedding, without training more than single backbone. In this section, the application of alternative distillation approaches is discussed. In particular, we implement two other approaches: first, the naive multi-teacher distillation (Figure [ middle), with independent specialist models as teachers (8 extra backbones), where each teacher is trained in its own domain, dubbed "8 separate teachers".. Second, one model with specialist heads is trained, i.e. I extra backbone followed by domain- specific projections (teacher embeddings). dubbed "1 separate teacher". In both cases, the teacher backhones are fixed during distillation, and the universal embedding (student) gets its own backbone. All backbones are initialized by ImageNer21 k in the experiments of Tables [ and For efficiency reasons, only relational distillation is performed in this experiment, and all the teacher embeddings are 256 dimensional, as in UDON.
Universal embedding performance. The results presented in Table indicating that our method is not only efficient, but also outperforms other variants. This finding indicates that UDON benefits significantly from sharing the backbone between the student and the teachers, even if that could limit the representation capacity of the teachers, given that they have fewer free parameters.
Compute cost reduction. The alternative approaches are less efficient in terms of the number of parameters, as well as in the number of steps needed to converge. More specifically, for this setup, UDON uses ~ 188 million (M) parameters, "I separate teacher" uses 4440M parameters, and "g

---

Fond2k
CARS196
SOP
InShopp
INat
Met
GLIV2
Ry2k
Mean
P675
P65
R@1
PG15
R
P605
R@1
R@1
P65
P6:5
sepurate teachers
47.6
60.8
58.0
09.0
62.8
79.4
67.9
87.3
70.1
75.2
12.5
15.8
26.0
38.4
74.7
87.6
52.4
64.2
soparate teacher
47.2
59.8
54.7
66.1
62.5
79.3
66.5
86.6
69.9
75.1
12.0
15.1
26.2
38.8
74.2
87.5
51.7
63.6
UDON (Ours)
49.2
61.8
60.1 70.4
62.6
79.2
68.0
87.2
70.3
75.3
12.5
15.6
28.7
41.0
74.9
87.8
53.3
64.8
Table 3: Performance comparison of the universal embedding using different distillation approaches, on the test set of UnED. "8 separate teachers" indicates the setting of 8 independent specialist models distilling to a universal model, while "I separate teacher" indicates the setting of independent model with 8 separate domain heads distilling to universal model.
Fond2k
CARS196
SOP
InShop
GL.D22
Rynk
Mean
Model
PG55
R@1
PG5
R@1
P@5
R@
PQS
R
PG5
R@
PG6
R
PGB
R@1
PG5
R
PG15
R@1
Separate Index Evaluation (Orncle)
separati leadherrs
66.4
71.8
81.1
74,9
87.0
77.7
92.6
76.7
81.3
74.7
separate teacher
52.5
65.3
$7.3
68.0
71.7
85.1
71.9
89.8
76.6
812
35.2
43.4
31.1
45.0
79.2
89.8
59.4
71.0
UDON teachers
62.4
71.6
72.7
85.6
75.2
91.8
74.8
79.6
29.0
79.5
59.9
70.4
Table 4: Performance comparison of the teacher embeddings (256D) that are used by the different distillation approaches, on the test set of UnED, but on the separate index setting, i.e. each query is only compared against the index of its own domain.
separate teachers" uses 8773M parameters, saving as much as 44.5 times in parameters, while improving performance. Additionally, UDON takes on average ~120 steps to converge, "I separate teacher" needs around 220k steps (sum of the 2 training phases), "8 separate teachers" 250k training steps (sum of the training phases), cutting the number of convergence steps in half. For reference, the no-distillation baseline USCRR converges at around the same steps UDON.
Teachers' performance. To gain a better insight, we also evaluate the performance of the teachers in their domains. The domain of test image is used as an oracle in these experiments in order to restrict the index to contain images of the same domain only, hence the reported numbers are not comparable to other reported results. Table that the independent specialists provide the best per-domain teachers. Interestingly, although being the best performing (teachers) in their domain, the latter do not provide the best distillation outcome, which is delivered by UDON, discussed in the previous paragraphs. We hypothesize that sharing the backbone between the teachers and the student in UDON, on the one hand limits the performance of the teachers on their individual domains, but, on the other hand, allows for more efficient distillation, as the specialist heads and the universal student operate on the same backbone embedding. Another related observation can be made from the ablation Table [ Row 2 ("USC w/ Dyn Sampler") and row 7 ("w/o any distillation") differ in the presence of separate domain classification heads top of the universal student backbone in UDON, without performing distillation. In the latter method, the specialist heads provide a regularization for the backbone training, which results in improved performance.
5 Conclusions and Limitations
Conclusions. In this work, a novel multi-teacher distillation approach Universal Dynamic Online distillatioN (UDON)- - is introduced to tackle the problem of learning a universal embedding. The universal embedding and the domain-specific teachers share the backbone parameters and are trained jointly, which proves to be very efficient both in time and resources. The proposed training approach is shown to deliver high efficacy distillation, in which the universal student performs even better than distilling from separate fixed teachers. The additionally proposed difficulty-based dynamic complex domains which are typically characterized by large number of classes and taig-tail distributions. The proposed method improves the tfte-off-the-nt performance on the recent UnED benchmark.
sampling results a significant boost of performance
Limitations. While UDON boosts universal embedding performance compared to the baseline method USCRR which only employs classification loss, it has 20% decrease in training throughput, given that it adds new parameters (in the teacher heads). Additionally, the proposed dynamic sampling significantly improves the performance in the difficult domains, such as Met and GI.Dv2, however, still at a cost of slightly decreased performance other simpler domains.
10

---

6 Acknowledgements
The authors acknowledge the support of the National Recovery Plan funded project MPO 60273/24221300022000 CEDMO 2.0 NPO, the Czech Technical University in Prague grant No. 36522/173/0HK3333VV13, and the CTU institutional support (Future fund).
References
[1] Bixby vision. Available online: bixby/vision/ (2024)
[2] Google Lens. Available online:
http://eeesn.gglee//00e0/ (2024)
[3] Almazan, J., Ko, B., Gu, G., Larlus, D., Kalantidis, Y.: Granularity-aaware Adaptation for Image Retrieval over Multiple Tasks. In: Proc. ECCV (2022)
[4] Bilen, H., Vedaldi, A.: Universal representations: The missing link between faces, text, plank- tons, and cat breeds. arXiv preprint
10t:::7710027777 (2017)
[5] Bradbury, J., Frostig, R., Hawkins, P., Johnson, MJ.,, Leary, C., Maclaurin, D., Necula, G., Paszke, A., VanderPlas, J., Wanderman-Milne, S., Zhang, Q.: JAX: composable transformations of Python+NumPP programs (2018), http:: //g thub. com/ foom//ooolejaax
[6] Chen, K. Salz, D., Chang, H., Sohn, K., Krishnan, D., Seyedhosseini, M.: Improve supervised representation learning with masked image modeling (2023)
[7] Dehghani, M., Gritsenko, A., Arnab, A., Minderer, M., Tay. Y:: Scenic: A jax library for com- puter vision research and beyond. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 21393-21398 (2022)
[8] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghami,
M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11 19929 (2020)
[9] Ermolov, A., Mirvakhabova, L., Khrulkov, V., Sebe, N., Oseledets, I.: Hyperbolic vision transformers: Combining improvements in metric learning. In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (2022)
[10] Feng, Y., Peng, F., Zhang, X., Zhu, W., Zhang, S., Zhou, H., Li, Z., Duerig, T., Chang, S.F., Luo, J.: Unifying Specialist Image Embedding into Universal Image Embedding. arXiv:2003.0370) (2020)
[11] Gordo, A., Almazan, J., Revaud, J., Larlus, D.: Deep Image Retrieval: Learning Global Representations for Image Search. In: Proc. ECCV (2016)
[12] Guo, Q., Wang, X., Wu, Y., Yu, Z., Liang, D.,, Hu, X., Luo, P.: Online knowledge distillation via collaborative learning. In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (2020)
[13] Heek, J., Levskaya, A., Oliver, A., Ritter, M., Rondepierre, B., Steiner, A., van Zee, M.: Flax: A neural network library and ecosystem for JAX (2023), http::/gtthhtt com/ google/flax
[14] Hinton, G., Vinyals, O., Dean, J.: Distilling the Knowledge in Neural Network. In: NIPS
Deep Learning and Representation Learning Workshop (2015)
[15] Jacob, G.M., Agarwal, V., Stenger, B.: Online knowledge distillation for multi-task learning. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (2023)
[16] Jouppi, N., Kurian, G., Li, S., Ma, P., Nagarajan, R., Nai, L., Patil, N., Subramanian, S., Swing, A., Towles, B., Young, C., Zhou, X., Zhou, Z., Patterson, D.A.: Tpu v4: An optically reconfigurable supercomputer for machine learning with hardware support for embeddings. In: Proceedings of the 50th Annual International Symposium on Computer Architecture. ISCA '23,
Association for Computing Machinery, New York, NY, USA (2023)
[17] Kim, S., Kim, D., Cho, M., Kwak, S.: Proxy anchor loss for deep metric learning. In: Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition (2020)
[18] Krause, J., Stark, M., Deng, J., Fei-Fei, L.: Object Representations for Fine-Grained Categorization. In: Proc. ICCV Workshops (2013)

---

[19] Laskar, Z.. Kannala, J.: Data-efficient ranking distillation for image retrieval. In: Proceedings of the Asian Conference on Computer Vision (ACCV) (November 2020)
[20] Li, W.H., Bilen, H.: Knowledge distillation for mulli-task learning. In: Computer Vision ECCV 2020 Workshops: Glasgow, UK, August 23-28, 2020, Proceedings, Part VI 16. pp. 163-176. Springer (2020)
[21] Likhosherstov, V., Arnab, A., Choromanski, K., Lucic, M., Tay, Y., Weller, A., Dehghani, M.: Polyvit: Co-training vision transformers on images, videos and audio. arXiv preprint arXiv:2111 1. 12993 (2021)
[22] Liu, Z., Luo, P., Qiu, S., Wang, X., Tang, X.: Deepfashion: Powering robust clothes recognition and retrieval with rich annotations. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)
[23] Lu, J., Goswami, V., Rohrbach, M., Parikh, D., Lee, S.: 12-in-1: Multi-task vision and language representation learning. In: Proceedings of the IEEE/CVF conference on computer vision and
pattern recognition (2020)
[24] Ma, Z., Dong. J. Ji, S., Liu, Z., Zhang, X., Wang, Z., He, S., Qian, F., Zhang, X., Yang, L.: Let all be whitened: Multi-teacher distillation for efficient visual retrieval (2023)
[25] Min, W., Wang. Z., Liu, Y., Luo, M., Kang, L., Wei, X., Wei, X., Jiang, S.: Large scale visual food recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence (2023)
[26] Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P.,
Haziza, D., Massa, F., El-Nouby, A., et al.: Dinov2: Learning robust visual features without supervision. arxiv preprint arXiv:2304.07193 (2023)
[27] Park, W. Kim, D., Lu, Y., Cho, M.: Relational knowledge distillation. In: Proceedings of the IEEE/CVE conference on computer vision and pattern recognition (2019)
[28] Passalis, N., Tefas, A.: Learning deep representations with probabilistic knowledge transfer. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 268-284 (2018) [29] Patel, Y., Tolias, G., Matas, J.: Recall @ surrogate loss with large batches and similarity mixup. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022)
[30] Peng, J., Xiao, C., Li, Y.: Rp2k: A large-scale retail product dataset forfine-grained image classification. arXiv preprint arXiv:2006. 1et222000.....44 12634 (2020)
[31] Piergiovanni. A., Kuo, W., Li, W., Angelova, A.: Dynamic pretraining of vision-language models (2022)
[32] Radenović, F., Tolias, G., Chum, O.: Fine-tuning CNN Image Retrieval with No Human Annotation. IEEE Transactions on Pattern Analysis and Machine Intelligence (2018)
[33] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748-8763. PMLR (2021)
[34] Ramzi, E.,, Thome, N., Rambour, C., Audebert, N., Bitot, X.: Robust and decomposable average precision for image retrieval. Advances in Neural Information Processing Systems (2021)
[35] Rebuffi, S.A., Bilen, H., Vedaldi, A.: Learning multiple visual domains with residual adapters. Advances in neural information processing systems 30 (2017)
[36] Roth, K., Milbich, T.. Ommer, B., Cohen, J.P.. Ghassemi, M.: Simultaneous similarity-basee self-distillation for deep metric learning. In: International Conference on Machine Learning (2021)
[37] Sariyildiz, M.B., Kalantidis, Y., Alahari, K., Larlus, D.: No reason for no supervision: Improved generalization in supervised models. In: ICLR Internnnetaaiinnll Conference on Learning Representations (2023)
[38] Sarryıldız M.B., Weinzaepfel, P., Lucas, T., Larlus, D., Kalantidis, Y.: Unic: Universal classification models via multi-teacher distillation. In: European Conference on Computer
Vision. Springer (2025)
[39] Schall, K., Barthel, K.U., Hezel, N., Jung, K.: GPR 1200: A Benchmark for Seeneral-Purposs Content-Based Image Retrieval. In: Proc. International Conference on Multimedia Modeling (2022)
12

---

[40] Schroff, F,, Kalenichenko, D., Philbin, J.: FaceNet: A Unified Embedding for Face Recognition and Clustering. In: Proc. CVPR (2015)
[41] Song, H.O., Xiang, Y., Jegelka, S., Savarese, S.: Deep Metric Learning via Lifted Structured Feature Embedding. In: Proc. CVPR (2016)
[42] Steiner, A., Kolesnikov, A., Zhai, X., Wightman, R., Uszkoreit, J., Beyer, L.: How to train your vit? data, augmentation, and regularization in vision transformers. arXiv preprint arXiv:2106. 10270 (2021)
[43] Van Horn, G., Mac Aodha, O., Song, Y., Cui, Y., Sun, C., Shepard, A., Adam, H., Perona, P., Belongie, S.: The inaturalist species classification and detection dataset. In: Proceedings of the
IEEE conference on computer vision and pattern recognition, pp. 8769-8778 (2018)
[44] Weyand, T., Araujo, A., Cao, B., Sim, J.: Google Landmarks Dataset v2 A Large-Scale Benchmark for Instance-Level Recognition and Retrieval. In: Proc. CVPR (2020)
[45] Ypsilantis, N.A., Chen, K., Cao, B., Lipovský, M., Jogan-Schöönergeer P., Makosa, G.,
Bluntschli, B., Scyedhosseini, M., Chum, O., Araujo. A.: Towards universal image embeddings: large-scale dataset and challenge for generic image representations. In: Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV) (2023)
[46] Ypsilantis, N.A., Garcia, N., Han, G., Ibrahimi, S., Van Noord, N., Tolias, G.: The met dataset:
Instance-level recognition for artworks. In: Thirty-fifti Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2) (2021)
[47] Zhai, A., Wu, H.Y., Tzeng, E., Park, D.H., Rosenberg, C.: Learning a Unified Embedding for Visual Search at Pinterest. Proc. SIGKDD (2019)
[48] Zhai, A., Wu, H.Y.: Classification is strong baseline for deep metric learning. arXiv preprint arXiv:1811. 12649 (2018)
[49] Zhai, X., Mustafa, B., Kolesnikov, A., Beyer, L.: Sigmoid loss for language image pre-training. In: Proceedings of the IEEE/CVF International Conference Computer Vision. pp. 11975- 11986 (2023)
[50] Zhu, X., Gong, S., al.: Knowledge distillation by on-the-fly native ensemble. Advances in
neural information processing systems (2018)
13

---

A Appendix
In the Appendix we present results from Tables to of the main paper that include the standard deviations calculated across 3 randomizations per experiment, Additionally, we present results for different backbone sizes, series of experiments justify the choice of the temperature value in the main paper, and study of the architecture of the embedding projectors used in the UDON pipeline. Lastly, additional qualitative results are presented.
A.1 Standard Deviations
the results from Table I of the main paper, along with the corresponding standard deviations are presented. Only the methods that were developed in this work shown, namely the baselines "USC w/ Dyn Sampler". "MLP baseline w/ Dyn Sampler" and the proposed UDON method, for ImageNet pretraining, as well the results for UDON method for CLIP pretraining. In Table the results of the ablations from Table@oot of the main paper are presented, with the corresponding standard deviations. In 7ahe results from Tf the main paper showcasing the performance of the universal embedding under different distillation methods are presented, with the corresponding standard deviations.
In Table 5]
USS Sunder
1808 barn
BOSAT
BC038
YE3.8 (230 BOGE7 30011
133122 20133
Table 5: Evaluation results for the methods developed in this work, with the corresponding standard deviations across the 3 randomizations, This table corresponds to Table I of the main paper.
P32
2000 Das Sumpler
5632ET
DITERY
10300
WOETS
Wil Sumple 1811
183180
429124
639214
12553
003103
TRASS
34385
2301A
768:11
MA20
452283
61223:2
663183
164553
A28385
192288
992233
122213
23:12
40831
340000
35663
08463
8 Dys Sieylle cel.
183444
(8461:
(05465
708461
(83665
304464
(85433
201664
253463
133-66
165-66
96611
453-05
Stan10 5694m6
Table 6: Evaluation results for the ablation studies from Table 2 of the main paper, along with the standard deviations calculated across the randomizations of each experiment.
Pook
CAUS
135004 26540
151416
175-03
Table 7: Evaluation results for the comparisons between distillation alternatives from Table 3 of the main paper, along with the standard deviations calculated across the 3 randomizations of each experiment.
A.2 UDON with a Smaller Backbone
We present additional results for the UDON method, for smaller backbone size, namely ViT- Small/16 in Ta where we compare with the baseline method UJCRR of [45), which is one of the best performing methods. Both methods start from ImageNet2 lk pretraining. The results indicate that our method is also applicable to smaller backbone sizes.
A.3 UDON with a Larger Backbone
In order to examine if the performance gain achieved by UDON with a ViT-Base (over the baseline USCRR) diminishes for larger backbone sizes, we performed additional experiments with the larger Vision Transformer variant, namely the ViT-Large model with ImageNet21k pre-training. The results are presented in Table 2 We observe that the USCRR baseline with a larger backbone achieves almost the same performance its smaller ViT-Base counterpart, indicating that larger backbone
14

---

Mean
Modcl
P@3
R@1
VIT-S (ImageNet21k)) UJCRR 45
48.3
58.9
VIT-S (ImageNer21k) + UDON (Ours)
49.1
61.1
Table 8: Performance comparison of the proposed UDON method with the UJCRR method of 45 on the test set of UnED, for the smaller backbone size of ViT-Small.
size doesn't 't necessarily mean better performance (note that similar observation is made in 45 Appendix A, section A.2). Additionally, the UDON-trained ViT-Large achieves performance that is better but close to the ViT-Base counterpart, indicating both the effectiveness of the UDON training procedure for the larger backbone size compared to the bascline training procedure, as well as the fact that the ViT-Base achieves very good pizzepeeformmncc tradeoff.
Mean
Model ViT-B USCRR
P@5
62.5
ViT-L USCRR 45
51.0
62.4
VT-+B+ UDON (Ours)
53.9
65.3
VIT-+ + UDON (Ours)
54.6
65,4
Table Performance comparison of the proposed UDON method and USCRR method of 45 on the test set of UnED, for two different backbone sizes, namely the VIT- ViT-Base and the larger backbone size of ViT-Large.
A.4 Larger Backbone off-the-shelf models
We provide some additional results for the larger off-the-shelf models of CLIP and DINOV2 pretrain- ing, which utilize the ViT-Large (ViT-L) backbone, in Table [0] Both utilize 1024D embeddings.
Mean
Model
R
Off-the-sheif
VIT-B CLIP (768-D)
39.8
53.5
VYT-I CLIP 33 (024-D))
44.5
58.3
VTT DINOv2 226 (768-D)
43.9
58.2
VIT-L DINOv2 26 (1024-D)
46.7
60.8
Table 10: Additional results on the test set of UnED for the off-the-shelf models CI.IP and DINOv2, which utilize the larger ViT-I. backbone variant. ViT-B results are shown as well for comparison.
A.5 Temperature logit distillation
The value of the temperature hyperparameter was tuned on the validation set of UnED independently of other hyperparameters and kept fixed. We provide additional results alternating the temperature value in the full UDON method (IN21k pre-trained) TT] The results indicate that the different values of (1,0.05,0.01) perform worse or cause the training diverge, than the one used in the main paper (0.1).
A.6 Architecture of the projectors in UDON
We performed an experiment where we replace the linear layers used as the projection for both the domain-specific teachers and the universal student by deeper network. More specifically, we use one hidden layer MLP with layernorm and GELU activation, with the same hidden dimension as output embedding dimension, i.e. 256 for the teachers and 64 for the student. The obtained results shown in Table T2 (IN2 pre-trained ViT-Base) indicate significant drop compared using the proposed linear layers.
15

---

Mean
T
P@5
R@1
0.1
53,9
65.3
53.6
0.05
53.2
0.01
Diverged
Study for different values the temperature hyperparameter used the UDON method. The results are shown on the test set of UnED.
Table 11:
Mean
Model
P
RØ I
UDON
53.9
65.3
UDON MLP projectors
51.8
63.5
Table 12: Performance comparison of the proposed UDON method to version of where the embedding projections are changed from linear layers to MLPs (UDON MLP projectors), on the test set of UnED.
A.7 Additional Qualitative Results
In Figure 4 additional qualitative results for queries that the UDON universal embedding outperforms the USCRR [45] baseline universal embedding are shown. Note that for queries whose class is represented by less than 5 positives in the index, present as many neighbors the number of positives, since only those are taken into account for the calculation of the metrics.
16

---

rank
2nd rank
3rd renk
4th rank
5th rank
Query
GLDvZ
GLDv2
GLDv2
UDON
GLDv2
GLDv2
GLDv2
GLDV2
GLDv2
rank
2nd rank
3rd rank
Query
Beseline
Met
Met
Met
UDON
Met
Met
Met
rank
Znd rank
3rd rank
4th rank
Sth rank
Query
Baseline
Rp2k
Rp2k
Rp2k
Rp2k
Rp2k
Ap2k
UDON
A1L VALR
221 10
RpZk
Rp2k
Rp2k
Rp2k
Rp2k
Figure 4: Additional qualitative results for UDON method. We present nearest neighbours that are retrieved by the baseline (USCRR) embedding (top row) and the proposed UDON embedding (bottom row), for queries that the proposed method improves domain it comes from (underneath it). The correct neighbors are in green border, the incorrect ones in red. For queries whose class represented by less than positives in the index, we present as many neighbors as the number of positives, since only those are taken into account for calculating the metries.
the baseline. Each image shows the
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims about the contributions of the paper made in the abstract and the introduction are supported by the experimental results provided.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Please refer the Limitations section after the Conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
18

---

Justification:
There are no theoretical results in the paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: All of the implementation details are described in the submission, so all the information needed to reproduce all the experimental results is provided. The code will be made public upon acceptance of the paper, to additionally aid reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
19

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: The code and the data used perform the experiments of this work not included as part of the submission. The code will be made available with Open Access in the case of the acceptance of the paper, while the data already available through the website of the UnED dataset.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
wwuuides/CodeSuuiissiiiii. sos.sssuubissiioooooocoyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurIPS code and data submission guidelines (https / ctltsps.ccpubbicc/// gwwe//ggieeeeeeeee sonddeSSbmissiiooPooe
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All of the details about training and testing are provided both in the "Experi- mental settings" section of the submission.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: In the Appendix, the mean values and standard deviations for all the metrics calculated for the main experiments of this work presented, to reflect the statistical significance the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
20

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide information about the type of Compute used in the Implementation details subsection in the submission.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
The research conducted in this paper conforms with the NeurIPS Code of
Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
The paper focuses on generalization of the retrieval method to a large variety of domains. This opens doors for number of application with positive societal impact. If there was any negative societal impact of retrieval application, existing domain-specitf approaches would be more suitable.
Justification:
21

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considcrations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning conrectly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
The code that is used to support the experiments are part of the Scenie frame- work repository (7]. which is cited in the submission. Additionally, the UnED dataset [45] is used, cited in the submission.
Justification:
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
22

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thcode com comddatasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: The paper does not release new assets at the time of submission. In case of acceptance, the Code with the corresponding licenses will be released.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
23

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "9HEr90Wy4q", "reviewer": "Reviewer_Mmnc", "summary": "The paper proposes an online distillation approach in a multi-teacher setup w/ weight sharing for efficiency. A strategic dynamic batch sampling process has been proposed to help domains w/ slower learning during training.", "questions": "Can you elaborate more on impact of the throughput drop in real-world industry applications? Also, specify how practical applications could benefit - please provide specific details.", "limitations": "Mentioned in paper", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The unified backbone and batch sampling strategy is novel and powerful. The paper is well written and experiments/datasets carefully chosen.", "weaknesses": "Did not spot any, though, it seems there is a throughput drop.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> __Can you elaborate more on impact of the throughput drop in real-world industry applications?__\n\nThe 20% throughput drop only impacts the model training stage. The inference time of our model is exactly the same as previous state-of-the-art models from [44], which makes our inference-time comparisons fair. For industry applications, generally the model will be trained during a development cycle when compute resources are used intensively one-off to produce a high-quality model with acceptable cost. Once the development cycle ends, the finalized model will be used for inference in production for a long period of time. For this reason, the inference time cost is the main concern when developing industry models, and a 20% training cost increase is not usually a problem, given the significant quality gains.\n\n> __Also, specify how practical applications could benefit - please provide specific details.__\n\nSeveral practical applications can directly benefit from an improved universal image embedding. In particular, generic visual recognition systems such as [46, 1, 2] need to handle queries depicting any type of object or scene. As per news reports, these systems today receive 10B+ searches per month and their popularity continues growing. This calls for a universal embedding, since it is not scalable to handle images of different domains with specialized, per-domain models. Besides, these embeddings can be useful in many other applications. For example, retrieval-augmented generation with large language models often requires access to specific visual information from an external database, which can searched with a universal image embedding. With the ever-growing number of images in many aspects of modern life, such embeddings also become critical for searching large private photo collections or street-level photos at planet-scale, for example.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "62uYVShTsY", "reviewer": "Reviewer_CCei", "summary": "The paper tackles the problem of multi-domain fine-grained instance recognition/retrieval. The authors propose to train a unified backbone for all modalities with online distillation with domain-specific teachers, improving the performance compared to naive single backbone baselines and being competitive with expensive methods that utilize a number of domain-specific specialists. The authors propose a number of technical contributions in addition to the online distillation, including dynamic resampling based on a proxy of the task difficulty. UDON exhibits favorable performance compared to a number of baselines on the UnED benchmark consisting of a high number of fine-grained domain-specific benchmarks.", "questions": "1) depending on the number of classes the magnitude of the loss can be very different. How is this handled in equation [5]?\n2) an important question is whether scaling is indeed the answer for most problems. For the off-the-shelf baselines, it can be easy to test much larger backbones (e.g. DinoV2-G, Clip-Large, …). It is interesting to see if simply scaling general-purpose pre-training would be sufficient to address the issues tackled in the paper.\n3) Related to the question above, given that the performance gain between a naive unified model vs UDON is significant but not enormous, I wonder if scaling the naive single model baseline’s capacity slightly would be sufficient to address the shortcomings of training one model with many fine-grained domains.", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is very well written with clarity and sufficient details about the implementations and the intuitive explanation of the main contributions.\n- UDON exhibits improvements over strong baselines for a variety of tasks.\n- The proposed dynamic domain sampling is interesting and achieves a clear boost to the performance for challenging benchmarks like Met and GLDv2.\n - The paper includes many ablations about the different design choices which helps in understanding where the gains stem from.", "weaknesses": "- [important] The paper tackles generalization for instance recognition/retrieval systems, but the scaling axis of this question has not been studied. This is lacking since in the past few years we have witnessed scalable pre-training in terms of data and parameters being a very effective solution to many generalization problems.\n- The performance gains provided by UDON seem highly sensitive to which dataset is used for the evaluation. For example, while the dynamic sampling contribution helps two datasets, it dropped the performance of the other five. This does not mean that this contribution should be dismissed but rather we might need more work to achieve its objectives without hurting the performance of other stable benchmarks.\n- Some ablations of the projections architecture as well as the design of the MLP baseline would be a welcome addition.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> __[important] The paper tackles generalization for instance recognition/retrieval systems, but the scaling axis of this question has not been studied. This is lacking since in the past few years we have witnessed scalable pre-training in terms of data and parameters being a very effective solution to many generalization problems. ... an important question is whether scaling is indeed the answer for most problems. For the off-the-shelf baselines, it can be easy to test much larger backbones (e.g. DinoV2-G, Clip-Large, …). It is interesting to see if simply scaling general-purpose pre-training would be sufficient to address the issues tackled in the paper.__\n\nWe examined larger variants of the off-the-shelf models used in this work, like the CLIP ViT-Large and the DINOv2 ViT-Large, both using a 1024 dimensional embedding. We present their results below:\n\n| off-the-shelf model|mean P@5(%)| mean R@1(%)|\n|--|--|--|\n| CLIP ViT-Base (768-D)    |39.8|53.5| \n| CLIP ViT-Large (1024-D)    |44.5| 58.3|\n| DINOv2 ViT-Base (768-D)  |43.9| 58.2| \n| DINOv2 ViT-Large (1024-D)  |46.7|60.8|\n| UDON ViT-Base (64-D, CLIP pretrained)|56.9|67.7|\n\nThose larger models are still underperforming the smaller ViT-Base UDON model with 64-dimensional embedding.\nThis result shows that simply scaling the general-purpose foundational models is not sufficient to address the issues tackled in the paper.\nWe will add the corresponding results and discussion in the final version of the paper.\n\n> __Related to the question above, given that the performance gain between a naive unified model vs UDON is significant but not enormous, I wonder if scaling the naive single model baseline’s capacity slightly would be sufficient to address the shortcomings of training one model with many fine-grained domains.__\n\nThis is a valid point raised by the reviewer.\nIn order to examine if the performance gain achieved by UDON with a ViT-Base (over the baseline USCRR) diminishes if we scale the backbone size, we performed additional experiment with the larger backbone size of ViT-Large (IN21k pre-trained).\nThe results are shown in the following table:\n\n|model| mean P@5(%)| mean R@1(%)|\n|----|-----|-----|\n| USCRR (baseline) ViT-Base |51.4|62.5|\n| USCRR (baseline) ViT-Large |51.0|62.4|\n| UDON ViT-Base|53.9|65.3|\n| UDON ViT-Large|54.6| 65.4| \n\nWe observe that the baseline with larger backbone achieves almost the same performance as its smaller ViT-Base counterpart, indicating that larger backbone size doesn't necessarily mean better performance (note that a similar observation is made in [44] Appendix A, section A.2). Additionally, the UDON trained ViT-Large achieves a performance which is a bit better but close to the ViT-Base counterpart, indicating both the effectiveness of the UDON training procedure for the larger backbone size compared to the baseline training procedure, as well as the fact that the ViT-Base achieves a very good size-performance tradeoff.\nWe will include this discussion in the final version of the paper.\n\n> __The performance gains provided by UDON seem highly sensitive to which dataset is used for the evaluation. For example, while the dynamic sampling contribution helps two datasets, it dropped the performance of the other five. This does not mean that this contribution should be dismissed but rather we might need more work to achieve its objectives without hurting the performance of other stable benchmarks.__\n\nAssume one (meaningful) sampling strategy, say Round Robin, which oversamples some domains and undersamples others. \nIf with a new strategy some domain is sampled more than it was before, the model is expected to perform better on that domain, and the other way around. The proposed Dynamic Sampling is not perfect, but it improves the difficult domains where previous work was performing poorly and leaves the others close to the original perfomance, and most importantly improves the average performance, moving closer towards the notion of a \"Universal Image Embedding\", which works well on all of the different subdomains.\n\n> __depending on the number of classes the magnitude of the loss can be very different. How is this handled in equation [5]?__\n\nThe Dynamic Sampling is designed to sample the difficult domains more often. There is no explicit guarantee that domains with more classes must have higher learning error. Large number of well separable classes may have smaller error that a few hard-to-distinguish classes. In general, domains with higher number of classes have higher chance to contain difficult sets of classes - and thus should be sampled more often. Therefore, the number of classes is not explicitly accounted for in equation [5].\nAdditionally, a number of sampling strategies and variations on the dynamic sampling have been tried, all yielding similar or worse performance on the validation set.\nAt the moment, the Dynamic sampling operates on the level of domains. Ideally, one would like to analyze the training data and sample difficult classes (and their confusers) more often, which might be time demanding. An efficient approach for such sampling is left for future work. \n\n> __Some ablations of the projections architecture as well as the design of the MLP baseline would be a welcome addition.__\n\nWe performed an experiment where we replace the linear layers used as the projection for both the domain-specific teachers and the universal student by a deeper network. More specifically, we use a one hidden layer MLP with layernorm and GELU activation (standard in the literature), with the same hidden dimension as the final dimension, i.e. 256 for the teachers and 64 for the student. The obtained results are shown below (IN21k pre-trained ViT-Base):\n\n| model | mean P@5(%) | mean R@1(%) |\n|---------|------|-----|\n| UDON |53.9|65.3|\n| UDON MLP projectors|51.8|63.5| \n\nThe results indicate a significant drop compared to using the proposed linear layers. We plan to include this study in the final version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y6TteSq62U", "reviewer": "Reviewer_xwWg", "summary": "The paper proposes Universal Dynamic Online distillatioN (UDON), which is a multi-teacher distillation method designed for universal image representations. UDON adopts a knowledge distillation strategy by distilling information from multiple teacher model trained for different domains to a student model to learn the universal embedding. It also proposes a dynamic domain sampling strategy for balancing the different domains. The provided experimental results verify its effectiveness.", "questions": "See weakness.", "limitations": "See weakness.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The proposed design is simple and elegant. The authors describe the design and implementation in details, which makes it easy to follow. \n\nThe experimental results also demonstrate its effectiveness. In addition, many ablation experiments are conducted to give insight. \n\nI suppose the paper is valuable enough to be accepted.", "weaknesses": "**The reason why the proposed method is better than the previous work (USC) is not straightforward and clear enough for me.**\nIn the paper, the main baseline is Universal Separate Classifier Training method (USC). It uses a backbone to learn an universe embedding where multiple classifier heads for different domains are trained separately with the universe embedding as input. Compared with USC, UDON introduces distillation and dynamic domain sampling. I understand the part about dynamic domain sampling but feel confused about the distillation. As shown in Figure 2, UDON uses multiple extra classifier heads as teachers for distillation. Since the teacher heads and student head are all based on the previous embedding $E_b$, why the distillation strategy can help to build an better universal embedding $E_u$?  \n\nThe authors explain the reason mainly on [Line 37 - Line 51] and [Line 164 - Line 169], while I am still confused about it.\nOn [Line 37 - Line 51], it says that \"it is difficult to encode detailed knowledge about many image domains in a single model\". Therefore, the authors propose to use knowledge distillation between a student and teachers model for different domains, while the student and teachers share the backbone. Does it mean that the difficulty remains for USC, but it is solved by UDON with the distillation? The story seems a bit conflicting to me. Because from my point of view, a backbone with more classifier heads is still a single model, since the heads are mostly very light-weight compared with the backbone.\n\nIn my opinion, the proposed distillation design is filtering the information from $E_b$, which keeps more useful and universal information for the student head to learn $E_u$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comment about how the UDON method works compared to the USC baseline. We will provide some clarifications and explain our interpretation of the key components that allow UDON to produce a better universal embedding than previous work, based on the findings from the experiments of the paper.\n\nThe extra classifier heads used in UDON as teachers, mentioned by the reviewer, each consist of a shallow domain-specific projection (which produces the domain-specific embeddings) plus a linear classifier. They stem from the same backbone embedding $E_b$ as the universal embedding $E_u$ does, and they are not necessarily lightweight (given the high dimensionality of the domain-specific embeddings + the large number of classes in the different subdomains). We will refer to them as domain-specific heads from now on. \n\nBuilding from USC + Dyn. Sampler for a fair comparison, appending these domain-specific heads on the side of the universal embedding that share the same backbone and training them with classification training loss from domain-specific data already results in a performance boost for the universal embedding; see the experiment in Ablation studies Table 2, rows 7 and 2. We hypothesize that these extra domain heads backpropagating through the shared backbone act as a form of regularization.\n\nOn top of that, adding the distillation losses to obtain the full UDON method results in a large performance boost (Table 2, rows 7 and 3 of the table). We hypothesize that the domain-specific embeddings have additionally captured features that the universal embedding would otherwise skip. These features are used by UDON as an extra supervisory signal (teachers) to the universal embedding in the form of similarities between embeddings and learnable class prototypes.\n\nIn UDON, not allowing the teachers to utilize their own backbone is not only efficient in terms of compute, but allows for a higher-performing distillation process. Our experiments show that the UDON teachers (the domain-specific embeddings) underperform domain-specific embeddings that utilize their own backbone (8 separate fixed teachers case) when evaluated on their own domain (see Table 4 row 1 vs 3). However, the resulting student universal embedding of UDON performs better than the corresponding universal embedding produced by the distillation from 8 separate fixed teachers (see Table 3 row 1 vs 3, subsection 4.4). We hypothesize that distilling from the UDON teachers is an easier task than distilling from different networks that utilize their own backbone, as the UDON teachers stem from the same embedding and are only a shallow transformation of it, so the domain-specific spaces they capture are more compatible with each other.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "slMPLl3S0u", "reviewer": "Reviewer_uq4A", "summary": "The paper introduces a novel method for enhancing universal image embeddings through a multi-teacher knowledge distillation approach. The method, named UDON, employs a dynamic sampling technique and a shared backbone across multiple domain-specific teachers and a universal student model to efficiently distill domain-specific knowledge into a universal embedding. This embedding model can work as a foundation for many downstream task models, such as classification, retrieval, generation, etc., with a high potential impact on the community.", "questions": "N/A", "limitations": "N/A", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1) The introduction of multi-teacher distillation with a shared backbone is novel and addresses significant challenges in universal image representation.\n\n(2) The dynamic sampling method that adapts based on domain-specific performance enhances learning efficiency and addresses the imbalance in training data distribution.\n\n(3) Extensive experiments demonstrate that UDON outperforms existing methods, showcasing the effectiveness of the proposed techniques.", "weaknesses": "(1) This paper misses a series of very important works for embedding learning [1,2]. The [1] proposed the Matryoshka representation learning (MRL) that the OpenAI's recent embedding model adopts. MRL would unfold the embedding model into multiple dimensions such as 64, 128, 512 or more. This paper simply takes the 64 and cannot be extended to higher dimensions which significantly limited its potential in applications.\n\n[1] Kusupati, Aditya, et al. \"Matryoshka representation learning.\" Advances in Neural Information Processing Systems 35 (2022): 30233-30249.\n\n[2] Cai, Mu, et al. \"Matryoshka Multimodal Models.\" arXiv preprint arXiv:2405.17430 (2024).\n\n(2) Tab 1 is pretty confusing. There are \"Off-the-shelf\", \"Specialist+Oracle\", \"ImageNet21k pretraining\" and \"CLIP pretraining\". It is hard to understand the exact meaning of them. Moreover, the evaluation is not convincing. It misses some recent SOTA embedding models, such as SigLIP.\n\n(3) Please unify the format of subtitles. In Line 253, there is \"Implementation Details\" but line 235 applies \"Compared methods\".\n\n(4) There is no ablation study over the temperature and embedding dimension, which should be critical hyperparameters to explore.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> __This paper misses a series of very important works for embedding learning [1,2]. The [1] proposed the Matryoshka representation learning (MRL) that the OpenAI's recent embedding model adopts. MRL would unfold the embedding model into multiple dimensions such as 64, 128, 512 or more. This paper simply takes the 64 and cannot be extended to higher dimensions which significantly limited its potential in applications.__\n\nThe main focus of our paper is on improving universal embedding models, following the setup proposed by [44]. Note that in the experimental setup of [44] the embedding dimension must be fixed to 64, which is the reason we chose this dimensionality. Regarding MRL, it is a technique proposed to enable adaptive embedding dimensions in deployment. We agree that this is an interesting direction, but it is not directly related to the topic explored in our paper, which proposes techniques to improve universal image embeddings in general. Combining UDON and MRL in the future would be an interesting exploration.\n\n> __Tab 1 is pretty confusing. There are \"Off-the-shelf\", \"Specialist+Oracle\", \"ImageNet21k pretraining\" and \"CLIP pretraining\". It is hard to understand the exact meaning of them.__\n\nThe meaning of different settings is explained in section 4.1 \"Compared methods\", 4.2 and the caption of Table 1. The concept of the table is taken from [44 - Table 4].\n\n> __Moreover, the evaluation is not convincing. It misses some recent SOTA embedding models, such as SigLIP.__\n\nAs mentioned in the \"Compared methods\" paragraph in Subsection 4.1 \"Experimental settings\", the Universal Embedding task and the corresponding UnED benchmark is only recently created, and we have done our best to include as many baselines as we can into the evaluation. We appreciate the reviewer additionally pointing us to the recent SigLIP model. We evaluated the SigLIP ViT-Base off-the-shelf model, which achieves a mean P@5 of 49.0% and a mean R@1 of 62.0% on the UnED benchmark. We will add it to the final version of the paper.\n\n> __Please unify the format of subtitles. In Line 253, there is \"Implementation Details\" but line 235 applies \"Compared methods\".__\n\nThank you for this one, good catch! We will fix this by replacing \"D\" with \"d\".\n\n> __There is no ablation study over the temperature and embedding dimension, which should be critical hyperparameters to explore.__\n\nThe value of the temperature hyperparameter was tuned on the validation set independently of other hyperparameters and kept fixed. For the rebuttal, we provide additional numbers alternating the temperature value in the full UDON method (IN21k pre-trained): \n\n| temperature value |mean P@5(%)|mean R@1(%)|\n|------|------|------|\n| 0.1 (paper)  |  53.9  | 65.3   |\n| 1.0    | 53.6  |  65.0  |\n| 0.05  | 53.2  | 64.8 |\n| 0.01  | - (Diverged)  | - (Diverged)  |\n\nall of which perform significantly worse than the one used in the paper. We will add this study to the final version of the paper.\n\nFor the teacher embedding dimension, there is already an ablation in the paper, see Table 2 row 5, and corresponding paragraph \"Online teacher dimensionality\" in Subsection 4.3.\nRegarding the student embedding dimensionality, we keep it fixed to 64, given the constraints of the UnED benchmark [44], as discussed above.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their constructive feedback.
We are encouraged that they recognize our contributions as novel (**Mmnc**, **uq4A**), interesting (**CCei**), and simple/elegant (**xwWg**). Reviewers also highlight the value of our experimental validation/ablations (**Mmnc**, **CCei**, **xwWg**, **uq4A**), demonstrating the method’s effectiveness/improvements against previous work (**CCei**, **xwWg**, **uq4A**).
We are also glad they found the paper well-written (**Mmnc**, **CCei**), making it easy to follow (**xwWg**).
We address concerns and comments in individual responses to each reviewer below.

Meta-review: This submission proposes an online distillation method to learn universal representations for multi-domain fine-grained image retrieval. The proposed method uses a shared vision transformer backbone to extract feature embeddings over images in different domains, then adds separate linear projection heads on each individual domain to construct a set of domain-specific teachers, which is used to guide the learning of a universal student representation (with another linear projection head) facilitated by a simple dynamic domain sampling strategy. The submission was finally scored (5,6,6,3) by three knowledgeable reviewers and another reviewer (scored 5 but with short and uninformative comments), who mostly acknowledged the novelty and the performance of the proposed method, and also raised some concerns about **1)** more comprehensive experiments, especially to consider data/model size scaling; **2)** more thorough experimental comparison with existing embedding learning methods; **3)** missing some ablations regarding the effect of different projector architectures in the head and different projected feature dimensions;  **4)** in-depth explanation of why multi-head domain-specific teachers are helpful to improve single-head multi-domain student, under the context of sharing a vision transformer backbone; **5)** in-depth analysis of why the proposed method is superior to counterpart methods.

The authors provided detailed responses to these concerns. Generally, two positive reviewers recognized that most of their major concerns are addressed. The negative reviewer uq4A acknowledged the authors' rebuttal, but criticized the paper's novelty and performance in final judgement, constituting a contradiction compared to the original judgement in which these points were commented as the strengths. The AC read the paper, the reviews, the rebuttal and the reviewers' feedback, and mostly agree with two positive reviewers' assessment. Therefore, I recommend to accept (poster), also considering that the novelty of the proposed method is not very strong (in online knowledge distillation research, the basic concept of using the shared network backbone and multi-head branches is not new, e.g., ONE in NeurIPS 2018 and many of its variants). The authors are encouraged to carefully consider the reviewers' comments/suggestions and their rebuttal in the final paper revision.
Decision: Accept (poster)