UDON: Universal Dynamic Online distillatioN for generic image representations

Authors: nikolaos-antonios  ypsilantis
kaifeng  chen
andré  araujo
ondřej  chum
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 23:01:02

Abstract: Universal image representations are critical in enabling real-world fine-grained and instance-level recognition applications, where objects and entities from any domain must be identified at large scale.
Despite recent advances, existing methods fail to capture important domain-specific knowledge, while also ignoring differences in data distribution across different domains.
This leads to a large performance gap between efficient universal solutions and expensive approaches utilising a collection of specialist models, one for each domain.
In this work, we make significant strides towards closing this gap, by introducing a new learning technique, dubbed UDON (Universal Dynamic Online distillatioN).
UDON employs multi-teacher distillation, where each teacher is specialized in one domain, to transfer detailed domain-specific knowledge into the student universal embedding.
UDON's distillation approach is not only effective, but also very efficient, by sharing most model parameters between the student and all teachers, where all models are jointly trained in an online manner.
UDON also comprises a sampling technique which adapts the training process to dynamically allocate batches to domains which are learned slower and require more frequent processing.
This boosts significantly the learning of complex domains which are characterised by a large number of classes and long-tail distributions.
With comprehensive experiments, we validate each component of UDON, and showcase significant improvements over the state of the art in the recent UnED benchmark.
Code: https://github.com/nikosips/UDON.

Main paper: 
 introduction: 
 Imagine you point your cellphone at anything, and it tells you what it is, be it tangerine chicken with rice, Mk1 Volkswagen Rabbit Cabriolet, statue of Aquaman, Pasadena City Hall, or Yorkshire Terrier. Such a product is the ultimate goal of fine-grained and instance-level visual recognition. The key component enabling such an application is a general-purpose image representation, or equivalently image embedding, designed to handle imagery of varied domains at scale. Traditionally, image embedding models have been developed for specific domains separately [29, 34, 17, 9] , such as landmarks [32] , products [41] , clothes [22] , faces [40] , to name just a few. However, as visual recognition applications grow in popularity and scope [47, 1, 2] , it is impractical to handle images of different object types with specialized, per-domain models. A potential solution for this problem is to leverage recent foundation models, such as CLIP [33] or DINOv2 [26] , which have been proposed to enable a wide variety of multimodal applications. Even though these models possess a broad visual understanding, they tend to lack detailed fine-grained knowledge off-the-shelf [45] , which is critical in practice. For this reason, recent efforts aim at developing universal embedding solutions that can generalize to handle multiple fine-grained object types with a single model [45, 39] , ensuring scalability in real-world scenarios. Training of a universal embedding on multiple fine-grained visual domains. The baseline approach of [45] (left) uses classification loss across training classes from all domains. It is prone to cancelling out contradicting cues from different domains. To overcome this issue, a naive multi-teacher distillation approach (middle) first trains one specialized teacher per domain (with a classification loss) to capture domain specifics, then distils them to the universal embedding (student).Our proposed Universal Dynamic Online distillatioN -UDON (right) jointly trains the specialized teacher embeddings and the universal embedding (student) with classification and, at the same time, distills the teacher embeddings to the universal embedding. Due to joint training of a shared backbone, UDON scales to a large number of domains.There are two main challenges in training such universal models addressed in the paper. First, it is difficult to encode detailed knowledge about many image domains in a single model. In several cases, features that are helpful for one object type may be useless for others. One example is the importance of color: while for food recognition it is critical to discriminate red curry from green curry, when recognizing car models both a red and a green Toyota Corolla LE 2024 would belong to the same class. This makes the training of universal models from data of different domains particularly hard, since the data may present conflicting peculiarities across domains, leading to sub-optimal learning. To address this issue, we propose a novel knowledge distillation approach, where one teacher model is trained for each domain individually, and a student universal embedding model learns from the collection of teachers. This setup allows the specialized teachers to capture domain-specific knowledge, which is then transferred into the universal embedding. However, if this distillation setup is deployed naively, one may incur substantial costs, as a separate teacher model would need to be trained for each domain.For this reason, we propose to share the backbone between all teachers and the universal student, reaching a solution that achieves high performance and incurs small additional training costs, as all models are jointly trained in an online manner -see Figure 1 .The second main challenge we highlight is that different domains may present data with vastly different distributions: e.g., while one domain may present a moderate number of classes with a roughly balanced number of samples, in others, the number of classes may be very large, and the distribution of samples long-tailed. This means that different domains may require different training curricula for a model to properly learn their characteristics. This leads us to propose a sampling technique that dynamically selects which domains will be processed at each point in the learning process based on the losses measured on the fly. With this method, we demonstrate significant improvements to the performance of the most challenging domains, which are learned slower and require more frequent processing compared to other domains.Contributions. To summarize, in this work we introduce the following contributions. (1) We leverage knowledge distillation to infuse the universal embedding model with the learnings of specialist, perdomain teacher models. In our novel training setup, the model backbone for all teacher embeddings and the student universal embedding is shared, leading to an efficient method where all models are jointly trained in an online manner. Our findings show that sharing the backbone facilitates the distillation process significantly, even reaching performance that surpasses the distillation from separate specialist teachers. (2) To enable an appropriate training regime in a universal embedding setup, we propose to adapt the learning process dynamically, adjusting the sampling of image domains based on their losses, which appears suitable to visual knowledge spanning a range of fine-grained domains. We show that this can help substantially with more complex domains that require more frequent model updates to enhance their performance. (3) We perform comprehensive experiments on the recent UnED [45] dataset, that highlight the value of each proposed component, as well as compare our technique against competitor approaches. Our complete method, named UDON (Universal Dynamic Online distillatioN), showcases state-of-the-art results that boost Recall@1 by up to 2.3%.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: The claims about the contributions of the paper made in the abstract and the introduction are supported by the experimental results provided. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Knowledge distillation (KD). Initially proposed to transfer the knowledge of large and complex models to smaller and faster ones [14] , standard KD trains a light student to mimic the softmax outputs produced by a heavy teacher. Tailoring KD to representation learning, [27, 28] distill relations between image representations and [19] focus on retrieval rankings. Online KD [12] lifts the need for separate two-stage training for the teacher and the student and trains them simultaneously. Multiteacher KD [20, 15] aims to transfer the knowledge of multiple teachers into one student model. [24] performs multi-teacher KD for a single domain visual retrieval, by aggregating the teacher relations in a single target that the student should mimic. Differently than [15] , which proposes an online multiteacher distillation approach that trains a different backbone for each teacher, we propose to share a common backbone between all the teachers and the student, showing improved performance while also being much more efficient. [50] combines online and multi-teacher knowledge distillation in a single multi-branch network, training an ensemble of teachers on the fly. Differently from [50, 24] , each of the teachers in our KD approach is specialized to only a fraction of the data (a single domain), being relevant only to part of the universal embedding task. Similarly to [50] , we also create an online multi-branch architecture, however the teachers are not updated by the other teachers' knowledge, as they are related to different visual domains. Additionally, our teachers transfer relational knowledge to the student, since our focus is on learning image embeddings, in contrast to [50] which only focuses on classification.Universal representation learning. Learning a representation that generalizes and can be reused efficiently across visual domains is a long-standing goal in computer vision. [4] introduces domainspecific normalization to make classification networks generalize to multiple visual domains, while [35] introduces adapter modules to improve the accuracy of domain-specific representations. [23] proposes a multi-task vision and language universal representation trained on 12 different datasets. However, this body of work assumes knowledge of the test time domain, which does not hold in our setup. Recent large visual foundational models [26, 33] that are trained on large amounts of data with diverse objectives show great zero-shot performance on a number of downstream visual tasks, making them great candidates for universal embedding applications. However, [45] shows that these models, even though generalizing to many diverse domains, cannot effectively handle instance-level and fine-grained domains (which are the focus of this work) without further fine-tuning. [37] shows that appending an MLP projector between the objective and the representation used for the downstream tasks improves the generalisability of the representation, inspiring a multi-domain variant we use as a baseline in this work. In [3] , a multi-domain representation for fine-grained retrieval is learned, which utilizes no labels for training. Differently from it, we focus on the supervised task setup of [45] , which constitutes a much larger-scale problem that additionally includes instance-level domains. [20] introduces distillation as a way to learn universal representations, while [10] tailors multi-teacher distillation to universal embedding learning. Differently from [20, 10] , we do not use task-specific backbones that are costly to scale across a large number of domains. While [10] tackles a universal embedding setup, the effectiveness of their method is only assessed on a small dataset, where three small domains at a time are distilled into a universal representation. In contrast, we tackle learning in a more practical large-scale setup, with an efficient approach that distills knowledge from eight diverse visual domains into the universal embedding. Recently, the UnED dataset was introduced in [45] as a new large-scale benchmark for universal embeddings. Their experiments considered the training of models only via classification objective, with different sampling and classifier configurations.Setting our approach apart is that we go beyond to capture detailed knowledge from diverse domains via distillation, besides proposing a more suitable training dynamic that can accommodate data from diverse domains. Concurrently, UNIC [38] proposes a universal classification model using multi-teacher distillation. While our approach trains a student embedding from multiple teachers specialized in fine-grained visual domains, UNIC distills foundational models trained for diverse tasks, such as semantic segmentation and classification, with both supervised and self-supervised objectives.Dynamic sampling. When training in a multi-task setting, the sampling frequency of the different domains can greatly affect final performance. Poly-ViT [21] explored different samplings tailored to their multi-modal multi-task model, and concluded that sampling each domain with a weight proportional to the number of training steps that a corresponding specialized model needs to achieve maximum performance works the best, while [45] additionally comes to the same conclusion for the task of universal image embedding. This approach is costly with an increasing number of domains in hand, as one model for each domain needs to be trained, which inspires us to discover more efficient sampling strategies. [23] proposes Dynamic Stop-and-Go sampling, which updates the sampling weight of each domain based on the validation set accuracy. Differently from them, we propose to calculate the sampling weights only based on training loss, which doesn't require the expensive feature extraction of the validation set but can happen on the fly. A similar idea has been explored by [31] in the context of pre-training vision-language models, which is far from this work's, as we focus on learning multi-domain fine-grained image embeddings.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: Please refer to the Limitations section after the Conclusion. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
proposed method: 
 This section presents our proposed training method, Universal Dynamic Online distillatioN (UDON), to learn the universal image embedding. UDON utilizes a pretrained Vision Transformer [8] as the image encoder, which is further fine-tuned with a combination of classification and distillation objectives. First, some preliminaries concerning the backbone architecture that we build upon are introduced, and afterward, the complete training pipeline is presented in detail.The Vision Transformer [8] backbone produces the [CLS] token as a global representation of the image. Let e b : X → R D denote the Vision Transformer as a function that takes an input image x ∈ X and maps it to the [CLS] token e b (x) ∈ R D , compactly denoted as e b . The dimensionality D is backbone dependent and usually higher than the one required in the downstream task, hence projection to lower dimensional space is introduced. The final vector after projection is the universal embedding, denoted as e u ∈ R d , d < D. Following standard practice used in image retrieval architectures [11] , e b and e u are ℓ 2 normalized. When referring to a batch of embeddings, we use capitalized notation, e.g., a batch of embeddings e u is denoted E u ∈ R d×B , where B is the batch size.For training with a classification loss on top of the universal embedding in the multi-domain setup, a Separate Classifier (SC) per domain is employed, classifying across the classes of that specific domain, an option justified by [45] . In the following, the word "head" denotes both the projection to the embedding space and the classifier to which the projected embedding is input.Our UDON training approach introduces an efficient multi-teacher distillation method, relying on a shared feature extraction backbone. The entire training pipeline is presented in Figure 2 . The backbone produces the initial high dimensional image embedding, e b , for the samples of all domains. Given e b , in addition to projecting it into the universal embedding space e u that is used at test time (as in [45] ), we also project e b to per-domain spaces, which constitute the teacher embeddings {e ti ∈ R Dt } i . Teacher i is only activated for samples of domain i. Both the universal and the domain-specific (teacher) projections are realized by linear layers, and there is a domain-specific projection for each domain.The universal embedding is trained with both classification and distillation objectives, calculated on batches containing a single domain at a time. While training with a classification objective allows grasping broad knowledge for several domains, it may lead to a sub-optimal model due to contradictory cues when combining data from all domains. Therefore, we employ distillation from the domain-specific teachers to infuse the universal model with domain-specific knowledge. The loss functions are presented below. Classification losses. The universal and the domain-specific embeddings are trained with classification losses (Normalized Softmax Loss [48] ), instantiated with separate classifiers for each domain. These losses update the backbone via gradients propagated through the teacher and the student heads, while teacher-specific or student-specific parameters are only updated via gradients from their respective heads. The i-th teacher (of domain i) classification loss L ti cls and the universal embedding (student) classification loss L u cls are defined as:Classifier W u ϵ R 64xC Classifier W u ϵ R 64xCEQUATIONwhere B is the batch size, y j is the one-hot ground truth vector of sample j, ŷti j is the predicted probability vector for sample j produced from teacher of domain i and ŷu j is the predicted probability vector for sample j produced from the universal embedding (student). It is important to note that the classifiers of the student and the teachers are different and that the student employs as many classifiers as the number of domains (SC).Distillation losses. The student is tasked to match the teacher embedding of the corresponding domain by enforcing two separate distillation losses. The first is a relational distillation loss, which acts on batch similarity matrices. Given the student's batch embeddings E u ∈ R d×B , its batch similarity matrix is formed asE ⊤ u E u ∈ R B×B .Similarly, for the i-th teacher's batch embeddings E ti ∈ R Dt×B , its batch similarity matrix is formed asE ⊤ ti E ti ∈ R B×B .The goal is for the student to learn detailed intra-domain similarities from the more powerful domain-specific teacher. Specifically, the student's intra-domain cosine similarities are encouraged to follow the i-th teacher's cosine similarities, when the batch of images comes from domain i:L ti rel = ||E ⊤ u E u -E ⊤ ti E ti || 2 . (2)Additionally, the student is tasked to match its logits to the teacher's, minimizing their KL divergence, after scaling with identical temperature T and softmax normalization of both:EQUATIONwhere l u is the logit vector produced by the classifier on the student's side and l ti is the logit vector produced by the classifier on the i-th teacher's side. The temperature T is shared across all teachers and the student. This loss provides a more global context, as it captures the similarities between an embedding and all of the class prototypes in the domain (which exist in the classifier), instead of only relating embeddings in a batch. Both distillation losses do not backpropagate through the domain-specific teacher head, as only the student should try to learn from the teacher. Distillation starts at the beginning of the training and it happens in an online manner, at the same time that the universal student and the teachers are trained with the classification losses. The total loss for a training batch, containing images of domain i, is as follows:EQUATIONDynamic domain sampling. Each domain comes with its own training data, which differ in the number of classes and in the number of examples. Balancing of the domains is performed through sampling of the training data. In [45] , training is performed with clean batches, i.e. each batch only contains examples from a single domain. Three different sampling schemes were compared in [45] .In Dataset Size sampling, the datasets are sampled proportionally to their size, biasing towards large datasets. In Round-Robin (RR) sampling, the domains are sampled equally often in a cyclic order. The Specialist Steps sampling selects the domains proportionally to the number of steps the specialist at the particular domain needs to achieve maximum validation performance. Only the last approach takes into account the difficulty of the classification task in each domain. However, such sampling is static, does not reflect possible correlations in similar domains, and requires the training of a specialist for each domain prior to training the universal model.We propose to sample each domain proportionally to the classification loss produced by the domainspecific embedding during training, relative to the corresponding loss of the other domain-specific embeddings. This way, the domains for which the training loss decreases slower than for other domains are sampled more for training. More specifically, we sample the domain of the next training batch out of the distribution which for each domain m assigns the probability:EQUATION)where N is the number of domains, train loss m is the classification loss produced by the embedding of domain m, and the denominator represents the sum of the corresponding losses across all domains. This distribution is updated after every S steps, a hyperparameter that is tuned on the validation set.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [No] Justification: The code and the data used to perform the experiments of this work are not included as part of the submission. The code will be made available with Open Access in the case of the acceptance of the paper, while the data is already available through the website of the UnED dataset. Guidelines:• The answer NA means that paper does not include experiments requiring code. Justification: The paper focuses on generalization of the retrieval method to a large variety of domains. This opens doors for a number of application with positive societal impact. If there was any negative societal impact of a retrieval application, existing domain-specific approaches would be more suitable.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
experimental settings: 
 Dataset. The proposed method is evaluated on the recent Universal Embeddings Dataset (UnED) [45] , the largest dataset for multi-domain fine-grained retrieval. It comprises 4.1M images, with 349k classes distributed across 8 image domains: food (Food2k dataset) [25] , cars (CARS196 dataset) [18] , online products (SOP dataset) [41] , clothing (InShop dataset) [22] , natural world (iNat dataset) [43] , artworks (Met dataset) [46] , landmarks (GLDv2 dataset) [44] and retail products (Rp2k dataset) [30] . We follow the train-validation-test splits and the evaluation protocol defined in [45] , a brief review follows. Each index image and each query in the test set are described by a 64-D (universal) embedding, and Euclidean nearest neighbors in the embedding space are found for each query among the index set. The index contains images from all 8 domains combined; hence, cross-domain false positives are possible. Performance is measured by two metrics: Recall@1 (R@1), which is equivalent to the correctness of the top neighbor, and modified Mean Precision@5 (P@5) [45] . The average performance over the queries in each domain is reported, as well as the balanced average of these values across all domains.Compared methods. The universal embedding task is relatively new and only a few baselines were published in [45, 6] . We extend these baselines by re-purposing the single-domain embedding method of [37] (Table 1 ). Additionally, we also compare to two straight-forward multi-domain distillation methods [10] (Table 3 ). The main baselines compared with this work are the best-performing methods from [45] , namely USCRR, UJCRR, and USCSS, which vary the classifier setup (SC: Separate Classifier, JC: Joint Classifier) and the sampling (RR: Round Robin, SS: Specialist Steps). We also evaluate a variant of USCRR, which uses the proposed dynamic sampling scheme, dubbed "USC w/ Dyn Sampler". The USCRR [45] method (USC w/ Dyn Sampler) is further expanded by Food2k CARS196 SOP InShop iNat Met GLDv2 Rp2k Mean Model P@5 R@1 P@5 R@1 P@5 R@1 P@5 R@1 P@5 R@1 P@5 R@1 P@5 R@1 P@5 R@1 P@5 R@1 Off-the-shelf IN [42] inserting an MLP projector [37] between the universal embedding and its a classifier for each domain.The projectors consist of three hidden layers of sizes 256, 256, and 512 respectively. This new baseline method is referred to as the "MLP baseline". We compare with the off-the-shelf embeddings from ImageNet21k (IN) [42] , finetuned ImageNet21k model with masked image modeling (IN+MIM) [6] , CLIP [33] , DINOv2 [26] , and SigLIP [49] , which utilize embeddings of much higher dimensionality (768D vs 64D). Lastly, we compare against the Specialist+Oracle baseline, a non-realistic model proposed in [45] to get a hypothetical estimate of the maximum performance that can be achieved on each individual domain, by choosing the specialist of the query's domain as the embedding for both the query and the index set. All of the methods (including UDON), use the ViT-Base/16 backbone, and additionally, apart from the off-the-shelf ones, are fine-tuned on the training set of UnED. Implementation details. For fair comparisons with the baselines of [45] , identical values for common hyperparameters are used. The newly introduced hyperparameters are tuned based on performance on the validation set of UnED. For the KL divergence loss (3), the value of temperature T is set to T = 0.1 (a discussion regarding this choice can be found in the Appendix); the teacher embeddings have dimensionality of D t = 256; the four loss components contribute equally to the total loss L total (no weights need to be tuned). We set the universal student embedding dimensionality to d = 64 for direct comparability against previous work. The batch size is set as B = 128. The hyperparameter S for the number of steps, after which the dynamic sampler is updated, is set to 1000. Each experiment is repeated 3 times with different seeds; the reported values are averaged over those runs. The standard deviations are reported in the Appendix. The ViT-Base/16 variant of the Vision Transformer is used as the backbone with ImageNet21k [42] and CLIP [33] initializations. The linear projections are initialized randomly, as well as the corresponding classifiers. Our implementation is based on the Scenic framework [7] , a library based on Jax [5] /Flax [13] . Experiments are executed on Google Cloud TPU v4s [16] .In Table 1 , we present the performance of UDON and the compared methods. For the full UDON method, we present results for two different pretrainings for more complete comparisons, namely ImageNet21k [42] , and CLIP [33] . For the "MLP baseline" and the " [45] USC w/ Dyn Sampler" baseline, we present results for ImageNet21k pretraining only.On average, as well as on most of the individual domains, the proposed UDON method achieves stateof-the-art performance, for both types of pretraining (ImageNet21k and CLIP). For ImageNet21k, it achieves an improvement of 1.5% and 2.3% on mean P@5 and R@1, respectively, over the previous state-of-the-art. For CLIP, it achieves an improvement of 1.9% and 1.8% on mean P@5 and R@1, respectively, over the previous state-of-the-art. The biggest improvements are observed in the Met, GLDv2 and iNat domains, all of which are characterized by a large number of classes and long-tail distribution. Our proposed method makes notable progress towards closing the performance gap to the Specialist+Oracle baseline, coming as close as 1%-1.3% for the respective metrics, for ImageNet21k pretraining. The MLP baseline [37] with the addition of dynamic sampling ("MLP baseline w/ Dyn Sampler") shows comparable performance on average with the previously reported state-of-the-art methods of [45] and the baseline method USCRR with Dyn Sampler instead of RR sampling ("USC w/ Dyn Sampler"), showing that it is not trivial to extend the conclusions of [37] to the multi-domain embedding setting. The proposed UDON method achieves an improvement of 2.9% and 2.5% on mean P@5 and R@1, respectively, over the "MLP baseline w/ Dyn Sampler", and 2.6% and 2.1% on mean P@5 and R@1 respectively, over the "USC w/ Dyn Sampler" baseline. We additionally perform an experiment where we combine the MLP baseline [37] with UDON, by appending an MLP projector between the classifier of every domain and the universal embedding. This underperforms the UDON method by 0.6% and 0.3% on mean P@5 and R@1 respectively, while bringing significant extra cost on the number of parameters of the model. In Figure 3 qualitative results for two queries of the UnED test set are shown, for which the UDON universal embedding exhibits better retrieval performance compared to the USCRR [45] baseline embedding. A spectrum of methods employing various components of UDON are evaluated to examine the impact of each individual block. The methods range from the baseline USCRR ( P@5 R@1 P@5 R@1 P@5 R@1 P@5 R@1 P@5 R@1 P@5 R@1 P@5 R@1 P@5 R@1 P@5 R@1 Online teacher dimensionality. We perform an experiment ("64-D teachers", row 5) with dimensionality of the specific domain teachers reduced to 64D (i.e. the same dimensionality as of the universal student embedding) as compared to 256D in the full method. This results in a performance drop, which aligns with previous observations that higher dimensional embeddings can be better teachers in a distillation setting [36] .Scheduling the dynamic sampler. The sampler probability is updated every 1000 optimization steps (UDON needs ∼120k steps to converge). Our experiments show that the method is not sensitive to the choice of this parameter. The probability is updated according to the training classification loss of the current model on each domain. In fact, there are two such losses in UDON. One provided by each domain teacher's classifier, and one provided by the classification loss on the universal student's separate classifier for each domain. Using the latter to update the sampler's weights incurs a small drop in performance, as seen in ("Dyn Sampler on univ.", row 9), compared to the Full UDON method, where the domain teacher's classification loss updates the sampler.The online distillation method of UDON provides a very efficient way of transferring domain-specific knowledge to the universal embedding, without training more than a single backbone. In this section, the application of alternative distillation approaches is discussed. In particular, we implement two other approaches: first, the naive multi-teacher distillation (Figure 1 middle), with independent specialist models as teachers (8 extra backbones), where each teacher is trained in its own domain, dubbed "8 separate teachers". Second, one model with specialist heads is trained, i.e. 1 extra backbone followed by domain-specific projections (teacher embeddings), dubbed "1 separate teacher". In both cases, the teacher backbones are fixed during distillation, and the universal embedding (student) gets its own backbone. All backbones are initialized by ImageNet21k in the experiments of Tables 3 and 4 .For efficiency reasons, only relational distillation is performed in this experiment, and all the teacher embeddings are 256 dimensional, as in UDON.Universal embedding performance. The results are presented in Table 3 , indicating that our method is not only efficient, but also outperforms other variants. This finding indicates that UDON benefits significantly from sharing the backbone between the student and the teachers, even if that could limit the representation capacity of the teachers, given that they have fewer free parameters.Compute cost reduction. The alternative approaches are less efficient in terms of the number of parameters, as well as in the number of steps needed to converge. More specifically, for this setup, UDON uses ∼188 million (M) parameters, "1 separate teacher" uses ∼440M parameters, and "8 Food2k CARS196 SOP InShop iNat Met GLDv2 Rp2k Mean Model P@5 R@1 P@5 R@1 P@5 R@1 P@5 R@1 P@5 R@1 P@5 R@1 P@5 R@1 P@5 R@1 P@5 R@1 P@5 R@1 P@5 R@1 P@5 R@1 P@5 R@1 P@5 R@1 P@5 R@1 P@5 R@1 P@5 R@1 P@5 R@1 Separate Index Evaluation (Oracle) Teachers' performance. To gain a better insight, we also evaluate the performance of the teachers in their domains. The domain of test image is used as an oracle in these experiments in order to restrict the index to contain images of the same domain only, hence the reported numbers are not comparable to other reported results. Table 4 shows that the independent specialists provide the best per-domain teachers. Interestingly, although being the best performing (teachers) in their domain, the latter do not provide the best distillation outcome, which is delivered by UDON, as discussed in the previous paragraphs. We hypothesize that sharing the backbone between the teachers and the student in UDON, on the one hand limits the performance of the teachers on their individual domains, but, on the other hand, allows for more efficient distillation, as the specialist heads and the universal student operate on the same backbone embedding. Another related observation can be made from the ablation Table 2 . Row 2 ("USC w/ Dyn Sampler") and row 7 ("w/o any distillation") differ in the presence of separate domain classification heads on top of the universal student backbone in UDON, without performing distillation. In the latter method, the specialist heads provide a regularization for the backbone training, which results in improved performance. 
conclusions and limitations: 
 Conclusions. In this work, a novel multi-teacher distillation approach -Universal Dynamic Online distillatioN (UDON) -is introduced to tackle the problem of learning a universal embedding. The universal embedding and the domain-specific teachers share the backbone parameters and are trained jointly, which proves to be very efficient both in time and resources. The proposed training approach is shown to deliver high efficacy distillation, in which the universal student performs even better than distilling from separate fixed teachers. The additionally proposed difficulty-based dynamic sampling results in a significant boost of performance in complex domains which are typically characterized by a large number of classes and long-tail distributions. The proposed method improves the state-of-the-art performance on the recent UnED benchmark.Limitations. While UDON boosts universal embedding performance compared to the baseline method USCRR which only employs classification loss, it has 20% decrease in training throughput, given that it adds new parameters (in the teacher heads). Additionally, the proposed dynamic sampling significantly improves the performance in the difficult domains, such as Met and GLDv2, however, still at a cost of slightly decreased performance on other simpler domains. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The paper poses no such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: The code that is used to support the experiments are part of the Scenic framework repository [7] , which is cited in the submission. Additionally, the UnED dataset [45] is used, cited in the submission. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [No] Justification: The paper does not release new assets at the time of submission. In case of acceptance, the Code with the corresponding licenses will be released. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: In the Appendix we present results from Tables 1 to 3 of the main paper that include the standard deviations calculated across 3 randomizations per experiment. Additionally, we present results for different backbone sizes, a series of experiments to justify the choice of the temperature value in the main paper, and a study of the architecture of the embedding projectors used in the UDON pipeline. Lastly, additional qualitative results are presented.In Table 5 , the results from Table 1 of the main paper, along with the corresponding standard deviations are presented. Only the methods that were developed in this work are shown, namely the baselines "USC w/ Dyn Sampler", "MLP baseline w/ Dyn Sampler" and the proposed UDON method, for ImageNet pretraining, as well as the results for UDON method for CLIP pretraining. In Table 6 the results of the ablations from Table 2 of the main paper are presented, with the corresponding standard deviations. In Table 7 the results from Table 3 of the main paper showcasing the performance of the universal embedding under different distillation methods are presented, with the corresponding standard deviations.Food2k CARS196 SOP InShop iNat Met GLDv2Rp2k Mean Model P@5 R@1 P@5 R@1 P@5 R@1 P@5 R@1 P@5 R@1 P@5 R@1 P@5 R@1 P@5 R@1 P@5 R@1 ImageNet21k pretraining USC w/ Dyn Sampler 1 of the main paper.Table 7 : Evaluation results for the comparisons between distillation alternatives from Table 3 of the main paper, along with the standard deviations calculated across the 3 randomizations of each experiment.We present additional results for the UDON method, for a smaller backbone size, namely ViT-Small/16 in Table 8 , where we compare with the baseline method UJCRR of [45] , which is one of the best performing methods. Both methods start from ImageNet21k pretraining. The results indicate that our method is also applicable to smaller backbone sizes.In order to examine if the performance gain achieved by UDON with a ViT-Base (over the baseline USCRR) diminishes for larger backbone sizes, we performed additional experiments with the larger Vision Transformer variant, namely the ViT-Large model with ImageNet21k pre-training. The results are presented in Table 9 . We observe that the USCRR baseline with a larger backbone achieves almost the same performance as its smaller ViT-Base counterpart, indicating that a larger backbone Figure 4 : Additional qualitative results for our UDON method. We present nearest neighbours that are retrieved by the baseline (USCRR) embedding (top row) and the proposed UDON embedding (bottom row), for queries that the proposed method improves over the baseline. Each image shows the domain it comes from (underneath it). The correct neighbors are in green border, the incorrect ones are in red. For queries whose class is represented by less than 5 positives in the index, we present as many neighbors as the number of positives, since only those are taken into account for calculating the metrics.NeurIPS Paper Checklist 


Decision: This submission proposes an online distillation method to learn universal representations for multi-domain fine-grained image retrieval. The proposed method uses a shared vision transformer backbone to extract feature embeddings over images in different domains, then adds separate linear projection heads on each individual domain to construct a set of domain-specific teachers, which is used to guide the learning of a universal student representation (with another linear projection head) facilitated by a simple dynamic domain sampling strategy. The submission was finally scored (5,6,6,3) by three knowledgeable reviewers and another reviewer (scored 5 but with short and uninformative comments), who mostly acknowledged the novelty and the performance of the proposed method, and also raised some concerns about **1)** more comprehensive experiments, especially to consider data/model size scaling; **2)** more thorough experimental comparison with existing embedding learning methods; **3)** missing some ablations regarding the effect of different projector architectures in the head and different projected feature dimensions;  **4)** in-depth explanation of why multi-head domain-specific teachers are helpful to improve single-head multi-domain student, under the context of sharing a vision transformer backbone; **5)** in-depth analysis of why the proposed method is superior to counterpart methods.

The authors provided detailed responses to these concerns. Generally, two positive reviewers recognized that most of their major concerns are addressed. The negative reviewer uq4A acknowledged the authors' rebuttal, but criticized the paper's novelty and performance in final judgement, constituting a contradiction compared to the original judgement in which these points were commented as the strengths. The AC read the paper, the reviews, the rebuttal and the reviewers' feedback, and mostly agree with two positive reviewers' assessment. Therefore, I recommend to accept (poster), also considering that the novelty of the proposed method is not very strong (in online knowledge distillation research, the basic concept of using the shared network backbone and multi-head branches is not new, e.g., ONE in NeurIPS 2018 and many of its variants). The authors are encouraged to carefully consider the reviewers' comments/suggestions and their rebuttal in the final paper revision.

Reviews: review: paper_summary: paper_summary: The paper proposes an online distillation approach in a multi-teacher setup w/ weight sharing for efficiency. A strategic dynamic batch sampling process has been proposed to help domains w/ slower learning during training.
strength_weakness: strength_weakness: Did not spot any, though, it seems there is a throughput drop.
questions: questions: Can you elaborate more on impact of the throughput drop in real-world industry applications? Also, specify how practical applications could benefit - please provide specific details.
limitations: limitations: Mentioned in paper
paper_summary: paper_summary: The paper tackles the problem of multi-domain fine-grained instance recognition/retrieval. The authors propose to train a unified backbone for all modalities with online distillation with domain-specific teachers, improving the performance compared to naive single backbone baselines and being competitive with expensive methods that utilize a number of domain-specific specialists. The authors propose a number of technical contributions in addition to the online distillation, including dynamic resampling based on a proxy of the task difficulty. UDON exhibits favorable performance compared to a number of baselines on the UnED benchmark consisting of a high number of fine-grained domain-specific benchmarks.
strength_weakness: strength_weakness: - [important] The paper tackles generalization for instance recognition/retrieval systems, but the scaling axis of this question has not been studied. This is lacking since in the past few years we have witnessed scalable pre-training in terms of data and parameters being a very effective solution to many generalization problems.
- The performance gains provided by UDON seem highly sensitive to which dataset is used for the evaluation. For example, while the dynamic sampling contribution helps two datasets, it dropped the performance of the other five. This does not mean that this contribution should be dismissed but rather we might need more work to achieve its objectives without hurting the performance of other stable benchmarks.
- Some ablations of the projections architecture as well as the design of the MLP baseline would be a welcome addition.
questions: questions: 1) depending on the number of classes the magnitude of the loss can be very different. How is this handled in equation [5]?
2) an important question is whether scaling is indeed the answer for most problems. For the off-the-shelf baselines, it can be easy to test much larger backbones (e.g. DinoV2-G, Clip-Large, …). It is interesting to see if simply scaling general-purpose pre-training would be sufficient to address the issues tackled in the paper.
3) Related to the question above, given that the performance gain between a naive unified model vs UDON is significant but not enormous, I wonder if scaling the naive single model baseline’s capacity slightly would be sufficient to address the shortcomings of training one model with many fine-grained domains.
limitations: limitations: Yes
paper_summary: paper_summary: The paper proposes Universal Dynamic Online distillatioN (UDON), which is a multi-teacher distillation method designed for universal image representations. UDON adopts a knowledge distillation strategy by distilling information from multiple teacher model trained for different domains to a student model to learn the universal embedding. It also proposes a dynamic domain sampling strategy for balancing the different domains. The provided experimental results verify its effectiveness.
strength_weakness: strength_weakness: **The reason why the proposed method is better than the previous work (USC) is not straightforward and clear enough for me.**
In the paper, the main baseline is Universal Separate Classifier Training method (USC). It uses a backbone to learn an universe embedding where multiple classifier heads for different domains are trained separately with the universe embedding as input. Compared with USC, UDON introduces distillation and dynamic domain sampling. I understand the part about dynamic domain sampling but feel confused about the distillation. As shown in Figure 2, UDON uses multiple extra classifier heads as teachers for distillation. Since the teacher heads and student head are all based on the previous embedding $E_b$, why the distillation strategy can help to build an better universal embedding $E_u$?  

The authors explain the reason mainly on [Line 37 - Line 51] and [Line 164 - Line 169], while I am still confused about it.
On [Line 37 - Line 51], it says that "it is difficult to encode detailed knowledge about many image domains in a single model". Therefore, the authors propose to use knowledge distillation between a student and teachers model for different domains, while the student and teachers share the backbone. Does it mean that the difficulty remains for USC, but it is solved by UDON with the distillation? The story seems a bit conflicting to me. Because from my point of view, a backbone with more classifier heads is still a single model, since the heads are mostly very light-weight compared with the backbone.

In my opinion, the proposed distillation design is filtering the information from $E_b$, which keeps more useful and universal information for the student head to learn $E_u$.
questions: questions: See weakness.
limitations: limitations: See weakness.
paper_summary: paper_summary: The paper introduces a novel method for enhancing universal image embeddings through a multi-teacher knowledge distillation approach. The method, named UDON, employs a dynamic sampling technique and a shared backbone across multiple domain-specific teachers and a universal student model to efficiently distill domain-specific knowledge into a universal embedding. This embedding model can work as a foundation for many downstream task models, such as classification, retrieval, generation, etc., with a high potential impact on the community.
strength_weakness: strength_weakness: (1) This paper misses a series of very important works for embedding learning [1,2]. The [1] proposed the Matryoshka representation learning (MRL) that the OpenAI's recent embedding model adopts. MRL would unfold the embedding model into multiple dimensions such as 64, 128, 512 or more. This paper simply takes the 64 and cannot be extended to higher dimensions which significantly limited its potential in applications.

[1] Kusupati, Aditya, et al. "Matryoshka representation learning." Advances in Neural Information Processing Systems 35 (2022): 30233-30249.

[2] Cai, Mu, et al. "Matryoshka Multimodal Models." arXiv preprint arXiv:2405.17430 (2024).

(2) Tab 1 is pretty confusing. There are "Off-the-shelf", "Specialist+Oracle", "ImageNet21k pretraining" and "CLIP pretraining". It is hard to understand the exact meaning of them. Moreover, the evaluation is not convincing. It misses some recent SOTA embedding models, such as SigLIP.

(3) Please unify the format of subtitles. In Line 253, there is "Implementation Details" but line 235 applies "Compared methods".

(4) There is no ablation study over the temperature and embedding dimension, which should be critical hyperparameters to explore.
questions: questions: N/A
limitations: limitations: N/A

Comments: title: After Rebuttal
comment: Thanks for your detailed response. After reading the response and other reviewers' comments, I will keep my original score.
I would like to thank the authors for the rebuttal, it has addressed the majority of my questions/concerns. Therefore, I will raise my score to 6.
Thank you for the detailed response from the authors. I will maintain my original score, as the novelty and impact of the work seem limited. The concept of multi-teacher distillation is not new, and testing solely on UnED with fixed dim size may not be sufficiently convincing.